Sep 18 16:15:58.599167 ip-10-0-133-147 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 16:16:02.197725 ip-10-0-133-147 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 16:16:02.197725 ip-10-0-133-147 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 16:16:02.197725 ip-10-0-133-147 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 16:16:02.197725 ip-10-0-133-147 kubenswrapper[2585]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 16:16:02.197725 ip-10-0-133-147 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 16:16:02.198452 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.197793    2585 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 16:16:02.201901 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.201884    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:16:02.201964 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.201901    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:16:02.201964 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.201907    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:16:02.201964 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.201926    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:16:02.201964 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.201929    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:16:02.201964 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.201932    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:16:02.201964 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.201935    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:16:02.201964 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.201938    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:16:02.201964 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.201941    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:16:02.201964 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.201943    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:16:02.201964 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.201946    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:16:02.201964 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.201949    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:16:02.201964 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.201952    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:16:02.201964 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.201954    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:16:02.201964 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.201962    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:16:02.201964 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.201966    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:16:02.201964 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.201968    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:16:02.201964 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.201971    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:16:02.201964 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.201974    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:16:02.202425 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.201977    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:16:02.202425 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.201980    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:16:02.202425 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.201983    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:16:02.202425 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.201985    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:16:02.202425 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.201990    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:16:02.202425 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.201993    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:16:02.202425 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.201996    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:16:02.202425 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202000    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:16:02.202425 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202003    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:16:02.202425 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202006    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:16:02.202425 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202008    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:16:02.202425 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202011    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:16:02.202425 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202014    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:16:02.202425 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202017    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:16:02.202425 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202019    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:16:02.202425 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202022    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:16:02.202425 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202024    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:16:02.202425 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202027    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:16:02.202425 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202030    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:16:02.202425 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202033    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:16:02.202907 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202035    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:16:02.202907 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202038    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:16:02.202907 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202040    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:16:02.202907 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202043    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:16:02.202907 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202045    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:16:02.202907 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202049    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:16:02.202907 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202052    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:16:02.202907 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202054    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:16:02.202907 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202057    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:16:02.202907 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202060    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:16:02.202907 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202062    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:16:02.202907 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202065    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:16:02.202907 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202067    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:16:02.202907 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202071    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:16:02.202907 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202074    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:16:02.202907 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202077    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:16:02.202907 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202079    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:16:02.202907 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202082    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:16:02.202907 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202085    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:16:02.202907 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202087    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:16:02.203410 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202093    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:16:02.203410 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202096    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:16:02.203410 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202099    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:16:02.203410 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202102    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:16:02.203410 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202105    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:16:02.203410 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202109    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:16:02.203410 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202112    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:16:02.203410 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202114    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:16:02.203410 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202117    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:16:02.203410 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202120    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:16:02.203410 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202122    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:16:02.203410 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202125    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:16:02.203410 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202127    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:16:02.203410 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202130    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:16:02.203410 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202133    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:16:02.203410 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202135    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:16:02.203410 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202138    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:16:02.203410 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202141    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:16:02.203410 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202143    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:16:02.203863 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202146    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:16:02.203863 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202148    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:16:02.203863 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202151    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:16:02.203863 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202154    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:16:02.203863 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202157    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:16:02.203863 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202160    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:16:02.203863 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202165    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:16:02.203863 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202167    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:16:02.203863 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202170    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:16:02.203863 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202172    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:16:02.203863 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202175    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:16:02.203863 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202178    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:16:02.203863 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202180    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:16:02.203863 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202183    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:16:02.203863 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202185    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:16:02.203863 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202188    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:16:02.203863 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202190    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:16:02.203863 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202193    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:16:02.203863 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202196    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:16:02.203863 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202198    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:16:02.204363 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202201    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:16:02.204363 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202864    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:16:02.204363 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202871    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:16:02.204363 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202874    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:16:02.204363 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202877    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:16:02.204363 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202881    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:16:02.204363 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202884    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:16:02.204363 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202887    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:16:02.204363 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202890    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:16:02.204363 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202893    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:16:02.204363 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202896    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:16:02.204363 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202899    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:16:02.204363 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202901    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:16:02.204363 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202904    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:16:02.204363 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202907    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:16:02.204363 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202927    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:16:02.204363 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202930    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:16:02.204363 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202933    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:16:02.204363 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202937    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:16:02.204906 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202940    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:16:02.204906 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202943    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:16:02.204906 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202946    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:16:02.204906 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202948    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:16:02.204906 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202951    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:16:02.204906 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202954    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:16:02.204906 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202957    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:16:02.204906 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202959    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:16:02.204906 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202962    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:16:02.204906 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202965    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:16:02.204906 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202968    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:16:02.204906 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202970    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:16:02.204906 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202973    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:16:02.204906 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202976    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:16:02.204906 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202979    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:16:02.204906 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202981    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:16:02.204906 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202984    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:16:02.204906 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202987    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:16:02.204906 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202990    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:16:02.205415 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202993    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:16:02.205415 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202995    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:16:02.205415 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.202998    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:16:02.205415 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203000    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:16:02.205415 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203004    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:16:02.205415 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203006    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:16:02.205415 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203009    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:16:02.205415 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203012    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:16:02.205415 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203014    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:16:02.205415 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203017    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:16:02.205415 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203020    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:16:02.205415 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203023    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:16:02.205415 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203026    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:16:02.205415 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203029    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:16:02.205415 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203031    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:16:02.205415 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203034    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:16:02.205415 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203037    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:16:02.205415 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203039    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:16:02.205415 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203042    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:16:02.205415 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203044    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:16:02.205938 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203047    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:16:02.205938 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203049    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:16:02.205938 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203052    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:16:02.205938 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203055    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:16:02.205938 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203057    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:16:02.205938 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203060    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:16:02.205938 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203062    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:16:02.205938 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203065    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:16:02.205938 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203068    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:16:02.205938 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203072    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:16:02.205938 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203075    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:16:02.205938 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203078    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:16:02.205938 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203081    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:16:02.205938 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203083    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:16:02.205938 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203086    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:16:02.205938 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203089    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:16:02.205938 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203092    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:16:02.205938 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203095    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:16:02.205938 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203097    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:16:02.206434 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203100    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:16:02.206434 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203103    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:16:02.206434 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203105    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:16:02.206434 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203108    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:16:02.206434 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203111    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:16:02.206434 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203115    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:16:02.206434 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203118    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:16:02.206434 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203121    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:16:02.206434 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203124    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:16:02.206434 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203127    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:16:02.206434 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203129    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:16:02.206434 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203132    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:16:02.206434 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203134    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:16:02.206434 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203137    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:16:02.206434 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203139    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:16:02.206434 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203143    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:16:02.206434 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203145    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:16:02.206434 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203148    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:16:02.206434 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203151    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:16:02.206434 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203153    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:16:02.206941 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203156    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:16:02.206941 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203160    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:16:02.206941 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203162    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:16:02.206941 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203546    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:16:02.206941 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203551    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:16:02.206941 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203555    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:16:02.206941 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203558    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:16:02.206941 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203561    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:16:02.206941 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203565    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:16:02.206941 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203567    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:16:02.206941 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203570    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:16:02.206941 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203575    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:16:02.206941 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203579    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:16:02.206941 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203582    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:16:02.206941 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203584    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:16:02.206941 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203587    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:16:02.206941 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203590    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:16:02.206941 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203593    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:16:02.206941 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203596    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:16:02.207405 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203599    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:16:02.207405 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203603    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:16:02.207405 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203606    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:16:02.207405 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203609    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:16:02.207405 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203612    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:16:02.207405 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203615    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:16:02.207405 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203618    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:16:02.207405 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203621    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:16:02.207405 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203624    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:16:02.207405 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203627    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:16:02.207405 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203629    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:16:02.207405 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203632    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:16:02.207405 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203635    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:16:02.207405 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203637    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:16:02.207405 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203640    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:16:02.207405 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203642    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:16:02.207405 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203645    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:16:02.207405 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203648    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:16:02.207405 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203651    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:16:02.207862 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203653    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:16:02.207862 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203656    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:16:02.207862 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203661    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:16:02.207862 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203663    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:16:02.207862 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203667    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:16:02.207862 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203670    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:16:02.207862 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203673    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:16:02.207862 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203675    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:16:02.207862 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203678    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:16:02.207862 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203681    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:16:02.207862 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203684    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:16:02.207862 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203687    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:16:02.207862 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203690    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:16:02.207862 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203693    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:16:02.207862 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203695    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:16:02.207862 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203698    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:16:02.207862 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203701    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:16:02.207862 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203703    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:16:02.207862 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203706    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:16:02.207862 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203708    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:16:02.208353 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203712    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:16:02.208353 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203715    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:16:02.208353 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203719    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:16:02.208353 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203722    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:16:02.208353 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203725    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:16:02.208353 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203728    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:16:02.208353 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203731    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:16:02.208353 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203734    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:16:02.208353 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203737    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:16:02.208353 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203740    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:16:02.208353 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203743    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:16:02.208353 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203746    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:16:02.208353 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203749    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:16:02.208353 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203751    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:16:02.208353 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203755    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:16:02.208353 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203758    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:16:02.208353 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203761    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:16:02.208353 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203763    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:16:02.208353 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203766    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:16:02.208803 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203769    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:16:02.208803 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203772    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:16:02.208803 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203775    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:16:02.208803 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203777    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:16:02.208803 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203780    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:16:02.208803 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203783    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:16:02.208803 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203786    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:16:02.208803 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203789    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:16:02.208803 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203791    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:16:02.208803 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203794    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:16:02.208803 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203797    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:16:02.208803 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203799    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:16:02.208803 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203802    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:16:02.208803 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203805    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:16:02.208803 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203807    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:16:02.208803 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203810    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:16:02.208803 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203813    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:16:02.208803 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203816    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:16:02.208803 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203818    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:16:02.208803 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203821    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:16:02.209305 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203823    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:16:02.209305 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203826    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:16:02.209305 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203828    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:16:02.209305 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203831    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:16:02.209305 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.203834    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:16:02.209305 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.203903    2585 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 16:16:02.209305 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.203952    2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 16:16:02.209305 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.203962    2585 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 16:16:02.209305 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.203967    2585 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 16:16:02.209305 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.203972    2585 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 16:16:02.209305 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.203975    2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 16:16:02.209305 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.203979    2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 16:16:02.209305 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.203983    2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 16:16:02.209305 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.203986    2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 16:16:02.209305 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.203989    2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 16:16:02.209305 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.203993    2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 16:16:02.209305 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.203996    2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 16:16:02.209305 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.203999    2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 16:16:02.209305 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204003    2585 flags.go:64] FLAG: --cgroup-root=""
Sep 18 16:16:02.209305 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204006    2585 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 16:16:02.209305 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204009    2585 flags.go:64] FLAG: --client-ca-file=""
Sep 18 16:16:02.209305 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204012    2585 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 16:16:02.209305 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204015    2585 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 16:16:02.209848 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204019    2585 flags.go:64] FLAG: --cluster-domain=""
Sep 18 16:16:02.209848 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204022    2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 16:16:02.209848 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204025    2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 16:16:02.209848 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204028    2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 16:16:02.209848 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204031    2585 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 16:16:02.209848 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204035    2585 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 16:16:02.209848 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204038    2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 16:16:02.209848 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204041    2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 16:16:02.209848 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204044    2585 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 16:16:02.209848 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204047    2585 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 16:16:02.209848 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204050    2585 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 16:16:02.209848 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204053    2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 16:16:02.209848 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204057    2585 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 16:16:02.209848 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204060    2585 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 16:16:02.209848 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204063    2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 16:16:02.209848 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204066    2585 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 16:16:02.209848 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204070    2585 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 16:16:02.209848 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204074    2585 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 16:16:02.209848 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204077    2585 flags.go:64] FLAG: --enable-server="true"
Sep 18 16:16:02.209848 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204080    2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 16:16:02.209848 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204085    2585 flags.go:64] FLAG: --event-burst="100"
Sep 18 16:16:02.209848 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204088    2585 flags.go:64] FLAG: --event-qps="50"
Sep 18 16:16:02.209848 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204091    2585 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 16:16:02.209848 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204094    2585 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 16:16:02.210478 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204097    2585 flags.go:64] FLAG: --eviction-hard=""
Sep 18 16:16:02.210478 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204101    2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 16:16:02.210478 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204104    2585 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 16:16:02.210478 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204107    2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 16:16:02.210478 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204110    2585 flags.go:64] FLAG: --eviction-soft=""
Sep 18 16:16:02.210478 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204113    2585 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 16:16:02.210478 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204116    2585 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 16:16:02.210478 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204119    2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 16:16:02.210478 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204122    2585 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 16:16:02.210478 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204125    2585 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 16:16:02.210478 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204127    2585 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 16:16:02.210478 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204130    2585 flags.go:64] FLAG: --feature-gates=""
Sep 18 16:16:02.210478 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204134    2585 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 16:16:02.210478 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204137    2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 16:16:02.210478 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204140    2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 16:16:02.210478 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204143    2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 16:16:02.210478 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204146    2585 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 16:16:02.210478 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204149    2585 flags.go:64] FLAG: --help="false"
Sep 18 16:16:02.210478 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204152    2585 flags.go:64] FLAG: --hostname-override="ip-10-0-133-147.ec2.internal"
Sep 18 16:16:02.210478 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204156    2585 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 16:16:02.210478 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204159    2585 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 16:16:02.210478 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204162    2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 16:16:02.210478 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204165    2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 16:16:02.210478 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204169    2585 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 16:16:02.211136 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204174    2585 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 16:16:02.211136 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204177    2585 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 16:16:02.211136 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204179    2585 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 16:16:02.211136 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204182    2585 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 16:16:02.211136 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204185    2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 16:16:02.211136 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204188    2585 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 16:16:02.211136 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204191    2585 flags.go:64] FLAG: --kube-reserved=""
Sep 18 16:16:02.211136 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204194    2585 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 16:16:02.211136 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204197    2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 16:16:02.211136 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204200    2585 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 16:16:02.211136 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204203    2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 16:16:02.211136 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204206    2585 flags.go:64] FLAG: --lock-file=""
Sep 18 16:16:02.211136 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204209    2585 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 16:16:02.211136 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204213    2585 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 16:16:02.211136 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204216    2585 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 16:16:02.211136 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204221    2585 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 16:16:02.211136 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204224    2585 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 16:16:02.211136 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204227    2585 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 16:16:02.211136 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204230    2585 flags.go:64] FLAG: --logging-format="text"
Sep 18 16:16:02.211136 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204232    2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 16:16:02.211136 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204236    2585 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 16:16:02.211136 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204239    2585 flags.go:64] FLAG: --manifest-url=""
Sep 18 16:16:02.211136 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204242    2585 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 16:16:02.211136 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204245    2585 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 16:16:02.211136 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204248    2585 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 16:16:02.211967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204252    2585 flags.go:64] FLAG: --max-pods="110"
Sep 18 16:16:02.211967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204256    2585 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 16:16:02.211967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204259    2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 16:16:02.211967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204261    2585 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 16:16:02.211967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204264    2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 16:16:02.211967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204267    2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 16:16:02.211967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204270    2585 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 16:16:02.211967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204274    2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 16:16:02.211967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204281    2585 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 16:16:02.211967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204284    2585 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 16:16:02.211967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204287    2585 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 16:16:02.211967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204291    2585 flags.go:64] FLAG: --pod-cidr=""
Sep 18 16:16:02.211967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204294    2585 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 16:16:02.211967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204299    2585 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 16:16:02.211967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204303    2585 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 16:16:02.211967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204306    2585 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 16:16:02.211967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204309    2585 flags.go:64] FLAG: --port="10250"
Sep 18 16:16:02.211967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204312    2585 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 16:16:02.211967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204315    2585 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e628d822f84cb08c"
Sep 18 16:16:02.211967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204318    2585 flags.go:64] FLAG: --qos-reserved=""
Sep 18 16:16:02.211967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204321    2585 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 16:16:02.211967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204325    2585 flags.go:64] FLAG: --register-node="true"
Sep 18 16:16:02.211967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204332    2585 flags.go:64] FLAG: --register-with-taints=""
Sep 18 16:16:02.211967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204336    2585 flags.go:64] FLAG: --registry-burst="10"
Sep 18 16:16:02.212839 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204339    2585 flags.go:64] FLAG: --registry-qps="5"
Sep 18 16:16:02.212839 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204342    2585 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 16:16:02.212839 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204345    2585 flags.go:64] FLAG: --reserved-memory=""
Sep 18 16:16:02.212839 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204351    2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 16:16:02.212839 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204354    2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 16:16:02.212839 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204357    2585 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 16:16:02.212839 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204360    2585 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 16:16:02.212839 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204363    2585 flags.go:64] FLAG: --runonce="false"
Sep 18 16:16:02.212839 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204366    2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 16:16:02.212839 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204371    2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 16:16:02.212839 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204375    2585 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 16:16:02.212839 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204378    2585 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 16:16:02.212839 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204380    2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 16:16:02.212839 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204384    2585 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 16:16:02.212839 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204387    2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 16:16:02.212839 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204390    2585 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 16:16:02.212839 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204393    2585 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 16:16:02.212839 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204396    2585 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 16:16:02.212839 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204399    2585 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 16:16:02.212839 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204402    2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 16:16:02.212839 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204405    2585 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 16:16:02.212839 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204408    2585 flags.go:64] FLAG: --system-cgroups=""
Sep 18 16:16:02.212839 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204411    2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 16:16:02.212839 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204416    2585 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 16:16:02.212839 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204419    2585 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 16:16:02.213942 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204422    2585 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 16:16:02.213942 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204426    2585 flags.go:64] FLAG: --tls-min-version=""
Sep 18 16:16:02.213942 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204429    2585 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 16:16:02.213942 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204432    2585 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 16:16:02.213942 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204434    2585 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 16:16:02.213942 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204438    2585 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 16:16:02.213942 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204441    2585 flags.go:64] FLAG: --v="2"
Sep 18 16:16:02.213942 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204445    2585 flags.go:64] FLAG: --version="false"
Sep 18 16:16:02.213942 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204448    2585 flags.go:64] FLAG: --vmodule=""
Sep 18 16:16:02.213942 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204452    2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 16:16:02.213942 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.204455    2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 16:16:02.213942 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204569    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:16:02.213942 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204573    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:16:02.213942 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204577    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:16:02.213942 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204581    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:16:02.213942 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204584    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:16:02.213942 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204589    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:16:02.213942 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204592    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:16:02.213942 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204595    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:16:02.213942 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204597    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:16:02.213942 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204600    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:16:02.213942 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204603    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:16:02.214884 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204605    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:16:02.214884 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204608    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:16:02.214884 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204611    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:16:02.214884 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204614    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:16:02.214884 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204617    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:16:02.214884 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204620    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:16:02.214884 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204622    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:16:02.214884 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204625    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:16:02.214884 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204628    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:16:02.214884 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204630    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:16:02.214884 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204633    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:16:02.214884 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204636    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:16:02.214884 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204639    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:16:02.214884 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204641    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:16:02.214884 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204644    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:16:02.214884 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204646    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:16:02.214884 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204649    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:16:02.214884 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204652    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:16:02.214884 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204655    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:16:02.215715 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204658    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:16:02.215715 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204661    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:16:02.215715 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204663    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:16:02.215715 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204666    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:16:02.215715 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204669    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:16:02.215715 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204671    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:16:02.215715 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204674    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:16:02.215715 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204678    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:16:02.215715 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204681    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:16:02.215715 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204683    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:16:02.215715 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204686    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:16:02.215715 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204688    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:16:02.215715 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204691    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:16:02.215715 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204694    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:16:02.215715 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204696    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:16:02.215715 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204699    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:16:02.215715 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204702    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:16:02.215715 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204704    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:16:02.215715 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204707    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:16:02.216366 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204709    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:16:02.216366 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204712    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:16:02.216366 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204715    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:16:02.216366 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204718    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:16:02.216366 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204720    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:16:02.216366 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204723    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:16:02.216366 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204726    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:16:02.216366 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204728    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:16:02.216366 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204731    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:16:02.216366 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204734    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:16:02.216366 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204736    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:16:02.216366 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204739    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:16:02.216366 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204743    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:16:02.216366 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204747    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:16:02.216366 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204750    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:16:02.216366 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204753    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:16:02.216366 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204756    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:16:02.216366 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204758    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:16:02.216366 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204761    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:16:02.216366 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204764    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:16:02.217233 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204768    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:16:02.217233 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204770    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:16:02.217233 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204773    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:16:02.217233 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204776    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:16:02.217233 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204778    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:16:02.217233 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204781    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:16:02.217233 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204783    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:16:02.217233 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204786    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:16:02.217233 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204789    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:16:02.217233 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204792    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:16:02.217233 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204795    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:16:02.217233 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204797    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:16:02.217233 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204800    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:16:02.217233 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204803    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:16:02.217233 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204805    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:16:02.217233 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204808    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:16:02.217233 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204810    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:16:02.217233 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204813    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:16:02.217233 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204816    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:16:02.217233 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204818    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:16:02.217960 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204821    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:16:02.217960 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204823    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:16:02.217960 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204826    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:16:02.217960 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204829    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:16:02.217960 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204832    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:16:02.217960 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204835    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:16:02.217960 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204837    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:16:02.217960 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204840    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:16:02.217960 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204843    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:16:02.217960 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.204847    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:16:02.217960 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.205598    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 16:16:02.217960 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.214181    2585 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 16:16:02.217960 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.214201    2585 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 16:16:02.217960 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.214228    2585 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 16:16:02.218440 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.214241    2585 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 16:16:02.218440 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214326    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:16:02.218440 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214333    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:16:02.218440 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214341    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:16:02.218440 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214347    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:16:02.218440 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214352    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:16:02.218440 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214356    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:16:02.218440 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214361    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:16:02.218440 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214366    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:16:02.218440 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214370    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:16:02.218440 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214375    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:16:02.218440 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214380    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:16:02.218440 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214385    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:16:02.218440 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214389    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:16:02.218440 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214393    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:16:02.218440 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214398    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:16:02.218440 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214402    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:16:02.218440 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214407    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:16:02.218440 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214412    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:16:02.218928 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214416    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:16:02.218928 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214421    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:16:02.218928 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214425    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:16:02.218928 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214429    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:16:02.218928 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214433    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:16:02.218928 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214437    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:16:02.218928 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214442    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:16:02.218928 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214446    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:16:02.218928 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214450    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:16:02.218928 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214455    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:16:02.218928 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214460    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:16:02.218928 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214465    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:16:02.218928 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214470    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:16:02.218928 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214474    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:16:02.218928 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214478    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:16:02.218928 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214483    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:16:02.218928 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214487    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:16:02.218928 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214491    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:16:02.218928 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214496    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:16:02.218928 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214500    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:16:02.219458 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214504    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:16:02.219458 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214509    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:16:02.219458 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214513    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:16:02.219458 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214517    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:16:02.219458 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214522    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:16:02.219458 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214526    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:16:02.219458 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214530    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:16:02.219458 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214536    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:16:02.219458 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214540    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:16:02.219458 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214544    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:16:02.219458 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214548    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:16:02.219458 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214552    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:16:02.219458 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214556    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:16:02.219458 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214561    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:16:02.219458 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214566    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:16:02.219458 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214570    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:16:02.219458 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214575    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:16:02.219458 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214579    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:16:02.219458 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214583    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:16:02.219458 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214590    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:16:02.220007 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214595    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:16:02.220007 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214600    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:16:02.220007 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214605    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:16:02.220007 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214610    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:16:02.220007 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214615    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:16:02.220007 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214619    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:16:02.220007 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214624    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:16:02.220007 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214629    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:16:02.220007 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214634    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:16:02.220007 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214638    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:16:02.220007 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214643    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:16:02.220007 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214647    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:16:02.220007 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214652    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:16:02.220007 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214657    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:16:02.220007 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214661    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:16:02.220007 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214666    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:16:02.220007 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214670    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:16:02.220007 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214675    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:16:02.220007 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214680    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:16:02.220007 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214684    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:16:02.220547 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214688    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:16:02.220547 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214694    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:16:02.220547 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214701    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:16:02.220547 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214707    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:16:02.220547 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214712    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:16:02.220547 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214716    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:16:02.220547 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214720    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:16:02.220547 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214725    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:16:02.220547 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214729    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:16:02.220547 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214733    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:16:02.220547 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214737    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:16:02.220547 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214742    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:16:02.220547 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214746    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:16:02.220547 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214750    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:16:02.220547 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214754    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:16:02.220547 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214759    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:16:02.220547 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214763    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:16:02.220547 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214768    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:16:02.220547 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214772    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:16:02.221060 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214776    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:16:02.221060 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.214781    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:16:02.221060 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.214788    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 16:16:02.221060 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215007    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:16:02.221060 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215017    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:16:02.221060 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215022    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:16:02.221060 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215030    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:16:02.221060 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215037    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:16:02.221060 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215043    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:16:02.221060 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215048    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:16:02.221060 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215052    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:16:02.221060 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215057    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:16:02.221060 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215062    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:16:02.221060 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215066    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:16:02.221060 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215071    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:16:02.221423 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215076    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:16:02.221423 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215081    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:16:02.221423 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215086    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:16:02.221423 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215090    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:16:02.221423 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215095    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:16:02.221423 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215099    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:16:02.221423 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215104    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:16:02.221423 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215108    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:16:02.221423 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215112    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:16:02.221423 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215117    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:16:02.221423 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215121    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:16:02.221423 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215125    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:16:02.221423 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215129    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:16:02.221423 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215134    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:16:02.221423 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215138    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:16:02.221423 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215142    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:16:02.221423 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215146    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:16:02.221423 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215151    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:16:02.221423 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215155    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:16:02.221423 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215159    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:16:02.222117 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215163    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:16:02.222117 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215167    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:16:02.222117 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215172    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:16:02.222117 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215176    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:16:02.222117 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215180    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:16:02.222117 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215184    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:16:02.222117 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215189    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:16:02.222117 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215193    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:16:02.222117 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215198    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:16:02.222117 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215202    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:16:02.222117 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215206    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:16:02.222117 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215210    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:16:02.222117 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215215    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:16:02.222117 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215221    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:16:02.222117 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215226    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:16:02.222117 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215230    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:16:02.222117 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215234    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:16:02.222117 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215238    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:16:02.222117 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215243    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:16:02.222117 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215247    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:16:02.222950 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215251    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:16:02.222950 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215256    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:16:02.222950 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215260    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:16:02.222950 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215264    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:16:02.222950 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215268    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:16:02.222950 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215272    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:16:02.222950 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215276    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:16:02.222950 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215280    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:16:02.222950 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215285    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:16:02.222950 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215289    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:16:02.222950 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215294    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:16:02.222950 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215298    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:16:02.222950 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215302    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:16:02.222950 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215306    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:16:02.222950 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215313    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:16:02.222950 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215319    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:16:02.222950 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215324    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:16:02.222950 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215328    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:16:02.222950 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215333    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:16:02.223747 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215337    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:16:02.223747 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215344    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:16:02.223747 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215349    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:16:02.223747 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215354    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:16:02.223747 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215359    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:16:02.223747 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215363    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:16:02.223747 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215367    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:16:02.223747 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215372    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:16:02.223747 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215377    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:16:02.223747 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215381    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:16:02.223747 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215387    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:16:02.223747 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215391    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:16:02.223747 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215396    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:16:02.223747 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215400    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:16:02.223747 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215404    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:16:02.223747 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215408    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:16:02.223747 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215413    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:16:02.223747 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215418    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:16:02.223747 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215422    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:16:02.224509 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215426    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:16:02.224509 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215430    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:16:02.224509 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215435    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:16:02.224509 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215439    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:16:02.224509 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215443    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:16:02.224509 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215447    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:16:02.224509 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215451    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:16:02.224509 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215456    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:16:02.224509 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:02.215461    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:16:02.224509 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.215468    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 16:16:02.224509 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.215650    2585 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 16:16:02.224509 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.218216    2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 16:16:02.224509 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.219354    2585 server.go:1019] "Starting client certificate rotation"
Sep 18 16:16:02.224509 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.219465    2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:16:02.225515 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.219500    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:16:02.225515 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.222284    2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 16:16:02.225515 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.224076    2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 16:16:02.227456 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.227442    2585 log.go:25] "Validated CRI v1 runtime API"
Sep 18 16:16:02.228889 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.228874    2585 log.go:25] "Validated CRI v1 image API"
Sep 18 16:16:02.231053 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.231038    2585 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 16:16:02.234551 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.234534    2585 fs.go:135] Filesystem UUIDs: map[07ddcfd7-2777-4e3c-b561-436ed02cce3a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e65c7775-74fe-4fc9-a57a-969c69bbb5f5:/dev/nvme0n1p3]
Sep 18 16:16:02.234611 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.234551    2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 16:16:02.239169 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.239062    2585 manager.go:216] Machine: {Timestamp:2026-09-18 16:16:02.238272782 +0000 UTC m=+2.021274799 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3112936 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec297f88b748ea6519414162e297e25f SystemUUID:ec297f88-b748-ea65-1941-4162e297e25f BootID:07c50887-c64c-4a70-b6bb-cbab1a490bf5 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d4:ef:39:95 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d4:ef:39:95 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0a:3a:af:36:03:c1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 16:16:02.239169 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.239160    2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 16:16:02.239282 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.239232    2585 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 16:16:02.240899 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.240875    2585 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 16:16:02.241045 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.240903    2585 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-147.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 16:16:02.241584 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.241574    2585 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 16:16:02.241616 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.241587    2585 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 16:16:02.241616 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.241594    2585 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 16:16:02.241616 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.241604    2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 16:16:02.242750 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.242740    2585 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 16:16:02.245606 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.245595    2585 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 16:16:02.245710 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.245701    2585 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 16:16:02.245744 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.245734    2585 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 16:16:02.245744 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.245743    2585 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 16:16:02.245797 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.245759    2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 16:16:02.245797 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.245768    2585 kubelet.go:398] "Adding apiserver pod source"
Sep 18 16:16:02.245797 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.245776    2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 16:16:02.246152 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.246141    2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 16:16:02.246221 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.246159    2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 16:16:02.248255 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.248236    2585 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 16:16:02.248626 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.248609    2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 16:16:02.250208 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.250194    2585 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 16:16:02.250265 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.250224    2585 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 16:16:02.251126 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.251114    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 16:16:02.251168 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.251136    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 16:16:02.251168 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.251145    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 16:16:02.251168 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.251153    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 16:16:02.251168 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.251162    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 16:16:02.251168 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.251167    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 16:16:02.251300 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.251181    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 16:16:02.251300 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.251186    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 16:16:02.251300 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.251193    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 16:16:02.251300 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.251198    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 16:16:02.251300 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.251207    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 16:16:02.251300 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.251212    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 16:16:02.251300 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.251238    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 16:16:02.251300 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.251247    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 16:16:02.255261 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.255232    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-147.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:16:02.255348 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.255272    2585 server.go:1268] "Started kubelet"
Sep 18 16:16:02.255348 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.255303    2585 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 16:16:02.255519 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.255498    2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 16:16:02.255597 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.255583    2585 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 16:16:02.255834 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:02.255807    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 16:16:02.255946 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:02.255805    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 16:16:02.256118 ip-10-0-133-147 systemd[1]: Started Kubernetes Kubelet.
Sep 18 16:16:02.256565 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.256392    2585 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 16:16:02.259506 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.259487    2585 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 16:16:02.260033 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:02.260015    2585 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 16:16:02.265248 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.265129    2585 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 16:16:02.265248 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.265158    2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 16:16:02.265248 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:02.265214    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-147.ec2.internal\" not found"
Sep 18 16:16:02.265944 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.265926    2585 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 16:16:02.266056 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.266045    2585 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 16:16:02.266136 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.266116    2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 16:16:02.266572 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.266555    2585 factory.go:55] Registering systemd factory
Sep 18 16:16:02.266643 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.266577    2585 factory.go:223] Registration of the systemd container factory successfully
Sep 18 16:16:02.266781 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.266757    2585 factory.go:153] Registering CRI-O factory
Sep 18 16:16:02.266781 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.266779    2585 factory.go:223] Registration of the crio container factory successfully
Sep 18 16:16:02.266866 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.266829    2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 16:16:02.266866 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.266850    2585 factory.go:103] Registering Raw factory
Sep 18 16:16:02.266866 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.266867    2585 manager.go:1196] Started watching for new ooms in manager
Sep 18 16:16:02.267085 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.267070    2585 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 16:16:02.267141 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.267095    2585 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 16:16:02.267452 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.267438    2585 manager.go:349] Starting recovery of all containers
Sep 18 16:16:02.285251 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.285225    2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hnbdd"
Sep 18 16:16:02.288761 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.288742    2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hnbdd"
Sep 18 16:16:02.292149 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:02.290950    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-147.ec2.internal.18d6767898a1dda3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-147.ec2.internal,UID:ip-10-0-133-147.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-147.ec2.internal,},FirstTimestamp:2026-09-18 16:16:02.255248803 +0000 UTC m=+2.038250881,LastTimestamp:2026-09-18 16:16:02.255248803 +0000 UTC m=+2.038250881,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-147.ec2.internal,}"
Sep 18 16:16:02.294061 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.294044    2585 manager.go:354] Recovery completed
Sep 18 16:16:02.298507 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.298489    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:16:02.298846 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.298833    2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:16:02.300193 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.300174    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:16:02.300286 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.300215    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:16:02.300286 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.300230    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:16:02.300691 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.300676    2585 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 16:16:02.300691 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.300687    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:16:02.300813 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.300690    2585 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 16:16:02.300813 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.300729    2585 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 16:16:02.301284 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.301268    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:16:02.301343 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.301294    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:16:02.301343 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.301306    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:16:02.301770 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.301759    2585 policy_none.go:49] "None policy: Start"
Sep 18 16:16:02.301806 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.301775    2585 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 16:16:02.301806 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.301785    2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 16:16:02.301924 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:02.301895    2585 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-147.ec2.internal\" not found" node="ip-10-0-133-147.ec2.internal"
Sep 18 16:16:02.302795 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.302784    2585 policy_none.go:47] "Start"
Sep 18 16:16:02.346495 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.346480    2585 manager.go:337] "Starting Device Plugin manager"
Sep 18 16:16:02.353887 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:02.346523    2585 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 16:16:02.353887 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.346534    2585 server.go:87] "Starting device plugin registration server"
Sep 18 16:16:02.353887 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.346760    2585 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 16:16:02.353887 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.346780    2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 16:16:02.353887 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.346840    2585 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 16:16:02.353887 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.346932    2585 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 16:16:02.353887 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.346941    2585 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 16:16:02.353887 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:02.347344    2585 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 16:16:02.353887 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:02.347400    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-147.ec2.internal\" not found"
Sep 18 16:16:02.384975 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.384949    2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 16:16:02.386313 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.386297    2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 16:16:02.386376 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.386352    2585 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 16:16:02.386423 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.386377    2585 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 16:16:02.386423 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:02.386418    2585 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 16:16:02.388895 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.388873    2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:16:02.446934 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.446902    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:16:02.447665 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.447653    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:16:02.447725 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.447680    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:16:02.447725 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.447691    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:16:02.447725 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.447711    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-147.ec2.internal"
Sep 18 16:16:02.456877 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.456840    2585 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-147.ec2.internal"
Sep 18 16:16:02.456877 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:02.456859    2585 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-147.ec2.internal\": node \"ip-10-0-133-147.ec2.internal\" not found"
Sep 18 16:16:02.482190 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:02.482174    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-147.ec2.internal\" not found"
Sep 18 16:16:02.487248 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.487234    2585 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-147.ec2.internal"]
Sep 18 16:16:02.487292 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.487286    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:16:02.487973 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.487957    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:16:02.488060 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.487982    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:16:02.488060 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.487993    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:16:02.489179 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.489167    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:16:02.489334 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.489323    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal"
Sep 18 16:16:02.489370 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.489350    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:16:02.489853 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.489834    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:16:02.489935 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.489853    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:16:02.489935 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.489873    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:16:02.489935 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.489884    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:16:02.490031 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.489874    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:16:02.490031 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.489959    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:16:02.490964 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.490947    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-147.ec2.internal"
Sep 18 16:16:02.491009 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.490979    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:16:02.491562 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.491545    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:16:02.491631 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.491576    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:16:02.491631 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.491589    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:16:02.511518 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:02.511492    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-147.ec2.internal\" not found" node="ip-10-0-133-147.ec2.internal"
Sep 18 16:16:02.515766 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:02.515750    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-147.ec2.internal\" not found" node="ip-10-0-133-147.ec2.internal"
Sep 18 16:16:02.568948 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.568907    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72458550952f102887362406515f2735-config\") pod \"kube-apiserver-proxy-ip-10-0-133-147.ec2.internal\" (UID: \"72458550952f102887362406515f2735\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-147.ec2.internal"
Sep 18 16:16:02.569052 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.568953    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4b278d3af8821d40e746f274a098db46-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal\" (UID: \"4b278d3af8821d40e746f274a098db46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal"
Sep 18 16:16:02.569052 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.568972    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b278d3af8821d40e746f274a098db46-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal\" (UID: \"4b278d3af8821d40e746f274a098db46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal"
Sep 18 16:16:02.582975 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:02.582957    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-147.ec2.internal\" not found"
Sep 18 16:16:02.670088 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.670062    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72458550952f102887362406515f2735-config\") pod \"kube-apiserver-proxy-ip-10-0-133-147.ec2.internal\" (UID: \"72458550952f102887362406515f2735\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-147.ec2.internal"
Sep 18 16:16:02.670231 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.670120    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4b278d3af8821d40e746f274a098db46-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal\" (UID: \"4b278d3af8821d40e746f274a098db46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal"
Sep 18 16:16:02.670231 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.670138    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b278d3af8821d40e746f274a098db46-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal\" (UID: \"4b278d3af8821d40e746f274a098db46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal"
Sep 18 16:16:02.670231 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.670069    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72458550952f102887362406515f2735-config\") pod \"kube-apiserver-proxy-ip-10-0-133-147.ec2.internal\" (UID: \"72458550952f102887362406515f2735\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-147.ec2.internal"
Sep 18 16:16:02.670231 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.670164    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b278d3af8821d40e746f274a098db46-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal\" (UID: \"4b278d3af8821d40e746f274a098db46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal"
Sep 18 16:16:02.670231 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.670199    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4b278d3af8821d40e746f274a098db46-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal\" (UID: \"4b278d3af8821d40e746f274a098db46\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal"
Sep 18 16:16:02.683154 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:02.683136    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-147.ec2.internal\" not found"
Sep 18 16:16:02.784004 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:02.783935    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-147.ec2.internal\" not found"
Sep 18 16:16:02.815127 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.815100    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal"
Sep 18 16:16:02.819775 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:02.819558    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-147.ec2.internal"
Sep 18 16:16:02.884384 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:02.884356    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-147.ec2.internal\" not found"
Sep 18 16:16:02.984900 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:02.984872    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-147.ec2.internal\" not found"
Sep 18 16:16:03.085448 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:03.085394    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-147.ec2.internal\" not found"
Sep 18 16:16:03.186041 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:03.186021    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-147.ec2.internal\" not found"
Sep 18 16:16:03.219230 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:03.219195    2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 16:16:03.219768 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:03.219336    2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 16:16:03.219768 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:03.219377    2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 16:16:03.286226 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:03.286197    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-147.ec2.internal\" not found"
Sep 18 16:16:03.294364 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:03.294328    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 16:11:02 +0000 UTC" deadline="2028-05-25 02:54:26.058873426 +0000 UTC"
Sep 18 16:16:03.294364 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:03.294363    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14746h38m22.764513664s"
Sep 18 16:16:03.387031 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:03.386969    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-147.ec2.internal\" not found"
Sep 18 16:16:03.487339 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:03.487310    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-147.ec2.internal\" not found"
Sep 18 16:16:03.569971 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:03.569937    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72458550952f102887362406515f2735.slice/crio-c0d9624776b5b177fb34596b6ea6e15d709a691c60564bf799ecdd5ad64fee96 WatchSource:0}: Error finding container c0d9624776b5b177fb34596b6ea6e15d709a691c60564bf799ecdd5ad64fee96: Status 404 returned error can't find the container with id c0d9624776b5b177fb34596b6ea6e15d709a691c60564bf799ecdd5ad64fee96
Sep 18 16:16:03.570543 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:03.570514    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b278d3af8821d40e746f274a098db46.slice/crio-39e99ab48d5fd34b868a691d72cf9319496b7dbf64c51c5f0a8d9f296a276eea WatchSource:0}: Error finding container 39e99ab48d5fd34b868a691d72cf9319496b7dbf64c51c5f0a8d9f296a276eea: Status 404 returned error can't find the container with id 39e99ab48d5fd34b868a691d72cf9319496b7dbf64c51c5f0a8d9f296a276eea
Sep 18 16:16:03.575884 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:03.575856    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:16:03.578942 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:03.578904    2585 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:16:03.587446 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:03.587427    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-147.ec2.internal\" not found"
Sep 18 16:16:03.688293 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:03.688217    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-147.ec2.internal\" not found"
Sep 18 16:16:03.782159 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:03.782136    2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:16:03.865897 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:03.865868    2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal"
Sep 18 16:16:03.884708 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:03.884689    2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 16:16:03.885879 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:03.885866    2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-147.ec2.internal"
Sep 18 16:16:03.900817 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:03.900795    2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 16:16:04.246676 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.246470    2585 apiserver.go:52] "Watching apiserver"
Sep 18 16:16:04.253014 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.252987    2585 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 16:16:04.253374 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.253357    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-c6t6w","openshift-network-diagnostics/network-check-target-bqkx2","openshift-network-operator/iptables-alerter-vl76v","openshift-ovn-kubernetes/ovnkube-node-vx9bt","kube-system/konnectivity-agent-rxjxh","kube-system/kube-apiserver-proxy-ip-10-0-133-147.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxsnh","openshift-cluster-node-tuning-operator/tuned-8nt4w","openshift-dns/node-resolver-q9svc","openshift-image-registry/node-ca-lh2tk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal","openshift-multus/multus-additional-cni-plugins-92ffq","kube-system/global-pull-secret-syncer-w8drz","openshift-multus/multus-tzzf6"]
Sep 18 16:16:04.255041 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.255020    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxsnh"
Sep 18 16:16:04.256800 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.256339    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:04.256800 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:04.256406    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8drz" podUID="d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d"
Sep 18 16:16:04.257539 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.257520    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 16:16:04.257849 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.257824    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 16:16:04.258024 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.258006    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 16:16:04.258104 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.258060    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pgjg8\""
Sep 18 16:16:04.258262 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.258244    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vl76v"
Sep 18 16:16:04.258941 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.258907    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.260077 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.260062    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rxjxh"
Sep 18 16:16:04.261242 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.261223    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 16:16:04.261345 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.261240    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:16:04.261345 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.261256    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 16:16:04.261457 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.261444    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.262181 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.261748    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 16:16:04.262181 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.261762    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zx62l\""
Sep 18 16:16:04.262181 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.261976    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 16:16:04.262181 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.262002    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bdccn\""
Sep 18 16:16:04.262584 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.262566    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 16:16:04.263001 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.262720    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 16:16:04.263001 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.262770    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 16:16:04.263001 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.262772    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 16:16:04.263196 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.263088    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 16:16:04.263806 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.263292    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hf8js\""
Sep 18 16:16:04.263806 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.263367    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 16:16:04.263981 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.263970    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 16:16:04.264129 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.264111    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 16:16:04.264406 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.264332    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-98cgp\""
Sep 18 16:16:04.264672 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.264654    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 16:16:04.264986 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.264959    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 16:16:04.265099 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.265081    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q9svc"
Sep 18 16:16:04.266389 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.266142    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:16:04.266389 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:04.266207    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c6t6w" podUID="53188ed0-31cd-4e9a-b72d-f21a8a738db6"
Sep 18 16:16:04.266389 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.266241    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 16:16:04.267430 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.267412    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 16:16:04.267529 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.267417    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-npkl6\""
Sep 18 16:16:04.267529 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.267458    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 16:16:04.267755 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.267740    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:16:04.267820 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:04.267803    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqkx2" podUID="068b8e2a-9f72-44c7-9340-783050326545"
Sep 18 16:16:04.273065 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.269994    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lh2tk"
Sep 18 16:16:04.273065 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.271534    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.273065 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.272234    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 16:16:04.273065 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.272607    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-92ffq"
Sep 18 16:16:04.273065 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.272993    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hth4q\""
Sep 18 16:16:04.273533 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.273514    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 16:16:04.273629 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.273523    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 16:16:04.274377 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.274358    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 16:16:04.274506 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.274375    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-b6sfz\""
Sep 18 16:16:04.274724 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.274706    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:16:04.275720 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.275490    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sz7kv\""
Sep 18 16:16:04.275720 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.275551    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 16:16:04.275720 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.275558    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 16:16:04.278130 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278105    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-node-log\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.278237 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278143    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-multus-cni-dir\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.278237 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278171    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e20b935d-4164-44ca-8b01-8bd08e10b3ac-konnectivity-ca\") pod \"konnectivity-agent-rxjxh\" (UID: \"e20b935d-4164-44ca-8b01-8bd08e10b3ac\") " pod="kube-system/konnectivity-agent-rxjxh"
Sep 18 16:16:04.278237 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278196    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aedb5565-e3e8-4be1-ac4d-545bdcd538dc-registration-dir\") pod \"aws-ebs-csi-driver-node-wxsnh\" (UID: \"aedb5565-e3e8-4be1-ac4d-545bdcd538dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxsnh"
Sep 18 16:16:04.278237 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278228    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d-kubelet-config\") pod \"global-pull-secret-syncer-w8drz\" (UID: \"d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d\") " pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:04.278447 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278249    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-host-run-netns\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.278447 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278270    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1c5e2210-6de0-4e88-ad43-8c99387acdb4-ovnkube-script-lib\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.278447 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278284    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-489w4\" (UniqueName: \"kubernetes.io/projected/de4c84a8-dfa3-4b78-ad71-e2d63345773c-kube-api-access-489w4\") pod \"iptables-alerter-vl76v\" (UID: \"de4c84a8-dfa3-4b78-ad71-e2d63345773c\") " pod="openshift-network-operator/iptables-alerter-vl76v"
Sep 18 16:16:04.278447 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278299    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-var-lib-openvswitch\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.278447 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278312    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-run-ovn\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.278447 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278326    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1c5e2210-6de0-4e88-ad43-8c99387acdb4-env-overrides\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.278447 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278340    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-host-run-k8s-cni-cncf-io\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.278447 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278389    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-host-run-netns\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.278447 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278422    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7dhl\" (UniqueName: \"kubernetes.io/projected/068b8e2a-9f72-44c7-9340-783050326545-kube-api-access-l7dhl\") pod \"network-check-target-bqkx2\" (UID: \"068b8e2a-9f72-44c7-9340-783050326545\") " pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:16:04.278811 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278448    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aedb5565-e3e8-4be1-ac4d-545bdcd538dc-socket-dir\") pod \"aws-ebs-csi-driver-node-wxsnh\" (UID: \"aedb5565-e3e8-4be1-ac4d-545bdcd538dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxsnh"
Sep 18 16:16:04.278811 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278481    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgrxp\" (UniqueName: \"kubernetes.io/projected/aedb5565-e3e8-4be1-ac4d-545bdcd538dc-kube-api-access-fgrxp\") pod \"aws-ebs-csi-driver-node-wxsnh\" (UID: \"aedb5565-e3e8-4be1-ac4d-545bdcd538dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxsnh"
Sep 18 16:16:04.278811 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278510    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/de4c84a8-dfa3-4b78-ad71-e2d63345773c-iptables-alerter-script\") pod \"iptables-alerter-vl76v\" (UID: \"de4c84a8-dfa3-4b78-ad71-e2d63345773c\") " pod="openshift-network-operator/iptables-alerter-vl76v"
Sep 18 16:16:04.278811 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278534    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-host-var-lib-cni-bin\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.278811 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278562    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-multus-socket-dir-parent\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.278811 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278585    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aedb5565-e3e8-4be1-ac4d-545bdcd538dc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wxsnh\" (UID: \"aedb5565-e3e8-4be1-ac4d-545bdcd538dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxsnh"
Sep 18 16:16:04.278811 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278607    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aedb5565-e3e8-4be1-ac4d-545bdcd538dc-etc-selinux\") pod \"aws-ebs-csi-driver-node-wxsnh\" (UID: \"aedb5565-e3e8-4be1-ac4d-545bdcd538dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxsnh"
Sep 18 16:16:04.278811 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278630    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsth2\" (UniqueName: \"kubernetes.io/projected/1c5e2210-6de0-4e88-ad43-8c99387acdb4-kube-api-access-lsth2\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.278811 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278655    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-cnibin\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.278811 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278675    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf86bb19-6c76-4d93-b36c-7a4078078ff3-cni-binary-copy\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.278811 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278694    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-host-var-lib-cni-multus\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.278811 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278715    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-hostroot\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.278811 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278751    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53188ed0-31cd-4e9a-b72d-f21a8a738db6-metrics-certs\") pod \"network-metrics-daemon-c6t6w\" (UID: \"53188ed0-31cd-4e9a-b72d-f21a8a738db6\") " pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:16:04.278811 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278791    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aedb5565-e3e8-4be1-ac4d-545bdcd538dc-sys-fs\") pod \"aws-ebs-csi-driver-node-wxsnh\" (UID: \"aedb5565-e3e8-4be1-ac4d-545bdcd538dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxsnh"
Sep 18 16:16:04.278811 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278815    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de4c84a8-dfa3-4b78-ad71-e2d63345773c-host-slash\") pod \"iptables-alerter-vl76v\" (UID: \"de4c84a8-dfa3-4b78-ad71-e2d63345773c\") " pod="openshift-network-operator/iptables-alerter-vl76v"
Sep 18 16:16:04.279535 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278845    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-host-slash\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.279535 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278867    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-run-openvswitch\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.279535 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278889    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-etc-kubernetes\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.279535 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278928    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn8cs\" (UniqueName: \"kubernetes.io/projected/cf86bb19-6c76-4d93-b36c-7a4078078ff3-kube-api-access-tn8cs\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.279535 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278952    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/895c0864-ed92-4326-b78f-3a85240437c2-hosts-file\") pod \"node-resolver-q9svc\" (UID: \"895c0864-ed92-4326-b78f-3a85240437c2\") " pod="openshift-dns/node-resolver-q9svc"
Sep 18 16:16:04.279535 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.278982    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d-original-pull-secret\") pod \"global-pull-secret-syncer-w8drz\" (UID: \"d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d\") " pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:04.279535 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.279021    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-etc-openvswitch\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.279535 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.279056    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1c5e2210-6de0-4e88-ad43-8c99387acdb4-ovnkube-config\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.279535 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.279078    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-system-cni-dir\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.279535 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.279099    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-multus-conf-dir\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.279535 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.279121    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d-dbus\") pod \"global-pull-secret-syncer-w8drz\" (UID: \"d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d\") " pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:04.279535 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.279144    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-systemd-units\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.279535 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.279166    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-host-run-ovn-kubernetes\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.279535 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.279181    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/895c0864-ed92-4326-b78f-3a85240437c2-tmp-dir\") pod \"node-resolver-q9svc\" (UID: \"895c0864-ed92-4326-b78f-3a85240437c2\") " pod="openshift-dns/node-resolver-q9svc"
Sep 18 16:16:04.279535 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.279194    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cf86bb19-6c76-4d93-b36c-7a4078078ff3-multus-daemon-config\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.279535 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.279207    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-host-run-multus-certs\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.279535 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.279231    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jwm4\" (UniqueName: \"kubernetes.io/projected/895c0864-ed92-4326-b78f-3a85240437c2-kube-api-access-4jwm4\") pod \"node-resolver-q9svc\" (UID: \"895c0864-ed92-4326-b78f-3a85240437c2\") " pod="openshift-dns/node-resolver-q9svc"
Sep 18 16:16:04.280324 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.279255    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-host-cni-bin\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.280324 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.279276    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-host-var-lib-kubelet\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.280324 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.279299    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e20b935d-4164-44ca-8b01-8bd08e10b3ac-agent-certs\") pod \"konnectivity-agent-rxjxh\" (UID: \"e20b935d-4164-44ca-8b01-8bd08e10b3ac\") " pod="kube-system/konnectivity-agent-rxjxh"
Sep 18 16:16:04.280324 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.279330    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aedb5565-e3e8-4be1-ac4d-545bdcd538dc-device-dir\") pod \"aws-ebs-csi-driver-node-wxsnh\" (UID: \"aedb5565-e3e8-4be1-ac4d-545bdcd538dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxsnh"
Sep 18 16:16:04.280324 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.279352    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-host-kubelet\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.280324 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.279402    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-run-systemd\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.280324 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.279435    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-log-socket\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.280324 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.279460    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.280324 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.279482    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-host-cni-netd\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.280324 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.279507    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5ccv\" (UniqueName: \"kubernetes.io/projected/53188ed0-31cd-4e9a-b72d-f21a8a738db6-kube-api-access-x5ccv\") pod \"network-metrics-daemon-c6t6w\" (UID: \"53188ed0-31cd-4e9a-b72d-f21a8a738db6\") " pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:16:04.280324 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.279529    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c5e2210-6de0-4e88-ad43-8c99387acdb4-ovn-node-metrics-cert\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.281705 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.281683    2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 16:16:04.285367 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.285345    2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:16:04.308075 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.308050    2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bnhkd"
Sep 18 16:16:04.317056 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.317033    2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bnhkd"
Sep 18 16:16:04.347400 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.347367    2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:16:04.367607 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.367583    2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 16:16:04.380082 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380054    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cf86bb19-6c76-4d93-b36c-7a4078078ff3-multus-daemon-config\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.380222 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380088    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-host-run-multus-certs\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.380222 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380113    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jwm4\" (UniqueName: \"kubernetes.io/projected/895c0864-ed92-4326-b78f-3a85240437c2-kube-api-access-4jwm4\") pod \"node-resolver-q9svc\" (UID: \"895c0864-ed92-4326-b78f-3a85240437c2\") " pod="openshift-dns/node-resolver-q9svc"
Sep 18 16:16:04.380222 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380133    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-host-cni-bin\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.380222 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380154    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-host-var-lib-kubelet\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.380222 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380173    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-host-run-multus-certs\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.380222 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380174    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e20b935d-4164-44ca-8b01-8bd08e10b3ac-agent-certs\") pod \"konnectivity-agent-rxjxh\" (UID: \"e20b935d-4164-44ca-8b01-8bd08e10b3ac\") " pod="kube-system/konnectivity-agent-rxjxh"
Sep 18 16:16:04.380505 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380238    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8e4139d-8ca0-4a9c-92b1-7a6998c5988b-host\") pod \"node-ca-lh2tk\" (UID: \"d8e4139d-8ca0-4a9c-92b1-7a6998c5988b\") " pod="openshift-image-registry/node-ca-lh2tk"
Sep 18 16:16:04.380505 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380275    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-etc-sysconfig\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.380505 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380306    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aedb5565-e3e8-4be1-ac4d-545bdcd538dc-device-dir\") pod \"aws-ebs-csi-driver-node-wxsnh\" (UID: \"aedb5565-e3e8-4be1-ac4d-545bdcd538dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxsnh"
Sep 18 16:16:04.380505 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380330    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-host-kubelet\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.380505 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380353    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-run-systemd\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.380505 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380375    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-log-socket\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.380505 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380399    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.380505 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380424    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70b1301b-d0c6-4fc3-aeac-661f45a32d79-cni-binary-copy\") pod \"multus-additional-cni-plugins-92ffq\" (UID: \"70b1301b-d0c6-4fc3-aeac-661f45a32d79\") " pod="openshift-multus/multus-additional-cni-plugins-92ffq"
Sep 18 16:16:04.380505 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380442    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aedb5565-e3e8-4be1-ac4d-545bdcd538dc-device-dir\") pod \"aws-ebs-csi-driver-node-wxsnh\" (UID: \"aedb5565-e3e8-4be1-ac4d-545bdcd538dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxsnh"
Sep 18 16:16:04.380505 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380450    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-host-cni-netd\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.380958 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380510    2585 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 16:16:04.380958 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380555    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-run-systemd\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.380958 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380528    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-host-kubelet\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.380958 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380497    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.380958 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380582    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-host-cni-netd\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.380958 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380583    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-host-cni-bin\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.380958 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380602    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5ccv\" (UniqueName: \"kubernetes.io/projected/53188ed0-31cd-4e9a-b72d-f21a8a738db6-kube-api-access-x5ccv\") pod \"network-metrics-daemon-c6t6w\" (UID: \"53188ed0-31cd-4e9a-b72d-f21a8a738db6\") " pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:16:04.380958 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380617    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-host-var-lib-kubelet\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.380958 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380645    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c5e2210-6de0-4e88-ad43-8c99387acdb4-ovn-node-metrics-cert\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.380958 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380661    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cf86bb19-6c76-4d93-b36c-7a4078078ff3-multus-daemon-config\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.380958 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380671    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-node-log\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.380958 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380699    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-multus-cni-dir\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.380958 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380723    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e20b935d-4164-44ca-8b01-8bd08e10b3ac-konnectivity-ca\") pod \"konnectivity-agent-rxjxh\" (UID: \"e20b935d-4164-44ca-8b01-8bd08e10b3ac\") " pod="kube-system/konnectivity-agent-rxjxh"
Sep 18 16:16:04.380958 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380739    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-node-log\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.380958 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380747    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-etc-systemd\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.380958 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380515    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-log-socket\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.380958 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380776    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-sys\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.380958 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380845    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-multus-cni-dir\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.381864 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380897    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-etc-tuned\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.381864 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380941    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70b1301b-d0c6-4fc3-aeac-661f45a32d79-system-cni-dir\") pod \"multus-additional-cni-plugins-92ffq\" (UID: \"70b1301b-d0c6-4fc3-aeac-661f45a32d79\") " pod="openshift-multus/multus-additional-cni-plugins-92ffq"
Sep 18 16:16:04.381864 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380970    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/70b1301b-d0c6-4fc3-aeac-661f45a32d79-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-92ffq\" (UID: \"70b1301b-d0c6-4fc3-aeac-661f45a32d79\") " pod="openshift-multus/multus-additional-cni-plugins-92ffq"
Sep 18 16:16:04.381864 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.380999    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aedb5565-e3e8-4be1-ac4d-545bdcd538dc-registration-dir\") pod \"aws-ebs-csi-driver-node-wxsnh\" (UID: \"aedb5565-e3e8-4be1-ac4d-545bdcd538dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxsnh"
Sep 18 16:16:04.381864 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.381021    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-host\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.381864 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.381043    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70b1301b-d0c6-4fc3-aeac-661f45a32d79-cnibin\") pod \"multus-additional-cni-plugins-92ffq\" (UID: \"70b1301b-d0c6-4fc3-aeac-661f45a32d79\") " pod="openshift-multus/multus-additional-cni-plugins-92ffq"
Sep 18 16:16:04.381864 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.381066    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl9bh\" (UniqueName: \"kubernetes.io/projected/70b1301b-d0c6-4fc3-aeac-661f45a32d79-kube-api-access-fl9bh\") pod \"multus-additional-cni-plugins-92ffq\" (UID: \"70b1301b-d0c6-4fc3-aeac-661f45a32d79\") " pod="openshift-multus/multus-additional-cni-plugins-92ffq"
Sep 18 16:16:04.381864 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.381094    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d-kubelet-config\") pod \"global-pull-secret-syncer-w8drz\" (UID: \"d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d\") " pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:04.381864 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.381118    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-host-run-netns\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.381864 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.381143    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1c5e2210-6de0-4e88-ad43-8c99387acdb4-ovnkube-script-lib\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.381864 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.381166    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-etc-sysctl-d\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.381864 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.381202    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e20b935d-4164-44ca-8b01-8bd08e10b3ac-konnectivity-ca\") pod \"konnectivity-agent-rxjxh\" (UID: \"e20b935d-4164-44ca-8b01-8bd08e10b3ac\") " pod="kube-system/konnectivity-agent-rxjxh"
Sep 18 16:16:04.381864 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.381269    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d-kubelet-config\") pod \"global-pull-secret-syncer-w8drz\" (UID: \"d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d\") " pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:04.381864 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.381307    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-run\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.381864 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.381337    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-489w4\" (UniqueName: \"kubernetes.io/projected/de4c84a8-dfa3-4b78-ad71-e2d63345773c-kube-api-access-489w4\") pod \"iptables-alerter-vl76v\" (UID: \"de4c84a8-dfa3-4b78-ad71-e2d63345773c\") " pod="openshift-network-operator/iptables-alerter-vl76v"
Sep 18 16:16:04.381864 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.381486    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aedb5565-e3e8-4be1-ac4d-545bdcd538dc-registration-dir\") pod \"aws-ebs-csi-driver-node-wxsnh\" (UID: \"aedb5565-e3e8-4be1-ac4d-545bdcd538dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxsnh"
Sep 18 16:16:04.382785 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.381544    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-var-lib-openvswitch\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.382785 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.381569    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-run-ovn\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.382785 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.381593    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1c5e2210-6de0-4e88-ad43-8c99387acdb4-env-overrides\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.382785 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.381613    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-var-lib-openvswitch\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.382785 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.381620    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-host-run-netns\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.382785 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.381639    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-host-run-k8s-cni-cncf-io\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.382785 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.381669    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1c5e2210-6de0-4e88-ad43-8c99387acdb4-ovnkube-script-lib\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.382785 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.381737    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-host-run-netns\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.382785 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.381747    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-host-run-k8s-cni-cncf-io\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.382785 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.381761    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7dhl\" (UniqueName: \"kubernetes.io/projected/068b8e2a-9f72-44c7-9340-783050326545-kube-api-access-l7dhl\") pod \"network-check-target-bqkx2\" (UID: \"068b8e2a-9f72-44c7-9340-783050326545\") " pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:16:04.382785 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.381789    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d8e4139d-8ca0-4a9c-92b1-7a6998c5988b-serviceca\") pod \"node-ca-lh2tk\" (UID: \"d8e4139d-8ca0-4a9c-92b1-7a6998c5988b\") " pod="openshift-image-registry/node-ca-lh2tk"
Sep 18 16:16:04.382785 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.381804    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aedb5565-e3e8-4be1-ac4d-545bdcd538dc-socket-dir\") pod \"aws-ebs-csi-driver-node-wxsnh\" (UID: \"aedb5565-e3e8-4be1-ac4d-545bdcd538dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxsnh"
Sep 18 16:16:04.382785 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.381815    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-run-ovn\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.382785 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.381843    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-host-run-netns\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.382785 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.381858    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgrxp\" (UniqueName: \"kubernetes.io/projected/aedb5565-e3e8-4be1-ac4d-545bdcd538dc-kube-api-access-fgrxp\") pod \"aws-ebs-csi-driver-node-wxsnh\" (UID: \"aedb5565-e3e8-4be1-ac4d-545bdcd538dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxsnh"
Sep 18 16:16:04.382785 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.381904    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aedb5565-e3e8-4be1-ac4d-545bdcd538dc-socket-dir\") pod \"aws-ebs-csi-driver-node-wxsnh\" (UID: \"aedb5565-e3e8-4be1-ac4d-545bdcd538dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxsnh"
Sep 18 16:16:04.382785 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.381907    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1c5e2210-6de0-4e88-ad43-8c99387acdb4-env-overrides\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.383621 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.381976    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/de4c84a8-dfa3-4b78-ad71-e2d63345773c-iptables-alerter-script\") pod \"iptables-alerter-vl76v\" (UID: \"de4c84a8-dfa3-4b78-ad71-e2d63345773c\") " pod="openshift-network-operator/iptables-alerter-vl76v"
Sep 18 16:16:04.383621 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382015    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-host-var-lib-cni-bin\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.383621 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382054    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-etc-kubernetes\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.383621 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382094    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-host-var-lib-cni-bin\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.383621 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382137    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/70b1301b-d0c6-4fc3-aeac-661f45a32d79-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-92ffq\" (UID: \"70b1301b-d0c6-4fc3-aeac-661f45a32d79\") " pod="openshift-multus/multus-additional-cni-plugins-92ffq"
Sep 18 16:16:04.383621 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382177    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-multus-socket-dir-parent\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.383621 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382209    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psv85\" (UniqueName: \"kubernetes.io/projected/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-kube-api-access-psv85\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.383621 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382235    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aedb5565-e3e8-4be1-ac4d-545bdcd538dc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wxsnh\" (UID: \"aedb5565-e3e8-4be1-ac4d-545bdcd538dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxsnh"
Sep 18 16:16:04.383621 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382262    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-multus-socket-dir-parent\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.383621 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382282    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aedb5565-e3e8-4be1-ac4d-545bdcd538dc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wxsnh\" (UID: \"aedb5565-e3e8-4be1-ac4d-545bdcd538dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxsnh"
Sep 18 16:16:04.383621 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382378    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aedb5565-e3e8-4be1-ac4d-545bdcd538dc-etc-selinux\") pod \"aws-ebs-csi-driver-node-wxsnh\" (UID: \"aedb5565-e3e8-4be1-ac4d-545bdcd538dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxsnh"
Sep 18 16:16:04.383621 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382412    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsth2\" (UniqueName: \"kubernetes.io/projected/1c5e2210-6de0-4e88-ad43-8c99387acdb4-kube-api-access-lsth2\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.383621 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382436    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-cnibin\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.383621 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382437    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/de4c84a8-dfa3-4b78-ad71-e2d63345773c-iptables-alerter-script\") pod \"iptables-alerter-vl76v\" (UID: \"de4c84a8-dfa3-4b78-ad71-e2d63345773c\") " pod="openshift-network-operator/iptables-alerter-vl76v"
Sep 18 16:16:04.383621 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382479    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf86bb19-6c76-4d93-b36c-7a4078078ff3-cni-binary-copy\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.383621 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382498    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-cnibin\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.383621 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382505    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-host-var-lib-cni-multus\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.384416 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382514    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aedb5565-e3e8-4be1-ac4d-545bdcd538dc-etc-selinux\") pod \"aws-ebs-csi-driver-node-wxsnh\" (UID: \"aedb5565-e3e8-4be1-ac4d-545bdcd538dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxsnh"
Sep 18 16:16:04.384416 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382540    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-hostroot\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.384416 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382543    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-host-var-lib-cni-multus\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.384416 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382567    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53188ed0-31cd-4e9a-b72d-f21a8a738db6-metrics-certs\") pod \"network-metrics-daemon-c6t6w\" (UID: \"53188ed0-31cd-4e9a-b72d-f21a8a738db6\") " pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:16:04.384416 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382587    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-hostroot\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.384416 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382592    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aedb5565-e3e8-4be1-ac4d-545bdcd538dc-sys-fs\") pod \"aws-ebs-csi-driver-node-wxsnh\" (UID: \"aedb5565-e3e8-4be1-ac4d-545bdcd538dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxsnh"
Sep 18 16:16:04.384416 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382615    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de4c84a8-dfa3-4b78-ad71-e2d63345773c-host-slash\") pod \"iptables-alerter-vl76v\" (UID: \"de4c84a8-dfa3-4b78-ad71-e2d63345773c\") " pod="openshift-network-operator/iptables-alerter-vl76v"
Sep 18 16:16:04.384416 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382656    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de4c84a8-dfa3-4b78-ad71-e2d63345773c-host-slash\") pod \"iptables-alerter-vl76v\" (UID: \"de4c84a8-dfa3-4b78-ad71-e2d63345773c\") " pod="openshift-network-operator/iptables-alerter-vl76v"
Sep 18 16:16:04.384416 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:04.382675    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:04.384416 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:04.382743    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53188ed0-31cd-4e9a-b72d-f21a8a738db6-metrics-certs podName:53188ed0-31cd-4e9a-b72d-f21a8a738db6 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:04.882716953 +0000 UTC m=+4.665718974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53188ed0-31cd-4e9a-b72d-f21a8a738db6-metrics-certs") pod "network-metrics-daemon-c6t6w" (UID: "53188ed0-31cd-4e9a-b72d-f21a8a738db6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:04.384416 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382758    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-host-slash\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.384416 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382661    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aedb5565-e3e8-4be1-ac4d-545bdcd538dc-sys-fs\") pod \"aws-ebs-csi-driver-node-wxsnh\" (UID: \"aedb5565-e3e8-4be1-ac4d-545bdcd538dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxsnh"
Sep 18 16:16:04.384416 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382780    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-run-openvswitch\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.384416 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382814    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-run-openvswitch\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.384416 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382825    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-etc-kubernetes\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.384416 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382854    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn8cs\" (UniqueName: \"kubernetes.io/projected/cf86bb19-6c76-4d93-b36c-7a4078078ff3-kube-api-access-tn8cs\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.384416 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382857    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-etc-kubernetes\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.385016 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382861    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-host-slash\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.385016 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382878    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/895c0864-ed92-4326-b78f-3a85240437c2-hosts-file\") pod \"node-resolver-q9svc\" (UID: \"895c0864-ed92-4326-b78f-3a85240437c2\") " pod="openshift-dns/node-resolver-q9svc"
Sep 18 16:16:04.385016 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382903    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwds2\" (UniqueName: \"kubernetes.io/projected/d8e4139d-8ca0-4a9c-92b1-7a6998c5988b-kube-api-access-jwds2\") pod \"node-ca-lh2tk\" (UID: \"d8e4139d-8ca0-4a9c-92b1-7a6998c5988b\") " pod="openshift-image-registry/node-ca-lh2tk"
Sep 18 16:16:04.385016 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382945    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/895c0864-ed92-4326-b78f-3a85240437c2-hosts-file\") pod \"node-resolver-q9svc\" (UID: \"895c0864-ed92-4326-b78f-3a85240437c2\") " pod="openshift-dns/node-resolver-q9svc"
Sep 18 16:16:04.385016 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.382946    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d-original-pull-secret\") pod \"global-pull-secret-syncer-w8drz\" (UID: \"d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d\") " pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:04.385016 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:04.383009    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:04.385016 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.383076    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-etc-openvswitch\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.385016 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:04.383110    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d-original-pull-secret podName:d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d nodeName:}" failed. No retries permitted until 2026-09-18 16:16:04.883092677 +0000 UTC m=+4.666094689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d-original-pull-secret") pod "global-pull-secret-syncer-w8drz" (UID: "d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:04.385016 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.383124    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1c5e2210-6de0-4e88-ad43-8c99387acdb4-ovnkube-config\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.385016 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.383146    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-etc-openvswitch\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.385016 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.383148    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-system-cni-dir\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.385016 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.383174    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-multus-conf-dir\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.385016 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.383211    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-system-cni-dir\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.385016 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.383221    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-etc-sysctl-conf\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.385016 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.383251    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf86bb19-6c76-4d93-b36c-7a4078078ff3-multus-conf-dir\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.385016 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.383274    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-var-lib-kubelet\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.385016 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.383296    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d-dbus\") pod \"global-pull-secret-syncer-w8drz\" (UID: \"d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d\") " pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:04.385693 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.383315    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-systemd-units\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.385693 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.383336    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-etc-modprobe-d\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.385693 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.383359    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-lib-modules\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.385693 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.383383    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-host-run-ovn-kubernetes\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.385693 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.383391    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d-dbus\") pod \"global-pull-secret-syncer-w8drz\" (UID: \"d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d\") " pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:04.385693 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.383406    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/895c0864-ed92-4326-b78f-3a85240437c2-tmp-dir\") pod \"node-resolver-q9svc\" (UID: \"895c0864-ed92-4326-b78f-3a85240437c2\") " pod="openshift-dns/node-resolver-q9svc"
Sep 18 16:16:04.385693 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.383424    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-systemd-units\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.385693 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.383427    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-tmp\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.385693 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.383447    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c5e2210-6de0-4e88-ad43-8c99387acdb4-host-run-ovn-kubernetes\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.385693 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.383451    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70b1301b-d0c6-4fc3-aeac-661f45a32d79-tuning-conf-dir\") pod \"multus-additional-cni-plugins-92ffq\" (UID: \"70b1301b-d0c6-4fc3-aeac-661f45a32d79\") " pod="openshift-multus/multus-additional-cni-plugins-92ffq"
Sep 18 16:16:04.385693 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.383520    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf86bb19-6c76-4d93-b36c-7a4078078ff3-cni-binary-copy\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.385693 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.383520    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1c5e2210-6de0-4e88-ad43-8c99387acdb4-ovnkube-config\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.385693 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.383634    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/895c0864-ed92-4326-b78f-3a85240437c2-tmp-dir\") pod \"node-resolver-q9svc\" (UID: \"895c0864-ed92-4326-b78f-3a85240437c2\") " pod="openshift-dns/node-resolver-q9svc"
Sep 18 16:16:04.385693 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.384524    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c5e2210-6de0-4e88-ad43-8c99387acdb4-ovn-node-metrics-cert\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.385693 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.384687    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e20b935d-4164-44ca-8b01-8bd08e10b3ac-agent-certs\") pod \"konnectivity-agent-rxjxh\" (UID: \"e20b935d-4164-44ca-8b01-8bd08e10b3ac\") " pod="kube-system/konnectivity-agent-rxjxh"
Sep 18 16:16:04.390900 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.390855    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal" event={"ID":"4b278d3af8821d40e746f274a098db46","Type":"ContainerStarted","Data":"39e99ab48d5fd34b868a691d72cf9319496b7dbf64c51c5f0a8d9f296a276eea"}
Sep 18 16:16:04.392038 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:04.392018    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:04.392038 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:04.392040    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:04.392189 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:04.392053    2585 projected.go:196] Error preparing data for projected volume kube-api-access-l7dhl for pod openshift-network-diagnostics/network-check-target-bqkx2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:04.396452 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:04.392116    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/068b8e2a-9f72-44c7-9340-783050326545-kube-api-access-l7dhl podName:068b8e2a-9f72-44c7-9340-783050326545 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:04.892099669 +0000 UTC m=+4.675101678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-l7dhl" (UniqueName: "kubernetes.io/projected/068b8e2a-9f72-44c7-9340-783050326545-kube-api-access-l7dhl") pod "network-check-target-bqkx2" (UID: "068b8e2a-9f72-44c7-9340-783050326545") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:04.396452 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.393886    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn8cs\" (UniqueName: \"kubernetes.io/projected/cf86bb19-6c76-4d93-b36c-7a4078078ff3-kube-api-access-tn8cs\") pod \"multus-tzzf6\" (UID: \"cf86bb19-6c76-4d93-b36c-7a4078078ff3\") " pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.396452 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.394588    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-489w4\" (UniqueName: \"kubernetes.io/projected/de4c84a8-dfa3-4b78-ad71-e2d63345773c-kube-api-access-489w4\") pod \"iptables-alerter-vl76v\" (UID: \"de4c84a8-dfa3-4b78-ad71-e2d63345773c\") " pod="openshift-network-operator/iptables-alerter-vl76v"
Sep 18 16:16:04.396833 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.396810    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsth2\" (UniqueName: \"kubernetes.io/projected/1c5e2210-6de0-4e88-ad43-8c99387acdb4-kube-api-access-lsth2\") pod \"ovnkube-node-vx9bt\" (UID: \"1c5e2210-6de0-4e88-ad43-8c99387acdb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.397569 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.397530    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jwm4\" (UniqueName: \"kubernetes.io/projected/895c0864-ed92-4326-b78f-3a85240437c2-kube-api-access-4jwm4\") pod \"node-resolver-q9svc\" (UID: \"895c0864-ed92-4326-b78f-3a85240437c2\") " pod="openshift-dns/node-resolver-q9svc"
Sep 18 16:16:04.398998 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.398974    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-147.ec2.internal" event={"ID":"72458550952f102887362406515f2735","Type":"ContainerStarted","Data":"c0d9624776b5b177fb34596b6ea6e15d709a691c60564bf799ecdd5ad64fee96"}
Sep 18 16:16:04.399161 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.399139    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5ccv\" (UniqueName: \"kubernetes.io/projected/53188ed0-31cd-4e9a-b72d-f21a8a738db6-kube-api-access-x5ccv\") pod \"network-metrics-daemon-c6t6w\" (UID: \"53188ed0-31cd-4e9a-b72d-f21a8a738db6\") " pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:16:04.399406 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.399384    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgrxp\" (UniqueName: \"kubernetes.io/projected/aedb5565-e3e8-4be1-ac4d-545bdcd538dc-kube-api-access-fgrxp\") pod \"aws-ebs-csi-driver-node-wxsnh\" (UID: \"aedb5565-e3e8-4be1-ac4d-545bdcd538dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxsnh"
Sep 18 16:16:04.484553 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.484510    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-etc-systemd\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.484553 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.484552    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-sys\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.484783 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.484576    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-etc-tuned\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.484783 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.484600    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70b1301b-d0c6-4fc3-aeac-661f45a32d79-system-cni-dir\") pod \"multus-additional-cni-plugins-92ffq\" (UID: \"70b1301b-d0c6-4fc3-aeac-661f45a32d79\") " pod="openshift-multus/multus-additional-cni-plugins-92ffq"
Sep 18 16:16:04.484783 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.484653    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70b1301b-d0c6-4fc3-aeac-661f45a32d79-system-cni-dir\") pod \"multus-additional-cni-plugins-92ffq\" (UID: \"70b1301b-d0c6-4fc3-aeac-661f45a32d79\") " pod="openshift-multus/multus-additional-cni-plugins-92ffq"
Sep 18 16:16:04.484783 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.484658    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-etc-systemd\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.484783 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.484658    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-sys\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.484783 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.484687    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/70b1301b-d0c6-4fc3-aeac-661f45a32d79-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-92ffq\" (UID: \"70b1301b-d0c6-4fc3-aeac-661f45a32d79\") " pod="openshift-multus/multus-additional-cni-plugins-92ffq"
Sep 18 16:16:04.484783 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.484723    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-host\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.484783 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.484748    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70b1301b-d0c6-4fc3-aeac-661f45a32d79-cnibin\") pod \"multus-additional-cni-plugins-92ffq\" (UID: \"70b1301b-d0c6-4fc3-aeac-661f45a32d79\") " pod="openshift-multus/multus-additional-cni-plugins-92ffq"
Sep 18 16:16:04.484783 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.484774    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl9bh\" (UniqueName: \"kubernetes.io/projected/70b1301b-d0c6-4fc3-aeac-661f45a32d79-kube-api-access-fl9bh\") pod \"multus-additional-cni-plugins-92ffq\" (UID: \"70b1301b-d0c6-4fc3-aeac-661f45a32d79\") " pod="openshift-multus/multus-additional-cni-plugins-92ffq"
Sep 18 16:16:04.485252 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.484785    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-host\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.485252 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.484801    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-etc-sysctl-d\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.485252 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.484822    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-run\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.485252 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.484834    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70b1301b-d0c6-4fc3-aeac-661f45a32d79-cnibin\") pod \"multus-additional-cni-plugins-92ffq\" (UID: \"70b1301b-d0c6-4fc3-aeac-661f45a32d79\") " pod="openshift-multus/multus-additional-cni-plugins-92ffq"
Sep 18 16:16:04.485252 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.484863    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d8e4139d-8ca0-4a9c-92b1-7a6998c5988b-serviceca\") pod \"node-ca-lh2tk\" (UID: \"d8e4139d-8ca0-4a9c-92b1-7a6998c5988b\") " pod="openshift-image-registry/node-ca-lh2tk"
Sep 18 16:16:04.485252 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.484889    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-etc-kubernetes\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.485252 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.484926    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/70b1301b-d0c6-4fc3-aeac-661f45a32d79-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-92ffq\" (UID: \"70b1301b-d0c6-4fc3-aeac-661f45a32d79\") " pod="openshift-multus/multus-additional-cni-plugins-92ffq"
Sep 18 16:16:04.485252 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.484951    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psv85\" (UniqueName: \"kubernetes.io/projected/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-kube-api-access-psv85\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.485252 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.484997    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwds2\" (UniqueName: \"kubernetes.io/projected/d8e4139d-8ca0-4a9c-92b1-7a6998c5988b-kube-api-access-jwds2\") pod \"node-ca-lh2tk\" (UID: \"d8e4139d-8ca0-4a9c-92b1-7a6998c5988b\") " pod="openshift-image-registry/node-ca-lh2tk"
Sep 18 16:16:04.485252 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.485034    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-etc-sysctl-conf\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.485252 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.485055    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-var-lib-kubelet\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.485252 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.485082    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-etc-modprobe-d\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.485252 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.485101    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-lib-modules\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.485252 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.485121    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-tmp\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.485252 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.485140    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70b1301b-d0c6-4fc3-aeac-661f45a32d79-tuning-conf-dir\") pod \"multus-additional-cni-plugins-92ffq\" (UID: \"70b1301b-d0c6-4fc3-aeac-661f45a32d79\") " pod="openshift-multus/multus-additional-cni-plugins-92ffq"
Sep 18 16:16:04.485252 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.485167    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8e4139d-8ca0-4a9c-92b1-7a6998c5988b-host\") pod \"node-ca-lh2tk\" (UID: \"d8e4139d-8ca0-4a9c-92b1-7a6998c5988b\") " pod="openshift-image-registry/node-ca-lh2tk"
Sep 18 16:16:04.485252 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.485183    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-etc-sysctl-d\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.486032 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.485192    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-etc-sysconfig\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.486032 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.485230    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70b1301b-d0c6-4fc3-aeac-661f45a32d79-cni-binary-copy\") pod \"multus-additional-cni-plugins-92ffq\" (UID: \"70b1301b-d0c6-4fc3-aeac-661f45a32d79\") " pod="openshift-multus/multus-additional-cni-plugins-92ffq"
Sep 18 16:16:04.486032 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.485268    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-var-lib-kubelet\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.486032 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.485233    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-run\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.486032 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.485303    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8e4139d-8ca0-4a9c-92b1-7a6998c5988b-host\") pod \"node-ca-lh2tk\" (UID: \"d8e4139d-8ca0-4a9c-92b1-7a6998c5988b\") " pod="openshift-image-registry/node-ca-lh2tk"
Sep 18 16:16:04.486032 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.485319    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-etc-kubernetes\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.486032 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.485288    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/70b1301b-d0c6-4fc3-aeac-661f45a32d79-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-92ffq\" (UID: \"70b1301b-d0c6-4fc3-aeac-661f45a32d79\") " pod="openshift-multus/multus-additional-cni-plugins-92ffq"
Sep 18 16:16:04.486032 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.485351    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-etc-sysconfig\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.486032 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.485429    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d8e4139d-8ca0-4a9c-92b1-7a6998c5988b-serviceca\") pod \"node-ca-lh2tk\" (UID: \"d8e4139d-8ca0-4a9c-92b1-7a6998c5988b\") " pod="openshift-image-registry/node-ca-lh2tk"
Sep 18 16:16:04.486032 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.485440    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-etc-modprobe-d\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.486032 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.485373    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70b1301b-d0c6-4fc3-aeac-661f45a32d79-tuning-conf-dir\") pod \"multus-additional-cni-plugins-92ffq\" (UID: \"70b1301b-d0c6-4fc3-aeac-661f45a32d79\") " pod="openshift-multus/multus-additional-cni-plugins-92ffq"
Sep 18 16:16:04.486032 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.485502    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-lib-modules\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.486032 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.485530    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-etc-sysctl-conf\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.486032 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.485748    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/70b1301b-d0c6-4fc3-aeac-661f45a32d79-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-92ffq\" (UID: \"70b1301b-d0c6-4fc3-aeac-661f45a32d79\") " pod="openshift-multus/multus-additional-cni-plugins-92ffq"
Sep 18 16:16:04.486444 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.486114    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70b1301b-d0c6-4fc3-aeac-661f45a32d79-cni-binary-copy\") pod \"multus-additional-cni-plugins-92ffq\" (UID: \"70b1301b-d0c6-4fc3-aeac-661f45a32d79\") " pod="openshift-multus/multus-additional-cni-plugins-92ffq"
Sep 18 16:16:04.487199 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.487171    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-etc-tuned\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.487878 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.487858    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-tmp\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.494413 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.494388    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwds2\" (UniqueName: \"kubernetes.io/projected/d8e4139d-8ca0-4a9c-92b1-7a6998c5988b-kube-api-access-jwds2\") pod \"node-ca-lh2tk\" (UID: \"d8e4139d-8ca0-4a9c-92b1-7a6998c5988b\") " pod="openshift-image-registry/node-ca-lh2tk"
Sep 18 16:16:04.495290 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.495263    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psv85\" (UniqueName: \"kubernetes.io/projected/6b80fdfe-7d1d-447b-ba5c-2f48515f18d0-kube-api-access-psv85\") pod \"tuned-8nt4w\" (UID: \"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0\") " pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.496806 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.496756    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl9bh\" (UniqueName: \"kubernetes.io/projected/70b1301b-d0c6-4fc3-aeac-661f45a32d79-kube-api-access-fl9bh\") pod \"multus-additional-cni-plugins-92ffq\" (UID: \"70b1301b-d0c6-4fc3-aeac-661f45a32d79\") " pod="openshift-multus/multus-additional-cni-plugins-92ffq"
Sep 18 16:16:04.569790 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.569748    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxsnh"
Sep 18 16:16:04.577031 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.577004    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vl76v"
Sep 18 16:16:04.586108 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.586085    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:04.591762 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.591737    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rxjxh"
Sep 18 16:16:04.598379 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.598358    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tzzf6"
Sep 18 16:16:04.605196 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.605176    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q9svc"
Sep 18 16:16:04.613875 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.613722    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lh2tk"
Sep 18 16:16:04.620452 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.620304    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8nt4w"
Sep 18 16:16:04.626170 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.626046    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-92ffq"
Sep 18 16:16:04.888593 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.888520    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53188ed0-31cd-4e9a-b72d-f21a8a738db6-metrics-certs\") pod \"network-metrics-daemon-c6t6w\" (UID: \"53188ed0-31cd-4e9a-b72d-f21a8a738db6\") " pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:16:04.888593 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.888564    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d-original-pull-secret\") pod \"global-pull-secret-syncer-w8drz\" (UID: \"d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d\") " pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:04.888796 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:04.888636    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:04.888796 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:04.888647    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:04.888796 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:04.888697    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d-original-pull-secret podName:d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d nodeName:}" failed. No retries permitted until 2026-09-18 16:16:05.888680205 +0000 UTC m=+5.671682207 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d-original-pull-secret") pod "global-pull-secret-syncer-w8drz" (UID: "d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:04.888796 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:04.888715    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53188ed0-31cd-4e9a-b72d-f21a8a738db6-metrics-certs podName:53188ed0-31cd-4e9a-b72d-f21a8a738db6 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:05.888705358 +0000 UTC m=+5.671707367 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53188ed0-31cd-4e9a-b72d-f21a8a738db6-metrics-certs") pod "network-metrics-daemon-c6t6w" (UID: "53188ed0-31cd-4e9a-b72d-f21a8a738db6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:04.989812 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:04.989778    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7dhl\" (UniqueName: \"kubernetes.io/projected/068b8e2a-9f72-44c7-9340-783050326545-kube-api-access-l7dhl\") pod \"network-check-target-bqkx2\" (UID: \"068b8e2a-9f72-44c7-9340-783050326545\") " pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:16:04.990008 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:04.989966    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:04.990008 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:04.989985    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:04.990008 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:04.989998    2585 projected.go:196] Error preparing data for projected volume kube-api-access-l7dhl for pod openshift-network-diagnostics/network-check-target-bqkx2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:04.990131 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:04.990066    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/068b8e2a-9f72-44c7-9340-783050326545-kube-api-access-l7dhl podName:068b8e2a-9f72-44c7-9340-783050326545 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:05.990050181 +0000 UTC m=+5.773052183 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-l7dhl" (UniqueName: "kubernetes.io/projected/068b8e2a-9f72-44c7-9340-783050326545-kube-api-access-l7dhl") pod "network-check-target-bqkx2" (UID: "068b8e2a-9f72-44c7-9340-783050326545") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:05.320186 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:05.320101    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 16:11:04 +0000 UTC" deadline="2028-03-31 09:02:18.679102017 +0000 UTC"
Sep 18 16:16:05.320186 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:05.320132    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13432h46m13.358974474s"
Sep 18 16:16:05.386722 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:05.386693    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:16:05.386895 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:05.386693    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:05.386895 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:05.386858    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqkx2" podUID="068b8e2a-9f72-44c7-9340-783050326545"
Sep 18 16:16:05.386895 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:05.386693    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:16:05.387057 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:05.386893    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8drz" podUID="d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d"
Sep 18 16:16:05.387057 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:05.387000    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c6t6w" podUID="53188ed0-31cd-4e9a-b72d-f21a8a738db6"
Sep 18 16:16:05.545989 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:05.545962    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode20b935d_4164_44ca_8b01_8bd08e10b3ac.slice/crio-44a81c690eb0c29ab8ecdeaa38429452687287e9644d8d42288d7fb09aadfeb8 WatchSource:0}: Error finding container 44a81c690eb0c29ab8ecdeaa38429452687287e9644d8d42288d7fb09aadfeb8: Status 404 returned error can't find the container with id 44a81c690eb0c29ab8ecdeaa38429452687287e9644d8d42288d7fb09aadfeb8
Sep 18 16:16:05.553242 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:05.553217    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b1301b_d0c6_4fc3_aeac_661f45a32d79.slice/crio-b57b34c72df56271270d5126c76ca3e4af9bc0cf90e6870a96e81ff728cff257 WatchSource:0}: Error finding container b57b34c72df56271270d5126c76ca3e4af9bc0cf90e6870a96e81ff728cff257: Status 404 returned error can't find the container with id b57b34c72df56271270d5126c76ca3e4af9bc0cf90e6870a96e81ff728cff257
Sep 18 16:16:05.554352 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:05.554328    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e4139d_8ca0_4a9c_92b1_7a6998c5988b.slice/crio-c1521ad97f225d0053da9602480753f8e97eb89eac0633503914b6f7c95d8a12 WatchSource:0}: Error finding container c1521ad97f225d0053da9602480753f8e97eb89eac0633503914b6f7c95d8a12: Status 404 returned error can't find the container with id c1521ad97f225d0053da9602480753f8e97eb89eac0633503914b6f7c95d8a12
Sep 18 16:16:05.896427 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:05.896131    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d-original-pull-secret\") pod \"global-pull-secret-syncer-w8drz\" (UID: \"d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d\") " pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:05.896583 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:05.896278    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:05.896583 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:05.896437    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53188ed0-31cd-4e9a-b72d-f21a8a738db6-metrics-certs\") pod \"network-metrics-daemon-c6t6w\" (UID: \"53188ed0-31cd-4e9a-b72d-f21a8a738db6\") " pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:16:05.896583 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:05.896511    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:05.896583 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:05.896524    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d-original-pull-secret podName:d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d nodeName:}" failed. No retries permitted until 2026-09-18 16:16:07.896498039 +0000 UTC m=+7.679500061 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d-original-pull-secret") pod "global-pull-secret-syncer-w8drz" (UID: "d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:05.896583 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:05.896551    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53188ed0-31cd-4e9a-b72d-f21a8a738db6-metrics-certs podName:53188ed0-31cd-4e9a-b72d-f21a8a738db6 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:07.89653928 +0000 UTC m=+7.679541296 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53188ed0-31cd-4e9a-b72d-f21a8a738db6-metrics-certs") pod "network-metrics-daemon-c6t6w" (UID: "53188ed0-31cd-4e9a-b72d-f21a8a738db6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:05.998360 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:05.997681    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7dhl\" (UniqueName: \"kubernetes.io/projected/068b8e2a-9f72-44c7-9340-783050326545-kube-api-access-l7dhl\") pod \"network-check-target-bqkx2\" (UID: \"068b8e2a-9f72-44c7-9340-783050326545\") " pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:16:05.998360 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:05.997905    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:05.998360 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:05.997959    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:05.998360 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:05.997973    2585 projected.go:196] Error preparing data for projected volume kube-api-access-l7dhl for pod openshift-network-diagnostics/network-check-target-bqkx2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:05.998360 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:05.998031    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/068b8e2a-9f72-44c7-9340-783050326545-kube-api-access-l7dhl podName:068b8e2a-9f72-44c7-9340-783050326545 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:07.998012636 +0000 UTC m=+7.781014645 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-l7dhl" (UniqueName: "kubernetes.io/projected/068b8e2a-9f72-44c7-9340-783050326545-kube-api-access-l7dhl") pod "network-check-target-bqkx2" (UID: "068b8e2a-9f72-44c7-9340-783050326545") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:06.322072 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:06.321773    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 16:11:04 +0000 UTC" deadline="2028-04-14 20:43:28.422721817 +0000 UTC"
Sep 18 16:16:06.322072 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:06.321835    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13780h27m22.10089081s"
Sep 18 16:16:06.415171 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:06.415078    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q9svc" event={"ID":"895c0864-ed92-4326-b78f-3a85240437c2","Type":"ContainerStarted","Data":"a8b8accd3a73416da7a5a45b897abe7dacccabf6e9aefa39650bf252b53424f3"}
Sep 18 16:16:06.440409 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:06.440374    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt" event={"ID":"1c5e2210-6de0-4e88-ad43-8c99387acdb4","Type":"ContainerStarted","Data":"6c736d275f70d007426b0c9e50f723e8bdac0cc1a3b5bd54cfca872dc870865d"}
Sep 18 16:16:06.466418 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:06.466375    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8nt4w" event={"ID":"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0","Type":"ContainerStarted","Data":"fc093a965b6d961092ed7a476d4891d93d0de0364f57d35c485e382f215d72ca"}
Sep 18 16:16:06.482081 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:06.482049    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxsnh" event={"ID":"aedb5565-e3e8-4be1-ac4d-545bdcd538dc","Type":"ContainerStarted","Data":"48ff481fc0a7d5a715ebab24bdeaae42f497276ea16fa0a2f83d01e373f58f12"}
Sep 18 16:16:06.502289 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:06.502251    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lh2tk" event={"ID":"d8e4139d-8ca0-4a9c-92b1-7a6998c5988b","Type":"ContainerStarted","Data":"c1521ad97f225d0053da9602480753f8e97eb89eac0633503914b6f7c95d8a12"}
Sep 18 16:16:06.516204 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:06.516174    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-92ffq" event={"ID":"70b1301b-d0c6-4fc3-aeac-661f45a32d79","Type":"ContainerStarted","Data":"b57b34c72df56271270d5126c76ca3e4af9bc0cf90e6870a96e81ff728cff257"}
Sep 18 16:16:06.527989 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:06.527933    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tzzf6" event={"ID":"cf86bb19-6c76-4d93-b36c-7a4078078ff3","Type":"ContainerStarted","Data":"61e3c6cae5b486126fac95aad3712bf53f8f3202f8d8e785208becb00a869fef"}
Sep 18 16:16:06.539149 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:06.539122    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rxjxh" event={"ID":"e20b935d-4164-44ca-8b01-8bd08e10b3ac","Type":"ContainerStarted","Data":"44a81c690eb0c29ab8ecdeaa38429452687287e9644d8d42288d7fb09aadfeb8"}
Sep 18 16:16:06.549017 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:06.548990    2585 generic.go:356] "Generic (PLEG): container finished" podID="4b278d3af8821d40e746f274a098db46" containerID="9deb6d4a6909864d64aa08a9a350c13159c9a35f555dc40d9a0d370633352e0c" exitCode=0
Sep 18 16:16:06.549119 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:06.549077    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal" event={"ID":"4b278d3af8821d40e746f274a098db46","Type":"ContainerDied","Data":"9deb6d4a6909864d64aa08a9a350c13159c9a35f555dc40d9a0d370633352e0c"}
Sep 18 16:16:06.560978 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:06.560538    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-147.ec2.internal" event={"ID":"72458550952f102887362406515f2735","Type":"ContainerStarted","Data":"88d7b2cf119718db76dac606ea0a74a2a55e7bf87e1ac04631fc8eae0daa0e81"}
Sep 18 16:16:06.571141 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:06.571076    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vl76v" event={"ID":"de4c84a8-dfa3-4b78-ad71-e2d63345773c","Type":"ContainerStarted","Data":"0b3e102a006b1df1125c99bbd9f5c91dfaca6042595db515b482d4029ad19208"}
Sep 18 16:16:07.386990 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:07.386951    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:16:07.387449 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:07.387109    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c6t6w" podUID="53188ed0-31cd-4e9a-b72d-f21a8a738db6"
Sep 18 16:16:07.387449 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:07.387189    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:16:07.387449 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:07.387269    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqkx2" podUID="068b8e2a-9f72-44c7-9340-783050326545"
Sep 18 16:16:07.387449 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:07.387330    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:07.387449 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:07.387415    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8drz" podUID="d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d"
Sep 18 16:16:07.601566 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:07.601491    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal" event={"ID":"4b278d3af8821d40e746f274a098db46","Type":"ContainerStarted","Data":"ee09d6b631c39ee5d82d3919a290d5295a8c6ec0f67e6afdac4487e833db3d43"}
Sep 18 16:16:07.616542 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:07.616465    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-147.ec2.internal" podStartSLOduration=4.616445857 podStartE2EDuration="4.616445857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:06.581014038 +0000 UTC m=+6.364016063" watchObservedRunningTime="2026-09-18 16:16:07.616445857 +0000 UTC m=+7.399447884"
Sep 18 16:16:07.912371 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:07.912334    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53188ed0-31cd-4e9a-b72d-f21a8a738db6-metrics-certs\") pod \"network-metrics-daemon-c6t6w\" (UID: \"53188ed0-31cd-4e9a-b72d-f21a8a738db6\") " pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:16:07.912586 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:07.912385    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d-original-pull-secret\") pod \"global-pull-secret-syncer-w8drz\" (UID: \"d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d\") " pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:07.912586 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:07.912508    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:07.912586 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:07.912570    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d-original-pull-secret podName:d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d nodeName:}" failed. No retries permitted until 2026-09-18 16:16:11.912549693 +0000 UTC m=+11.695551695 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d-original-pull-secret") pod "global-pull-secret-syncer-w8drz" (UID: "d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:07.913123 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:07.912978    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:07.913123 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:07.913032    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53188ed0-31cd-4e9a-b72d-f21a8a738db6-metrics-certs podName:53188ed0-31cd-4e9a-b72d-f21a8a738db6 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:11.913015441 +0000 UTC m=+11.696017455 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53188ed0-31cd-4e9a-b72d-f21a8a738db6-metrics-certs") pod "network-metrics-daemon-c6t6w" (UID: "53188ed0-31cd-4e9a-b72d-f21a8a738db6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:08.013974 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:08.013359    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7dhl\" (UniqueName: \"kubernetes.io/projected/068b8e2a-9f72-44c7-9340-783050326545-kube-api-access-l7dhl\") pod \"network-check-target-bqkx2\" (UID: \"068b8e2a-9f72-44c7-9340-783050326545\") " pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:16:08.013974 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:08.013551    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:08.013974 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:08.013569    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:08.013974 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:08.013581    2585 projected.go:196] Error preparing data for projected volume kube-api-access-l7dhl for pod openshift-network-diagnostics/network-check-target-bqkx2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:08.013974 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:08.013635    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/068b8e2a-9f72-44c7-9340-783050326545-kube-api-access-l7dhl podName:068b8e2a-9f72-44c7-9340-783050326545 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:12.013617134 +0000 UTC m=+11.796619140 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-l7dhl" (UniqueName: "kubernetes.io/projected/068b8e2a-9f72-44c7-9340-783050326545-kube-api-access-l7dhl") pod "network-check-target-bqkx2" (UID: "068b8e2a-9f72-44c7-9340-783050326545") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:09.387370 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:09.386595    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:16:09.387370 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:09.386721    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqkx2" podUID="068b8e2a-9f72-44c7-9340-783050326545"
Sep 18 16:16:09.387370 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:09.386796    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:09.387370 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:09.386864    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8drz" podUID="d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d"
Sep 18 16:16:09.387370 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:09.386933    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:16:09.387370 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:09.387010    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c6t6w" podUID="53188ed0-31cd-4e9a-b72d-f21a8a738db6"
Sep 18 16:16:11.387533 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:11.387505    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:16:11.387986 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:11.387667    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c6t6w" podUID="53188ed0-31cd-4e9a-b72d-f21a8a738db6"
Sep 18 16:16:11.387986 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:11.387735    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:16:11.387986 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:11.387805    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqkx2" podUID="068b8e2a-9f72-44c7-9340-783050326545"
Sep 18 16:16:11.387986 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:11.387847    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:11.388328 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:11.387907    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8drz" podUID="d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d"
Sep 18 16:16:11.948897 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:11.948859    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53188ed0-31cd-4e9a-b72d-f21a8a738db6-metrics-certs\") pod \"network-metrics-daemon-c6t6w\" (UID: \"53188ed0-31cd-4e9a-b72d-f21a8a738db6\") " pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:16:11.949079 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:11.948924    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d-original-pull-secret\") pod \"global-pull-secret-syncer-w8drz\" (UID: \"d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d\") " pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:11.949079 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:11.949042    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:11.949079 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:11.949057    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:11.949245 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:11.949111    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d-original-pull-secret podName:d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d nodeName:}" failed. No retries permitted until 2026-09-18 16:16:19.949096337 +0000 UTC m=+19.732098339 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d-original-pull-secret") pod "global-pull-secret-syncer-w8drz" (UID: "d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:11.949245 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:11.949128    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53188ed0-31cd-4e9a-b72d-f21a8a738db6-metrics-certs podName:53188ed0-31cd-4e9a-b72d-f21a8a738db6 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:19.949119572 +0000 UTC m=+19.732121574 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53188ed0-31cd-4e9a-b72d-f21a8a738db6-metrics-certs") pod "network-metrics-daemon-c6t6w" (UID: "53188ed0-31cd-4e9a-b72d-f21a8a738db6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:12.050261 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:12.050221    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7dhl\" (UniqueName: \"kubernetes.io/projected/068b8e2a-9f72-44c7-9340-783050326545-kube-api-access-l7dhl\") pod \"network-check-target-bqkx2\" (UID: \"068b8e2a-9f72-44c7-9340-783050326545\") " pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:16:12.050412 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:12.050372    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:12.050412 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:12.050393    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:12.050412 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:12.050406    2585 projected.go:196] Error preparing data for projected volume kube-api-access-l7dhl for pod openshift-network-diagnostics/network-check-target-bqkx2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:12.050575 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:12.050464    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/068b8e2a-9f72-44c7-9340-783050326545-kube-api-access-l7dhl podName:068b8e2a-9f72-44c7-9340-783050326545 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:20.050446516 +0000 UTC m=+19.833448523 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-l7dhl" (UniqueName: "kubernetes.io/projected/068b8e2a-9f72-44c7-9340-783050326545-kube-api-access-l7dhl") pod "network-check-target-bqkx2" (UID: "068b8e2a-9f72-44c7-9340-783050326545") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:13.387193 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:13.387151    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:16:13.387733 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:13.387273    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqkx2" podUID="068b8e2a-9f72-44c7-9340-783050326545"
Sep 18 16:16:13.387733 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:13.387623    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:13.387733 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:13.387687    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8drz" podUID="d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d"
Sep 18 16:16:13.387855 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:13.387739    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:16:13.387855 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:13.387814    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c6t6w" podUID="53188ed0-31cd-4e9a-b72d-f21a8a738db6"
Sep 18 16:16:15.386543 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:15.386509    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:15.386959 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:15.386510    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:16:15.386959 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:15.386647    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8drz" podUID="d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d"
Sep 18 16:16:15.386959 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:15.386510    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:16:15.386959 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:15.386717    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c6t6w" podUID="53188ed0-31cd-4e9a-b72d-f21a8a738db6"
Sep 18 16:16:15.386959 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:15.386784    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqkx2" podUID="068b8e2a-9f72-44c7-9340-783050326545"
Sep 18 16:16:17.387279 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:17.387246    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:16:17.387710 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:17.387351    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:17.387710 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:17.387361    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqkx2" podUID="068b8e2a-9f72-44c7-9340-783050326545"
Sep 18 16:16:17.387710 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:17.387465    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8drz" podUID="d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d"
Sep 18 16:16:17.387710 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:17.387497    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:16:17.387710 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:17.387581    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c6t6w" podUID="53188ed0-31cd-4e9a-b72d-f21a8a738db6"
Sep 18 16:16:19.386984 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:19.386895    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:16:19.386984 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:19.386958    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:19.386984 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:19.386966    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:16:19.387692 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:19.387067    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqkx2" podUID="068b8e2a-9f72-44c7-9340-783050326545"
Sep 18 16:16:19.387692 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:19.387166    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8drz" podUID="d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d"
Sep 18 16:16:19.387692 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:19.387258    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c6t6w" podUID="53188ed0-31cd-4e9a-b72d-f21a8a738db6"
Sep 18 16:16:20.007182 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:20.007145    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53188ed0-31cd-4e9a-b72d-f21a8a738db6-metrics-certs\") pod \"network-metrics-daemon-c6t6w\" (UID: \"53188ed0-31cd-4e9a-b72d-f21a8a738db6\") " pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:16:20.007349 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:20.007197    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d-original-pull-secret\") pod \"global-pull-secret-syncer-w8drz\" (UID: \"d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d\") " pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:20.007349 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:20.007283    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:20.007349 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:20.007324    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:20.007349 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:20.007347    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53188ed0-31cd-4e9a-b72d-f21a8a738db6-metrics-certs podName:53188ed0-31cd-4e9a-b72d-f21a8a738db6 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:36.007331002 +0000 UTC m=+35.790333007 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53188ed0-31cd-4e9a-b72d-f21a8a738db6-metrics-certs") pod "network-metrics-daemon-c6t6w" (UID: "53188ed0-31cd-4e9a-b72d-f21a8a738db6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:20.007515 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:20.007374    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d-original-pull-secret podName:d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d nodeName:}" failed. No retries permitted until 2026-09-18 16:16:36.007357889 +0000 UTC m=+35.790359897 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d-original-pull-secret") pod "global-pull-secret-syncer-w8drz" (UID: "d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:20.107835 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:20.107794    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7dhl\" (UniqueName: \"kubernetes.io/projected/068b8e2a-9f72-44c7-9340-783050326545-kube-api-access-l7dhl\") pod \"network-check-target-bqkx2\" (UID: \"068b8e2a-9f72-44c7-9340-783050326545\") " pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:16:20.108040 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:20.107968    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:20.108040 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:20.107988    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:20.108040 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:20.108001    2585 projected.go:196] Error preparing data for projected volume kube-api-access-l7dhl for pod openshift-network-diagnostics/network-check-target-bqkx2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:20.108171 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:20.108056    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/068b8e2a-9f72-44c7-9340-783050326545-kube-api-access-l7dhl podName:068b8e2a-9f72-44c7-9340-783050326545 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:36.108034908 +0000 UTC m=+35.891036910 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-l7dhl" (UniqueName: "kubernetes.io/projected/068b8e2a-9f72-44c7-9340-783050326545-kube-api-access-l7dhl") pod "network-check-target-bqkx2" (UID: "068b8e2a-9f72-44c7-9340-783050326545") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:21.386726 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:21.386687    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:16:21.387181 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:21.386730    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:21.387181 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:21.386730    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:16:21.387181 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:21.386821    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqkx2" podUID="068b8e2a-9f72-44c7-9340-783050326545"
Sep 18 16:16:21.387181 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:21.386934    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8drz" podUID="d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d"
Sep 18 16:16:21.387181 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:21.387020    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c6t6w" podUID="53188ed0-31cd-4e9a-b72d-f21a8a738db6"
Sep 18 16:16:23.386770 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:23.386741    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:23.387150 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:23.386859    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8drz" podUID="d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d"
Sep 18 16:16:23.387232 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:23.387210    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:16:23.387356 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:23.387325    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c6t6w" podUID="53188ed0-31cd-4e9a-b72d-f21a8a738db6"
Sep 18 16:16:23.387420 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:23.387385    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:16:23.387469 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:23.387453    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqkx2" podUID="068b8e2a-9f72-44c7-9340-783050326545"
Sep 18 16:16:23.637312 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:23.637116    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8nt4w" event={"ID":"6b80fdfe-7d1d-447b-ba5c-2f48515f18d0","Type":"ContainerStarted","Data":"ed6066119ad0a77dfafbab590496e44dd466a6e1b1c746f2ca3db3f5bc344d0d"}
Sep 18 16:16:23.639781 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:23.638650    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxsnh" event={"ID":"aedb5565-e3e8-4be1-ac4d-545bdcd538dc","Type":"ContainerStarted","Data":"7a676db91ce1abbd4b9b66c78e81810f44650eafe02ec2ce7bd07b3da13e5903"}
Sep 18 16:16:23.640214 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:23.640186    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lh2tk" event={"ID":"d8e4139d-8ca0-4a9c-92b1-7a6998c5988b","Type":"ContainerStarted","Data":"294bd4461e0c239e7dd542ac6521baa3d231d638aafeddaedc98edc25e0bae91"}
Sep 18 16:16:23.642581 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:23.642559    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-92ffq" event={"ID":"70b1301b-d0c6-4fc3-aeac-661f45a32d79","Type":"ContainerStarted","Data":"3733f65a04a3fe7bf86bb79d395d34e899f7a252c0bf4ddc4319aadca9e4b3e3"}
Sep 18 16:16:23.643775 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:23.643756    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tzzf6" event={"ID":"cf86bb19-6c76-4d93-b36c-7a4078078ff3","Type":"ContainerStarted","Data":"09904c0b3cc620f1a521b1a63ac80202b088729e1c6d7a9a7bf94ceca097414e"}
Sep 18 16:16:23.644763 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:23.644747    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rxjxh" event={"ID":"e20b935d-4164-44ca-8b01-8bd08e10b3ac","Type":"ContainerStarted","Data":"58e49f8c146dd150bd6873998a21dc9ee8119f880d7cce41c933a74d64877198"}
Sep 18 16:16:23.645773 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:23.645757    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q9svc" event={"ID":"895c0864-ed92-4326-b78f-3a85240437c2","Type":"ContainerStarted","Data":"67117cf15a6f002429a9b08d6c98418308993cdd227199072b8c14d6a8883fe1"}
Sep 18 16:16:23.647272 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:23.647254    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt" event={"ID":"1c5e2210-6de0-4e88-ad43-8c99387acdb4","Type":"ContainerStarted","Data":"78733a1d045776473b3b334009e2b627a9d687cc3c8fcf154cecf685aced0ac7"}
Sep 18 16:16:23.647366 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:23.647322    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt" event={"ID":"1c5e2210-6de0-4e88-ad43-8c99387acdb4","Type":"ContainerStarted","Data":"938dfc7de36034bf2886ab1db2c126721c645c330bdd34b310524242785c8112"}
Sep 18 16:16:23.656415 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:23.656379    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8nt4w" podStartSLOduration=3.935039269 podStartE2EDuration="21.65636718s" totalImagesPullingTime="17.721327911s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:23.656049908 +0000 UTC m=+23.439051934" watchObservedRunningTime="2026-09-18 16:16:23.65636718 +0000 UTC m=+23.439369233"
Sep 18 16:16:23.656580 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:23.656556    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal" podStartSLOduration=20.656549763 podStartE2EDuration="20.656549763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:07.617199537 +0000 UTC m=+7.400201563" watchObservedRunningTime="2026-09-18 16:16:23.656549763 +0000 UTC m=+23.439551786"
Sep 18 16:16:23.689604 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:23.689558    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lh2tk" podStartSLOduration=3.919225478 podStartE2EDuration="21.689544537s" totalImagesPullingTime="17.770319059s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:23.689402271 +0000 UTC m=+23.472404295" watchObservedRunningTime="2026-09-18 16:16:23.689544537 +0000 UTC m=+23.472546618"
Sep 18 16:16:23.703781 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:23.703748    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-q9svc" podStartSLOduration=3.926217403 podStartE2EDuration="21.7037312s" totalImagesPullingTime="17.777513797s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:23.703669026 +0000 UTC m=+23.486671050" watchObservedRunningTime="2026-09-18 16:16:23.7037312 +0000 UTC m=+23.486733224"
Sep 18 16:16:23.720414 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:23.720369    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rxjxh" podStartSLOduration=4.064316659 podStartE2EDuration="21.720351263s" totalImagesPullingTime="17.656034604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:23.719708676 +0000 UTC m=+23.502710700" watchObservedRunningTime="2026-09-18 16:16:23.720351263 +0000 UTC m=+23.503353290"
Sep 18 16:16:23.736111 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:23.736070    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-tzzf6" podStartSLOduration=3.9175897280000003 podStartE2EDuration="21.73606033s" totalImagesPullingTime="17.818470602s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:23.735890404 +0000 UTC m=+23.518892428" watchObservedRunningTime="2026-09-18 16:16:23.73606033 +0000 UTC m=+23.519062354"
Sep 18 16:16:23.756438 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:23.756411    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rxjxh"
Sep 18 16:16:23.756944 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:23.756899    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rxjxh"
Sep 18 16:16:24.502092 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:24.501954    2585 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 16:16:24.651285 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:24.651253    2585 generic.go:356] "Generic (PLEG): container finished" podID="70b1301b-d0c6-4fc3-aeac-661f45a32d79" containerID="3733f65a04a3fe7bf86bb79d395d34e899f7a252c0bf4ddc4319aadca9e4b3e3" exitCode=0
Sep 18 16:16:24.651380 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:24.651317    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-92ffq" event={"ID":"70b1301b-d0c6-4fc3-aeac-661f45a32d79","Type":"ContainerDied","Data":"3733f65a04a3fe7bf86bb79d395d34e899f7a252c0bf4ddc4319aadca9e4b3e3"}
Sep 18 16:16:24.652935 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:24.652893    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vl76v" event={"ID":"de4c84a8-dfa3-4b78-ad71-e2d63345773c","Type":"ContainerStarted","Data":"d0a74e5f08c25cc5028637065c791826a5a40ce11dc4b1246a39018d58d1c161"}
Sep 18 16:16:24.655929 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:24.655894    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 16:16:24.656305 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:24.656286    2585 generic.go:356] "Generic (PLEG): container finished" podID="1c5e2210-6de0-4e88-ad43-8c99387acdb4" containerID="78733a1d045776473b3b334009e2b627a9d687cc3c8fcf154cecf685aced0ac7" exitCode=1
Sep 18 16:16:24.656379 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:24.656360    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt" event={"ID":"1c5e2210-6de0-4e88-ad43-8c99387acdb4","Type":"ContainerDied","Data":"78733a1d045776473b3b334009e2b627a9d687cc3c8fcf154cecf685aced0ac7"}
Sep 18 16:16:24.656429 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:24.656417    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt" event={"ID":"1c5e2210-6de0-4e88-ad43-8c99387acdb4","Type":"ContainerStarted","Data":"19b9d9bacce161b5752bfeadbcae913365a63185edcf2a9114ddc35fd9af2eac"}
Sep 18 16:16:24.656476 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:24.656466    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt" event={"ID":"1c5e2210-6de0-4e88-ad43-8c99387acdb4","Type":"ContainerStarted","Data":"e9a1ea3f82977a5a35157c00ca83ddfa4789383cb5d65794a7475ad87441fab9"}
Sep 18 16:16:24.656515 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:24.656495    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt" event={"ID":"1c5e2210-6de0-4e88-ad43-8c99387acdb4","Type":"ContainerStarted","Data":"4bbf3083b2b01b189d1f46097da1e4222b8e004aee7b68acd2185b08d66928b4"}
Sep 18 16:16:24.656551 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:24.656532    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt" event={"ID":"1c5e2210-6de0-4e88-ad43-8c99387acdb4","Type":"ContainerStarted","Data":"9b376d38f558a4a3b081eff7b75c1b873ed4951b44fbf7485a80665268724289"}
Sep 18 16:16:24.658051 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:24.658032    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxsnh" event={"ID":"aedb5565-e3e8-4be1-ac4d-545bdcd538dc","Type":"ContainerStarted","Data":"c333b7ffec9e5a51bdc656fb491686afd3e257a0a260debb560c5b7127af0e3c"}
Sep 18 16:16:24.658781 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:24.658767    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rxjxh"
Sep 18 16:16:24.658881 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:24.658808    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rxjxh"
Sep 18 16:16:24.688767 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:24.688725    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vl76v" podStartSLOduration=4.969150639 podStartE2EDuration="22.68871377s" totalImagesPullingTime="17.719563131s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:24.688395554 +0000 UTC m=+24.471397580" watchObservedRunningTime="2026-09-18 16:16:24.68871377 +0000 UTC m=+24.471715793"
Sep 18 16:16:25.360499 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:25.360341    2585 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T16:16:24.502089958Z","UUID":"3bd074fd-75cd-4ec1-951e-081ccebabe80","Handler":null,"Name":"","Endpoint":""}
Sep 18 16:16:25.361963 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:25.361945    2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 16:16:25.362076 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:25.361983    2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 16:16:25.387271 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:25.387246    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:16:25.387271 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:25.387262    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:16:25.387465 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:25.387282    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:25.387465 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:25.387378    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c6t6w" podUID="53188ed0-31cd-4e9a-b72d-f21a8a738db6"
Sep 18 16:16:25.387465 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:25.387424    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqkx2" podUID="068b8e2a-9f72-44c7-9340-783050326545"
Sep 18 16:16:25.387603 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:25.387499    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8drz" podUID="d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d"
Sep 18 16:16:26.667682 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:26.667527    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 16:16:26.668129 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:26.668062    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt" event={"ID":"1c5e2210-6de0-4e88-ad43-8c99387acdb4","Type":"ContainerStarted","Data":"67c4c69633b363a0b7b955c2ce3285e7d996a709a5364b4b01118b24baf6a9a8"}
Sep 18 16:16:26.670107 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:26.670084    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxsnh" event={"ID":"aedb5565-e3e8-4be1-ac4d-545bdcd538dc","Type":"ContainerStarted","Data":"8ac9bfa1fb21ef78cb6fde065fb1107d7fb973d1ba7f6cced5820cad31d9b2fe"}
Sep 18 16:16:26.691599 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:26.691566    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxsnh" podStartSLOduration=5.030918428 podStartE2EDuration="24.691554613s" totalImagesPullingTime="19.660636185s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:26.691551778 +0000 UTC m=+26.474553802" watchObservedRunningTime="2026-09-18 16:16:26.691554613 +0000 UTC m=+26.474556634"
Sep 18 16:16:27.387162 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:27.387134    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:16:27.387372 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:27.387237    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:27.387372 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:27.387247    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqkx2" podUID="068b8e2a-9f72-44c7-9340-783050326545"
Sep 18 16:16:27.387372 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:27.387257    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:16:27.387492 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:27.387382    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8drz" podUID="d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d"
Sep 18 16:16:27.387492 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:27.387457    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c6t6w" podUID="53188ed0-31cd-4e9a-b72d-f21a8a738db6"
Sep 18 16:16:29.387090 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:29.386886    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:16:29.388389 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:29.387175    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqkx2" podUID="068b8e2a-9f72-44c7-9340-783050326545"
Sep 18 16:16:29.388389 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:29.386928    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:29.388389 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:29.387267    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8drz" podUID="d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d"
Sep 18 16:16:29.388389 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:29.386888    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:16:29.388389 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:29.387359    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c6t6w" podUID="53188ed0-31cd-4e9a-b72d-f21a8a738db6"
Sep 18 16:16:29.679153 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:29.679123    2585 generic.go:356] "Generic (PLEG): container finished" podID="70b1301b-d0c6-4fc3-aeac-661f45a32d79" containerID="c4564108e115c41e929ed840763f60f25ca27914c2ead090491f78a19076f898" exitCode=0
Sep 18 16:16:29.679307 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:29.679212    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-92ffq" event={"ID":"70b1301b-d0c6-4fc3-aeac-661f45a32d79","Type":"ContainerDied","Data":"c4564108e115c41e929ed840763f60f25ca27914c2ead090491f78a19076f898"}
Sep 18 16:16:29.682795 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:29.682775    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 16:16:29.683170 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:29.683149    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt" event={"ID":"1c5e2210-6de0-4e88-ad43-8c99387acdb4","Type":"ContainerStarted","Data":"a6fca91a010ae06d231e343ae14eea671e7d8e408b46a046daa1c872bb330ffc"}
Sep 18 16:16:29.683375 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:29.683357    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:29.683446 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:29.683411    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:29.683446 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:29.683442    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:29.683603 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:29.683588    2585 scope.go:117] "RemoveContainer" containerID="78733a1d045776473b3b334009e2b627a9d687cc3c8fcf154cecf685aced0ac7"
Sep 18 16:16:29.700993 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:29.700971    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:29.701111 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:29.701073    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:16:30.689179 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:30.689154    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 16:16:30.689596 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:30.689485    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt" event={"ID":"1c5e2210-6de0-4e88-ad43-8c99387acdb4","Type":"ContainerStarted","Data":"def00c6aafb1b839d87d9568f738f03bb9fe220766357ae736719d5a4d1bfdfa"}
Sep 18 16:16:30.727503 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:30.727451    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt" podStartSLOduration=10.958395592 podStartE2EDuration="28.727432404s" totalImagesPullingTime="17.769036812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:30.725891153 +0000 UTC m=+30.508893177" watchObservedRunningTime="2026-09-18 16:16:30.727432404 +0000 UTC m=+30.510434476"
Sep 18 16:16:31.155644 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:31.155612    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w8drz"]
Sep 18 16:16:31.155850 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:31.155730    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:31.155850 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:31.155836    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8drz" podUID="d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d"
Sep 18 16:16:31.157256 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:31.157171    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bqkx2"]
Sep 18 16:16:31.157363 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:31.157270    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:16:31.157363 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:31.157344    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqkx2" podUID="068b8e2a-9f72-44c7-9340-783050326545"
Sep 18 16:16:31.158354 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:31.158331    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c6t6w"]
Sep 18 16:16:31.158460 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:31.158448    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:16:31.158580 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:31.158559    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c6t6w" podUID="53188ed0-31cd-4e9a-b72d-f21a8a738db6"
Sep 18 16:16:31.694047 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:31.694014    2585 generic.go:356] "Generic (PLEG): container finished" podID="70b1301b-d0c6-4fc3-aeac-661f45a32d79" containerID="dd45657c2927d650a2a00edbef3620c4d982a5ce2393da3cc6bcfad1be4b0e1e" exitCode=0
Sep 18 16:16:31.694494 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:31.694102    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-92ffq" event={"ID":"70b1301b-d0c6-4fc3-aeac-661f45a32d79","Type":"ContainerDied","Data":"dd45657c2927d650a2a00edbef3620c4d982a5ce2393da3cc6bcfad1be4b0e1e"}
Sep 18 16:16:32.388119 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:32.387868    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:16:32.388298 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:32.388209    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqkx2" podUID="068b8e2a-9f72-44c7-9340-783050326545"
Sep 18 16:16:33.387073 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:33.387044    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:16:33.387619 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:33.387084    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:33.387619 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:33.387177    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c6t6w" podUID="53188ed0-31cd-4e9a-b72d-f21a8a738db6"
Sep 18 16:16:33.387619 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:33.387303    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8drz" podUID="d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d"
Sep 18 16:16:33.701855 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:33.701769    2585 generic.go:356] "Generic (PLEG): container finished" podID="70b1301b-d0c6-4fc3-aeac-661f45a32d79" containerID="531dc6da47af5909d9d53e57812a4352a0bd193888f47853bbb71d9be86b234d" exitCode=0
Sep 18 16:16:33.701855 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:33.701814    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-92ffq" event={"ID":"70b1301b-d0c6-4fc3-aeac-661f45a32d79","Type":"ContainerDied","Data":"531dc6da47af5909d9d53e57812a4352a0bd193888f47853bbb71d9be86b234d"}
Sep 18 16:16:34.387574 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:34.387543    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:16:34.388040 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:34.387662    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqkx2" podUID="068b8e2a-9f72-44c7-9340-783050326545"
Sep 18 16:16:35.386899 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:35.386868    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:35.387077 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:35.386874    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:16:35.387077 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:35.386997    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8drz" podUID="d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d"
Sep 18 16:16:35.387077 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:35.387060    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c6t6w" podUID="53188ed0-31cd-4e9a-b72d-f21a8a738db6"
Sep 18 16:16:36.031693 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.031642    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53188ed0-31cd-4e9a-b72d-f21a8a738db6-metrics-certs\") pod \"network-metrics-daemon-c6t6w\" (UID: \"53188ed0-31cd-4e9a-b72d-f21a8a738db6\") " pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:16:36.031693 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.031698    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d-original-pull-secret\") pod \"global-pull-secret-syncer-w8drz\" (UID: \"d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d\") " pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:36.032482 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:36.031814    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:36.032482 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:36.031865    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:36.032482 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:36.031883    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53188ed0-31cd-4e9a-b72d-f21a8a738db6-metrics-certs podName:53188ed0-31cd-4e9a-b72d-f21a8a738db6 nodeName:}" failed. No retries permitted until 2026-09-18 16:17:08.031861729 +0000 UTC m=+67.814863748 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53188ed0-31cd-4e9a-b72d-f21a8a738db6-metrics-certs") pod "network-metrics-daemon-c6t6w" (UID: "53188ed0-31cd-4e9a-b72d-f21a8a738db6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:36.032482 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:36.031931    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d-original-pull-secret podName:d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d nodeName:}" failed. No retries permitted until 2026-09-18 16:17:08.031895728 +0000 UTC m=+67.814897743 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d-original-pull-secret") pod "global-pull-secret-syncer-w8drz" (UID: "d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:36.133142 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.132568    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7dhl\" (UniqueName: \"kubernetes.io/projected/068b8e2a-9f72-44c7-9340-783050326545-kube-api-access-l7dhl\") pod \"network-check-target-bqkx2\" (UID: \"068b8e2a-9f72-44c7-9340-783050326545\") " pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:16:36.133142 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:36.132710    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:36.133142 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:36.132727    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:36.133142 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:36.132740    2585 projected.go:196] Error preparing data for projected volume kube-api-access-l7dhl for pod openshift-network-diagnostics/network-check-target-bqkx2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:36.133142 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:36.132799    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/068b8e2a-9f72-44c7-9340-783050326545-kube-api-access-l7dhl podName:068b8e2a-9f72-44c7-9340-783050326545 nodeName:}" failed. No retries permitted until 2026-09-18 16:17:08.132779714 +0000 UTC m=+67.915781721 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-l7dhl" (UniqueName: "kubernetes.io/projected/068b8e2a-9f72-44c7-9340-783050326545-kube-api-access-l7dhl") pod "network-check-target-bqkx2" (UID: "068b8e2a-9f72-44c7-9340-783050326545") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:36.387026 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.386948    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:16:36.387186 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:36.387071    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bqkx2" podUID="068b8e2a-9f72-44c7-9340-783050326545"
Sep 18 16:16:36.532749 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.532715    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeReady"
Sep 18 16:16:36.533010 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.532846    2585 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 16:16:36.578649 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.578621    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-glsl2"]
Sep 18 16:16:36.581087 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.581064    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-glsl2"
Sep 18 16:16:36.583357 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.583336    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fxkzj"]
Sep 18 16:16:36.583999 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.583963    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 16:16:36.583999 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.583997    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f68fb\""
Sep 18 16:16:36.584168 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.584021    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 16:16:36.587712 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.587695    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fxkzj"
Sep 18 16:16:36.590967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.590448    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 16:16:36.590967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.590490    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6h8pt\""
Sep 18 16:16:36.590967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.590622    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 16:16:36.590967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.590696    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 16:16:36.593093 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.593073    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-glsl2"]
Sep 18 16:16:36.595176 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.595159    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fxkzj"]
Sep 18 16:16:36.737096 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.737072    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1157877-3544-413a-8893-f143ad8e234d-metrics-tls\") pod \"dns-default-glsl2\" (UID: \"b1157877-3544-413a-8893-f143ad8e234d\") " pod="openshift-dns/dns-default-glsl2"
Sep 18 16:16:36.737096 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.737100    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1157877-3544-413a-8893-f143ad8e234d-tmp-dir\") pod \"dns-default-glsl2\" (UID: \"b1157877-3544-413a-8893-f143ad8e234d\") " pod="openshift-dns/dns-default-glsl2"
Sep 18 16:16:36.737314 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.737121    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb9dt\" (UniqueName: \"kubernetes.io/projected/b1157877-3544-413a-8893-f143ad8e234d-kube-api-access-nb9dt\") pod \"dns-default-glsl2\" (UID: \"b1157877-3544-413a-8893-f143ad8e234d\") " pod="openshift-dns/dns-default-glsl2"
Sep 18 16:16:36.737314 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.737142    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e73cb49-af8f-408b-aaeb-db96fabe4998-cert\") pod \"ingress-canary-fxkzj\" (UID: \"4e73cb49-af8f-408b-aaeb-db96fabe4998\") " pod="openshift-ingress-canary/ingress-canary-fxkzj"
Sep 18 16:16:36.737314 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.737227    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vflpc\" (UniqueName: \"kubernetes.io/projected/4e73cb49-af8f-408b-aaeb-db96fabe4998-kube-api-access-vflpc\") pod \"ingress-canary-fxkzj\" (UID: \"4e73cb49-af8f-408b-aaeb-db96fabe4998\") " pod="openshift-ingress-canary/ingress-canary-fxkzj"
Sep 18 16:16:36.737314 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.737300    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b1157877-3544-413a-8893-f143ad8e234d-config-volume\") pod \"dns-default-glsl2\" (UID: \"b1157877-3544-413a-8893-f143ad8e234d\") " pod="openshift-dns/dns-default-glsl2"
Sep 18 16:16:36.838518 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.838492    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1157877-3544-413a-8893-f143ad8e234d-metrics-tls\") pod \"dns-default-glsl2\" (UID: \"b1157877-3544-413a-8893-f143ad8e234d\") " pod="openshift-dns/dns-default-glsl2"
Sep 18 16:16:36.838518 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.838521    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1157877-3544-413a-8893-f143ad8e234d-tmp-dir\") pod \"dns-default-glsl2\" (UID: \"b1157877-3544-413a-8893-f143ad8e234d\") " pod="openshift-dns/dns-default-glsl2"
Sep 18 16:16:36.838699 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.838545    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb9dt\" (UniqueName: \"kubernetes.io/projected/b1157877-3544-413a-8893-f143ad8e234d-kube-api-access-nb9dt\") pod \"dns-default-glsl2\" (UID: \"b1157877-3544-413a-8893-f143ad8e234d\") " pod="openshift-dns/dns-default-glsl2"
Sep 18 16:16:36.838750 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:36.838703    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:16:36.838801 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:36.838760    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1157877-3544-413a-8893-f143ad8e234d-metrics-tls podName:b1157877-3544-413a-8893-f143ad8e234d nodeName:}" failed. No retries permitted until 2026-09-18 16:16:37.338741835 +0000 UTC m=+37.121743840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1157877-3544-413a-8893-f143ad8e234d-metrics-tls") pod "dns-default-glsl2" (UID: "b1157877-3544-413a-8893-f143ad8e234d") : secret "dns-default-metrics-tls" not found
Sep 18 16:16:36.838801 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.838752    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e73cb49-af8f-408b-aaeb-db96fabe4998-cert\") pod \"ingress-canary-fxkzj\" (UID: \"4e73cb49-af8f-408b-aaeb-db96fabe4998\") " pod="openshift-ingress-canary/ingress-canary-fxkzj"
Sep 18 16:16:36.839005 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.838799    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vflpc\" (UniqueName: \"kubernetes.io/projected/4e73cb49-af8f-408b-aaeb-db96fabe4998-kube-api-access-vflpc\") pod \"ingress-canary-fxkzj\" (UID: \"4e73cb49-af8f-408b-aaeb-db96fabe4998\") " pod="openshift-ingress-canary/ingress-canary-fxkzj"
Sep 18 16:16:36.839005 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.838821    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1157877-3544-413a-8893-f143ad8e234d-tmp-dir\") pod \"dns-default-glsl2\" (UID: \"b1157877-3544-413a-8893-f143ad8e234d\") " pod="openshift-dns/dns-default-glsl2"
Sep 18 16:16:36.839005 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.838853    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b1157877-3544-413a-8893-f143ad8e234d-config-volume\") pod \"dns-default-glsl2\" (UID: \"b1157877-3544-413a-8893-f143ad8e234d\") " pod="openshift-dns/dns-default-glsl2"
Sep 18 16:16:36.839339 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.839319    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b1157877-3544-413a-8893-f143ad8e234d-config-volume\") pod \"dns-default-glsl2\" (UID: \"b1157877-3544-413a-8893-f143ad8e234d\") " pod="openshift-dns/dns-default-glsl2"
Sep 18 16:16:36.843079 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.843058    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e73cb49-af8f-408b-aaeb-db96fabe4998-cert\") pod \"ingress-canary-fxkzj\" (UID: \"4e73cb49-af8f-408b-aaeb-db96fabe4998\") " pod="openshift-ingress-canary/ingress-canary-fxkzj"
Sep 18 16:16:36.850590 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.850571    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vflpc\" (UniqueName: \"kubernetes.io/projected/4e73cb49-af8f-408b-aaeb-db96fabe4998-kube-api-access-vflpc\") pod \"ingress-canary-fxkzj\" (UID: \"4e73cb49-af8f-408b-aaeb-db96fabe4998\") " pod="openshift-ingress-canary/ingress-canary-fxkzj"
Sep 18 16:16:36.850985 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.850965    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb9dt\" (UniqueName: \"kubernetes.io/projected/b1157877-3544-413a-8893-f143ad8e234d-kube-api-access-nb9dt\") pod \"dns-default-glsl2\" (UID: \"b1157877-3544-413a-8893-f143ad8e234d\") " pod="openshift-dns/dns-default-glsl2"
Sep 18 16:16:36.901111 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:36.901079    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fxkzj"
Sep 18 16:16:37.342894 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:37.342856    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1157877-3544-413a-8893-f143ad8e234d-metrics-tls\") pod \"dns-default-glsl2\" (UID: \"b1157877-3544-413a-8893-f143ad8e234d\") " pod="openshift-dns/dns-default-glsl2"
Sep 18 16:16:37.343513 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:37.342994    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:16:37.343513 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:37.343064    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1157877-3544-413a-8893-f143ad8e234d-metrics-tls podName:b1157877-3544-413a-8893-f143ad8e234d nodeName:}" failed. No retries permitted until 2026-09-18 16:16:38.343044602 +0000 UTC m=+38.126046618 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1157877-3544-413a-8893-f143ad8e234d-metrics-tls") pod "dns-default-glsl2" (UID: "b1157877-3544-413a-8893-f143ad8e234d") : secret "dns-default-metrics-tls" not found
Sep 18 16:16:37.386893 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:37.386866    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:16:37.387048 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:37.387032    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:16:37.390039 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:37.389861    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 16:16:37.390895 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:37.390875    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 16:16:37.391035 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:37.390875    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-px2dk\""
Sep 18 16:16:38.349687 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:38.349654    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1157877-3544-413a-8893-f143ad8e234d-metrics-tls\") pod \"dns-default-glsl2\" (UID: \"b1157877-3544-413a-8893-f143ad8e234d\") " pod="openshift-dns/dns-default-glsl2"
Sep 18 16:16:38.350096 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:38.349801    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:16:38.350096 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:38.349885    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1157877-3544-413a-8893-f143ad8e234d-metrics-tls podName:b1157877-3544-413a-8893-f143ad8e234d nodeName:}" failed. No retries permitted until 2026-09-18 16:16:40.349863919 +0000 UTC m=+40.132865928 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1157877-3544-413a-8893-f143ad8e234d-metrics-tls") pod "dns-default-glsl2" (UID: "b1157877-3544-413a-8893-f143ad8e234d") : secret "dns-default-metrics-tls" not found
Sep 18 16:16:38.389937 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:38.389186    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:16:38.394349 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:38.393815    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 16:16:38.394349 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:38.393815    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fcv6z\""
Sep 18 16:16:38.397322 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:38.393826    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 16:16:38.599682 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:38.599610    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fxkzj"]
Sep 18 16:16:38.603279 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:16:38.603252    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e73cb49_af8f_408b_aaeb_db96fabe4998.slice/crio-d885555e5ffaf23e84897f45c9bba307be010c1bdb6dd7708f525f1cab56f5b8 WatchSource:0}: Error finding container d885555e5ffaf23e84897f45c9bba307be010c1bdb6dd7708f525f1cab56f5b8: Status 404 returned error can't find the container with id d885555e5ffaf23e84897f45c9bba307be010c1bdb6dd7708f525f1cab56f5b8
Sep 18 16:16:38.719889 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:38.719859    2585 generic.go:356] "Generic (PLEG): container finished" podID="70b1301b-d0c6-4fc3-aeac-661f45a32d79" containerID="050d1ced537054eb8c7eb3ed6d136ca0a6e7604ef6db9abeebbe0f23355edcff" exitCode=0
Sep 18 16:16:38.720059 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:38.719948    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-92ffq" event={"ID":"70b1301b-d0c6-4fc3-aeac-661f45a32d79","Type":"ContainerDied","Data":"050d1ced537054eb8c7eb3ed6d136ca0a6e7604ef6db9abeebbe0f23355edcff"}
Sep 18 16:16:38.721104 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:38.721085    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fxkzj" event={"ID":"4e73cb49-af8f-408b-aaeb-db96fabe4998","Type":"ContainerStarted","Data":"d885555e5ffaf23e84897f45c9bba307be010c1bdb6dd7708f525f1cab56f5b8"}
Sep 18 16:16:39.727099 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:39.727063    2585 generic.go:356] "Generic (PLEG): container finished" podID="70b1301b-d0c6-4fc3-aeac-661f45a32d79" containerID="210b281983c9e320336233615618b3104e3efbc7ea5b3c90056d20968b6d1759" exitCode=0
Sep 18 16:16:39.727581 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:39.727124    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-92ffq" event={"ID":"70b1301b-d0c6-4fc3-aeac-661f45a32d79","Type":"ContainerDied","Data":"210b281983c9e320336233615618b3104e3efbc7ea5b3c90056d20968b6d1759"}
Sep 18 16:16:40.365899 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:40.365672    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1157877-3544-413a-8893-f143ad8e234d-metrics-tls\") pod \"dns-default-glsl2\" (UID: \"b1157877-3544-413a-8893-f143ad8e234d\") " pod="openshift-dns/dns-default-glsl2"
Sep 18 16:16:40.366107 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:40.365842    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:16:40.366107 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:40.366050    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1157877-3544-413a-8893-f143ad8e234d-metrics-tls podName:b1157877-3544-413a-8893-f143ad8e234d nodeName:}" failed. No retries permitted until 2026-09-18 16:16:44.366026988 +0000 UTC m=+44.149028993 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1157877-3544-413a-8893-f143ad8e234d-metrics-tls") pod "dns-default-glsl2" (UID: "b1157877-3544-413a-8893-f143ad8e234d") : secret "dns-default-metrics-tls" not found
Sep 18 16:16:40.734093 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:40.734060    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-92ffq" event={"ID":"70b1301b-d0c6-4fc3-aeac-661f45a32d79","Type":"ContainerStarted","Data":"6bb1a5a594faa132eb173f17a4f08be45c964059b399729c95c49b1c759b9f3a"}
Sep 18 16:16:40.760738 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:40.760686    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-92ffq" podStartSLOduration=9.542801236 podStartE2EDuration="38.760664269s" totalImagesPullingTime="29.217863033s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:40.759543109 +0000 UTC m=+40.542545178" watchObservedRunningTime="2026-09-18 16:16:40.760664269 +0000 UTC m=+40.543666322"
Sep 18 16:16:41.737793 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:41.737754    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fxkzj" event={"ID":"4e73cb49-af8f-408b-aaeb-db96fabe4998","Type":"ContainerStarted","Data":"89b45928fd7d782de31b52eb3eb42bf8c4d9bae78b7af9727d4b36b31fdcbb09"}
Sep 18 16:16:41.754259 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:41.754209    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fxkzj" podStartSLOduration=3.156403611 podStartE2EDuration="5.754192335s" totalImagesPullingTime="2.597788724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:41.753778883 +0000 UTC m=+41.536780916" watchObservedRunningTime="2026-09-18 16:16:41.754192335 +0000 UTC m=+41.537194362"
Sep 18 16:16:44.393268 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:44.393233    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1157877-3544-413a-8893-f143ad8e234d-metrics-tls\") pod \"dns-default-glsl2\" (UID: \"b1157877-3544-413a-8893-f143ad8e234d\") " pod="openshift-dns/dns-default-glsl2"
Sep 18 16:16:44.393675 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:44.393340    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:16:44.393675 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:44.393389    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1157877-3544-413a-8893-f143ad8e234d-metrics-tls podName:b1157877-3544-413a-8893-f143ad8e234d nodeName:}" failed. No retries permitted until 2026-09-18 16:16:52.393376747 +0000 UTC m=+52.176378750 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1157877-3544-413a-8893-f143ad8e234d-metrics-tls") pod "dns-default-glsl2" (UID: "b1157877-3544-413a-8893-f143ad8e234d") : secret "dns-default-metrics-tls" not found
Sep 18 16:16:52.444442 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:16:52.444411    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1157877-3544-413a-8893-f143ad8e234d-metrics-tls\") pod \"dns-default-glsl2\" (UID: \"b1157877-3544-413a-8893-f143ad8e234d\") " pod="openshift-dns/dns-default-glsl2"
Sep 18 16:16:52.444804 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:52.444525    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:16:52.444804 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:16:52.444580    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1157877-3544-413a-8893-f143ad8e234d-metrics-tls podName:b1157877-3544-413a-8893-f143ad8e234d nodeName:}" failed. No retries permitted until 2026-09-18 16:17:08.444566852 +0000 UTC m=+68.227568854 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1157877-3544-413a-8893-f143ad8e234d-metrics-tls") pod "dns-default-glsl2" (UID: "b1157877-3544-413a-8893-f143ad8e234d") : secret "dns-default-metrics-tls" not found
Sep 18 16:17:01.705633 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:17:01.705599    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vx9bt"
Sep 18 16:17:08.045487 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:17:08.045436    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53188ed0-31cd-4e9a-b72d-f21a8a738db6-metrics-certs\") pod \"network-metrics-daemon-c6t6w\" (UID: \"53188ed0-31cd-4e9a-b72d-f21a8a738db6\") " pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:17:08.045487 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:17:08.045487    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d-original-pull-secret\") pod \"global-pull-secret-syncer-w8drz\" (UID: \"d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d\") " pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:17:08.049845 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:17:08.049814    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 16:17:08.049906 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:17:08.049844    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 16:17:08.056369 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:17:08.056349    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 16:17:08.056537 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:17:08.056429    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53188ed0-31cd-4e9a-b72d-f21a8a738db6-metrics-certs podName:53188ed0-31cd-4e9a-b72d-f21a8a738db6 nodeName:}" failed. No retries permitted until 2026-09-18 16:18:12.056408193 +0000 UTC m=+131.839410210 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53188ed0-31cd-4e9a-b72d-f21a8a738db6-metrics-certs") pod "network-metrics-daemon-c6t6w" (UID: "53188ed0-31cd-4e9a-b72d-f21a8a738db6") : secret "metrics-daemon-secret" not found
Sep 18 16:17:08.060121 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:17:08.060100    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d-original-pull-secret\") pod \"global-pull-secret-syncer-w8drz\" (UID: \"d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d\") " pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:17:08.146009 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:17:08.145972    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7dhl\" (UniqueName: \"kubernetes.io/projected/068b8e2a-9f72-44c7-9340-783050326545-kube-api-access-l7dhl\") pod \"network-check-target-bqkx2\" (UID: \"068b8e2a-9f72-44c7-9340-783050326545\") " pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:17:08.148797 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:17:08.148778    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 16:17:08.159417 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:17:08.159397    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 16:17:08.169364 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:17:08.169343    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7dhl\" (UniqueName: \"kubernetes.io/projected/068b8e2a-9f72-44c7-9340-783050326545-kube-api-access-l7dhl\") pod \"network-check-target-bqkx2\" (UID: \"068b8e2a-9f72-44c7-9340-783050326545\") " pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:17:08.299345 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:17:08.299277    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8drz"
Sep 18 16:17:08.408704 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:17:08.408678    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fcv6z\""
Sep 18 16:17:08.415585 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:17:08.415567    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:17:08.448452 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:17:08.448422    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1157877-3544-413a-8893-f143ad8e234d-metrics-tls\") pod \"dns-default-glsl2\" (UID: \"b1157877-3544-413a-8893-f143ad8e234d\") " pod="openshift-dns/dns-default-glsl2"
Sep 18 16:17:08.448590 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:17:08.448544    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:17:08.448633 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:17:08.448595    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1157877-3544-413a-8893-f143ad8e234d-metrics-tls podName:b1157877-3544-413a-8893-f143ad8e234d nodeName:}" failed. No retries permitted until 2026-09-18 16:17:40.448583097 +0000 UTC m=+100.231585099 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1157877-3544-413a-8893-f143ad8e234d-metrics-tls") pod "dns-default-glsl2" (UID: "b1157877-3544-413a-8893-f143ad8e234d") : secret "dns-default-metrics-tls" not found
Sep 18 16:17:08.569890 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:17:08.569770    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w8drz"]
Sep 18 16:17:08.573449 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:17:08.573425    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ab8ecb_15ee_4992_96d1_eb456a8a0d1d.slice/crio-9459044f51a530983cdfe2f4fa38c3749c44fc5143845e4ccef4529f850f8f06 WatchSource:0}: Error finding container 9459044f51a530983cdfe2f4fa38c3749c44fc5143845e4ccef4529f850f8f06: Status 404 returned error can't find the container with id 9459044f51a530983cdfe2f4fa38c3749c44fc5143845e4ccef4529f850f8f06
Sep 18 16:17:08.688569 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:17:08.688543    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bqkx2"]
Sep 18 16:17:08.691255 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:17:08.691225    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod068b8e2a_9f72_44c7_9340_783050326545.slice/crio-f37e7a353a359fc07554b8f234685c5d1489204af53a418655faa673f2ac5a97 WatchSource:0}: Error finding container f37e7a353a359fc07554b8f234685c5d1489204af53a418655faa673f2ac5a97: Status 404 returned error can't find the container with id f37e7a353a359fc07554b8f234685c5d1489204af53a418655faa673f2ac5a97
Sep 18 16:17:08.799007 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:17:08.798981    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bqkx2" event={"ID":"068b8e2a-9f72-44c7-9340-783050326545","Type":"ContainerStarted","Data":"f37e7a353a359fc07554b8f234685c5d1489204af53a418655faa673f2ac5a97"}
Sep 18 16:17:08.800044 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:17:08.800022    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w8drz" event={"ID":"d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d","Type":"ContainerStarted","Data":"9459044f51a530983cdfe2f4fa38c3749c44fc5143845e4ccef4529f850f8f06"}
Sep 18 16:17:13.815650 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:17:13.815613    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bqkx2" event={"ID":"068b8e2a-9f72-44c7-9340-783050326545","Type":"ContainerStarted","Data":"f7277573f3fc85ed10584bfeae9d245b75b3780654fd7a0af7c15091bd6aa162"}
Sep 18 16:17:13.816133 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:17:13.815709    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:17:13.817176 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:17:13.817156    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w8drz" event={"ID":"d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d","Type":"ContainerStarted","Data":"9bf76cac44eca56d91628452ffedcc1affad752a91e377e5042a34c593dda3c1"}
Sep 18 16:17:13.842337 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:17:13.842291    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bqkx2" podStartSLOduration=67.312640586 podStartE2EDuration="1m11.842279299s" totalImagesPullingTime="4.529638713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:17:13.841884297 +0000 UTC m=+73.624886320" watchObservedRunningTime="2026-09-18 16:17:13.842279299 +0000 UTC m=+73.625281322"
Sep 18 16:17:40.460319 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:17:40.460280    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1157877-3544-413a-8893-f143ad8e234d-metrics-tls\") pod \"dns-default-glsl2\" (UID: \"b1157877-3544-413a-8893-f143ad8e234d\") " pod="openshift-dns/dns-default-glsl2"
Sep 18 16:17:40.460829 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:17:40.460420    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:17:40.460829 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:17:40.460477    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1157877-3544-413a-8893-f143ad8e234d-metrics-tls podName:b1157877-3544-413a-8893-f143ad8e234d nodeName:}" failed. No retries permitted until 2026-09-18 16:18:44.460461801 +0000 UTC m=+164.243463804 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1157877-3544-413a-8893-f143ad8e234d-metrics-tls") pod "dns-default-glsl2" (UID: "b1157877-3544-413a-8893-f143ad8e234d") : secret "dns-default-metrics-tls" not found
Sep 18 16:17:44.822138 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:17:44.822104    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bqkx2"
Sep 18 16:17:44.837611 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:17:44.837563    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-w8drz" podStartSLOduration=98.18706834 podStartE2EDuration="1m42.837549973s" totalImagesPullingTime="4.650481633s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:17:13.863013468 +0000 UTC m=+73.646015493" watchObservedRunningTime="2026-09-18 16:17:44.837549973 +0000 UTC m=+104.620552019"
Sep 18 16:18:12.069171 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:18:12.069130    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53188ed0-31cd-4e9a-b72d-f21a8a738db6-metrics-certs\") pod \"network-metrics-daemon-c6t6w\" (UID: \"53188ed0-31cd-4e9a-b72d-f21a8a738db6\") " pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:18:12.069657 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:18:12.069248    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 16:18:12.069657 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:18:12.069308    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53188ed0-31cd-4e9a-b72d-f21a8a738db6-metrics-certs podName:53188ed0-31cd-4e9a-b72d-f21a8a738db6 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:14.069293201 +0000 UTC m=+253.852295204 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53188ed0-31cd-4e9a-b72d-f21a8a738db6-metrics-certs") pod "network-metrics-daemon-c6t6w" (UID: "53188ed0-31cd-4e9a-b72d-f21a8a738db6") : secret "metrics-daemon-secret" not found
Sep 18 16:18:39.594285 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:18:39.594239    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-glsl2" podUID="b1157877-3544-413a-8893-f143ad8e234d"
Sep 18 16:18:40.020079 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:18:40.020053    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-glsl2"
Sep 18 16:18:40.405264 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:18:40.405184    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-c6t6w" podUID="53188ed0-31cd-4e9a-b72d-f21a8a738db6"
Sep 18 16:18:44.468164 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:18:44.468083    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1157877-3544-413a-8893-f143ad8e234d-metrics-tls\") pod \"dns-default-glsl2\" (UID: \"b1157877-3544-413a-8893-f143ad8e234d\") " pod="openshift-dns/dns-default-glsl2"
Sep 18 16:18:44.468527 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:18:44.468227    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:18:44.468527 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:18:44.468285    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1157877-3544-413a-8893-f143ad8e234d-metrics-tls podName:b1157877-3544-413a-8893-f143ad8e234d nodeName:}" failed. No retries permitted until 2026-09-18 16:20:46.468270513 +0000 UTC m=+286.251272528 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1157877-3544-413a-8893-f143ad8e234d-metrics-tls") pod "dns-default-glsl2" (UID: "b1157877-3544-413a-8893-f143ad8e234d") : secret "dns-default-metrics-tls" not found
Sep 18 16:18:53.387612 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:18:53.387562    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:19:53.573132 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.573099    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-rsb8k"]
Sep 18 16:19:53.574934 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.574897    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-cfzmp"]
Sep 18 16:19:53.575068 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.575018    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-rsb8k"
Sep 18 16:19:53.576316 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.576297    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-674b5879-vn5cz"]
Sep 18 16:19:53.576455 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.576434    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp"
Sep 18 16:19:53.578072 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.578045    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xkd4k\""
Sep 18 16:19:53.578189 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.578054    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:19:53.578806 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.578791    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 18 16:19:53.578806 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.578792    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 18 16:19:53.578958 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.578873    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 16:19:53.579273 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.579259    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 18 16:19:53.579370 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.579355    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 18 16:19:53.579518 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.579503    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-gjgtd\""
Sep 18 16:19:53.579676 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.579661    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 18 16:19:53.579769 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.579756    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:19:53.581256 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.581232    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hlk9j\""
Sep 18 16:19:53.581743 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.581726    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 16:19:53.583666 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.583194    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 16:19:53.584882 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.584820    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 16:19:53.585234 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.585211    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 16:19:53.589499 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.589475    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 18 16:19:53.590026 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.590008    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 18 16:19:53.590729 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.590714    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 16:19:53.594754 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.594736    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-674b5879-vn5cz"]
Sep 18 16:19:53.598428 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.598404    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-cfzmp"]
Sep 18 16:19:53.599215 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.599195    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-rsb8k"]
Sep 18 16:19:53.716998 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.716972    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f505047-5eda-41d3-8df8-b8c2a31d0001-tmp\") pod \"insights-operator-6f9c644dd9-rsb8k\" (UID: \"6f505047-5eda-41d3-8df8-b8c2a31d0001\") " pod="openshift-insights/insights-operator-6f9c644dd9-rsb8k"
Sep 18 16:19:53.716998 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.717000    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjgqz\" (UniqueName: \"kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-kube-api-access-wjgqz\") pod \"image-registry-674b5879-vn5cz\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") " pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:19:53.717160 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.717018    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f505047-5eda-41d3-8df8-b8c2a31d0001-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-rsb8k\" (UID: \"6f505047-5eda-41d3-8df8-b8c2a31d0001\") " pod="openshift-insights/insights-operator-6f9c644dd9-rsb8k"
Sep 18 16:19:53.717160 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.717044    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f0cae29c-d3b9-4c3f-ac8d-d151e05916af-serving-cert\") pod \"console-operator-fd4bb46c-cfzmp\" (UID: \"f0cae29c-d3b9-4c3f-ac8d-d151e05916af\") " pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp"
Sep 18 16:19:53.717160 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.717135    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f505047-5eda-41d3-8df8-b8c2a31d0001-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-rsb8k\" (UID: \"6f505047-5eda-41d3-8df8-b8c2a31d0001\") " pod="openshift-insights/insights-operator-6f9c644dd9-rsb8k"
Sep 18 16:19:53.717255 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.717160    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f0cae29c-d3b9-4c3f-ac8d-d151e05916af-config\") pod \"console-operator-fd4bb46c-cfzmp\" (UID: \"f0cae29c-d3b9-4c3f-ac8d-d151e05916af\") " pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp"
Sep 18 16:19:53.717255 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.717178    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0cae29c-d3b9-4c3f-ac8d-d151e05916af-trusted-ca\") pod \"console-operator-fd4bb46c-cfzmp\" (UID: \"f0cae29c-d3b9-4c3f-ac8d-d151e05916af\") " pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp"
Sep 18 16:19:53.717255 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.717204    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ef1dfae-0866-44a8-ace5-9ad83a6df587-image-registry-private-configuration\") pod \"image-registry-674b5879-vn5cz\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") " pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:19:53.717255 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.717219    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ef1dfae-0866-44a8-ace5-9ad83a6df587-registry-certificates\") pod \"image-registry-674b5879-vn5cz\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") " pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:19:53.717255 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.717240    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-registry-tls\") pod \"image-registry-674b5879-vn5cz\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") " pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:19:53.717255 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.717254    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ef1dfae-0866-44a8-ace5-9ad83a6df587-ca-trust-extracted\") pod \"image-registry-674b5879-vn5cz\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") " pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:19:53.717430 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.717268    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6f505047-5eda-41d3-8df8-b8c2a31d0001-snapshots\") pod \"insights-operator-6f9c644dd9-rsb8k\" (UID: \"6f505047-5eda-41d3-8df8-b8c2a31d0001\") " pod="openshift-insights/insights-operator-6f9c644dd9-rsb8k"
Sep 18 16:19:53.717430 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.717282    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ef1dfae-0866-44a8-ace5-9ad83a6df587-trusted-ca\") pod \"image-registry-674b5879-vn5cz\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") " pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:19:53.717430 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.717319    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9zcf\" (UniqueName: \"kubernetes.io/projected/f0cae29c-d3b9-4c3f-ac8d-d151e05916af-kube-api-access-r9zcf\") pod \"console-operator-fd4bb46c-cfzmp\" (UID: \"f0cae29c-d3b9-4c3f-ac8d-d151e05916af\") " pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp"
Sep 18 16:19:53.717430 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.717362    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ef1dfae-0866-44a8-ace5-9ad83a6df587-installation-pull-secrets\") pod \"image-registry-674b5879-vn5cz\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") " pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:19:53.717430 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.717384    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-bound-sa-token\") pod \"image-registry-674b5879-vn5cz\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") " pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:19:53.717430 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.717401    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f505047-5eda-41d3-8df8-b8c2a31d0001-serving-cert\") pod \"insights-operator-6f9c644dd9-rsb8k\" (UID: \"6f505047-5eda-41d3-8df8-b8c2a31d0001\") " pod="openshift-insights/insights-operator-6f9c644dd9-rsb8k"
Sep 18 16:19:53.717430 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.717414    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2glgt\" (UniqueName: \"kubernetes.io/projected/6f505047-5eda-41d3-8df8-b8c2a31d0001-kube-api-access-2glgt\") pod \"insights-operator-6f9c644dd9-rsb8k\" (UID: \"6f505047-5eda-41d3-8df8-b8c2a31d0001\") " pod="openshift-insights/insights-operator-6f9c644dd9-rsb8k"
Sep 18 16:19:53.818640 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.818603    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f505047-5eda-41d3-8df8-b8c2a31d0001-tmp\") pod \"insights-operator-6f9c644dd9-rsb8k\" (UID: \"6f505047-5eda-41d3-8df8-b8c2a31d0001\") " pod="openshift-insights/insights-operator-6f9c644dd9-rsb8k"
Sep 18 16:19:53.818640 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.818644    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjgqz\" (UniqueName: \"kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-kube-api-access-wjgqz\") pod \"image-registry-674b5879-vn5cz\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") " pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:19:53.818814 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.818671    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f505047-5eda-41d3-8df8-b8c2a31d0001-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-rsb8k\" (UID: \"6f505047-5eda-41d3-8df8-b8c2a31d0001\") " pod="openshift-insights/insights-operator-6f9c644dd9-rsb8k"
Sep 18 16:19:53.818814 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.818694    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f0cae29c-d3b9-4c3f-ac8d-d151e05916af-serving-cert\") pod \"console-operator-fd4bb46c-cfzmp\" (UID: \"f0cae29c-d3b9-4c3f-ac8d-d151e05916af\") " pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp"
Sep 18 16:19:53.818814 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.818745    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f505047-5eda-41d3-8df8-b8c2a31d0001-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-rsb8k\" (UID: \"6f505047-5eda-41d3-8df8-b8c2a31d0001\") " pod="openshift-insights/insights-operator-6f9c644dd9-rsb8k"
Sep 18 16:19:53.818814 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.818766    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f0cae29c-d3b9-4c3f-ac8d-d151e05916af-config\") pod \"console-operator-fd4bb46c-cfzmp\" (UID: \"f0cae29c-d3b9-4c3f-ac8d-d151e05916af\") " pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp"
Sep 18 16:19:53.818814 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.818785    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0cae29c-d3b9-4c3f-ac8d-d151e05916af-trusted-ca\") pod \"console-operator-fd4bb46c-cfzmp\" (UID: \"f0cae29c-d3b9-4c3f-ac8d-d151e05916af\") " pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp"
Sep 18 16:19:53.819091 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.818819    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ef1dfae-0866-44a8-ace5-9ad83a6df587-image-registry-private-configuration\") pod \"image-registry-674b5879-vn5cz\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") " pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:19:53.819091 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.818846    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ef1dfae-0866-44a8-ace5-9ad83a6df587-registry-certificates\") pod \"image-registry-674b5879-vn5cz\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") " pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:19:53.819091 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.818880    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-registry-tls\") pod \"image-registry-674b5879-vn5cz\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") " pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:19:53.819091 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.818902    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ef1dfae-0866-44a8-ace5-9ad83a6df587-ca-trust-extracted\") pod \"image-registry-674b5879-vn5cz\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") " pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:19:53.819091 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.818956    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6f505047-5eda-41d3-8df8-b8c2a31d0001-snapshots\") pod \"insights-operator-6f9c644dd9-rsb8k\" (UID: \"6f505047-5eda-41d3-8df8-b8c2a31d0001\") " pod="openshift-insights/insights-operator-6f9c644dd9-rsb8k"
Sep 18 16:19:53.819091 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.818980    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ef1dfae-0866-44a8-ace5-9ad83a6df587-trusted-ca\") pod \"image-registry-674b5879-vn5cz\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") " pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:19:53.819091 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.819005    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9zcf\" (UniqueName: \"kubernetes.io/projected/f0cae29c-d3b9-4c3f-ac8d-d151e05916af-kube-api-access-r9zcf\") pod \"console-operator-fd4bb46c-cfzmp\" (UID: \"f0cae29c-d3b9-4c3f-ac8d-d151e05916af\") " pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp"
Sep 18 16:19:53.819091 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.819038    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ef1dfae-0866-44a8-ace5-9ad83a6df587-installation-pull-secrets\") pod \"image-registry-674b5879-vn5cz\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") " pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:19:53.819091 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.819060    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-bound-sa-token\") pod \"image-registry-674b5879-vn5cz\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") " pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:19:53.819091 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.819089    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f505047-5eda-41d3-8df8-b8c2a31d0001-serving-cert\") pod \"insights-operator-6f9c644dd9-rsb8k\" (UID: \"6f505047-5eda-41d3-8df8-b8c2a31d0001\") " pod="openshift-insights/insights-operator-6f9c644dd9-rsb8k"
Sep 18 16:19:53.819546 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.819108    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f505047-5eda-41d3-8df8-b8c2a31d0001-tmp\") pod \"insights-operator-6f9c644dd9-rsb8k\" (UID: \"6f505047-5eda-41d3-8df8-b8c2a31d0001\") " pod="openshift-insights/insights-operator-6f9c644dd9-rsb8k"
Sep 18 16:19:53.819546 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.819113    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2glgt\" (UniqueName: \"kubernetes.io/projected/6f505047-5eda-41d3-8df8-b8c2a31d0001-kube-api-access-2glgt\") pod \"insights-operator-6f9c644dd9-rsb8k\" (UID: \"6f505047-5eda-41d3-8df8-b8c2a31d0001\") " pod="openshift-insights/insights-operator-6f9c644dd9-rsb8k"
Sep 18 16:19:53.819546 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:19:53.819315    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 16:19:53.819546 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:19:53.819334    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-674b5879-vn5cz: secret "image-registry-tls" not found
Sep 18 16:19:53.819546 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.819472    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ef1dfae-0866-44a8-ace5-9ad83a6df587-registry-certificates\") pod \"image-registry-674b5879-vn5cz\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") " pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:19:53.819546 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:19:53.819517    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-registry-tls podName:8ef1dfae-0866-44a8-ace5-9ad83a6df587 nodeName:}" failed. No retries permitted until 2026-09-18 16:19:54.31949689 +0000 UTC m=+234.102498892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-registry-tls") pod "image-registry-674b5879-vn5cz" (UID: "8ef1dfae-0866-44a8-ace5-9ad83a6df587") : secret "image-registry-tls" not found
Sep 18 16:19:53.819853 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.819629    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f0cae29c-d3b9-4c3f-ac8d-d151e05916af-config\") pod \"console-operator-fd4bb46c-cfzmp\" (UID: \"f0cae29c-d3b9-4c3f-ac8d-d151e05916af\") " pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp"
Sep 18 16:19:53.819853 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.819739    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0cae29c-d3b9-4c3f-ac8d-d151e05916af-trusted-ca\") pod \"console-operator-fd4bb46c-cfzmp\" (UID: \"f0cae29c-d3b9-4c3f-ac8d-d151e05916af\") " pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp"
Sep 18 16:19:53.819853 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.819835    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f505047-5eda-41d3-8df8-b8c2a31d0001-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-rsb8k\" (UID: \"6f505047-5eda-41d3-8df8-b8c2a31d0001\") " pod="openshift-insights/insights-operator-6f9c644dd9-rsb8k"
Sep 18 16:19:53.820027 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.819960    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ef1dfae-0866-44a8-ace5-9ad83a6df587-ca-trust-extracted\") pod \"image-registry-674b5879-vn5cz\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") " pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:19:53.820027 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.819963    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ef1dfae-0866-44a8-ace5-9ad83a6df587-trusted-ca\") pod \"image-registry-674b5879-vn5cz\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") " pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:19:53.820358 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.820307    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f505047-5eda-41d3-8df8-b8c2a31d0001-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-rsb8k\" (UID: \"6f505047-5eda-41d3-8df8-b8c2a31d0001\") " pod="openshift-insights/insights-operator-6f9c644dd9-rsb8k"
Sep 18 16:19:53.820571 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.820554    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6f505047-5eda-41d3-8df8-b8c2a31d0001-snapshots\") pod \"insights-operator-6f9c644dd9-rsb8k\" (UID: \"6f505047-5eda-41d3-8df8-b8c2a31d0001\") " pod="openshift-insights/insights-operator-6f9c644dd9-rsb8k"
Sep 18 16:19:53.821819 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.821801    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f0cae29c-d3b9-4c3f-ac8d-d151e05916af-serving-cert\") pod \"console-operator-fd4bb46c-cfzmp\" (UID: \"f0cae29c-d3b9-4c3f-ac8d-d151e05916af\") " pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp"
Sep 18 16:19:53.822134 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.822110    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ef1dfae-0866-44a8-ace5-9ad83a6df587-installation-pull-secrets\") pod \"image-registry-674b5879-vn5cz\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") " pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:19:53.822241 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.822197    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f505047-5eda-41d3-8df8-b8c2a31d0001-serving-cert\") pod \"insights-operator-6f9c644dd9-rsb8k\" (UID: \"6f505047-5eda-41d3-8df8-b8c2a31d0001\") " pod="openshift-insights/insights-operator-6f9c644dd9-rsb8k"
Sep 18 16:19:53.822312 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.822293    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ef1dfae-0866-44a8-ace5-9ad83a6df587-image-registry-private-configuration\") pod \"image-registry-674b5879-vn5cz\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") " pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:19:53.829673 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.829613    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-bound-sa-token\") pod \"image-registry-674b5879-vn5cz\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") " pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:19:53.830017 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.829996    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjgqz\" (UniqueName: \"kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-kube-api-access-wjgqz\") pod \"image-registry-674b5879-vn5cz\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") " pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:19:53.830068 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.830000    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2glgt\" (UniqueName: \"kubernetes.io/projected/6f505047-5eda-41d3-8df8-b8c2a31d0001-kube-api-access-2glgt\") pod \"insights-operator-6f9c644dd9-rsb8k\" (UID: \"6f505047-5eda-41d3-8df8-b8c2a31d0001\") " pod="openshift-insights/insights-operator-6f9c644dd9-rsb8k"
Sep 18 16:19:53.830746 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.830731    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9zcf\" (UniqueName: \"kubernetes.io/projected/f0cae29c-d3b9-4c3f-ac8d-d151e05916af-kube-api-access-r9zcf\") pod \"console-operator-fd4bb46c-cfzmp\" (UID: \"f0cae29c-d3b9-4c3f-ac8d-d151e05916af\") " pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp"
Sep 18 16:19:53.886753 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.886732    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-rsb8k"
Sep 18 16:19:53.893165 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:53.893144    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp"
Sep 18 16:19:54.160130 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:54.160067    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-rsb8k"]
Sep 18 16:19:54.162690 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:19:54.162663    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f505047_5eda_41d3_8df8_b8c2a31d0001.slice/crio-449a2420cc2dd2d990142f66320257db994f2255edd552d5a9bec3ebc0df42a5 WatchSource:0}: Error finding container 449a2420cc2dd2d990142f66320257db994f2255edd552d5a9bec3ebc0df42a5: Status 404 returned error can't find the container with id 449a2420cc2dd2d990142f66320257db994f2255edd552d5a9bec3ebc0df42a5
Sep 18 16:19:54.165837 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:54.165812    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-cfzmp"]
Sep 18 16:19:54.168712 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:19:54.168686    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0cae29c_d3b9_4c3f_ac8d_d151e05916af.slice/crio-2d02d85738446f721567e9fd5855bb55687f8f21108a52a2ef0961a00d87d95f WatchSource:0}: Error finding container 2d02d85738446f721567e9fd5855bb55687f8f21108a52a2ef0961a00d87d95f: Status 404 returned error can't find the container with id 2d02d85738446f721567e9fd5855bb55687f8f21108a52a2ef0961a00d87d95f
Sep 18 16:19:54.197604 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:54.197572    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-rsb8k" event={"ID":"6f505047-5eda-41d3-8df8-b8c2a31d0001","Type":"ContainerStarted","Data":"449a2420cc2dd2d990142f66320257db994f2255edd552d5a9bec3ebc0df42a5"}
Sep 18 16:19:54.198658 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:54.198638    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp" event={"ID":"f0cae29c-d3b9-4c3f-ac8d-d151e05916af","Type":"ContainerStarted","Data":"2d02d85738446f721567e9fd5855bb55687f8f21108a52a2ef0961a00d87d95f"}
Sep 18 16:19:54.323046 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:54.323022    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-registry-tls\") pod \"image-registry-674b5879-vn5cz\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") " pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:19:54.323138 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:19:54.323124    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 16:19:54.323181 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:19:54.323143    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-674b5879-vn5cz: secret "image-registry-tls" not found
Sep 18 16:19:54.323213 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:19:54.323192    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-registry-tls podName:8ef1dfae-0866-44a8-ace5-9ad83a6df587 nodeName:}" failed. No retries permitted until 2026-09-18 16:19:55.323177702 +0000 UTC m=+235.106179705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-registry-tls") pod "image-registry-674b5879-vn5cz" (UID: "8ef1dfae-0866-44a8-ace5-9ad83a6df587") : secret "image-registry-tls" not found
Sep 18 16:19:55.330967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:55.330934    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-registry-tls\") pod \"image-registry-674b5879-vn5cz\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") " pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:19:55.331400 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:19:55.331108    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 16:19:55.331400 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:19:55.331129    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-674b5879-vn5cz: secret "image-registry-tls" not found
Sep 18 16:19:55.331400 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:19:55.331198    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-registry-tls podName:8ef1dfae-0866-44a8-ace5-9ad83a6df587 nodeName:}" failed. No retries permitted until 2026-09-18 16:19:57.331176282 +0000 UTC m=+237.114178301 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-registry-tls") pod "image-registry-674b5879-vn5cz" (UID: "8ef1dfae-0866-44a8-ace5-9ad83a6df587") : secret "image-registry-tls" not found
Sep 18 16:19:57.209600 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:57.209568    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/0.log"
Sep 18 16:19:57.210057 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:57.209613    2585 generic.go:356] "Generic (PLEG): container finished" podID="f0cae29c-d3b9-4c3f-ac8d-d151e05916af" containerID="a1a75284cbfb90dfa014dd760dae2ee42add18e4930702b05f8dc677122e5bdd" exitCode=255
Sep 18 16:19:57.210057 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:57.209721    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp" event={"ID":"f0cae29c-d3b9-4c3f-ac8d-d151e05916af","Type":"ContainerDied","Data":"a1a75284cbfb90dfa014dd760dae2ee42add18e4930702b05f8dc677122e5bdd"}
Sep 18 16:19:57.210057 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:57.209958    2585 scope.go:117] "RemoveContainer" containerID="a1a75284cbfb90dfa014dd760dae2ee42add18e4930702b05f8dc677122e5bdd"
Sep 18 16:19:57.211256 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:57.211231    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-rsb8k" event={"ID":"6f505047-5eda-41d3-8df8-b8c2a31d0001","Type":"ContainerStarted","Data":"22e01b2bd06e6cca4e210b8ae3cc30d22baaf520b7a13b5df3038c87ac69bbe5"}
Sep 18 16:19:57.246291 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:57.246251    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-rsb8k" podStartSLOduration=1.792813505 podStartE2EDuration="4.246236611s" totalImagesPullingTime="2.453423106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:57.24548457 +0000 UTC m=+237.028486595" watchObservedRunningTime="2026-09-18 16:19:57.246236611 +0000 UTC m=+237.029238651"
Sep 18 16:19:57.345829 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:57.345797    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-registry-tls\") pod \"image-registry-674b5879-vn5cz\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") " pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:19:57.346000 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:19:57.345976    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 16:19:57.346000 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:19:57.346003    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-674b5879-vn5cz: secret "image-registry-tls" not found
Sep 18 16:19:57.346139 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:19:57.346060    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-registry-tls podName:8ef1dfae-0866-44a8-ace5-9ad83a6df587 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:01.346041481 +0000 UTC m=+241.129043488 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-registry-tls") pod "image-registry-674b5879-vn5cz" (UID: "8ef1dfae-0866-44a8-ace5-9ad83a6df587") : secret "image-registry-tls" not found
Sep 18 16:19:57.643677 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:57.643648    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gvzrz"]
Sep 18 16:19:57.645656 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:57.645633    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gvzrz"
Sep 18 16:19:57.648173 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:57.648147    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 16:19:57.648315 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:57.648298    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 16:19:57.648384 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:57.648304    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rpr2n\""
Sep 18 16:19:57.660660 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:57.660639    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gvzrz"]
Sep 18 16:19:57.748362 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:57.748329    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-crio-socket\") pod \"insights-runtime-extractor-gvzrz\" (UID: \"eb3be0d0-05e3-4f51-8bd0-1cff06299f5a\") " pod="openshift-insights/insights-runtime-extractor-gvzrz"
Sep 18 16:19:57.748509 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:57.748364    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwttz\" (UniqueName: \"kubernetes.io/projected/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-kube-api-access-zwttz\") pod \"insights-runtime-extractor-gvzrz\" (UID: \"eb3be0d0-05e3-4f51-8bd0-1cff06299f5a\") " pod="openshift-insights/insights-runtime-extractor-gvzrz"
Sep 18 16:19:57.748509 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:57.748436    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-data-volume\") pod \"insights-runtime-extractor-gvzrz\" (UID: \"eb3be0d0-05e3-4f51-8bd0-1cff06299f5a\") " pod="openshift-insights/insights-runtime-extractor-gvzrz"
Sep 18 16:19:57.748509 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:57.748468    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gvzrz\" (UID: \"eb3be0d0-05e3-4f51-8bd0-1cff06299f5a\") " pod="openshift-insights/insights-runtime-extractor-gvzrz"
Sep 18 16:19:57.748615 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:57.748569    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gvzrz\" (UID: \"eb3be0d0-05e3-4f51-8bd0-1cff06299f5a\") " pod="openshift-insights/insights-runtime-extractor-gvzrz"
Sep 18 16:19:57.849214 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:57.849185    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gvzrz\" (UID: \"eb3be0d0-05e3-4f51-8bd0-1cff06299f5a\") " pod="openshift-insights/insights-runtime-extractor-gvzrz"
Sep 18 16:19:57.849357 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:57.849229    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-crio-socket\") pod \"insights-runtime-extractor-gvzrz\" (UID: \"eb3be0d0-05e3-4f51-8bd0-1cff06299f5a\") " pod="openshift-insights/insights-runtime-extractor-gvzrz"
Sep 18 16:19:57.849357 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:57.849247    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwttz\" (UniqueName: \"kubernetes.io/projected/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-kube-api-access-zwttz\") pod \"insights-runtime-extractor-gvzrz\" (UID: \"eb3be0d0-05e3-4f51-8bd0-1cff06299f5a\") " pod="openshift-insights/insights-runtime-extractor-gvzrz"
Sep 18 16:19:57.849357 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:57.849284    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-data-volume\") pod \"insights-runtime-extractor-gvzrz\" (UID: \"eb3be0d0-05e3-4f51-8bd0-1cff06299f5a\") " pod="openshift-insights/insights-runtime-extractor-gvzrz"
Sep 18 16:19:57.849357 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:19:57.849304    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:19:57.849357 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:57.849328    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-crio-socket\") pod \"insights-runtime-extractor-gvzrz\" (UID: \"eb3be0d0-05e3-4f51-8bd0-1cff06299f5a\") " pod="openshift-insights/insights-runtime-extractor-gvzrz"
Sep 18 16:19:57.849357 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:57.849308    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gvzrz\" (UID: \"eb3be0d0-05e3-4f51-8bd0-1cff06299f5a\") " pod="openshift-insights/insights-runtime-extractor-gvzrz"
Sep 18 16:19:57.849627 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:19:57.849377    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-kube-rbac-proxy-cm podName:eb3be0d0-05e3-4f51-8bd0-1cff06299f5a nodeName:}" failed. No retries permitted until 2026-09-18 16:19:58.349361854 +0000 UTC m=+238.132363856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gvzrz" (UID: "eb3be0d0-05e3-4f51-8bd0-1cff06299f5a") : configmap "kube-rbac-proxy" not found
Sep 18 16:19:57.849627 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:19:57.849387    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:19:57.849627 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:19:57.849440    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-insights-runtime-extractor-tls podName:eb3be0d0-05e3-4f51-8bd0-1cff06299f5a nodeName:}" failed. No retries permitted until 2026-09-18 16:19:58.349424188 +0000 UTC m=+238.132426204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gvzrz" (UID: "eb3be0d0-05e3-4f51-8bd0-1cff06299f5a") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:19:57.849627 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:57.849601    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-data-volume\") pod \"insights-runtime-extractor-gvzrz\" (UID: \"eb3be0d0-05e3-4f51-8bd0-1cff06299f5a\") " pod="openshift-insights/insights-runtime-extractor-gvzrz"
Sep 18 16:19:57.858639 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:57.858608    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwttz\" (UniqueName: \"kubernetes.io/projected/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-kube-api-access-zwttz\") pod \"insights-runtime-extractor-gvzrz\" (UID: \"eb3be0d0-05e3-4f51-8bd0-1cff06299f5a\") " pod="openshift-insights/insights-runtime-extractor-gvzrz"
Sep 18 16:19:58.216353 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:58.216322    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/1.log"
Sep 18 16:19:58.216722 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:58.216708    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/0.log"
Sep 18 16:19:58.216766 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:58.216744    2585 generic.go:356] "Generic (PLEG): container finished" podID="f0cae29c-d3b9-4c3f-ac8d-d151e05916af" containerID="e6ec259a1da05e643f035b116d41f7941e492d100b81ae135da5d7e34f6babad" exitCode=255
Sep 18 16:19:58.216845 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:58.216824    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp" event={"ID":"f0cae29c-d3b9-4c3f-ac8d-d151e05916af","Type":"ContainerDied","Data":"e6ec259a1da05e643f035b116d41f7941e492d100b81ae135da5d7e34f6babad"}
Sep 18 16:19:58.216890 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:58.216879    2585 scope.go:117] "RemoveContainer" containerID="a1a75284cbfb90dfa014dd760dae2ee42add18e4930702b05f8dc677122e5bdd"
Sep 18 16:19:58.217109 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:58.217093    2585 scope.go:117] "RemoveContainer" containerID="e6ec259a1da05e643f035b116d41f7941e492d100b81ae135da5d7e34f6babad"
Sep 18 16:19:58.217277 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:19:58.217259    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-cfzmp_openshift-console-operator(f0cae29c-d3b9-4c3f-ac8d-d151e05916af)\"" pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp" podUID="f0cae29c-d3b9-4c3f-ac8d-d151e05916af"
Sep 18 16:19:58.353188 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:58.353160    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gvzrz\" (UID: \"eb3be0d0-05e3-4f51-8bd0-1cff06299f5a\") " pod="openshift-insights/insights-runtime-extractor-gvzrz"
Sep 18 16:19:58.353464 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:58.353420    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gvzrz\" (UID: \"eb3be0d0-05e3-4f51-8bd0-1cff06299f5a\") " pod="openshift-insights/insights-runtime-extractor-gvzrz"
Sep 18 16:19:58.353542 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:19:58.353431    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:19:58.353600 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:19:58.353533    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:19:58.353600 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:19:58.353588    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-insights-runtime-extractor-tls podName:eb3be0d0-05e3-4f51-8bd0-1cff06299f5a nodeName:}" failed. No retries permitted until 2026-09-18 16:19:59.353569175 +0000 UTC m=+239.136571177 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gvzrz" (UID: "eb3be0d0-05e3-4f51-8bd0-1cff06299f5a") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:19:58.353702 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:19:58.353607    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-kube-rbac-proxy-cm podName:eb3be0d0-05e3-4f51-8bd0-1cff06299f5a nodeName:}" failed. No retries permitted until 2026-09-18 16:19:59.353599233 +0000 UTC m=+239.136601237 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gvzrz" (UID: "eb3be0d0-05e3-4f51-8bd0-1cff06299f5a") : configmap "kube-rbac-proxy" not found
Sep 18 16:19:59.221619 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:59.221591    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/1.log"
Sep 18 16:19:59.222229 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:59.222024    2585 scope.go:117] "RemoveContainer" containerID="e6ec259a1da05e643f035b116d41f7941e492d100b81ae135da5d7e34f6babad"
Sep 18 16:19:59.222300 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:19:59.222215    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-cfzmp_openshift-console-operator(f0cae29c-d3b9-4c3f-ac8d-d151e05916af)\"" pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp" podUID="f0cae29c-d3b9-4c3f-ac8d-d151e05916af"
Sep 18 16:19:59.360492 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:59.360458    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gvzrz\" (UID: \"eb3be0d0-05e3-4f51-8bd0-1cff06299f5a\") " pod="openshift-insights/insights-runtime-extractor-gvzrz"
Sep 18 16:19:59.360649 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:59.360521    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gvzrz\" (UID: \"eb3be0d0-05e3-4f51-8bd0-1cff06299f5a\") " pod="openshift-insights/insights-runtime-extractor-gvzrz"
Sep 18 16:19:59.360649 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:19:59.360599    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:19:59.360649 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:19:59.360638    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:19:59.360820 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:19:59.360671    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-insights-runtime-extractor-tls podName:eb3be0d0-05e3-4f51-8bd0-1cff06299f5a nodeName:}" failed. No retries permitted until 2026-09-18 16:20:01.360651475 +0000 UTC m=+241.143653483 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gvzrz" (UID: "eb3be0d0-05e3-4f51-8bd0-1cff06299f5a") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:19:59.360820 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:19:59.360688    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-kube-rbac-proxy-cm podName:eb3be0d0-05e3-4f51-8bd0-1cff06299f5a nodeName:}" failed. No retries permitted until 2026-09-18 16:20:01.360680066 +0000 UTC m=+241.143682068 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gvzrz" (UID: "eb3be0d0-05e3-4f51-8bd0-1cff06299f5a") : configmap "kube-rbac-proxy" not found
Sep 18 16:19:59.685063 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:59.685030    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-kpvvg"]
Sep 18 16:19:59.687541 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:59.687518    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-kpvvg"
Sep 18 16:19:59.690391 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:59.690364    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 18 16:19:59.690499 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:59.690395    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 18 16:19:59.690499 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:59.690434    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 18 16:19:59.690499 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:59.690458    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 18 16:19:59.690673 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:59.690588    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-b2wjf\""
Sep 18 16:19:59.695582 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:59.695558    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-kpvvg"]
Sep 18 16:19:59.812775 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:59.812755    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q9svc_895c0864-ed92-4326-b78f-3a85240437c2/dns-node-resolver/0.log"
Sep 18 16:19:59.864196 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:59.864159    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/af7ba602-1387-437c-946e-7a264d9f1f59-signing-cabundle\") pod \"service-ca-55f6fc7b86-kpvvg\" (UID: \"af7ba602-1387-437c-946e-7a264d9f1f59\") " pod="openshift-service-ca/service-ca-55f6fc7b86-kpvvg"
Sep 18 16:19:59.864314 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:59.864242    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tbfd\" (UniqueName: \"kubernetes.io/projected/af7ba602-1387-437c-946e-7a264d9f1f59-kube-api-access-5tbfd\") pod \"service-ca-55f6fc7b86-kpvvg\" (UID: \"af7ba602-1387-437c-946e-7a264d9f1f59\") " pod="openshift-service-ca/service-ca-55f6fc7b86-kpvvg"
Sep 18 16:19:59.864314 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:59.864262    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/af7ba602-1387-437c-946e-7a264d9f1f59-signing-key\") pod \"service-ca-55f6fc7b86-kpvvg\" (UID: \"af7ba602-1387-437c-946e-7a264d9f1f59\") " pod="openshift-service-ca/service-ca-55f6fc7b86-kpvvg"
Sep 18 16:19:59.964697 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:59.964620    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/af7ba602-1387-437c-946e-7a264d9f1f59-signing-cabundle\") pod \"service-ca-55f6fc7b86-kpvvg\" (UID: \"af7ba602-1387-437c-946e-7a264d9f1f59\") " pod="openshift-service-ca/service-ca-55f6fc7b86-kpvvg"
Sep 18 16:19:59.964829 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:59.964714    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tbfd\" (UniqueName: \"kubernetes.io/projected/af7ba602-1387-437c-946e-7a264d9f1f59-kube-api-access-5tbfd\") pod \"service-ca-55f6fc7b86-kpvvg\" (UID: \"af7ba602-1387-437c-946e-7a264d9f1f59\") " pod="openshift-service-ca/service-ca-55f6fc7b86-kpvvg"
Sep 18 16:19:59.964829 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:59.964742    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/af7ba602-1387-437c-946e-7a264d9f1f59-signing-key\") pod \"service-ca-55f6fc7b86-kpvvg\" (UID: \"af7ba602-1387-437c-946e-7a264d9f1f59\") " pod="openshift-service-ca/service-ca-55f6fc7b86-kpvvg"
Sep 18 16:19:59.965252 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:59.965232    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/af7ba602-1387-437c-946e-7a264d9f1f59-signing-cabundle\") pod \"service-ca-55f6fc7b86-kpvvg\" (UID: \"af7ba602-1387-437c-946e-7a264d9f1f59\") " pod="openshift-service-ca/service-ca-55f6fc7b86-kpvvg"
Sep 18 16:19:59.967202 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:59.967181    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/af7ba602-1387-437c-946e-7a264d9f1f59-signing-key\") pod \"service-ca-55f6fc7b86-kpvvg\" (UID: \"af7ba602-1387-437c-946e-7a264d9f1f59\") " pod="openshift-service-ca/service-ca-55f6fc7b86-kpvvg"
Sep 18 16:19:59.973265 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:59.973247    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tbfd\" (UniqueName: \"kubernetes.io/projected/af7ba602-1387-437c-946e-7a264d9f1f59-kube-api-access-5tbfd\") pod \"service-ca-55f6fc7b86-kpvvg\" (UID: \"af7ba602-1387-437c-946e-7a264d9f1f59\") " pod="openshift-service-ca/service-ca-55f6fc7b86-kpvvg"
Sep 18 16:19:59.998696 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:19:59.998677    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-kpvvg"
Sep 18 16:20:00.265067 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:00.265035    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-kpvvg"]
Sep 18 16:20:00.269280 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:20:00.269258    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf7ba602_1387_437c_946e_7a264d9f1f59.slice/crio-c90cc757f6ac8049b50821da72a0e7136b02758d888f7bc5e21dac527cd5c5cc WatchSource:0}: Error finding container c90cc757f6ac8049b50821da72a0e7136b02758d888f7bc5e21dac527cd5c5cc: Status 404 returned error can't find the container with id c90cc757f6ac8049b50821da72a0e7136b02758d888f7bc5e21dac527cd5c5cc
Sep 18 16:20:00.613017 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:00.612950    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lh2tk_d8e4139d-8ca0-4a9c-92b1-7a6998c5988b/node-ca/0.log"
Sep 18 16:20:01.229929 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:01.229878    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-kpvvg" event={"ID":"af7ba602-1387-437c-946e-7a264d9f1f59","Type":"ContainerStarted","Data":"c90cc757f6ac8049b50821da72a0e7136b02758d888f7bc5e21dac527cd5c5cc"}
Sep 18 16:20:01.375061 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:01.374999    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gvzrz\" (UID: \"eb3be0d0-05e3-4f51-8bd0-1cff06299f5a\") " pod="openshift-insights/insights-runtime-extractor-gvzrz"
Sep 18 16:20:01.375061 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:01.375042    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-registry-tls\") pod \"image-registry-674b5879-vn5cz\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") " pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:20:01.375699 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:01.375089    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gvzrz\" (UID: \"eb3be0d0-05e3-4f51-8bd0-1cff06299f5a\") " pod="openshift-insights/insights-runtime-extractor-gvzrz"
Sep 18 16:20:01.375699 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:20:01.375133    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:20:01.375699 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:20:01.375190    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 16:20:01.375699 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:20:01.375210    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-674b5879-vn5cz: secret "image-registry-tls" not found
Sep 18 16:20:01.375699 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:20:01.375193    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:01.375699 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:20:01.375209    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-kube-rbac-proxy-cm podName:eb3be0d0-05e3-4f51-8bd0-1cff06299f5a nodeName:}" failed. No retries permitted until 2026-09-18 16:20:05.375183611 +0000 UTC m=+245.158185635 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gvzrz" (UID: "eb3be0d0-05e3-4f51-8bd0-1cff06299f5a") : configmap "kube-rbac-proxy" not found
Sep 18 16:20:01.375699 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:20:01.375298    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-registry-tls podName:8ef1dfae-0866-44a8-ace5-9ad83a6df587 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:09.375280132 +0000 UTC m=+249.158282138 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-registry-tls") pod "image-registry-674b5879-vn5cz" (UID: "8ef1dfae-0866-44a8-ace5-9ad83a6df587") : secret "image-registry-tls" not found
Sep 18 16:20:01.375699 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:20:01.375315    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-insights-runtime-extractor-tls podName:eb3be0d0-05e3-4f51-8bd0-1cff06299f5a nodeName:}" failed. No retries permitted until 2026-09-18 16:20:05.375303883 +0000 UTC m=+245.158305888 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gvzrz" (UID: "eb3be0d0-05e3-4f51-8bd0-1cff06299f5a") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:01.613808 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:01.613725    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fxkzj_4e73cb49-af8f-408b-aaeb-db96fabe4998/serve-healthcheck-canary/0.log"
Sep 18 16:20:02.238604 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:02.238566    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-kpvvg" event={"ID":"af7ba602-1387-437c-946e-7a264d9f1f59","Type":"ContainerStarted","Data":"33e8da0cbfccd9661c6a20793e850feeebd03bb6adbd95d8669bdeb50b4bcd4b"}
Sep 18 16:20:02.256715 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:02.256674    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-kpvvg" podStartSLOduration=1.3825065140000001 podStartE2EDuration="3.256660367s" totalImagesPullingTime="1.874153853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:02.255732573 +0000 UTC m=+242.038734596" watchObservedRunningTime="2026-09-18 16:20:02.256660367 +0000 UTC m=+242.039662406"
Sep 18 16:20:03.893783 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:03.893746    2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp"
Sep 18 16:20:03.894255 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:03.893799    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp"
Sep 18 16:20:03.894255 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:03.894103    2585 scope.go:117] "RemoveContainer" containerID="e6ec259a1da05e643f035b116d41f7941e492d100b81ae135da5d7e34f6babad"
Sep 18 16:20:03.894362 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:20:03.894250    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-cfzmp_openshift-console-operator(f0cae29c-d3b9-4c3f-ac8d-d151e05916af)\"" pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp" podUID="f0cae29c-d3b9-4c3f-ac8d-d151e05916af"
Sep 18 16:20:05.403954 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:05.403923    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gvzrz\" (UID: \"eb3be0d0-05e3-4f51-8bd0-1cff06299f5a\") " pod="openshift-insights/insights-runtime-extractor-gvzrz"
Sep 18 16:20:05.404394 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:05.403979    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gvzrz\" (UID: \"eb3be0d0-05e3-4f51-8bd0-1cff06299f5a\") " pod="openshift-insights/insights-runtime-extractor-gvzrz"
Sep 18 16:20:05.404394 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:20:05.404075    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:20:05.404394 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:20:05.404102    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:05.404394 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:20:05.404127    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-kube-rbac-proxy-cm podName:eb3be0d0-05e3-4f51-8bd0-1cff06299f5a nodeName:}" failed. No retries permitted until 2026-09-18 16:20:13.404112673 +0000 UTC m=+253.187114675 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gvzrz" (UID: "eb3be0d0-05e3-4f51-8bd0-1cff06299f5a") : configmap "kube-rbac-proxy" not found
Sep 18 16:20:05.404394 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:20:05.404169    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-insights-runtime-extractor-tls podName:eb3be0d0-05e3-4f51-8bd0-1cff06299f5a nodeName:}" failed. No retries permitted until 2026-09-18 16:20:13.404150341 +0000 UTC m=+253.187152344 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gvzrz" (UID: "eb3be0d0-05e3-4f51-8bd0-1cff06299f5a") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:09.434290 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:09.434255    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-registry-tls\") pod \"image-registry-674b5879-vn5cz\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") " pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:20:09.436817 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:09.436798    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-registry-tls\") pod \"image-registry-674b5879-vn5cz\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") " pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:20:09.501247 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:09.501221    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hlk9j\""
Sep 18 16:20:09.508670 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:09.508652    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:20:09.789858 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:09.789721    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-674b5879-vn5cz"]
Sep 18 16:20:09.792206 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:20:09.792182    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ef1dfae_0866_44a8_ace5_9ad83a6df587.slice/crio-0849fa8fd629ab7e7c7d964f618f0d8ee830f5379c10081bf5e5ab058318747e WatchSource:0}: Error finding container 0849fa8fd629ab7e7c7d964f618f0d8ee830f5379c10081bf5e5ab058318747e: Status 404 returned error can't find the container with id 0849fa8fd629ab7e7c7d964f618f0d8ee830f5379c10081bf5e5ab058318747e
Sep 18 16:20:10.264079 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:10.264052    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-674b5879-vn5cz" event={"ID":"8ef1dfae-0866-44a8-ace5-9ad83a6df587","Type":"ContainerStarted","Data":"54f19968f897713e6b1b386c437905e5d1955f1590963aab76bdebbeec77c301"}
Sep 18 16:20:10.264273 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:10.264099    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-674b5879-vn5cz" event={"ID":"8ef1dfae-0866-44a8-ace5-9ad83a6df587","Type":"ContainerStarted","Data":"0849fa8fd629ab7e7c7d964f618f0d8ee830f5379c10081bf5e5ab058318747e"}
Sep 18 16:20:10.264273 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:10.264209    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:20:10.291400 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:10.291356    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-674b5879-vn5cz" podStartSLOduration=17.291343335 podStartE2EDuration="17.291343335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:10.29013069 +0000 UTC m=+250.073132715" watchObservedRunningTime="2026-09-18 16:20:10.291343335 +0000 UTC m=+250.074345355"
Sep 18 16:20:13.462457 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:13.462373    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gvzrz\" (UID: \"eb3be0d0-05e3-4f51-8bd0-1cff06299f5a\") " pod="openshift-insights/insights-runtime-extractor-gvzrz"
Sep 18 16:20:13.462457 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:13.462431    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gvzrz\" (UID: \"eb3be0d0-05e3-4f51-8bd0-1cff06299f5a\") " pod="openshift-insights/insights-runtime-extractor-gvzrz"
Sep 18 16:20:13.462881 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:13.462861    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gvzrz\" (UID: \"eb3be0d0-05e3-4f51-8bd0-1cff06299f5a\") " pod="openshift-insights/insights-runtime-extractor-gvzrz"
Sep 18 16:20:13.464907 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:13.464890    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb3be0d0-05e3-4f51-8bd0-1cff06299f5a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gvzrz\" (UID: \"eb3be0d0-05e3-4f51-8bd0-1cff06299f5a\") " pod="openshift-insights/insights-runtime-extractor-gvzrz"
Sep 18 16:20:13.557103 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:13.557078    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rpr2n\""
Sep 18 16:20:13.565078 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:13.565056    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gvzrz"
Sep 18 16:20:13.842544 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:13.842514    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gvzrz"]
Sep 18 16:20:13.845693 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:20:13.845673    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb3be0d0_05e3_4f51_8bd0_1cff06299f5a.slice/crio-df28d321b98320040896e1775d2515a2eaccbcccabe0023dc5d7b488b1c9062d WatchSource:0}: Error finding container df28d321b98320040896e1775d2515a2eaccbcccabe0023dc5d7b488b1c9062d: Status 404 returned error can't find the container with id df28d321b98320040896e1775d2515a2eaccbcccabe0023dc5d7b488b1c9062d
Sep 18 16:20:14.166887 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:14.166815    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53188ed0-31cd-4e9a-b72d-f21a8a738db6-metrics-certs\") pod \"network-metrics-daemon-c6t6w\" (UID: \"53188ed0-31cd-4e9a-b72d-f21a8a738db6\") " pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:20:14.169110 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:14.169087    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53188ed0-31cd-4e9a-b72d-f21a8a738db6-metrics-certs\") pod \"network-metrics-daemon-c6t6w\" (UID: \"53188ed0-31cd-4e9a-b72d-f21a8a738db6\") " pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:20:14.282073 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:14.282045    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gvzrz" event={"ID":"eb3be0d0-05e3-4f51-8bd0-1cff06299f5a","Type":"ContainerStarted","Data":"a34c0c0eaab3a1c2b66b8b2c10f9ff7b470b1e5bb3e4f2ade099ad3b5bc77bb1"}
Sep 18 16:20:14.282183 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:14.282103    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gvzrz" event={"ID":"eb3be0d0-05e3-4f51-8bd0-1cff06299f5a","Type":"ContainerStarted","Data":"df28d321b98320040896e1775d2515a2eaccbcccabe0023dc5d7b488b1c9062d"}
Sep 18 16:20:14.391227 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:14.391197    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-px2dk\""
Sep 18 16:20:14.398512 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:14.398487    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c6t6w"
Sep 18 16:20:14.737696 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:14.737609    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c6t6w"]
Sep 18 16:20:14.741365 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:20:14.741338    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53188ed0_31cd_4e9a_b72d_f21a8a738db6.slice/crio-5ccbbd2850b401b767a2c01f8bc9d7f77b7d2ae1c4fbd257ff71dd2e3b941ce6 WatchSource:0}: Error finding container 5ccbbd2850b401b767a2c01f8bc9d7f77b7d2ae1c4fbd257ff71dd2e3b941ce6: Status 404 returned error can't find the container with id 5ccbbd2850b401b767a2c01f8bc9d7f77b7d2ae1c4fbd257ff71dd2e3b941ce6
Sep 18 16:20:15.287576 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:15.287544    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gvzrz" event={"ID":"eb3be0d0-05e3-4f51-8bd0-1cff06299f5a","Type":"ContainerStarted","Data":"d56032743c955ccd813679073388a39951fa6e0ff557eedcd3ea3b3419f498bd"}
Sep 18 16:20:15.288791 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:15.288759    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c6t6w" event={"ID":"53188ed0-31cd-4e9a-b72d-f21a8a738db6","Type":"ContainerStarted","Data":"5ccbbd2850b401b767a2c01f8bc9d7f77b7d2ae1c4fbd257ff71dd2e3b941ce6"}
Sep 18 16:20:17.296448 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:17.296407    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gvzrz" event={"ID":"eb3be0d0-05e3-4f51-8bd0-1cff06299f5a","Type":"ContainerStarted","Data":"5bdd277d031d7df9145253ac8d289c7c2cd60c4d2c5cbdf58ac1cd88c0432871"}
Sep 18 16:20:17.298254 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:17.298231    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c6t6w" event={"ID":"53188ed0-31cd-4e9a-b72d-f21a8a738db6","Type":"ContainerStarted","Data":"21c6cfbcb95c2e2bb7cabc148af5f9bb3e2480a9b95ef16d9512e31c97f85f82"}
Sep 18 16:20:17.298375 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:17.298270    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c6t6w" event={"ID":"53188ed0-31cd-4e9a-b72d-f21a8a738db6","Type":"ContainerStarted","Data":"c254010796841525c9d81438ce7f1a32d0be56211947c43a2c22708d225c5a87"}
Sep 18 16:20:17.316861 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:17.316811    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gvzrz" podStartSLOduration=17.812808272 podStartE2EDuration="20.316796711s" totalImagesPullingTime="2.503988439s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:19:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:17.315897309 +0000 UTC m=+257.098899332" watchObservedRunningTime="2026-09-18 16:20:17.316796711 +0000 UTC m=+257.099798736"
Sep 18 16:20:17.333284 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:17.333247    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-c6t6w" podStartSLOduration=253.623302899 podStartE2EDuration="4m15.333236621s" totalImagesPullingTime="1.709933722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:17.332433733 +0000 UTC m=+257.115435757" watchObservedRunningTime="2026-09-18 16:20:17.333236621 +0000 UTC m=+257.116238704"
Sep 18 16:20:17.387148 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:17.387129    2585 scope.go:117] "RemoveContainer" containerID="e6ec259a1da05e643f035b116d41f7941e492d100b81ae135da5d7e34f6babad"
Sep 18 16:20:18.302779 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:18.302751    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 16:20:18.303201 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:18.303158    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/1.log"
Sep 18 16:20:18.303201 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:18.303193    2585 generic.go:356] "Generic (PLEG): container finished" podID="f0cae29c-d3b9-4c3f-ac8d-d151e05916af" containerID="377c1e564c19a6500ce5ef9418fd3767db76b59b5425aac5d38de0f2d0050198" exitCode=255
Sep 18 16:20:18.303292 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:18.303269    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp" event={"ID":"f0cae29c-d3b9-4c3f-ac8d-d151e05916af","Type":"ContainerDied","Data":"377c1e564c19a6500ce5ef9418fd3767db76b59b5425aac5d38de0f2d0050198"}
Sep 18 16:20:18.303356 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:18.303335    2585 scope.go:117] "RemoveContainer" containerID="e6ec259a1da05e643f035b116d41f7941e492d100b81ae135da5d7e34f6babad"
Sep 18 16:20:18.303811 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:18.303786    2585 scope.go:117] "RemoveContainer" containerID="377c1e564c19a6500ce5ef9418fd3767db76b59b5425aac5d38de0f2d0050198"
Sep 18 16:20:18.303994 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:20:18.303975    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-cfzmp_openshift-console-operator(f0cae29c-d3b9-4c3f-ac8d-d151e05916af)\"" pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp" podUID="f0cae29c-d3b9-4c3f-ac8d-d151e05916af"
Sep 18 16:20:19.310038 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:19.310013    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 16:20:23.572288 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.572259    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-674b5879-vn5cz"]
Sep 18 16:20:23.622677 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.622647    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5876889fd5-2dfl4"]
Sep 18 16:20:23.627099 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.627082    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5876889fd5-2dfl4"
Sep 18 16:20:23.645414 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.645383    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5876889fd5-2dfl4"]
Sep 18 16:20:23.729689 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.729654    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4adc28b1-0f25-4090-9014-7ca2c46f57cf-installation-pull-secrets\") pod \"image-registry-5876889fd5-2dfl4\" (UID: \"4adc28b1-0f25-4090-9014-7ca2c46f57cf\") " pod="openshift-image-registry/image-registry-5876889fd5-2dfl4"
Sep 18 16:20:23.729689 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.729690    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4adc28b1-0f25-4090-9014-7ca2c46f57cf-ca-trust-extracted\") pod \"image-registry-5876889fd5-2dfl4\" (UID: \"4adc28b1-0f25-4090-9014-7ca2c46f57cf\") " pod="openshift-image-registry/image-registry-5876889fd5-2dfl4"
Sep 18 16:20:23.729878 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.729709    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4adc28b1-0f25-4090-9014-7ca2c46f57cf-trusted-ca\") pod \"image-registry-5876889fd5-2dfl4\" (UID: \"4adc28b1-0f25-4090-9014-7ca2c46f57cf\") " pod="openshift-image-registry/image-registry-5876889fd5-2dfl4"
Sep 18 16:20:23.729878 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.729791    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4adc28b1-0f25-4090-9014-7ca2c46f57cf-image-registry-private-configuration\") pod \"image-registry-5876889fd5-2dfl4\" (UID: \"4adc28b1-0f25-4090-9014-7ca2c46f57cf\") " pod="openshift-image-registry/image-registry-5876889fd5-2dfl4"
Sep 18 16:20:23.729878 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.729840    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4adc28b1-0f25-4090-9014-7ca2c46f57cf-bound-sa-token\") pod \"image-registry-5876889fd5-2dfl4\" (UID: \"4adc28b1-0f25-4090-9014-7ca2c46f57cf\") " pod="openshift-image-registry/image-registry-5876889fd5-2dfl4"
Sep 18 16:20:23.729878 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.729859    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4adc28b1-0f25-4090-9014-7ca2c46f57cf-registry-tls\") pod \"image-registry-5876889fd5-2dfl4\" (UID: \"4adc28b1-0f25-4090-9014-7ca2c46f57cf\") " pod="openshift-image-registry/image-registry-5876889fd5-2dfl4"
Sep 18 16:20:23.730029 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.729901    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztdgq\" (UniqueName: \"kubernetes.io/projected/4adc28b1-0f25-4090-9014-7ca2c46f57cf-kube-api-access-ztdgq\") pod \"image-registry-5876889fd5-2dfl4\" (UID: \"4adc28b1-0f25-4090-9014-7ca2c46f57cf\") " pod="openshift-image-registry/image-registry-5876889fd5-2dfl4"
Sep 18 16:20:23.730029 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.729953    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4adc28b1-0f25-4090-9014-7ca2c46f57cf-registry-certificates\") pod \"image-registry-5876889fd5-2dfl4\" (UID: \"4adc28b1-0f25-4090-9014-7ca2c46f57cf\") " pod="openshift-image-registry/image-registry-5876889fd5-2dfl4"
Sep 18 16:20:23.831157 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.831094    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztdgq\" (UniqueName: \"kubernetes.io/projected/4adc28b1-0f25-4090-9014-7ca2c46f57cf-kube-api-access-ztdgq\") pod \"image-registry-5876889fd5-2dfl4\" (UID: \"4adc28b1-0f25-4090-9014-7ca2c46f57cf\") " pod="openshift-image-registry/image-registry-5876889fd5-2dfl4"
Sep 18 16:20:23.831157 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.831127    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4adc28b1-0f25-4090-9014-7ca2c46f57cf-registry-certificates\") pod \"image-registry-5876889fd5-2dfl4\" (UID: \"4adc28b1-0f25-4090-9014-7ca2c46f57cf\") " pod="openshift-image-registry/image-registry-5876889fd5-2dfl4"
Sep 18 16:20:23.831332 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.831158    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4adc28b1-0f25-4090-9014-7ca2c46f57cf-installation-pull-secrets\") pod \"image-registry-5876889fd5-2dfl4\" (UID: \"4adc28b1-0f25-4090-9014-7ca2c46f57cf\") " pod="openshift-image-registry/image-registry-5876889fd5-2dfl4"
Sep 18 16:20:23.831332 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.831186    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4adc28b1-0f25-4090-9014-7ca2c46f57cf-ca-trust-extracted\") pod \"image-registry-5876889fd5-2dfl4\" (UID: \"4adc28b1-0f25-4090-9014-7ca2c46f57cf\") " pod="openshift-image-registry/image-registry-5876889fd5-2dfl4"
Sep 18 16:20:23.831332 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.831210    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4adc28b1-0f25-4090-9014-7ca2c46f57cf-trusted-ca\") pod \"image-registry-5876889fd5-2dfl4\" (UID: \"4adc28b1-0f25-4090-9014-7ca2c46f57cf\") " pod="openshift-image-registry/image-registry-5876889fd5-2dfl4"
Sep 18 16:20:23.831332 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.831237    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4adc28b1-0f25-4090-9014-7ca2c46f57cf-image-registry-private-configuration\") pod \"image-registry-5876889fd5-2dfl4\" (UID: \"4adc28b1-0f25-4090-9014-7ca2c46f57cf\") " pod="openshift-image-registry/image-registry-5876889fd5-2dfl4"
Sep 18 16:20:23.831332 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.831275    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4adc28b1-0f25-4090-9014-7ca2c46f57cf-bound-sa-token\") pod \"image-registry-5876889fd5-2dfl4\" (UID: \"4adc28b1-0f25-4090-9014-7ca2c46f57cf\") " pod="openshift-image-registry/image-registry-5876889fd5-2dfl4"
Sep 18 16:20:23.831332 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.831297    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4adc28b1-0f25-4090-9014-7ca2c46f57cf-registry-tls\") pod \"image-registry-5876889fd5-2dfl4\" (UID: \"4adc28b1-0f25-4090-9014-7ca2c46f57cf\") " pod="openshift-image-registry/image-registry-5876889fd5-2dfl4"
Sep 18 16:20:23.831787 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.831758    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4adc28b1-0f25-4090-9014-7ca2c46f57cf-ca-trust-extracted\") pod \"image-registry-5876889fd5-2dfl4\" (UID: \"4adc28b1-0f25-4090-9014-7ca2c46f57cf\") " pod="openshift-image-registry/image-registry-5876889fd5-2dfl4"
Sep 18 16:20:23.832155 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.832131    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4adc28b1-0f25-4090-9014-7ca2c46f57cf-trusted-ca\") pod \"image-registry-5876889fd5-2dfl4\" (UID: \"4adc28b1-0f25-4090-9014-7ca2c46f57cf\") " pod="openshift-image-registry/image-registry-5876889fd5-2dfl4"
Sep 18 16:20:23.832220 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.832153    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4adc28b1-0f25-4090-9014-7ca2c46f57cf-registry-certificates\") pod \"image-registry-5876889fd5-2dfl4\" (UID: \"4adc28b1-0f25-4090-9014-7ca2c46f57cf\") " pod="openshift-image-registry/image-registry-5876889fd5-2dfl4"
Sep 18 16:20:23.833789 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.833766    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4adc28b1-0f25-4090-9014-7ca2c46f57cf-image-registry-private-configuration\") pod \"image-registry-5876889fd5-2dfl4\" (UID: \"4adc28b1-0f25-4090-9014-7ca2c46f57cf\") " pod="openshift-image-registry/image-registry-5876889fd5-2dfl4"
Sep 18 16:20:23.834190 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.834174    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4adc28b1-0f25-4090-9014-7ca2c46f57cf-registry-tls\") pod \"image-registry-5876889fd5-2dfl4\" (UID: \"4adc28b1-0f25-4090-9014-7ca2c46f57cf\") " pod="openshift-image-registry/image-registry-5876889fd5-2dfl4"
Sep 18 16:20:23.834259 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.834217    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4adc28b1-0f25-4090-9014-7ca2c46f57cf-installation-pull-secrets\") pod \"image-registry-5876889fd5-2dfl4\" (UID: \"4adc28b1-0f25-4090-9014-7ca2c46f57cf\") " pod="openshift-image-registry/image-registry-5876889fd5-2dfl4"
Sep 18 16:20:23.853428 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.853407    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztdgq\" (UniqueName: \"kubernetes.io/projected/4adc28b1-0f25-4090-9014-7ca2c46f57cf-kube-api-access-ztdgq\") pod \"image-registry-5876889fd5-2dfl4\" (UID: \"4adc28b1-0f25-4090-9014-7ca2c46f57cf\") " pod="openshift-image-registry/image-registry-5876889fd5-2dfl4"
Sep 18 16:20:23.853706 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.853685    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4adc28b1-0f25-4090-9014-7ca2c46f57cf-bound-sa-token\") pod \"image-registry-5876889fd5-2dfl4\" (UID: \"4adc28b1-0f25-4090-9014-7ca2c46f57cf\") " pod="openshift-image-registry/image-registry-5876889fd5-2dfl4"
Sep 18 16:20:23.893577 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.893556    2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp"
Sep 18 16:20:23.893683 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.893623    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp"
Sep 18 16:20:23.893879 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.893866    2585 scope.go:117] "RemoveContainer" containerID="377c1e564c19a6500ce5ef9418fd3767db76b59b5425aac5d38de0f2d0050198"
Sep 18 16:20:23.894045 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:20:23.894027    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-cfzmp_openshift-console-operator(f0cae29c-d3b9-4c3f-ac8d-d151e05916af)\"" pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp" podUID="f0cae29c-d3b9-4c3f-ac8d-d151e05916af"
Sep 18 16:20:23.935388 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:23.935366    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5876889fd5-2dfl4"
Sep 18 16:20:24.207555 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:24.207475    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5876889fd5-2dfl4"]
Sep 18 16:20:24.210214 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:20:24.210187    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4adc28b1_0f25_4090_9014_7ca2c46f57cf.slice/crio-6d9a1968b2aa15dbca5a6f35c8abf4499a4a0798eaedc5e0494001543353d8f9 WatchSource:0}: Error finding container 6d9a1968b2aa15dbca5a6f35c8abf4499a4a0798eaedc5e0494001543353d8f9: Status 404 returned error can't find the container with id 6d9a1968b2aa15dbca5a6f35c8abf4499a4a0798eaedc5e0494001543353d8f9
Sep 18 16:20:24.327382 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:24.327343    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5876889fd5-2dfl4" event={"ID":"4adc28b1-0f25-4090-9014-7ca2c46f57cf","Type":"ContainerStarted","Data":"f7079a60a65d66b062204693ec42dde99af7bb044925974db1e695edfbac12b9"}
Sep 18 16:20:24.327513 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:24.327411    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5876889fd5-2dfl4" event={"ID":"4adc28b1-0f25-4090-9014-7ca2c46f57cf","Type":"ContainerStarted","Data":"6d9a1968b2aa15dbca5a6f35c8abf4499a4a0798eaedc5e0494001543353d8f9"}
Sep 18 16:20:24.327648 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:24.327631    2585 scope.go:117] "RemoveContainer" containerID="377c1e564c19a6500ce5ef9418fd3767db76b59b5425aac5d38de0f2d0050198"
Sep 18 16:20:24.327807 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:20:24.327790    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-cfzmp_openshift-console-operator(f0cae29c-d3b9-4c3f-ac8d-d151e05916af)\"" pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp" podUID="f0cae29c-d3b9-4c3f-ac8d-d151e05916af"
Sep 18 16:20:24.355371 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:24.355330    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5876889fd5-2dfl4" podStartSLOduration=1.35531531 podStartE2EDuration="1.35531531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:24.354659862 +0000 UTC m=+264.137661888" watchObservedRunningTime="2026-09-18 16:20:24.35531531 +0000 UTC m=+264.138317333"
Sep 18 16:20:25.331241 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:25.331205    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5876889fd5-2dfl4"
Sep 18 16:20:33.578058 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:33.578026    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:20:35.734731 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.734694    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-strvg"]
Sep 18 16:20:35.740028 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.740009    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-strvg"
Sep 18 16:20:35.742805 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.742778    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 18 16:20:35.742934 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.742878    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 16:20:35.742934 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.742895    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 18 16:20:35.743869 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.743852    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 16:20:35.745352 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.745335    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 16:20:35.746205 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.746185    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-q4v6p\""
Sep 18 16:20:35.754576 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.754552    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-strvg"]
Sep 18 16:20:35.775202 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.775184    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5gwsw"]
Sep 18 16:20:35.778184 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.778170    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5gwsw"
Sep 18 16:20:35.782250 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.782234    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 16:20:35.782372 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.782242    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 16:20:35.782433 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.782287    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-57lgg\""
Sep 18 16:20:35.782496 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.782477    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 16:20:35.811808 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.811787    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a2a7a976-02e8-47ac-8d0e-b081f8ceb225-node-exporter-textfile\") pod \"node-exporter-5gwsw\" (UID: \"a2a7a976-02e8-47ac-8d0e-b081f8ceb225\") " pod="openshift-monitoring/node-exporter-5gwsw"
Sep 18 16:20:35.811955 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.811817    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a2a7a976-02e8-47ac-8d0e-b081f8ceb225-node-exporter-accelerators-collector-config\") pod \"node-exporter-5gwsw\" (UID: \"a2a7a976-02e8-47ac-8d0e-b081f8ceb225\") " pod="openshift-monitoring/node-exporter-5gwsw"
Sep 18 16:20:35.811955 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.811837    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2a7a976-02e8-47ac-8d0e-b081f8ceb225-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5gwsw\" (UID: \"a2a7a976-02e8-47ac-8d0e-b081f8ceb225\") " pod="openshift-monitoring/node-exporter-5gwsw"
Sep 18 16:20:35.811955 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.811862    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2vk5\" (UniqueName: \"kubernetes.io/projected/a2a7a976-02e8-47ac-8d0e-b081f8ceb225-kube-api-access-q2vk5\") pod \"node-exporter-5gwsw\" (UID: \"a2a7a976-02e8-47ac-8d0e-b081f8ceb225\") " pod="openshift-monitoring/node-exporter-5gwsw"
Sep 18 16:20:35.811955 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.811892    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a2a7a976-02e8-47ac-8d0e-b081f8ceb225-node-exporter-tls\") pod \"node-exporter-5gwsw\" (UID: \"a2a7a976-02e8-47ac-8d0e-b081f8ceb225\") " pod="openshift-monitoring/node-exporter-5gwsw"
Sep 18 16:20:35.812158 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.811965    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2a7a976-02e8-47ac-8d0e-b081f8ceb225-sys\") pod \"node-exporter-5gwsw\" (UID: \"a2a7a976-02e8-47ac-8d0e-b081f8ceb225\") " pod="openshift-monitoring/node-exporter-5gwsw"
Sep 18 16:20:35.812158 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.811986    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/96d83cf1-2b3e-46e0-9deb-a666cc9fa27d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-strvg\" (UID: \"96d83cf1-2b3e-46e0-9deb-a666cc9fa27d\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-strvg"
Sep 18 16:20:35.812158 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.812013    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2a7a976-02e8-47ac-8d0e-b081f8ceb225-metrics-client-ca\") pod \"node-exporter-5gwsw\" (UID: \"a2a7a976-02e8-47ac-8d0e-b081f8ceb225\") " pod="openshift-monitoring/node-exporter-5gwsw"
Sep 18 16:20:35.812158 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.812029    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a2a7a976-02e8-47ac-8d0e-b081f8ceb225-root\") pod \"node-exporter-5gwsw\" (UID: \"a2a7a976-02e8-47ac-8d0e-b081f8ceb225\") " pod="openshift-monitoring/node-exporter-5gwsw"
Sep 18 16:20:35.812158 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.812119    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/96d83cf1-2b3e-46e0-9deb-a666cc9fa27d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-strvg\" (UID: \"96d83cf1-2b3e-46e0-9deb-a666cc9fa27d\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-strvg"
Sep 18 16:20:35.812351 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.812157    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96d83cf1-2b3e-46e0-9deb-a666cc9fa27d-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-strvg\" (UID: \"96d83cf1-2b3e-46e0-9deb-a666cc9fa27d\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-strvg"
Sep 18 16:20:35.812351 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.812185    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a2a7a976-02e8-47ac-8d0e-b081f8ceb225-node-exporter-wtmp\") pod \"node-exporter-5gwsw\" (UID: \"a2a7a976-02e8-47ac-8d0e-b081f8ceb225\") " pod="openshift-monitoring/node-exporter-5gwsw"
Sep 18 16:20:35.812351 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.812212    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtd6q\" (UniqueName: \"kubernetes.io/projected/96d83cf1-2b3e-46e0-9deb-a666cc9fa27d-kube-api-access-xtd6q\") pod \"openshift-state-metrics-86c5bc79d6-strvg\" (UID: \"96d83cf1-2b3e-46e0-9deb-a666cc9fa27d\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-strvg"
Sep 18 16:20:35.913310 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.913274    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2a7a976-02e8-47ac-8d0e-b081f8ceb225-metrics-client-ca\") pod \"node-exporter-5gwsw\" (UID: \"a2a7a976-02e8-47ac-8d0e-b081f8ceb225\") " pod="openshift-monitoring/node-exporter-5gwsw"
Sep 18 16:20:35.913456 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.913318    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a2a7a976-02e8-47ac-8d0e-b081f8ceb225-root\") pod \"node-exporter-5gwsw\" (UID: \"a2a7a976-02e8-47ac-8d0e-b081f8ceb225\") " pod="openshift-monitoring/node-exporter-5gwsw"
Sep 18 16:20:35.913456 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.913387    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/96d83cf1-2b3e-46e0-9deb-a666cc9fa27d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-strvg\" (UID: \"96d83cf1-2b3e-46e0-9deb-a666cc9fa27d\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-strvg"
Sep 18 16:20:35.913456 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.913421    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a2a7a976-02e8-47ac-8d0e-b081f8ceb225-root\") pod \"node-exporter-5gwsw\" (UID: \"a2a7a976-02e8-47ac-8d0e-b081f8ceb225\") " pod="openshift-monitoring/node-exporter-5gwsw"
Sep 18 16:20:35.913456 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.913424    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96d83cf1-2b3e-46e0-9deb-a666cc9fa27d-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-strvg\" (UID: \"96d83cf1-2b3e-46e0-9deb-a666cc9fa27d\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-strvg"
Sep 18 16:20:35.913630 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.913466    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a2a7a976-02e8-47ac-8d0e-b081f8ceb225-node-exporter-wtmp\") pod \"node-exporter-5gwsw\" (UID: \"a2a7a976-02e8-47ac-8d0e-b081f8ceb225\") " pod="openshift-monitoring/node-exporter-5gwsw"
Sep 18 16:20:35.913630 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.913496    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtd6q\" (UniqueName: \"kubernetes.io/projected/96d83cf1-2b3e-46e0-9deb-a666cc9fa27d-kube-api-access-xtd6q\") pod \"openshift-state-metrics-86c5bc79d6-strvg\" (UID: \"96d83cf1-2b3e-46e0-9deb-a666cc9fa27d\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-strvg"
Sep 18 16:20:35.913630 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:20:35.913523    2585 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 18 16:20:35.913630 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:20:35.913591    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96d83cf1-2b3e-46e0-9deb-a666cc9fa27d-openshift-state-metrics-tls podName:96d83cf1-2b3e-46e0-9deb-a666cc9fa27d nodeName:}" failed. No retries permitted until 2026-09-18 16:20:36.413571132 +0000 UTC m=+276.196573148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/96d83cf1-2b3e-46e0-9deb-a666cc9fa27d-openshift-state-metrics-tls") pod "openshift-state-metrics-86c5bc79d6-strvg" (UID: "96d83cf1-2b3e-46e0-9deb-a666cc9fa27d") : secret "openshift-state-metrics-tls" not found
Sep 18 16:20:35.913850 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.913529    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a2a7a976-02e8-47ac-8d0e-b081f8ceb225-node-exporter-textfile\") pod \"node-exporter-5gwsw\" (UID: \"a2a7a976-02e8-47ac-8d0e-b081f8ceb225\") " pod="openshift-monitoring/node-exporter-5gwsw"
Sep 18 16:20:35.913964 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.913879    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a2a7a976-02e8-47ac-8d0e-b081f8ceb225-node-exporter-accelerators-collector-config\") pod \"node-exporter-5gwsw\" (UID: \"a2a7a976-02e8-47ac-8d0e-b081f8ceb225\") " pod="openshift-monitoring/node-exporter-5gwsw"
Sep 18 16:20:35.913964 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.913898    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a2a7a976-02e8-47ac-8d0e-b081f8ceb225-node-exporter-textfile\") pod \"node-exporter-5gwsw\" (UID: \"a2a7a976-02e8-47ac-8d0e-b081f8ceb225\") " pod="openshift-monitoring/node-exporter-5gwsw"
Sep 18 16:20:35.913964 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.913934    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2a7a976-02e8-47ac-8d0e-b081f8ceb225-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5gwsw\" (UID: \"a2a7a976-02e8-47ac-8d0e-b081f8ceb225\") " pod="openshift-monitoring/node-exporter-5gwsw"
Sep 18 16:20:35.914114 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.913986    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2vk5\" (UniqueName: \"kubernetes.io/projected/a2a7a976-02e8-47ac-8d0e-b081f8ceb225-kube-api-access-q2vk5\") pod \"node-exporter-5gwsw\" (UID: \"a2a7a976-02e8-47ac-8d0e-b081f8ceb225\") " pod="openshift-monitoring/node-exporter-5gwsw"
Sep 18 16:20:35.914114 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.914014    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a2a7a976-02e8-47ac-8d0e-b081f8ceb225-node-exporter-tls\") pod \"node-exporter-5gwsw\" (UID: \"a2a7a976-02e8-47ac-8d0e-b081f8ceb225\") " pod="openshift-monitoring/node-exporter-5gwsw"
Sep 18 16:20:35.914114 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.914068    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2a7a976-02e8-47ac-8d0e-b081f8ceb225-sys\") pod \"node-exporter-5gwsw\" (UID: \"a2a7a976-02e8-47ac-8d0e-b081f8ceb225\") " pod="openshift-monitoring/node-exporter-5gwsw"
Sep 18 16:20:35.914114 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.914089    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a2a7a976-02e8-47ac-8d0e-b081f8ceb225-node-exporter-wtmp\") pod \"node-exporter-5gwsw\" (UID: \"a2a7a976-02e8-47ac-8d0e-b081f8ceb225\") " pod="openshift-monitoring/node-exporter-5gwsw"
Sep 18 16:20:35.914114 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.914096    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/96d83cf1-2b3e-46e0-9deb-a666cc9fa27d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-strvg\" (UID: \"96d83cf1-2b3e-46e0-9deb-a666cc9fa27d\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-strvg"
Sep 18 16:20:35.914340 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.914143    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2a7a976-02e8-47ac-8d0e-b081f8ceb225-metrics-client-ca\") pod \"node-exporter-5gwsw\" (UID: \"a2a7a976-02e8-47ac-8d0e-b081f8ceb225\") " pod="openshift-monitoring/node-exporter-5gwsw"
Sep 18 16:20:35.914340 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.914191    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2a7a976-02e8-47ac-8d0e-b081f8ceb225-sys\") pod \"node-exporter-5gwsw\" (UID: \"a2a7a976-02e8-47ac-8d0e-b081f8ceb225\") " pod="openshift-monitoring/node-exporter-5gwsw"
Sep 18 16:20:35.914635 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.914614    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96d83cf1-2b3e-46e0-9deb-a666cc9fa27d-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-strvg\" (UID: \"96d83cf1-2b3e-46e0-9deb-a666cc9fa27d\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-strvg"
Sep 18 16:20:35.915056 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.914810    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a2a7a976-02e8-47ac-8d0e-b081f8ceb225-node-exporter-accelerators-collector-config\") pod \"node-exporter-5gwsw\" (UID: \"a2a7a976-02e8-47ac-8d0e-b081f8ceb225\") " pod="openshift-monitoring/node-exporter-5gwsw"
Sep 18 16:20:35.917177 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.917136    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/96d83cf1-2b3e-46e0-9deb-a666cc9fa27d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-strvg\" (UID: \"96d83cf1-2b3e-46e0-9deb-a666cc9fa27d\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-strvg"
Sep 18 16:20:35.917390 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.917372    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a2a7a976-02e8-47ac-8d0e-b081f8ceb225-node-exporter-tls\") pod \"node-exporter-5gwsw\" (UID: \"a2a7a976-02e8-47ac-8d0e-b081f8ceb225\") " pod="openshift-monitoring/node-exporter-5gwsw"
Sep 18 16:20:35.917756 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.917734    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2a7a976-02e8-47ac-8d0e-b081f8ceb225-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5gwsw\" (UID: \"a2a7a976-02e8-47ac-8d0e-b081f8ceb225\") " pod="openshift-monitoring/node-exporter-5gwsw"
Sep 18 16:20:35.925629 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.925567    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2vk5\" (UniqueName: \"kubernetes.io/projected/a2a7a976-02e8-47ac-8d0e-b081f8ceb225-kube-api-access-q2vk5\") pod \"node-exporter-5gwsw\" (UID: \"a2a7a976-02e8-47ac-8d0e-b081f8ceb225\") " pod="openshift-monitoring/node-exporter-5gwsw"
Sep 18 16:20:35.926538 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:35.926518    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtd6q\" (UniqueName: \"kubernetes.io/projected/96d83cf1-2b3e-46e0-9deb-a666cc9fa27d-kube-api-access-xtd6q\") pod \"openshift-state-metrics-86c5bc79d6-strvg\" (UID: \"96d83cf1-2b3e-46e0-9deb-a666cc9fa27d\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-strvg"
Sep 18 16:20:36.086780 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:36.086705    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5gwsw"
Sep 18 16:20:36.097997 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:20:36.097972    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a7a976_02e8_47ac_8d0e_b081f8ceb225.slice/crio-b6bcb78ff344a554042817995f8f4a31ebcc5f4c764feb0551e34ed57094da87 WatchSource:0}: Error finding container b6bcb78ff344a554042817995f8f4a31ebcc5f4c764feb0551e34ed57094da87: Status 404 returned error can't find the container with id b6bcb78ff344a554042817995f8f4a31ebcc5f4c764feb0551e34ed57094da87
Sep 18 16:20:36.366949 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:36.366846    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5gwsw" event={"ID":"a2a7a976-02e8-47ac-8d0e-b081f8ceb225","Type":"ContainerStarted","Data":"b6bcb78ff344a554042817995f8f4a31ebcc5f4c764feb0551e34ed57094da87"}
Sep 18 16:20:36.387392 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:36.387370    2585 scope.go:117] "RemoveContainer" containerID="377c1e564c19a6500ce5ef9418fd3767db76b59b5425aac5d38de0f2d0050198"
Sep 18 16:20:36.387526 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:20:36.387514    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-cfzmp_openshift-console-operator(f0cae29c-d3b9-4c3f-ac8d-d151e05916af)\"" pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp" podUID="f0cae29c-d3b9-4c3f-ac8d-d151e05916af"
Sep 18 16:20:36.417850 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:36.417821    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/96d83cf1-2b3e-46e0-9deb-a666cc9fa27d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-strvg\" (UID: \"96d83cf1-2b3e-46e0-9deb-a666cc9fa27d\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-strvg"
Sep 18 16:20:36.420322 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:36.420294    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/96d83cf1-2b3e-46e0-9deb-a666cc9fa27d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-strvg\" (UID: \"96d83cf1-2b3e-46e0-9deb-a666cc9fa27d\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-strvg"
Sep 18 16:20:36.651565 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:36.651491    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-strvg"
Sep 18 16:20:36.924664 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:36.924542    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-strvg"]
Sep 18 16:20:36.927996 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:20:36.927958    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d83cf1_2b3e_46e0_9deb_a666cc9fa27d.slice/crio-72dee2b28f2e5011aa647b6b08dee6233c8fc8b606e6be3179eb5d7b257bfc39 WatchSource:0}: Error finding container 72dee2b28f2e5011aa647b6b08dee6233c8fc8b606e6be3179eb5d7b257bfc39: Status 404 returned error can't find the container with id 72dee2b28f2e5011aa647b6b08dee6233c8fc8b606e6be3179eb5d7b257bfc39
Sep 18 16:20:37.371575 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:37.371546    2585 generic.go:356] "Generic (PLEG): container finished" podID="a2a7a976-02e8-47ac-8d0e-b081f8ceb225" containerID="a1bad325fdfc251a360a7c2c5e5ff812773b0407a60c0c3b5a71686d80c21d64" exitCode=0
Sep 18 16:20:37.371695 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:37.371628    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5gwsw" event={"ID":"a2a7a976-02e8-47ac-8d0e-b081f8ceb225","Type":"ContainerDied","Data":"a1bad325fdfc251a360a7c2c5e5ff812773b0407a60c0c3b5a71686d80c21d64"}
Sep 18 16:20:37.373599 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:37.373576    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-strvg" event={"ID":"96d83cf1-2b3e-46e0-9deb-a666cc9fa27d","Type":"ContainerStarted","Data":"183e1bd104ed2884dab12b97e057c69268eccf81bda3f0752490fbd998d93258"}
Sep 18 16:20:37.373698 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:37.373619    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-strvg" event={"ID":"96d83cf1-2b3e-46e0-9deb-a666cc9fa27d","Type":"ContainerStarted","Data":"125fd42d594a09f4f4b6b00b5fd89455b6193fb4f09f07f6df9798ef2af32dfc"}
Sep 18 16:20:37.373698 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:37.373649    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-strvg" event={"ID":"96d83cf1-2b3e-46e0-9deb-a666cc9fa27d","Type":"ContainerStarted","Data":"72dee2b28f2e5011aa647b6b08dee6233c8fc8b606e6be3179eb5d7b257bfc39"}
Sep 18 16:20:38.379193 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:38.379153    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-strvg" event={"ID":"96d83cf1-2b3e-46e0-9deb-a666cc9fa27d","Type":"ContainerStarted","Data":"f6452d7c9e2b92a5fdcd02929e1821780c1fb1cc0de54c3c5558bb066244934f"}
Sep 18 16:20:38.381222 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:38.381198    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5gwsw" event={"ID":"a2a7a976-02e8-47ac-8d0e-b081f8ceb225","Type":"ContainerStarted","Data":"b404bf6bdde34a860fad054a01ffda092b16d6138ac700059d1ef93ebaaf96da"}
Sep 18 16:20:38.381343 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:38.381248    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5gwsw" event={"ID":"a2a7a976-02e8-47ac-8d0e-b081f8ceb225","Type":"ContainerStarted","Data":"65a06629f49203a1e61ad3d975f3958a2be763221fbc32a86fa09dc7bfd23f4b"}
Sep 18 16:20:38.415397 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:38.415356    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-strvg" podStartSLOduration=2.585913375 podStartE2EDuration="3.41534148s" totalImagesPullingTime="829.428105ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:38.414233085 +0000 UTC m=+278.197235109" watchObservedRunningTime="2026-09-18 16:20:38.41534148 +0000 UTC m=+278.198343514"
Sep 18 16:20:38.455044 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:38.455001    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5gwsw" podStartSLOduration=2.290577728 podStartE2EDuration="3.45498635s" totalImagesPullingTime="1.095503063s" totalInitContainerRuntime="68.905559ms" isStatefulPod=true podCreationTimestamp="2026-09-18 16:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:38.450983183 +0000 UTC m=+278.233985207" watchObservedRunningTime="2026-09-18 16:20:38.45498635 +0000 UTC m=+278.237988375"
Sep 18 16:20:43.020944 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:20:43.020882    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-glsl2" podUID="b1157877-3544-413a-8893-f143ad8e234d"
Sep 18 16:20:43.397668 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:43.397588    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-glsl2"
Sep 18 16:20:46.338687 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:46.338661    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5876889fd5-2dfl4"
Sep 18 16:20:46.488192 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:46.488166    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1157877-3544-413a-8893-f143ad8e234d-metrics-tls\") pod \"dns-default-glsl2\" (UID: \"b1157877-3544-413a-8893-f143ad8e234d\") " pod="openshift-dns/dns-default-glsl2"
Sep 18 16:20:46.490495 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:46.490479    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1157877-3544-413a-8893-f143ad8e234d-metrics-tls\") pod \"dns-default-glsl2\" (UID: \"b1157877-3544-413a-8893-f143ad8e234d\") " pod="openshift-dns/dns-default-glsl2"
Sep 18 16:20:46.701112 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:46.701044    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f68fb\""
Sep 18 16:20:46.708651 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:46.708633    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-glsl2"
Sep 18 16:20:46.977670 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:46.977636    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-glsl2"]
Sep 18 16:20:46.980862 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:20:46.980829    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1157877_3544_413a_8893_f143ad8e234d.slice/crio-55d0a630c6d4450f1f922a379a0ffc1d25e6cb11d5dfaf3f18c22d7034098a58 WatchSource:0}: Error finding container 55d0a630c6d4450f1f922a379a0ffc1d25e6cb11d5dfaf3f18c22d7034098a58: Status 404 returned error can't find the container with id 55d0a630c6d4450f1f922a379a0ffc1d25e6cb11d5dfaf3f18c22d7034098a58
Sep 18 16:20:47.412141 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:47.412069    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-glsl2" event={"ID":"b1157877-3544-413a-8893-f143ad8e234d","Type":"ContainerStarted","Data":"55d0a630c6d4450f1f922a379a0ffc1d25e6cb11d5dfaf3f18c22d7034098a58"}
Sep 18 16:20:48.592268 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:48.592222    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-674b5879-vn5cz" podUID="8ef1dfae-0866-44a8-ace5-9ad83a6df587" containerName="registry" containerID="cri-o://54f19968f897713e6b1b386c437905e5d1955f1590963aab76bdebbeec77c301" gracePeriod=30
Sep 18 16:20:48.912563 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:48.912541    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:20:49.008313 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.008284    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ef1dfae-0866-44a8-ace5-9ad83a6df587-trusted-ca\") pod \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") "
Sep 18 16:20:49.008433 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.008329    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjgqz\" (UniqueName: \"kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-kube-api-access-wjgqz\") pod \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") "
Sep 18 16:20:49.008433 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.008366    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-registry-tls\") pod \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") "
Sep 18 16:20:49.008433 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.008416    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ef1dfae-0866-44a8-ace5-9ad83a6df587-registry-certificates\") pod \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") "
Sep 18 16:20:49.008599 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.008574    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ef1dfae-0866-44a8-ace5-9ad83a6df587-installation-pull-secrets\") pod \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") "
Sep 18 16:20:49.008661 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.008622    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ef1dfae-0866-44a8-ace5-9ad83a6df587-ca-trust-extracted\") pod \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") "
Sep 18 16:20:49.008712 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.008657    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ef1dfae-0866-44a8-ace5-9ad83a6df587-image-registry-private-configuration\") pod \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") "
Sep 18 16:20:49.008712 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.008690    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-bound-sa-token\") pod \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\" (UID: \"8ef1dfae-0866-44a8-ace5-9ad83a6df587\") "
Sep 18 16:20:49.008812 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.008759    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ef1dfae-0866-44a8-ace5-9ad83a6df587-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8ef1dfae-0866-44a8-ace5-9ad83a6df587" (UID: "8ef1dfae-0866-44a8-ace5-9ad83a6df587"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:20:49.008880 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.008856    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ef1dfae-0866-44a8-ace5-9ad83a6df587-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8ef1dfae-0866-44a8-ace5-9ad83a6df587" (UID: "8ef1dfae-0866-44a8-ace5-9ad83a6df587"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:20:49.009044 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.009017    2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ef1dfae-0866-44a8-ace5-9ad83a6df587-trusted-ca\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:20:49.009044 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.009039    2585 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8ef1dfae-0866-44a8-ace5-9ad83a6df587-registry-certificates\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:20:49.011205 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.011171    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ef1dfae-0866-44a8-ace5-9ad83a6df587-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8ef1dfae-0866-44a8-ace5-9ad83a6df587" (UID: "8ef1dfae-0866-44a8-ace5-9ad83a6df587"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:20:49.011277 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.011254    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ef1dfae-0866-44a8-ace5-9ad83a6df587-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8ef1dfae-0866-44a8-ace5-9ad83a6df587" (UID: "8ef1dfae-0866-44a8-ace5-9ad83a6df587"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:20:49.011331 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.011278    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8ef1dfae-0866-44a8-ace5-9ad83a6df587" (UID: "8ef1dfae-0866-44a8-ace5-9ad83a6df587"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:20:49.011331 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.011294    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8ef1dfae-0866-44a8-ace5-9ad83a6df587" (UID: "8ef1dfae-0866-44a8-ace5-9ad83a6df587"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:20:49.011404 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.011370    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-kube-api-access-wjgqz" (OuterVolumeSpecName: "kube-api-access-wjgqz") pod "8ef1dfae-0866-44a8-ace5-9ad83a6df587" (UID: "8ef1dfae-0866-44a8-ace5-9ad83a6df587"). InnerVolumeSpecName "kube-api-access-wjgqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:20:49.017143 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.017115    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef1dfae-0866-44a8-ace5-9ad83a6df587-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8ef1dfae-0866-44a8-ace5-9ad83a6df587" (UID: "8ef1dfae-0866-44a8-ace5-9ad83a6df587"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:20:49.109595 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.109566    2585 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-registry-tls\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:20:49.109595 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.109591    2585 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8ef1dfae-0866-44a8-ace5-9ad83a6df587-installation-pull-secrets\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:20:49.109595 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.109601    2585 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8ef1dfae-0866-44a8-ace5-9ad83a6df587-ca-trust-extracted\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:20:49.109792 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.109611    2585 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8ef1dfae-0866-44a8-ace5-9ad83a6df587-image-registry-private-configuration\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:20:49.109792 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.109620    2585 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-bound-sa-token\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:20:49.109792 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.109629    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjgqz\" (UniqueName: \"kubernetes.io/projected/8ef1dfae-0866-44a8-ace5-9ad83a6df587-kube-api-access-wjgqz\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:20:49.419657 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.419625    2585 generic.go:356] "Generic (PLEG): container finished" podID="8ef1dfae-0866-44a8-ace5-9ad83a6df587" containerID="54f19968f897713e6b1b386c437905e5d1955f1590963aab76bdebbeec77c301" exitCode=0
Sep 18 16:20:49.419889 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.419672    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-674b5879-vn5cz"
Sep 18 16:20:49.419889 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.419710    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-674b5879-vn5cz" event={"ID":"8ef1dfae-0866-44a8-ace5-9ad83a6df587","Type":"ContainerDied","Data":"54f19968f897713e6b1b386c437905e5d1955f1590963aab76bdebbeec77c301"}
Sep 18 16:20:49.419889 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.419750    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-674b5879-vn5cz" event={"ID":"8ef1dfae-0866-44a8-ace5-9ad83a6df587","Type":"ContainerDied","Data":"0849fa8fd629ab7e7c7d964f618f0d8ee830f5379c10081bf5e5ab058318747e"}
Sep 18 16:20:49.419889 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.419768    2585 scope.go:117] "RemoveContainer" containerID="54f19968f897713e6b1b386c437905e5d1955f1590963aab76bdebbeec77c301"
Sep 18 16:20:49.421780 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.421750    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-glsl2" event={"ID":"b1157877-3544-413a-8893-f143ad8e234d","Type":"ContainerStarted","Data":"aa2389a1027ffe2c38127536301aed125b5d65cff73147f872d8681547f5e363"}
Sep 18 16:20:49.421896 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.421802    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-glsl2" event={"ID":"b1157877-3544-413a-8893-f143ad8e234d","Type":"ContainerStarted","Data":"edf17e9cbaa723f638a6d130040173ad702484e53b187ba896ed9fe440617362"}
Sep 18 16:20:49.421896 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.421854    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-glsl2"
Sep 18 16:20:49.428090 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.428074    2585 scope.go:117] "RemoveContainer" containerID="54f19968f897713e6b1b386c437905e5d1955f1590963aab76bdebbeec77c301"
Sep 18 16:20:49.428361 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:20:49.428345    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54f19968f897713e6b1b386c437905e5d1955f1590963aab76bdebbeec77c301\": container with ID starting with 54f19968f897713e6b1b386c437905e5d1955f1590963aab76bdebbeec77c301 not found: ID does not exist" containerID="54f19968f897713e6b1b386c437905e5d1955f1590963aab76bdebbeec77c301"
Sep 18 16:20:49.428405 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.428368    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54f19968f897713e6b1b386c437905e5d1955f1590963aab76bdebbeec77c301"} err="failed to get container status \"54f19968f897713e6b1b386c437905e5d1955f1590963aab76bdebbeec77c301\": rpc error: code = NotFound desc = could not find container \"54f19968f897713e6b1b386c437905e5d1955f1590963aab76bdebbeec77c301\": container with ID starting with 54f19968f897713e6b1b386c437905e5d1955f1590963aab76bdebbeec77c301 not found: ID does not exist"
Sep 18 16:20:49.441762 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.441724    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-glsl2" podStartSLOduration=251.613711716 podStartE2EDuration="4m13.441711602s" totalImagesPullingTime="1.827999886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:49.441335227 +0000 UTC m=+289.224337250" watchObservedRunningTime="2026-09-18 16:20:49.441711602 +0000 UTC m=+289.224713625"
Sep 18 16:20:49.453500 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.453480    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-674b5879-vn5cz"]
Sep 18 16:20:49.458976 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:49.458955    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-674b5879-vn5cz"]
Sep 18 16:20:50.391582 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:50.391554    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ef1dfae-0866-44a8-ace5-9ad83a6df587" path="/var/lib/kubelet/pods/8ef1dfae-0866-44a8-ace5-9ad83a6df587/volumes"
Sep 18 16:20:51.387674 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:51.387642    2585 scope.go:117] "RemoveContainer" containerID="377c1e564c19a6500ce5ef9418fd3767db76b59b5425aac5d38de0f2d0050198"
Sep 18 16:20:52.438948 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:52.438899    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 16:20:52.439295 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:52.438967    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp" event={"ID":"f0cae29c-d3b9-4c3f-ac8d-d151e05916af","Type":"ContainerStarted","Data":"16ebd2dca94f754d32f150d768b9dbc54de829fba5cead5294c0e89342753744"}
Sep 18 16:20:52.439295 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:52.439236    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp"
Sep 18 16:20:52.444063 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:52.444043    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp"
Sep 18 16:20:52.460084 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:52.460044    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-cfzmp" podStartSLOduration=57.010956767 podStartE2EDuration="59.460028928s" totalImagesPullingTime="2.449072161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:52.458448557 +0000 UTC m=+292.241450582" watchObservedRunningTime="2026-09-18 16:20:52.460028928 +0000 UTC m=+292.243030953"
Sep 18 16:20:59.428362 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:20:59.428331    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-glsl2"
Sep 18 16:21:02.273043 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:02.273020    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 16:21:02.276628 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:02.276604    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 16:21:02.277134 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:02.277113    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 16:21:02.280788 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:02.280767    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 16:21:02.284166 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:02.284150    2585 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 16:21:03.219369 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.219338    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cd8f78b9b-vb49r"]
Sep 18 16:21:03.221502 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.219575    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ef1dfae-0866-44a8-ace5-9ad83a6df587" containerName="registry"
Sep 18 16:21:03.221502 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.219586    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ef1dfae-0866-44a8-ace5-9ad83a6df587" containerName="registry"
Sep 18 16:21:03.221502 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.219623    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ef1dfae-0866-44a8-ace5-9ad83a6df587" containerName="registry"
Sep 18 16:21:03.222338 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.222323    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd8f78b9b-vb49r"
Sep 18 16:21:03.226235 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.226204    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 16:21:03.226235 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.226211    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 16:21:03.226408 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.226263    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 16:21:03.226408 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.226274    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 16:21:03.226544 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.226529    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 16:21:03.226605 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.226546    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kdwv8\""
Sep 18 16:21:03.226605 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.226548    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 16:21:03.226605 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.226585    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 16:21:03.236827 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.236808    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cd8f78b9b-vb49r"]
Sep 18 16:21:03.306343 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.306324    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e1e4b2e-249d-4cd7-bafd-b895d796b634-console-oauth-config\") pod \"console-5cd8f78b9b-vb49r\" (UID: \"7e1e4b2e-249d-4cd7-bafd-b895d796b634\") " pod="openshift-console/console-5cd8f78b9b-vb49r"
Sep 18 16:21:03.306696 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.306352    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwvhr\" (UniqueName: \"kubernetes.io/projected/7e1e4b2e-249d-4cd7-bafd-b895d796b634-kube-api-access-lwvhr\") pod \"console-5cd8f78b9b-vb49r\" (UID: \"7e1e4b2e-249d-4cd7-bafd-b895d796b634\") " pod="openshift-console/console-5cd8f78b9b-vb49r"
Sep 18 16:21:03.306696 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.306394    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e1e4b2e-249d-4cd7-bafd-b895d796b634-service-ca\") pod \"console-5cd8f78b9b-vb49r\" (UID: \"7e1e4b2e-249d-4cd7-bafd-b895d796b634\") " pod="openshift-console/console-5cd8f78b9b-vb49r"
Sep 18 16:21:03.306696 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.306521    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e1e4b2e-249d-4cd7-bafd-b895d796b634-oauth-serving-cert\") pod \"console-5cd8f78b9b-vb49r\" (UID: \"7e1e4b2e-249d-4cd7-bafd-b895d796b634\") " pod="openshift-console/console-5cd8f78b9b-vb49r"
Sep 18 16:21:03.306696 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.306558    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e1e4b2e-249d-4cd7-bafd-b895d796b634-console-serving-cert\") pod \"console-5cd8f78b9b-vb49r\" (UID: \"7e1e4b2e-249d-4cd7-bafd-b895d796b634\") " pod="openshift-console/console-5cd8f78b9b-vb49r"
Sep 18 16:21:03.306696 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.306587    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e1e4b2e-249d-4cd7-bafd-b895d796b634-console-config\") pod \"console-5cd8f78b9b-vb49r\" (UID: \"7e1e4b2e-249d-4cd7-bafd-b895d796b634\") " pod="openshift-console/console-5cd8f78b9b-vb49r"
Sep 18 16:21:03.407191 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.407159    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e1e4b2e-249d-4cd7-bafd-b895d796b634-oauth-serving-cert\") pod \"console-5cd8f78b9b-vb49r\" (UID: \"7e1e4b2e-249d-4cd7-bafd-b895d796b634\") " pod="openshift-console/console-5cd8f78b9b-vb49r"
Sep 18 16:21:03.407191 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.407191    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e1e4b2e-249d-4cd7-bafd-b895d796b634-console-serving-cert\") pod \"console-5cd8f78b9b-vb49r\" (UID: \"7e1e4b2e-249d-4cd7-bafd-b895d796b634\") " pod="openshift-console/console-5cd8f78b9b-vb49r"
Sep 18 16:21:03.407373 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.407206    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e1e4b2e-249d-4cd7-bafd-b895d796b634-console-config\") pod \"console-5cd8f78b9b-vb49r\" (UID: \"7e1e4b2e-249d-4cd7-bafd-b895d796b634\") " pod="openshift-console/console-5cd8f78b9b-vb49r"
Sep 18 16:21:03.407373 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.407227    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e1e4b2e-249d-4cd7-bafd-b895d796b634-console-oauth-config\") pod \"console-5cd8f78b9b-vb49r\" (UID: \"7e1e4b2e-249d-4cd7-bafd-b895d796b634\") " pod="openshift-console/console-5cd8f78b9b-vb49r"
Sep 18 16:21:03.407373 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.407252    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwvhr\" (UniqueName: \"kubernetes.io/projected/7e1e4b2e-249d-4cd7-bafd-b895d796b634-kube-api-access-lwvhr\") pod \"console-5cd8f78b9b-vb49r\" (UID: \"7e1e4b2e-249d-4cd7-bafd-b895d796b634\") " pod="openshift-console/console-5cd8f78b9b-vb49r"
Sep 18 16:21:03.407373 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.407301    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e1e4b2e-249d-4cd7-bafd-b895d796b634-service-ca\") pod \"console-5cd8f78b9b-vb49r\" (UID: \"7e1e4b2e-249d-4cd7-bafd-b895d796b634\") " pod="openshift-console/console-5cd8f78b9b-vb49r"
Sep 18 16:21:03.407881 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.407860    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e1e4b2e-249d-4cd7-bafd-b895d796b634-oauth-serving-cert\") pod \"console-5cd8f78b9b-vb49r\" (UID: \"7e1e4b2e-249d-4cd7-bafd-b895d796b634\") " pod="openshift-console/console-5cd8f78b9b-vb49r"
Sep 18 16:21:03.408042 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.408022    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e1e4b2e-249d-4cd7-bafd-b895d796b634-console-config\") pod \"console-5cd8f78b9b-vb49r\" (UID: \"7e1e4b2e-249d-4cd7-bafd-b895d796b634\") " pod="openshift-console/console-5cd8f78b9b-vb49r"
Sep 18 16:21:03.408085 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.408022    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e1e4b2e-249d-4cd7-bafd-b895d796b634-service-ca\") pod \"console-5cd8f78b9b-vb49r\" (UID: \"7e1e4b2e-249d-4cd7-bafd-b895d796b634\") " pod="openshift-console/console-5cd8f78b9b-vb49r"
Sep 18 16:21:03.409716 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.409687    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e1e4b2e-249d-4cd7-bafd-b895d796b634-console-serving-cert\") pod \"console-5cd8f78b9b-vb49r\" (UID: \"7e1e4b2e-249d-4cd7-bafd-b895d796b634\") " pod="openshift-console/console-5cd8f78b9b-vb49r"
Sep 18 16:21:03.409814 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.409725    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e1e4b2e-249d-4cd7-bafd-b895d796b634-console-oauth-config\") pod \"console-5cd8f78b9b-vb49r\" (UID: \"7e1e4b2e-249d-4cd7-bafd-b895d796b634\") " pod="openshift-console/console-5cd8f78b9b-vb49r"
Sep 18 16:21:03.416323 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.416302    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwvhr\" (UniqueName: \"kubernetes.io/projected/7e1e4b2e-249d-4cd7-bafd-b895d796b634-kube-api-access-lwvhr\") pod \"console-5cd8f78b9b-vb49r\" (UID: \"7e1e4b2e-249d-4cd7-bafd-b895d796b634\") " pod="openshift-console/console-5cd8f78b9b-vb49r"
Sep 18 16:21:03.531377 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.531354    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd8f78b9b-vb49r"
Sep 18 16:21:03.805353 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.805273    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cd8f78b9b-vb49r"]
Sep 18 16:21:03.808564 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:21:03.808531    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e1e4b2e_249d_4cd7_bafd_b895d796b634.slice/crio-c21143714e725586d45a2ef89df14791b82da1df8cb84250d17b9e56cb13b94c WatchSource:0}: Error finding container c21143714e725586d45a2ef89df14791b82da1df8cb84250d17b9e56cb13b94c: Status 404 returned error can't find the container with id c21143714e725586d45a2ef89df14791b82da1df8cb84250d17b9e56cb13b94c
Sep 18 16:21:03.810641 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:03.810625    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:21:04.482564 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:04.482526    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd8f78b9b-vb49r" event={"ID":"7e1e4b2e-249d-4cd7-bafd-b895d796b634","Type":"ContainerStarted","Data":"c21143714e725586d45a2ef89df14791b82da1df8cb84250d17b9e56cb13b94c"}
Sep 18 16:21:08.499773 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:08.499734    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd8f78b9b-vb49r" event={"ID":"7e1e4b2e-249d-4cd7-bafd-b895d796b634","Type":"ContainerStarted","Data":"4d3548537a70d32860216c863ab883bd5009a15f1caee35052da8b4d7fd10360"}
Sep 18 16:21:08.518174 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:08.518130    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5cd8f78b9b-vb49r" podStartSLOduration=1.610094382 podStartE2EDuration="5.518113257s" totalImagesPullingTime="3.908018875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:21:08.517223066 +0000 UTC m=+308.300225102" watchObservedRunningTime="2026-09-18 16:21:08.518113257 +0000 UTC m=+308.301115282"
Sep 18 16:21:13.531775 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:13.531745    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cd8f78b9b-vb49r"
Sep 18 16:21:13.532230 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:13.531820    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5cd8f78b9b-vb49r"
Sep 18 16:21:13.536444 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:13.536419    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5cd8f78b9b-vb49r"
Sep 18 16:21:14.522558 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:14.522529    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5cd8f78b9b-vb49r"
Sep 18 16:21:22.546494 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:22.546463    2585 generic.go:356] "Generic (PLEG): container finished" podID="6f505047-5eda-41d3-8df8-b8c2a31d0001" containerID="22e01b2bd06e6cca4e210b8ae3cc30d22baaf520b7a13b5df3038c87ac69bbe5" exitCode=0
Sep 18 16:21:22.546879 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:22.546515    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-rsb8k" event={"ID":"6f505047-5eda-41d3-8df8-b8c2a31d0001","Type":"ContainerDied","Data":"22e01b2bd06e6cca4e210b8ae3cc30d22baaf520b7a13b5df3038c87ac69bbe5"}
Sep 18 16:21:22.546879 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:22.546857    2585 scope.go:117] "RemoveContainer" containerID="22e01b2bd06e6cca4e210b8ae3cc30d22baaf520b7a13b5df3038c87ac69bbe5"
Sep 18 16:21:23.551961 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:23.551907    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-rsb8k" event={"ID":"6f505047-5eda-41d3-8df8-b8c2a31d0001","Type":"ContainerStarted","Data":"658a00a3b865caf9b104b643d822d290b0f4508d1585ef3b46240663efa4351b"}
Sep 18 16:21:24.281834 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:24.281799    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cd8f78b9b-vb49r"]
Sep 18 16:21:25.547474 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:25.547444    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5gwsw_a2a7a976-02e8-47ac-8d0e-b081f8ceb225/init-textfile/0.log"
Sep 18 16:21:25.748339 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:25.748308    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5gwsw_a2a7a976-02e8-47ac-8d0e-b081f8ceb225/node-exporter/0.log"
Sep 18 16:21:25.947467 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:25.947399    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5gwsw_a2a7a976-02e8-47ac-8d0e-b081f8ceb225/kube-rbac-proxy/0.log"
Sep 18 16:21:27.347499 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:27.347472    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-strvg_96d83cf1-2b3e-46e0-9deb-a666cc9fa27d/kube-rbac-proxy-main/0.log"
Sep 18 16:21:27.547292 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:27.547268    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-strvg_96d83cf1-2b3e-46e0-9deb-a666cc9fa27d/kube-rbac-proxy-self/0.log"
Sep 18 16:21:27.750045 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:27.750008    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-strvg_96d83cf1-2b3e-46e0-9deb-a666cc9fa27d/openshift-state-metrics/0.log"
Sep 18 16:21:29.353187 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:29.353156    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 16:21:29.549001 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:29.548971    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/3.log"
Sep 18 16:21:29.747523 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:29.747501    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5cd8f78b9b-vb49r_7e1e4b2e-249d-4cd7-bafd-b895d796b634/console/0.log"
Sep 18 16:21:30.947661 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:30.947613    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fxkzj_4e73cb49-af8f-408b-aaeb-db96fabe4998/serve-healthcheck-canary/0.log"
Sep 18 16:21:49.301813 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:49.301718    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5cd8f78b9b-vb49r" podUID="7e1e4b2e-249d-4cd7-bafd-b895d796b634" containerName="console" containerID="cri-o://4d3548537a70d32860216c863ab883bd5009a15f1caee35052da8b4d7fd10360" gracePeriod=15
Sep 18 16:21:49.540734 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:49.540711    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd8f78b9b-vb49r"
Sep 18 16:21:49.630496 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:49.630431    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwvhr\" (UniqueName: \"kubernetes.io/projected/7e1e4b2e-249d-4cd7-bafd-b895d796b634-kube-api-access-lwvhr\") pod \"7e1e4b2e-249d-4cd7-bafd-b895d796b634\" (UID: \"7e1e4b2e-249d-4cd7-bafd-b895d796b634\") "
Sep 18 16:21:49.630496 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:49.630471    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e1e4b2e-249d-4cd7-bafd-b895d796b634-console-config\") pod \"7e1e4b2e-249d-4cd7-bafd-b895d796b634\" (UID: \"7e1e4b2e-249d-4cd7-bafd-b895d796b634\") "
Sep 18 16:21:49.630669 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:49.630500    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e1e4b2e-249d-4cd7-bafd-b895d796b634-console-oauth-config\") pod \"7e1e4b2e-249d-4cd7-bafd-b895d796b634\" (UID: \"7e1e4b2e-249d-4cd7-bafd-b895d796b634\") "
Sep 18 16:21:49.630669 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:49.630514    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e1e4b2e-249d-4cd7-bafd-b895d796b634-oauth-serving-cert\") pod \"7e1e4b2e-249d-4cd7-bafd-b895d796b634\" (UID: \"7e1e4b2e-249d-4cd7-bafd-b895d796b634\") "
Sep 18 16:21:49.630828 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:49.630792    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e1e4b2e-249d-4cd7-bafd-b895d796b634-console-config" (OuterVolumeSpecName: "console-config") pod "7e1e4b2e-249d-4cd7-bafd-b895d796b634" (UID: "7e1e4b2e-249d-4cd7-bafd-b895d796b634"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:21:49.630957 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:49.630904    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e1e4b2e-249d-4cd7-bafd-b895d796b634-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7e1e4b2e-249d-4cd7-bafd-b895d796b634" (UID: "7e1e4b2e-249d-4cd7-bafd-b895d796b634"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:21:49.632864 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:49.632841    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e1e4b2e-249d-4cd7-bafd-b895d796b634-kube-api-access-lwvhr" (OuterVolumeSpecName: "kube-api-access-lwvhr") pod "7e1e4b2e-249d-4cd7-bafd-b895d796b634" (UID: "7e1e4b2e-249d-4cd7-bafd-b895d796b634"). InnerVolumeSpecName "kube-api-access-lwvhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:21:49.632965 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:49.632898    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e1e4b2e-249d-4cd7-bafd-b895d796b634-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7e1e4b2e-249d-4cd7-bafd-b895d796b634" (UID: "7e1e4b2e-249d-4cd7-bafd-b895d796b634"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:21:49.638137 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:49.638114    2585 generic.go:356] "Generic (PLEG): container finished" podID="7e1e4b2e-249d-4cd7-bafd-b895d796b634" containerID="4d3548537a70d32860216c863ab883bd5009a15f1caee35052da8b4d7fd10360" exitCode=0
Sep 18 16:21:49.638232 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:49.638158    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd8f78b9b-vb49r"
Sep 18 16:21:49.638232 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:49.638200    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd8f78b9b-vb49r" event={"ID":"7e1e4b2e-249d-4cd7-bafd-b895d796b634","Type":"ContainerDied","Data":"4d3548537a70d32860216c863ab883bd5009a15f1caee35052da8b4d7fd10360"}
Sep 18 16:21:49.638350 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:49.638254    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd8f78b9b-vb49r" event={"ID":"7e1e4b2e-249d-4cd7-bafd-b895d796b634","Type":"ContainerDied","Data":"c21143714e725586d45a2ef89df14791b82da1df8cb84250d17b9e56cb13b94c"}
Sep 18 16:21:49.638350 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:49.638282    2585 scope.go:117] "RemoveContainer" containerID="4d3548537a70d32860216c863ab883bd5009a15f1caee35052da8b4d7fd10360"
Sep 18 16:21:49.645773 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:49.645754    2585 scope.go:117] "RemoveContainer" containerID="4d3548537a70d32860216c863ab883bd5009a15f1caee35052da8b4d7fd10360"
Sep 18 16:21:49.646024 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:21:49.646004    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d3548537a70d32860216c863ab883bd5009a15f1caee35052da8b4d7fd10360\": container with ID starting with 4d3548537a70d32860216c863ab883bd5009a15f1caee35052da8b4d7fd10360 not found: ID does not exist" containerID="4d3548537a70d32860216c863ab883bd5009a15f1caee35052da8b4d7fd10360"
Sep 18 16:21:49.646088 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:49.646031    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d3548537a70d32860216c863ab883bd5009a15f1caee35052da8b4d7fd10360"} err="failed to get container status \"4d3548537a70d32860216c863ab883bd5009a15f1caee35052da8b4d7fd10360\": rpc error: code = NotFound desc = could not find container \"4d3548537a70d32860216c863ab883bd5009a15f1caee35052da8b4d7fd10360\": container with ID starting with 4d3548537a70d32860216c863ab883bd5009a15f1caee35052da8b4d7fd10360 not found: ID does not exist"
Sep 18 16:21:49.731280 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:49.731257    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e1e4b2e-249d-4cd7-bafd-b895d796b634-service-ca\") pod \"7e1e4b2e-249d-4cd7-bafd-b895d796b634\" (UID: \"7e1e4b2e-249d-4cd7-bafd-b895d796b634\") "
Sep 18 16:21:49.731406 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:49.731287    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e1e4b2e-249d-4cd7-bafd-b895d796b634-console-serving-cert\") pod \"7e1e4b2e-249d-4cd7-bafd-b895d796b634\" (UID: \"7e1e4b2e-249d-4cd7-bafd-b895d796b634\") "
Sep 18 16:21:49.731474 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:49.731426    2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7e1e4b2e-249d-4cd7-bafd-b895d796b634-console-oauth-config\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:21:49.731474 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:49.731436    2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7e1e4b2e-249d-4cd7-bafd-b895d796b634-oauth-serving-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:21:49.731474 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:49.731445    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwvhr\" (UniqueName: \"kubernetes.io/projected/7e1e4b2e-249d-4cd7-bafd-b895d796b634-kube-api-access-lwvhr\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:21:49.731474 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:49.731454    2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7e1e4b2e-249d-4cd7-bafd-b895d796b634-console-config\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:21:49.731669 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:49.731544    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e1e4b2e-249d-4cd7-bafd-b895d796b634-service-ca" (OuterVolumeSpecName: "service-ca") pod "7e1e4b2e-249d-4cd7-bafd-b895d796b634" (UID: "7e1e4b2e-249d-4cd7-bafd-b895d796b634"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:21:49.733304 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:49.733282    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e1e4b2e-249d-4cd7-bafd-b895d796b634-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7e1e4b2e-249d-4cd7-bafd-b895d796b634" (UID: "7e1e4b2e-249d-4cd7-bafd-b895d796b634"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:21:49.832353 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:49.832331    2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7e1e4b2e-249d-4cd7-bafd-b895d796b634-service-ca\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:21:49.832353 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:49.832355    2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e1e4b2e-249d-4cd7-bafd-b895d796b634-console-serving-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:21:49.959236 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:49.959206    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cd8f78b9b-vb49r"]
Sep 18 16:21:49.964978 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:49.964954    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5cd8f78b9b-vb49r"]
Sep 18 16:21:50.391544 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:21:50.391517    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e1e4b2e-249d-4cd7-bafd-b895d796b634" path="/var/lib/kubelet/pods/7e1e4b2e-249d-4cd7-bafd-b895d796b634/volumes"
Sep 18 16:26:02.299124 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:02.299093    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 16:26:02.300483 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:02.300456    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 16:26:02.303074 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:02.303056    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 16:26:02.304620 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:02.304603    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 16:26:56.911515 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:56.911478    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-794bb4599-lpdxz"]
Sep 18 16:26:56.913848 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:56.911735    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e1e4b2e-249d-4cd7-bafd-b895d796b634" containerName="console"
Sep 18 16:26:56.913848 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:56.911748    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e1e4b2e-249d-4cd7-bafd-b895d796b634" containerName="console"
Sep 18 16:26:56.913848 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:56.911792    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e1e4b2e-249d-4cd7-bafd-b895d796b634" containerName="console"
Sep 18 16:26:56.914646 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:56.914632    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-794bb4599-lpdxz"
Sep 18 16:26:56.919986 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:56.919957    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 16:26:56.920171 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:56.920153    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 16:26:56.920237 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:56.920201    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 16:26:56.920289 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:56.920253    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 16:26:56.920344 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:56.920296    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 16:26:56.920344 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:56.920315    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 16:26:56.921148 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:56.921073    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 16:26:56.921148 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:56.921113    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kdwv8\""
Sep 18 16:26:56.929072 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:56.929048    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 18 16:26:56.942968 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:56.942941    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-794bb4599-lpdxz"]
Sep 18 16:26:56.976661 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:56.976635    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7422290-2994-42fc-8ef4-3f39f099080d-trusted-ca-bundle\") pod \"console-794bb4599-lpdxz\" (UID: \"a7422290-2994-42fc-8ef4-3f39f099080d\") " pod="openshift-console/console-794bb4599-lpdxz"
Sep 18 16:26:56.976783 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:56.976678    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wh6b\" (UniqueName: \"kubernetes.io/projected/a7422290-2994-42fc-8ef4-3f39f099080d-kube-api-access-6wh6b\") pod \"console-794bb4599-lpdxz\" (UID: \"a7422290-2994-42fc-8ef4-3f39f099080d\") " pod="openshift-console/console-794bb4599-lpdxz"
Sep 18 16:26:56.976783 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:56.976700    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7422290-2994-42fc-8ef4-3f39f099080d-console-oauth-config\") pod \"console-794bb4599-lpdxz\" (UID: \"a7422290-2994-42fc-8ef4-3f39f099080d\") " pod="openshift-console/console-794bb4599-lpdxz"
Sep 18 16:26:56.976783 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:56.976715    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7422290-2994-42fc-8ef4-3f39f099080d-service-ca\") pod \"console-794bb4599-lpdxz\" (UID: \"a7422290-2994-42fc-8ef4-3f39f099080d\") " pod="openshift-console/console-794bb4599-lpdxz"
Sep 18 16:26:56.976783 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:56.976734    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7422290-2994-42fc-8ef4-3f39f099080d-oauth-serving-cert\") pod \"console-794bb4599-lpdxz\" (UID: \"a7422290-2994-42fc-8ef4-3f39f099080d\") " pod="openshift-console/console-794bb4599-lpdxz"
Sep 18 16:26:56.976944 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:56.976787    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7422290-2994-42fc-8ef4-3f39f099080d-console-serving-cert\") pod \"console-794bb4599-lpdxz\" (UID: \"a7422290-2994-42fc-8ef4-3f39f099080d\") " pod="openshift-console/console-794bb4599-lpdxz"
Sep 18 16:26:56.976944 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:56.976817    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7422290-2994-42fc-8ef4-3f39f099080d-console-config\") pod \"console-794bb4599-lpdxz\" (UID: \"a7422290-2994-42fc-8ef4-3f39f099080d\") " pod="openshift-console/console-794bb4599-lpdxz"
Sep 18 16:26:57.078212 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:57.078187    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wh6b\" (UniqueName: \"kubernetes.io/projected/a7422290-2994-42fc-8ef4-3f39f099080d-kube-api-access-6wh6b\") pod \"console-794bb4599-lpdxz\" (UID: \"a7422290-2994-42fc-8ef4-3f39f099080d\") " pod="openshift-console/console-794bb4599-lpdxz"
Sep 18 16:26:57.078352 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:57.078219    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7422290-2994-42fc-8ef4-3f39f099080d-console-oauth-config\") pod \"console-794bb4599-lpdxz\" (UID: \"a7422290-2994-42fc-8ef4-3f39f099080d\") " pod="openshift-console/console-794bb4599-lpdxz"
Sep 18 16:26:57.078352 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:57.078234    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7422290-2994-42fc-8ef4-3f39f099080d-service-ca\") pod \"console-794bb4599-lpdxz\" (UID: \"a7422290-2994-42fc-8ef4-3f39f099080d\") " pod="openshift-console/console-794bb4599-lpdxz"
Sep 18 16:26:57.078352 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:57.078251    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7422290-2994-42fc-8ef4-3f39f099080d-oauth-serving-cert\") pod \"console-794bb4599-lpdxz\" (UID: \"a7422290-2994-42fc-8ef4-3f39f099080d\") " pod="openshift-console/console-794bb4599-lpdxz"
Sep 18 16:26:57.078352 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:57.078271    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7422290-2994-42fc-8ef4-3f39f099080d-console-serving-cert\") pod \"console-794bb4599-lpdxz\" (UID: \"a7422290-2994-42fc-8ef4-3f39f099080d\") " pod="openshift-console/console-794bb4599-lpdxz"
Sep 18 16:26:57.078352 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:57.078292    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7422290-2994-42fc-8ef4-3f39f099080d-console-config\") pod \"console-794bb4599-lpdxz\" (UID: \"a7422290-2994-42fc-8ef4-3f39f099080d\") " pod="openshift-console/console-794bb4599-lpdxz"
Sep 18 16:26:57.078352 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:57.078317    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7422290-2994-42fc-8ef4-3f39f099080d-trusted-ca-bundle\") pod \"console-794bb4599-lpdxz\" (UID: \"a7422290-2994-42fc-8ef4-3f39f099080d\") " pod="openshift-console/console-794bb4599-lpdxz"
Sep 18 16:26:57.078967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:57.078948    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7422290-2994-42fc-8ef4-3f39f099080d-oauth-serving-cert\") pod \"console-794bb4599-lpdxz\" (UID: \"a7422290-2994-42fc-8ef4-3f39f099080d\") " pod="openshift-console/console-794bb4599-lpdxz"
Sep 18 16:26:57.079372 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:57.079351    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7422290-2994-42fc-8ef4-3f39f099080d-service-ca\") pod \"console-794bb4599-lpdxz\" (UID: \"a7422290-2994-42fc-8ef4-3f39f099080d\") " pod="openshift-console/console-794bb4599-lpdxz"
Sep 18 16:26:57.079461 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:57.079351    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7422290-2994-42fc-8ef4-3f39f099080d-console-config\") pod \"console-794bb4599-lpdxz\" (UID: \"a7422290-2994-42fc-8ef4-3f39f099080d\") " pod="openshift-console/console-794bb4599-lpdxz"
Sep 18 16:26:57.079738 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:57.079717    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7422290-2994-42fc-8ef4-3f39f099080d-trusted-ca-bundle\") pod \"console-794bb4599-lpdxz\" (UID: \"a7422290-2994-42fc-8ef4-3f39f099080d\") " pod="openshift-console/console-794bb4599-lpdxz"
Sep 18 16:26:57.081036 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:57.081016    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7422290-2994-42fc-8ef4-3f39f099080d-console-serving-cert\") pod \"console-794bb4599-lpdxz\" (UID: \"a7422290-2994-42fc-8ef4-3f39f099080d\") " pod="openshift-console/console-794bb4599-lpdxz"
Sep 18 16:26:57.081122 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:57.081035    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7422290-2994-42fc-8ef4-3f39f099080d-console-oauth-config\") pod \"console-794bb4599-lpdxz\" (UID: \"a7422290-2994-42fc-8ef4-3f39f099080d\") " pod="openshift-console/console-794bb4599-lpdxz"
Sep 18 16:26:57.089891 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:57.089872    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wh6b\" (UniqueName: \"kubernetes.io/projected/a7422290-2994-42fc-8ef4-3f39f099080d-kube-api-access-6wh6b\") pod \"console-794bb4599-lpdxz\" (UID: \"a7422290-2994-42fc-8ef4-3f39f099080d\") " pod="openshift-console/console-794bb4599-lpdxz"
Sep 18 16:26:57.224302 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:57.224236    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-794bb4599-lpdxz"
Sep 18 16:26:57.494669 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:57.494637    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-794bb4599-lpdxz"]
Sep 18 16:26:57.497770 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:26:57.497734    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7422290_2994_42fc_8ef4_3f39f099080d.slice/crio-96b7a7715b376e614862797d0d7aa7027a0d3cfe8ed5cd2c52dbe7ea6a663bf8 WatchSource:0}: Error finding container 96b7a7715b376e614862797d0d7aa7027a0d3cfe8ed5cd2c52dbe7ea6a663bf8: Status 404 returned error can't find the container with id 96b7a7715b376e614862797d0d7aa7027a0d3cfe8ed5cd2c52dbe7ea6a663bf8
Sep 18 16:26:57.652320 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:57.652284    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-794bb4599-lpdxz" event={"ID":"a7422290-2994-42fc-8ef4-3f39f099080d","Type":"ContainerStarted","Data":"92e1db9dcb8c47646cef3de99b3ead36e1db7917b314761e69e39689b5159b51"}
Sep 18 16:26:57.652459 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:57.652338    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-794bb4599-lpdxz" event={"ID":"a7422290-2994-42fc-8ef4-3f39f099080d","Type":"ContainerStarted","Data":"96b7a7715b376e614862797d0d7aa7027a0d3cfe8ed5cd2c52dbe7ea6a663bf8"}
Sep 18 16:26:57.671354 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:26:57.671308    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-794bb4599-lpdxz" podStartSLOduration=1.671294047 podStartE2EDuration="1.671294047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:26:57.670182708 +0000 UTC m=+657.453184736" watchObservedRunningTime="2026-09-18 16:26:57.671294047 +0000 UTC m=+657.454296070"
Sep 18 16:27:07.224927 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:07.224887    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-794bb4599-lpdxz"
Sep 18 16:27:07.225470 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:07.224974    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-794bb4599-lpdxz"
Sep 18 16:27:07.229394 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:07.229371    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-794bb4599-lpdxz"
Sep 18 16:27:07.691403 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:07.691372    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-794bb4599-lpdxz"
Sep 18 16:27:47.420330 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.420246    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78d76dcff6-cmm6w"]
Sep 18 16:27:47.423096 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.423080    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78d76dcff6-cmm6w"
Sep 18 16:27:47.427182 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.427158    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-5jlnv\""
Sep 18 16:27:47.427326 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.427217    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 18 16:27:47.427326 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.427252    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 18 16:27:47.427326 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.427179    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 18 16:27:47.427326 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.427158    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 16:27:47.434868 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.434849    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78d76dcff6-cmm6w"]
Sep 18 16:27:47.439137 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.439115    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xzrn\" (UniqueName: \"kubernetes.io/projected/31e25497-6656-4fd2-83cb-4b5102263ea0-kube-api-access-7xzrn\") pod \"managed-serviceaccount-addon-agent-78d76dcff6-cmm6w\" (UID: \"31e25497-6656-4fd2-83cb-4b5102263ea0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78d76dcff6-cmm6w"
Sep 18 16:27:47.439240 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.439169    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/31e25497-6656-4fd2-83cb-4b5102263ea0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-78d76dcff6-cmm6w\" (UID: \"31e25497-6656-4fd2-83cb-4b5102263ea0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78d76dcff6-cmm6w"
Sep 18 16:27:47.540252 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.540218    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xzrn\" (UniqueName: \"kubernetes.io/projected/31e25497-6656-4fd2-83cb-4b5102263ea0-kube-api-access-7xzrn\") pod \"managed-serviceaccount-addon-agent-78d76dcff6-cmm6w\" (UID: \"31e25497-6656-4fd2-83cb-4b5102263ea0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78d76dcff6-cmm6w"
Sep 18 16:27:47.540438 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.540269    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/31e25497-6656-4fd2-83cb-4b5102263ea0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-78d76dcff6-cmm6w\" (UID: \"31e25497-6656-4fd2-83cb-4b5102263ea0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78d76dcff6-cmm6w"
Sep 18 16:27:47.543009 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.542986    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/31e25497-6656-4fd2-83cb-4b5102263ea0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-78d76dcff6-cmm6w\" (UID: \"31e25497-6656-4fd2-83cb-4b5102263ea0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78d76dcff6-cmm6w"
Sep 18 16:27:47.551976 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.551945    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xzrn\" (UniqueName: \"kubernetes.io/projected/31e25497-6656-4fd2-83cb-4b5102263ea0-kube-api-access-7xzrn\") pod \"managed-serviceaccount-addon-agent-78d76dcff6-cmm6w\" (UID: \"31e25497-6656-4fd2-83cb-4b5102263ea0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78d76dcff6-cmm6w"
Sep 18 16:27:47.558267 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.558238    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cb9cccb-fw799"]
Sep 18 16:27:47.561276 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.561262    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cb9cccb-fw799"
Sep 18 16:27:47.563899 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.563881    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 18 16:27:47.564010 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.563900    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 18 16:27:47.564068 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.564012    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 18 16:27:47.564312 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.564297    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 18 16:27:47.572693 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.572674    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cb9cccb-fw799"]
Sep 18 16:27:47.641550 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.641516    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cc17fcb3-355f-4a52-889d-62160627aae0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-84cb9cccb-fw799\" (UID: \"cc17fcb3-355f-4a52-889d-62160627aae0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cb9cccb-fw799"
Sep 18 16:27:47.641724 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.641567    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cc17fcb3-355f-4a52-889d-62160627aae0-ca\") pod \"cluster-proxy-proxy-agent-84cb9cccb-fw799\" (UID: \"cc17fcb3-355f-4a52-889d-62160627aae0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cb9cccb-fw799"
Sep 18 16:27:47.641724 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.641593    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cc17fcb3-355f-4a52-889d-62160627aae0-hub\") pod \"cluster-proxy-proxy-agent-84cb9cccb-fw799\" (UID: \"cc17fcb3-355f-4a52-889d-62160627aae0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cb9cccb-fw799"
Sep 18 16:27:47.641724 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.641608    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc17fcb3-355f-4a52-889d-62160627aae0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-84cb9cccb-fw799\" (UID: \"cc17fcb3-355f-4a52-889d-62160627aae0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cb9cccb-fw799"
Sep 18 16:27:47.641724 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.641681    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpjc8\" (UniqueName: \"kubernetes.io/projected/cc17fcb3-355f-4a52-889d-62160627aae0-kube-api-access-lpjc8\") pod \"cluster-proxy-proxy-agent-84cb9cccb-fw799\" (UID: \"cc17fcb3-355f-4a52-889d-62160627aae0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cb9cccb-fw799"
Sep 18 16:27:47.641724 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.641719    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc17fcb3-355f-4a52-889d-62160627aae0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-84cb9cccb-fw799\" (UID: \"cc17fcb3-355f-4a52-889d-62160627aae0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cb9cccb-fw799"
Sep 18 16:27:47.742763 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.742734    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cc17fcb3-355f-4a52-889d-62160627aae0-ca\") pod \"cluster-proxy-proxy-agent-84cb9cccb-fw799\" (UID: \"cc17fcb3-355f-4a52-889d-62160627aae0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cb9cccb-fw799"
Sep 18 16:27:47.742960 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.742771    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cc17fcb3-355f-4a52-889d-62160627aae0-hub\") pod \"cluster-proxy-proxy-agent-84cb9cccb-fw799\" (UID: \"cc17fcb3-355f-4a52-889d-62160627aae0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cb9cccb-fw799"
Sep 18 16:27:47.742960 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.742792    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc17fcb3-355f-4a52-889d-62160627aae0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-84cb9cccb-fw799\" (UID: \"cc17fcb3-355f-4a52-889d-62160627aae0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cb9cccb-fw799"
Sep 18 16:27:47.742960 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.742822    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpjc8\" (UniqueName: \"kubernetes.io/projected/cc17fcb3-355f-4a52-889d-62160627aae0-kube-api-access-lpjc8\") pod \"cluster-proxy-proxy-agent-84cb9cccb-fw799\" (UID: \"cc17fcb3-355f-4a52-889d-62160627aae0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cb9cccb-fw799"
Sep 18 16:27:47.742960 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.742858    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc17fcb3-355f-4a52-889d-62160627aae0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-84cb9cccb-fw799\" (UID: \"cc17fcb3-355f-4a52-889d-62160627aae0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cb9cccb-fw799"
Sep 18 16:27:47.742960 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.742907    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cc17fcb3-355f-4a52-889d-62160627aae0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-84cb9cccb-fw799\" (UID: \"cc17fcb3-355f-4a52-889d-62160627aae0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cb9cccb-fw799"
Sep 18 16:27:47.743621 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.743544    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc17fcb3-355f-4a52-889d-62160627aae0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-84cb9cccb-fw799\" (UID: \"cc17fcb3-355f-4a52-889d-62160627aae0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cb9cccb-fw799"
Sep 18 16:27:47.745558 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.745527    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cc17fcb3-355f-4a52-889d-62160627aae0-hub\") pod \"cluster-proxy-proxy-agent-84cb9cccb-fw799\" (UID: \"cc17fcb3-355f-4a52-889d-62160627aae0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cb9cccb-fw799"
Sep 18 16:27:47.745664 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.745588    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cc17fcb3-355f-4a52-889d-62160627aae0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-84cb9cccb-fw799\" (UID: \"cc17fcb3-355f-4a52-889d-62160627aae0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cb9cccb-fw799"
Sep 18 16:27:47.745664 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.745594    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cc17fcb3-355f-4a52-889d-62160627aae0-ca\") pod \"cluster-proxy-proxy-agent-84cb9cccb-fw799\" (UID: \"cc17fcb3-355f-4a52-889d-62160627aae0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cb9cccb-fw799"
Sep 18 16:27:47.745797 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.745781    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc17fcb3-355f-4a52-889d-62160627aae0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-84cb9cccb-fw799\" (UID: \"cc17fcb3-355f-4a52-889d-62160627aae0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cb9cccb-fw799"
Sep 18 16:27:47.747514 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.747497    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78d76dcff6-cmm6w"
Sep 18 16:27:47.752012 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.751977    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpjc8\" (UniqueName: \"kubernetes.io/projected/cc17fcb3-355f-4a52-889d-62160627aae0-kube-api-access-lpjc8\") pod \"cluster-proxy-proxy-agent-84cb9cccb-fw799\" (UID: \"cc17fcb3-355f-4a52-889d-62160627aae0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cb9cccb-fw799"
Sep 18 16:27:47.870427 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:47.870393    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cb9cccb-fw799"
Sep 18 16:27:48.014662 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:48.014582    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78d76dcff6-cmm6w"]
Sep 18 16:27:48.018459 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:27:48.018430    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31e25497_6656_4fd2_83cb_4b5102263ea0.slice/crio-27694d89c44b69bcd1c8aabbb68bdfc89856a9ecb6f0cbb99023e9992fe7cad2 WatchSource:0}: Error finding container 27694d89c44b69bcd1c8aabbb68bdfc89856a9ecb6f0cbb99023e9992fe7cad2: Status 404 returned error can't find the container with id 27694d89c44b69bcd1c8aabbb68bdfc89856a9ecb6f0cbb99023e9992fe7cad2
Sep 18 16:27:48.020362 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:48.020345    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:27:48.142428 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:48.142396    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cb9cccb-fw799"]
Sep 18 16:27:48.145465 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:27:48.145435    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc17fcb3_355f_4a52_889d_62160627aae0.slice/crio-a625527ddc96caddeb29b3985613a31f68cba0c1b4ff481f3d71e4304d8483fc WatchSource:0}: Error finding container a625527ddc96caddeb29b3985613a31f68cba0c1b4ff481f3d71e4304d8483fc: Status 404 returned error can't find the container with id a625527ddc96caddeb29b3985613a31f68cba0c1b4ff481f3d71e4304d8483fc
Sep 18 16:27:48.829688 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:48.829648    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cb9cccb-fw799" event={"ID":"cc17fcb3-355f-4a52-889d-62160627aae0","Type":"ContainerStarted","Data":"a625527ddc96caddeb29b3985613a31f68cba0c1b4ff481f3d71e4304d8483fc"}
Sep 18 16:27:48.831273 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:48.831230    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78d76dcff6-cmm6w" event={"ID":"31e25497-6656-4fd2-83cb-4b5102263ea0","Type":"ContainerStarted","Data":"27694d89c44b69bcd1c8aabbb68bdfc89856a9ecb6f0cbb99023e9992fe7cad2"}
Sep 18 16:27:50.841881 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:50.841744    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78d76dcff6-cmm6w" event={"ID":"31e25497-6656-4fd2-83cb-4b5102263ea0","Type":"ContainerStarted","Data":"2bbf1c8e137e2ca90bb855146b49a759faadfa56ac861f9ea7c4c5bbddc6e8f1"}
Sep 18 16:27:50.860322 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:50.860097    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78d76dcff6-cmm6w" podStartSLOduration=1.370880546 podStartE2EDuration="3.860077425s" totalImagesPullingTime="2.489196879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:27:50.859655005 +0000 UTC m=+710.642657028" watchObservedRunningTime="2026-09-18 16:27:50.860077425 +0000 UTC m=+710.643079449"
Sep 18 16:27:53.855794 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:53.855758    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cb9cccb-fw799" event={"ID":"cc17fcb3-355f-4a52-889d-62160627aae0","Type":"ContainerStarted","Data":"dd2c5c92a65bb7efd7c36204e00415d177f6fb353cd4d8eebb2eccab438cf8f6"}
Sep 18 16:27:53.856196 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:53.855807    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cb9cccb-fw799" event={"ID":"cc17fcb3-355f-4a52-889d-62160627aae0","Type":"ContainerStarted","Data":"082ba8e742e762539975f843da9563b39bf50961af161895f5b8b64e303be07b"}
Sep 18 16:27:53.856196 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:53.855825    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cb9cccb-fw799" event={"ID":"cc17fcb3-355f-4a52-889d-62160627aae0","Type":"ContainerStarted","Data":"db3d6b9196d38aa41724c82dc20d621ebc4014535c241fb78752cb387987edbc"}
Sep 18 16:27:53.878667 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:27:53.878621    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84cb9cccb-fw799" podStartSLOduration=1.846400088 podStartE2EDuration="6.878606904s" totalImagesPullingTime="5.032206816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:27:53.876881435 +0000 UTC m=+713.659883461" watchObservedRunningTime="2026-09-18 16:27:53.878606904 +0000 UTC m=+713.661608927"
Sep 18 16:31:02.323207 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:02.323135    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 16:31:02.323770 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:02.323370    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 16:31:02.327534 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:02.327512    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 16:31:02.327812 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:02.327797    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 16:31:27.435389 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:27.435356    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-dkb7p"]
Sep 18 16:31:27.438250 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:27.438235    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-dkb7p"
Sep 18 16:31:27.440823 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:27.440803    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 18 16:31:27.441973 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:27.441941    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 18 16:31:27.442076 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:27.441989    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-6zjh8\""
Sep 18 16:31:27.448751 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:27.448732    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-dkb7p"]
Sep 18 16:31:27.472156 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:27.472124    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2528a644-8fe1-4373-b86c-ba586af40639-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-dkb7p\" (UID: \"2528a644-8fe1-4373-b86c-ba586af40639\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dkb7p"
Sep 18 16:31:27.472314 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:27.472175    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk7v7\" (UniqueName: \"kubernetes.io/projected/2528a644-8fe1-4373-b86c-ba586af40639-kube-api-access-hk7v7\") pod \"cert-manager-webhook-6486ffd997-dkb7p\" (UID: \"2528a644-8fe1-4373-b86c-ba586af40639\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dkb7p"
Sep 18 16:31:27.572652 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:27.572621    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2528a644-8fe1-4373-b86c-ba586af40639-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-dkb7p\" (UID: \"2528a644-8fe1-4373-b86c-ba586af40639\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dkb7p"
Sep 18 16:31:27.572876 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:27.572671    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk7v7\" (UniqueName: \"kubernetes.io/projected/2528a644-8fe1-4373-b86c-ba586af40639-kube-api-access-hk7v7\") pod \"cert-manager-webhook-6486ffd997-dkb7p\" (UID: \"2528a644-8fe1-4373-b86c-ba586af40639\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dkb7p"
Sep 18 16:31:27.580935 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:27.580896    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2528a644-8fe1-4373-b86c-ba586af40639-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-dkb7p\" (UID: \"2528a644-8fe1-4373-b86c-ba586af40639\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dkb7p"
Sep 18 16:31:27.581075 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:27.581057    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk7v7\" (UniqueName: \"kubernetes.io/projected/2528a644-8fe1-4373-b86c-ba586af40639-kube-api-access-hk7v7\") pod \"cert-manager-webhook-6486ffd997-dkb7p\" (UID: \"2528a644-8fe1-4373-b86c-ba586af40639\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dkb7p"
Sep 18 16:31:27.747021 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:27.746990    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-dkb7p"
Sep 18 16:31:28.017423 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:28.017398    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-dkb7p"]
Sep 18 16:31:28.020000 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:31:28.019975    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2528a644_8fe1_4373_b86c_ba586af40639.slice/crio-3d22060671a87a242751fbff58c5a513da7c54f2e8989200581f7f690081a08a WatchSource:0}: Error finding container 3d22060671a87a242751fbff58c5a513da7c54f2e8989200581f7f690081a08a: Status 404 returned error can't find the container with id 3d22060671a87a242751fbff58c5a513da7c54f2e8989200581f7f690081a08a
Sep 18 16:31:28.537223 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:28.537185    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-794bb4599-lpdxz"]
Sep 18 16:31:28.607150 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:28.607120    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-dkb7p" event={"ID":"2528a644-8fe1-4373-b86c-ba586af40639","Type":"ContainerStarted","Data":"3d22060671a87a242751fbff58c5a513da7c54f2e8989200581f7f690081a08a"}
Sep 18 16:31:31.621696 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:31.621655    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-dkb7p" event={"ID":"2528a644-8fe1-4373-b86c-ba586af40639","Type":"ContainerStarted","Data":"570e350d428044fc8df58a44e167ccb322c7a9f23899fdc78290476dee0752b6"}
Sep 18 16:31:31.622124 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:31.621786    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-dkb7p"
Sep 18 16:31:31.638967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:31.638901    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-dkb7p" podStartSLOduration=1.513787809 podStartE2EDuration="4.638887402s" totalImagesPullingTime="3.125099593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:31:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:31:31.637012881 +0000 UTC m=+931.420014904" watchObservedRunningTime="2026-09-18 16:31:31.638887402 +0000 UTC m=+931.421889425"
Sep 18 16:31:37.627631 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:37.627596    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-dkb7p"
Sep 18 16:31:53.556614 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:53.556560    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-794bb4599-lpdxz" podUID="a7422290-2994-42fc-8ef4-3f39f099080d" containerName="console" containerID="cri-o://92e1db9dcb8c47646cef3de99b3ead36e1db7917b314761e69e39689b5159b51" gracePeriod=15
Sep 18 16:31:53.704319 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:53.704286    2585 generic.go:356] "Generic (PLEG): container finished" podID="a7422290-2994-42fc-8ef4-3f39f099080d" containerID="92e1db9dcb8c47646cef3de99b3ead36e1db7917b314761e69e39689b5159b51" exitCode=0
Sep 18 16:31:53.704477 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:53.704358    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-794bb4599-lpdxz" event={"ID":"a7422290-2994-42fc-8ef4-3f39f099080d","Type":"ContainerDied","Data":"92e1db9dcb8c47646cef3de99b3ead36e1db7917b314761e69e39689b5159b51"}
Sep 18 16:31:53.795791 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:53.795769    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-794bb4599-lpdxz"
Sep 18 16:31:53.864857 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:53.864779    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7422290-2994-42fc-8ef4-3f39f099080d-console-serving-cert\") pod \"a7422290-2994-42fc-8ef4-3f39f099080d\" (UID: \"a7422290-2994-42fc-8ef4-3f39f099080d\") "
Sep 18 16:31:53.864857 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:53.864830    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wh6b\" (UniqueName: \"kubernetes.io/projected/a7422290-2994-42fc-8ef4-3f39f099080d-kube-api-access-6wh6b\") pod \"a7422290-2994-42fc-8ef4-3f39f099080d\" (UID: \"a7422290-2994-42fc-8ef4-3f39f099080d\") "
Sep 18 16:31:53.865084 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:53.864862    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7422290-2994-42fc-8ef4-3f39f099080d-console-oauth-config\") pod \"a7422290-2994-42fc-8ef4-3f39f099080d\" (UID: \"a7422290-2994-42fc-8ef4-3f39f099080d\") "
Sep 18 16:31:53.865084 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:53.864984    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7422290-2994-42fc-8ef4-3f39f099080d-trusted-ca-bundle\") pod \"a7422290-2994-42fc-8ef4-3f39f099080d\" (UID: \"a7422290-2994-42fc-8ef4-3f39f099080d\") "
Sep 18 16:31:53.865084 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:53.865048    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7422290-2994-42fc-8ef4-3f39f099080d-service-ca\") pod \"a7422290-2994-42fc-8ef4-3f39f099080d\" (UID: \"a7422290-2994-42fc-8ef4-3f39f099080d\") "
Sep 18 16:31:53.865232 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:53.865084    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7422290-2994-42fc-8ef4-3f39f099080d-oauth-serving-cert\") pod \"a7422290-2994-42fc-8ef4-3f39f099080d\" (UID: \"a7422290-2994-42fc-8ef4-3f39f099080d\") "
Sep 18 16:31:53.865232 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:53.865121    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7422290-2994-42fc-8ef4-3f39f099080d-console-config\") pod \"a7422290-2994-42fc-8ef4-3f39f099080d\" (UID: \"a7422290-2994-42fc-8ef4-3f39f099080d\") "
Sep 18 16:31:53.865428 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:53.865398    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7422290-2994-42fc-8ef4-3f39f099080d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a7422290-2994-42fc-8ef4-3f39f099080d" (UID: "a7422290-2994-42fc-8ef4-3f39f099080d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:31:53.865724 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:53.865702    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7422290-2994-42fc-8ef4-3f39f099080d-console-config" (OuterVolumeSpecName: "console-config") pod "a7422290-2994-42fc-8ef4-3f39f099080d" (UID: "a7422290-2994-42fc-8ef4-3f39f099080d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:31:53.865803 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:53.865458    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7422290-2994-42fc-8ef4-3f39f099080d-service-ca" (OuterVolumeSpecName: "service-ca") pod "a7422290-2994-42fc-8ef4-3f39f099080d" (UID: "a7422290-2994-42fc-8ef4-3f39f099080d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:31:53.865803 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:53.865755    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7422290-2994-42fc-8ef4-3f39f099080d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a7422290-2994-42fc-8ef4-3f39f099080d" (UID: "a7422290-2994-42fc-8ef4-3f39f099080d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:31:53.867203 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:53.867180    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7422290-2994-42fc-8ef4-3f39f099080d-kube-api-access-6wh6b" (OuterVolumeSpecName: "kube-api-access-6wh6b") pod "a7422290-2994-42fc-8ef4-3f39f099080d" (UID: "a7422290-2994-42fc-8ef4-3f39f099080d"). InnerVolumeSpecName "kube-api-access-6wh6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:31:53.867203 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:53.867199    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7422290-2994-42fc-8ef4-3f39f099080d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a7422290-2994-42fc-8ef4-3f39f099080d" (UID: "a7422290-2994-42fc-8ef4-3f39f099080d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:31:53.867315 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:53.867257    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7422290-2994-42fc-8ef4-3f39f099080d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a7422290-2994-42fc-8ef4-3f39f099080d" (UID: "a7422290-2994-42fc-8ef4-3f39f099080d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:31:53.966552 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:53.966519    2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7422290-2994-42fc-8ef4-3f39f099080d-trusted-ca-bundle\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:31:53.966552 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:53.966547    2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7422290-2994-42fc-8ef4-3f39f099080d-service-ca\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:31:53.966552 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:53.966556    2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7422290-2994-42fc-8ef4-3f39f099080d-oauth-serving-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:31:53.966552 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:53.966564    2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7422290-2994-42fc-8ef4-3f39f099080d-console-config\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:31:53.966805 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:53.966572    2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7422290-2994-42fc-8ef4-3f39f099080d-console-serving-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:31:53.966805 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:53.966582    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wh6b\" (UniqueName: \"kubernetes.io/projected/a7422290-2994-42fc-8ef4-3f39f099080d-kube-api-access-6wh6b\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:31:53.966805 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:53.966590    2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7422290-2994-42fc-8ef4-3f39f099080d-console-oauth-config\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:31:54.709816 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:54.709724    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-794bb4599-lpdxz" event={"ID":"a7422290-2994-42fc-8ef4-3f39f099080d","Type":"ContainerDied","Data":"96b7a7715b376e614862797d0d7aa7027a0d3cfe8ed5cd2c52dbe7ea6a663bf8"}
Sep 18 16:31:54.709816 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:54.709752    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-794bb4599-lpdxz"
Sep 18 16:31:54.709816 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:54.709793    2585 scope.go:117] "RemoveContainer" containerID="92e1db9dcb8c47646cef3de99b3ead36e1db7917b314761e69e39689b5159b51"
Sep 18 16:31:54.753667 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:54.753639    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-794bb4599-lpdxz"]
Sep 18 16:31:54.757472 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:54.757446    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-794bb4599-lpdxz"]
Sep 18 16:31:55.557215 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.557180    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"]
Sep 18 16:31:55.557512 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.557496    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7422290-2994-42fc-8ef4-3f39f099080d" containerName="console"
Sep 18 16:31:55.557592 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.557515    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7422290-2994-42fc-8ef4-3f39f099080d" containerName="console"
Sep 18 16:31:55.557648 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.557604    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7422290-2994-42fc-8ef4-3f39f099080d" containerName="console"
Sep 18 16:31:55.560708 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.560687    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:31:55.563212 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.563192    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 16:31:55.563326 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.563194    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-7vltq\""
Sep 18 16:31:55.563326 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.563313    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 16:31:55.563447 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.563355    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 16:31:55.573302 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.573282    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"]
Sep 18 16:31:55.679125 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.679090    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a522b31-c301-4372-a729-93e898489653-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d\" (UID: \"3a522b31-c301-4372-a729-93e898489653\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:31:55.679125 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.679125    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a522b31-c301-4372-a729-93e898489653-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d\" (UID: \"3a522b31-c301-4372-a729-93e898489653\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:31:55.679359 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.679146    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a522b31-c301-4372-a729-93e898489653-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d\" (UID: \"3a522b31-c301-4372-a729-93e898489653\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:31:55.679359 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.679163    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a522b31-c301-4372-a729-93e898489653-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d\" (UID: \"3a522b31-c301-4372-a729-93e898489653\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:31:55.679359 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.679230    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a522b31-c301-4372-a729-93e898489653-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d\" (UID: \"3a522b31-c301-4372-a729-93e898489653\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:31:55.679359 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.679262    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjcrr\" (UniqueName: \"kubernetes.io/projected/3a522b31-c301-4372-a729-93e898489653-kube-api-access-bjcrr\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d\" (UID: \"3a522b31-c301-4372-a729-93e898489653\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:31:55.679359 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.679294    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a522b31-c301-4372-a729-93e898489653-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d\" (UID: \"3a522b31-c301-4372-a729-93e898489653\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:31:55.679359 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.679312    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a522b31-c301-4372-a729-93e898489653-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d\" (UID: \"3a522b31-c301-4372-a729-93e898489653\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:31:55.679359 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.679339    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a522b31-c301-4372-a729-93e898489653-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d\" (UID: \"3a522b31-c301-4372-a729-93e898489653\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:31:55.780562 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.780524    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a522b31-c301-4372-a729-93e898489653-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d\" (UID: \"3a522b31-c301-4372-a729-93e898489653\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:31:55.780562 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.780561    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjcrr\" (UniqueName: \"kubernetes.io/projected/3a522b31-c301-4372-a729-93e898489653-kube-api-access-bjcrr\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d\" (UID: \"3a522b31-c301-4372-a729-93e898489653\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:31:55.781103 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.780696    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a522b31-c301-4372-a729-93e898489653-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d\" (UID: \"3a522b31-c301-4372-a729-93e898489653\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:31:55.781103 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.780745    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a522b31-c301-4372-a729-93e898489653-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d\" (UID: \"3a522b31-c301-4372-a729-93e898489653\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:31:55.781103 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.780789    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a522b31-c301-4372-a729-93e898489653-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d\" (UID: \"3a522b31-c301-4372-a729-93e898489653\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:31:55.781103 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.780878    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a522b31-c301-4372-a729-93e898489653-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d\" (UID: \"3a522b31-c301-4372-a729-93e898489653\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:31:55.781103 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.780906    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a522b31-c301-4372-a729-93e898489653-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d\" (UID: \"3a522b31-c301-4372-a729-93e898489653\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:31:55.781103 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.780976    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a522b31-c301-4372-a729-93e898489653-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d\" (UID: \"3a522b31-c301-4372-a729-93e898489653\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:31:55.781103 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.780996    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a522b31-c301-4372-a729-93e898489653-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d\" (UID: \"3a522b31-c301-4372-a729-93e898489653\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:31:55.781103 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.781003    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a522b31-c301-4372-a729-93e898489653-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d\" (UID: \"3a522b31-c301-4372-a729-93e898489653\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:31:55.781103 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.781106    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a522b31-c301-4372-a729-93e898489653-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d\" (UID: \"3a522b31-c301-4372-a729-93e898489653\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:31:55.781479 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.781371    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a522b31-c301-4372-a729-93e898489653-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d\" (UID: \"3a522b31-c301-4372-a729-93e898489653\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:31:55.781479 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.781418    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a522b31-c301-4372-a729-93e898489653-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d\" (UID: \"3a522b31-c301-4372-a729-93e898489653\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:31:55.781479 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.781444    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a522b31-c301-4372-a729-93e898489653-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d\" (UID: \"3a522b31-c301-4372-a729-93e898489653\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:31:55.783238 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.783217    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a522b31-c301-4372-a729-93e898489653-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d\" (UID: \"3a522b31-c301-4372-a729-93e898489653\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:31:55.783379 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.783339    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a522b31-c301-4372-a729-93e898489653-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d\" (UID: \"3a522b31-c301-4372-a729-93e898489653\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:31:55.790145 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.790115    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a522b31-c301-4372-a729-93e898489653-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d\" (UID: \"3a522b31-c301-4372-a729-93e898489653\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:31:55.790392 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.790372    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjcrr\" (UniqueName: \"kubernetes.io/projected/3a522b31-c301-4372-a729-93e898489653-kube-api-access-bjcrr\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d\" (UID: \"3a522b31-c301-4372-a729-93e898489653\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:31:55.872043 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:55.871945    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:31:56.145238 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:56.145213    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"]
Sep 18 16:31:56.147384 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:31:56.147358    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a522b31_c301_4372_a729_93e898489653.slice/crio-15fa595bbad7dbfbb8ab65c4e772a18688662f9d23c4883ce94cd2be7a6b4069 WatchSource:0}: Error finding container 15fa595bbad7dbfbb8ab65c4e772a18688662f9d23c4883ce94cd2be7a6b4069: Status 404 returned error can't find the container with id 15fa595bbad7dbfbb8ab65c4e772a18688662f9d23c4883ce94cd2be7a6b4069
Sep 18 16:31:56.391282 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:56.391248    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7422290-2994-42fc-8ef4-3f39f099080d" path="/var/lib/kubelet/pods/a7422290-2994-42fc-8ef4-3f39f099080d/volumes"
Sep 18 16:31:56.719573 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:56.719538    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d" event={"ID":"3a522b31-c301-4372-a729-93e898489653","Type":"ContainerStarted","Data":"15fa595bbad7dbfbb8ab65c4e772a18688662f9d23c4883ce94cd2be7a6b4069"}
Sep 18 16:31:59.068231 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:59.068195    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 16:31:59.068543 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:59.068270    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 16:31:59.068543 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:59.068301    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 16:31:59.734769 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:59.734739    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d" event={"ID":"3a522b31-c301-4372-a729-93e898489653","Type":"ContainerStarted","Data":"3076a97e832449dfa1acd9f6d48709b3cbd421c0f70926f720ba98650a558300"}
Sep 18 16:31:59.757659 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:59.757615    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d" podStartSLOduration=1.843429141 podStartE2EDuration="4.757601621s" totalImagesPullingTime="2.91417248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:31:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:31:59.756104573 +0000 UTC m=+959.539106609" watchObservedRunningTime="2026-09-18 16:31:59.757601621 +0000 UTC m=+959.540603644"
Sep 18 16:31:59.872985 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:31:59.872958    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:32:00.877294 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:00.877263    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:32:01.742576 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:01.742549    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:32:01.743748 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:01.743725    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:32:07.490613 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:07.490577    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6fd4df8954-lhqzz"]
Sep 18 16:32:07.494980 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:07.494958    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-lhqzz"
Sep 18 16:32:07.499100 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:07.499078    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 18 16:32:07.500309 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:07.500288    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 18 16:32:07.500569 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:07.500549    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-n5fqj\""
Sep 18 16:32:07.500569 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:07.500564    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:32:07.500712 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:07.500559    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 18 16:32:07.500712 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:07.500631    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 18 16:32:07.513694 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:07.513663    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6fd4df8954-lhqzz"]
Sep 18 16:32:07.576769 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:07.576737    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c098bf08-79fd-499a-88b0-deec4754cc75-manager-config\") pod \"lws-controller-manager-6fd4df8954-lhqzz\" (UID: \"c098bf08-79fd-499a-88b0-deec4754cc75\") " pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-lhqzz"
Sep 18 16:32:07.576769 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:07.576771    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c098bf08-79fd-499a-88b0-deec4754cc75-metrics-cert\") pod \"lws-controller-manager-6fd4df8954-lhqzz\" (UID: \"c098bf08-79fd-499a-88b0-deec4754cc75\") " pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-lhqzz"
Sep 18 16:32:07.577010 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:07.576794    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c098bf08-79fd-499a-88b0-deec4754cc75-cert\") pod \"lws-controller-manager-6fd4df8954-lhqzz\" (UID: \"c098bf08-79fd-499a-88b0-deec4754cc75\") " pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-lhqzz"
Sep 18 16:32:07.577010 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:07.576885    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6lzr\" (UniqueName: \"kubernetes.io/projected/c098bf08-79fd-499a-88b0-deec4754cc75-kube-api-access-k6lzr\") pod \"lws-controller-manager-6fd4df8954-lhqzz\" (UID: \"c098bf08-79fd-499a-88b0-deec4754cc75\") " pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-lhqzz"
Sep 18 16:32:07.677449 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:07.677406    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6lzr\" (UniqueName: \"kubernetes.io/projected/c098bf08-79fd-499a-88b0-deec4754cc75-kube-api-access-k6lzr\") pod \"lws-controller-manager-6fd4df8954-lhqzz\" (UID: \"c098bf08-79fd-499a-88b0-deec4754cc75\") " pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-lhqzz"
Sep 18 16:32:07.677676 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:07.677500    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c098bf08-79fd-499a-88b0-deec4754cc75-manager-config\") pod \"lws-controller-manager-6fd4df8954-lhqzz\" (UID: \"c098bf08-79fd-499a-88b0-deec4754cc75\") " pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-lhqzz"
Sep 18 16:32:07.677676 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:07.677529    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c098bf08-79fd-499a-88b0-deec4754cc75-metrics-cert\") pod \"lws-controller-manager-6fd4df8954-lhqzz\" (UID: \"c098bf08-79fd-499a-88b0-deec4754cc75\") " pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-lhqzz"
Sep 18 16:32:07.677676 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:07.677557    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c098bf08-79fd-499a-88b0-deec4754cc75-cert\") pod \"lws-controller-manager-6fd4df8954-lhqzz\" (UID: \"c098bf08-79fd-499a-88b0-deec4754cc75\") " pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-lhqzz"
Sep 18 16:32:07.678201 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:07.678180    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c098bf08-79fd-499a-88b0-deec4754cc75-manager-config\") pod \"lws-controller-manager-6fd4df8954-lhqzz\" (UID: \"c098bf08-79fd-499a-88b0-deec4754cc75\") " pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-lhqzz"
Sep 18 16:32:07.680247 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:07.680226    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c098bf08-79fd-499a-88b0-deec4754cc75-cert\") pod \"lws-controller-manager-6fd4df8954-lhqzz\" (UID: \"c098bf08-79fd-499a-88b0-deec4754cc75\") " pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-lhqzz"
Sep 18 16:32:07.680326 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:07.680272    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c098bf08-79fd-499a-88b0-deec4754cc75-metrics-cert\") pod \"lws-controller-manager-6fd4df8954-lhqzz\" (UID: \"c098bf08-79fd-499a-88b0-deec4754cc75\") " pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-lhqzz"
Sep 18 16:32:07.688063 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:07.688042    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6lzr\" (UniqueName: \"kubernetes.io/projected/c098bf08-79fd-499a-88b0-deec4754cc75-kube-api-access-k6lzr\") pod \"lws-controller-manager-6fd4df8954-lhqzz\" (UID: \"c098bf08-79fd-499a-88b0-deec4754cc75\") " pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-lhqzz"
Sep 18 16:32:07.804374 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:07.804285    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-lhqzz"
Sep 18 16:32:08.079380 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:08.079266    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6fd4df8954-lhqzz"]
Sep 18 16:32:08.082176 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:32:08.082142    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc098bf08_79fd_499a_88b0_deec4754cc75.slice/crio-0244fafdde3d565a25dfcda3375135165f5ac88b4025fbcfa0dd553acd0406b7 WatchSource:0}: Error finding container 0244fafdde3d565a25dfcda3375135165f5ac88b4025fbcfa0dd553acd0406b7: Status 404 returned error can't find the container with id 0244fafdde3d565a25dfcda3375135165f5ac88b4025fbcfa0dd553acd0406b7
Sep 18 16:32:08.767734 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:08.767700    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-lhqzz" event={"ID":"c098bf08-79fd-499a-88b0-deec4754cc75","Type":"ContainerStarted","Data":"0244fafdde3d565a25dfcda3375135165f5ac88b4025fbcfa0dd553acd0406b7"}
Sep 18 16:32:11.780705 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:11.780617    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-lhqzz" event={"ID":"c098bf08-79fd-499a-88b0-deec4754cc75","Type":"ContainerStarted","Data":"55233a78c552f6d3869991157936621d02f4a4a72f859920fad56bf5881d594c"}
Sep 18 16:32:11.781174 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:11.780744    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-lhqzz"
Sep 18 16:32:11.798273 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:11.798228    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-lhqzz" podStartSLOduration=2.130576896 podStartE2EDuration="4.798214308s" totalImagesPullingTime="2.667637412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:32:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:32:11.7973041 +0000 UTC m=+971.580306123" watchObservedRunningTime="2026-09-18 16:32:11.798214308 +0000 UTC m=+971.581216331"
Sep 18 16:32:22.786411 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:22.786381    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6fd4df8954-lhqzz"
Sep 18 16:32:34.730704 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:34.730664    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-r28wr"]
Sep 18 16:32:34.736097 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:34.736076    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r28wr"
Sep 18 16:32:34.739088 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:34.739064    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 18 16:32:34.739228 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:34.739098    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 16:32:34.739228 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:34.739073    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 18 16:32:34.739228 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:34.739134    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-24526\""
Sep 18 16:32:34.740091 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:34.740071    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 16:32:34.742692 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:34.742672    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-r28wr"]
Sep 18 16:32:34.783839 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:34.783802    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr7d6\" (UniqueName: \"kubernetes.io/projected/dee1080d-684c-452a-a15f-5f39b67634e9-kube-api-access-zr7d6\") pod \"kuadrant-console-plugin-fbdd4f445-r28wr\" (UID: \"dee1080d-684c-452a-a15f-5f39b67634e9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r28wr"
Sep 18 16:32:34.783990 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:34.783874    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/dee1080d-684c-452a-a15f-5f39b67634e9-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-r28wr\" (UID: \"dee1080d-684c-452a-a15f-5f39b67634e9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r28wr"
Sep 18 16:32:34.783990 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:34.783955    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dee1080d-684c-452a-a15f-5f39b67634e9-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-r28wr\" (UID: \"dee1080d-684c-452a-a15f-5f39b67634e9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r28wr"
Sep 18 16:32:34.884641 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:34.884609    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr7d6\" (UniqueName: \"kubernetes.io/projected/dee1080d-684c-452a-a15f-5f39b67634e9-kube-api-access-zr7d6\") pod \"kuadrant-console-plugin-fbdd4f445-r28wr\" (UID: \"dee1080d-684c-452a-a15f-5f39b67634e9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r28wr"
Sep 18 16:32:34.884778 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:34.884660    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/dee1080d-684c-452a-a15f-5f39b67634e9-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-r28wr\" (UID: \"dee1080d-684c-452a-a15f-5f39b67634e9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r28wr"
Sep 18 16:32:34.884778 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:34.884692    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dee1080d-684c-452a-a15f-5f39b67634e9-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-r28wr\" (UID: \"dee1080d-684c-452a-a15f-5f39b67634e9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r28wr"
Sep 18 16:32:34.885348 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:34.885320    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dee1080d-684c-452a-a15f-5f39b67634e9-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-r28wr\" (UID: \"dee1080d-684c-452a-a15f-5f39b67634e9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r28wr"
Sep 18 16:32:34.887332 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:34.887314    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/dee1080d-684c-452a-a15f-5f39b67634e9-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-r28wr\" (UID: \"dee1080d-684c-452a-a15f-5f39b67634e9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r28wr"
Sep 18 16:32:34.894245 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:34.894216    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr7d6\" (UniqueName: \"kubernetes.io/projected/dee1080d-684c-452a-a15f-5f39b67634e9-kube-api-access-zr7d6\") pod \"kuadrant-console-plugin-fbdd4f445-r28wr\" (UID: \"dee1080d-684c-452a-a15f-5f39b67634e9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r28wr"
Sep 18 16:32:35.045993 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:35.045927    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r28wr"
Sep 18 16:32:35.316191 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:35.316167    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-r28wr"]
Sep 18 16:32:35.318945 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:32:35.318896    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddee1080d_684c_452a_a15f_5f39b67634e9.slice/crio-93ce7d80fa6ea3919344966807a29384b2b2ec30af805bfc4797a1b2bb7f9c9f WatchSource:0}: Error finding container 93ce7d80fa6ea3919344966807a29384b2b2ec30af805bfc4797a1b2bb7f9c9f: Status 404 returned error can't find the container with id 93ce7d80fa6ea3919344966807a29384b2b2ec30af805bfc4797a1b2bb7f9c9f
Sep 18 16:32:35.867693 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:35.867659    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r28wr" event={"ID":"dee1080d-684c-452a-a15f-5f39b67634e9","Type":"ContainerStarted","Data":"93ce7d80fa6ea3919344966807a29384b2b2ec30af805bfc4797a1b2bb7f9c9f"}
Sep 18 16:32:42.898008 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:42.897971    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r28wr" event={"ID":"dee1080d-684c-452a-a15f-5f39b67634e9","Type":"ContainerStarted","Data":"7ba68b27ab2f4a315c960321b48020565394e49a474d26671fc71cc35fd9e2cb"}
Sep 18 16:32:42.916600 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:32:42.916549    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r28wr" podStartSLOduration=2.211584592 podStartE2EDuration="8.916535157s" totalImagesPullingTime="6.704950565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:32:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:32:42.915614661 +0000 UTC m=+1002.698616685" watchObservedRunningTime="2026-09-18 16:32:42.916535157 +0000 UTC m=+1002.699537181"
Sep 18 16:33:20.607248 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:33:20.607207    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-qsv4l"]
Sep 18 16:33:20.609531 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:33:20.609513    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qsv4l"
Sep 18 16:33:20.612725 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:33:20.612706    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cxxmj\""
Sep 18 16:33:20.618774 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:33:20.618755    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qsv4l"]
Sep 18 16:33:20.741389 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:33:20.741348    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7nps\" (UniqueName: \"kubernetes.io/projected/a8cefe33-c85c-452e-a52c-1f9017ad72bb-kube-api-access-z7nps\") pod \"authorino-59c974bd48-qsv4l\" (UID: \"a8cefe33-c85c-452e-a52c-1f9017ad72bb\") " pod="kuadrant-system/authorino-59c974bd48-qsv4l"
Sep 18 16:33:20.842505 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:33:20.842468    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7nps\" (UniqueName: \"kubernetes.io/projected/a8cefe33-c85c-452e-a52c-1f9017ad72bb-kube-api-access-z7nps\") pod \"authorino-59c974bd48-qsv4l\" (UID: \"a8cefe33-c85c-452e-a52c-1f9017ad72bb\") " pod="kuadrant-system/authorino-59c974bd48-qsv4l"
Sep 18 16:33:20.851932 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:33:20.851883    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7nps\" (UniqueName: \"kubernetes.io/projected/a8cefe33-c85c-452e-a52c-1f9017ad72bb-kube-api-access-z7nps\") pod \"authorino-59c974bd48-qsv4l\" (UID: \"a8cefe33-c85c-452e-a52c-1f9017ad72bb\") " pod="kuadrant-system/authorino-59c974bd48-qsv4l"
Sep 18 16:33:20.918219 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:33:20.918119    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qsv4l"
Sep 18 16:33:21.186073 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:33:21.185988    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qsv4l"]
Sep 18 16:33:21.188244 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:33:21.188203    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8cefe33_c85c_452e_a52c_1f9017ad72bb.slice/crio-3a22d4dcb750e6542e701e4850e314057fc777bcd1ad12316dccfead83c434f4 WatchSource:0}: Error finding container 3a22d4dcb750e6542e701e4850e314057fc777bcd1ad12316dccfead83c434f4: Status 404 returned error can't find the container with id 3a22d4dcb750e6542e701e4850e314057fc777bcd1ad12316dccfead83c434f4
Sep 18 16:33:21.189564 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:33:21.189544    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:33:22.048403 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:33:22.048373    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qsv4l" event={"ID":"a8cefe33-c85c-452e-a52c-1f9017ad72bb","Type":"ContainerStarted","Data":"3a22d4dcb750e6542e701e4850e314057fc777bcd1ad12316dccfead83c434f4"}
Sep 18 16:33:29.077299 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:33:29.077262    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qsv4l" event={"ID":"a8cefe33-c85c-452e-a52c-1f9017ad72bb","Type":"ContainerStarted","Data":"5b43972037c694fc7ba5689f12e0b6967fafcec4310f2e482ca3c84ef1b93add"}
Sep 18 16:33:29.094367 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:33:29.094310    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-qsv4l" podStartSLOduration=2.033583831 podStartE2EDuration="9.094295002s" totalImagesPullingTime="7.060711171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:33:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:33:29.093460246 +0000 UTC m=+1048.876462281" watchObservedRunningTime="2026-09-18 16:33:29.094295002 +0000 UTC m=+1048.877297069"
Sep 18 16:33:44.134414 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:33:44.134328    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qsv4l"]
Sep 18 16:33:44.134801 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:33:44.134500    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-qsv4l" podUID="a8cefe33-c85c-452e-a52c-1f9017ad72bb" containerName="authorino" containerID="cri-o://5b43972037c694fc7ba5689f12e0b6967fafcec4310f2e482ca3c84ef1b93add" gracePeriod=30
Sep 18 16:33:44.372370 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:33:44.372346    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qsv4l"
Sep 18 16:33:44.429798 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:33:44.429735    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7nps\" (UniqueName: \"kubernetes.io/projected/a8cefe33-c85c-452e-a52c-1f9017ad72bb-kube-api-access-z7nps\") pod \"a8cefe33-c85c-452e-a52c-1f9017ad72bb\" (UID: \"a8cefe33-c85c-452e-a52c-1f9017ad72bb\") "
Sep 18 16:33:44.432008 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:33:44.431983    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8cefe33-c85c-452e-a52c-1f9017ad72bb-kube-api-access-z7nps" (OuterVolumeSpecName: "kube-api-access-z7nps") pod "a8cefe33-c85c-452e-a52c-1f9017ad72bb" (UID: "a8cefe33-c85c-452e-a52c-1f9017ad72bb"). InnerVolumeSpecName "kube-api-access-z7nps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:33:44.530555 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:33:44.530531    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7nps\" (UniqueName: \"kubernetes.io/projected/a8cefe33-c85c-452e-a52c-1f9017ad72bb-kube-api-access-z7nps\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:33:45.141035 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:33:45.140998    2585 generic.go:356] "Generic (PLEG): container finished" podID="a8cefe33-c85c-452e-a52c-1f9017ad72bb" containerID="5b43972037c694fc7ba5689f12e0b6967fafcec4310f2e482ca3c84ef1b93add" exitCode=0
Sep 18 16:33:45.141035 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:33:45.141037    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qsv4l"
Sep 18 16:33:45.141495 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:33:45.141084    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qsv4l" event={"ID":"a8cefe33-c85c-452e-a52c-1f9017ad72bb","Type":"ContainerDied","Data":"5b43972037c694fc7ba5689f12e0b6967fafcec4310f2e482ca3c84ef1b93add"}
Sep 18 16:33:45.141495 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:33:45.141118    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qsv4l" event={"ID":"a8cefe33-c85c-452e-a52c-1f9017ad72bb","Type":"ContainerDied","Data":"3a22d4dcb750e6542e701e4850e314057fc777bcd1ad12316dccfead83c434f4"}
Sep 18 16:33:45.141495 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:33:45.141138    2585 scope.go:117] "RemoveContainer" containerID="5b43972037c694fc7ba5689f12e0b6967fafcec4310f2e482ca3c84ef1b93add"
Sep 18 16:33:45.150180 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:33:45.150162    2585 scope.go:117] "RemoveContainer" containerID="5b43972037c694fc7ba5689f12e0b6967fafcec4310f2e482ca3c84ef1b93add"
Sep 18 16:33:45.150473 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:33:45.150455    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b43972037c694fc7ba5689f12e0b6967fafcec4310f2e482ca3c84ef1b93add\": container with ID starting with 5b43972037c694fc7ba5689f12e0b6967fafcec4310f2e482ca3c84ef1b93add not found: ID does not exist" containerID="5b43972037c694fc7ba5689f12e0b6967fafcec4310f2e482ca3c84ef1b93add"
Sep 18 16:33:45.150513 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:33:45.150482    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b43972037c694fc7ba5689f12e0b6967fafcec4310f2e482ca3c84ef1b93add"} err="failed to get container status \"5b43972037c694fc7ba5689f12e0b6967fafcec4310f2e482ca3c84ef1b93add\": rpc error: code = NotFound desc = could not find container \"5b43972037c694fc7ba5689f12e0b6967fafcec4310f2e482ca3c84ef1b93add\": container with ID starting with 5b43972037c694fc7ba5689f12e0b6967fafcec4310f2e482ca3c84ef1b93add not found: ID does not exist"
Sep 18 16:33:45.161794 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:33:45.161764    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qsv4l"]
Sep 18 16:33:45.167484 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:33:45.167463    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qsv4l"]
Sep 18 16:33:46.391723 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:33:46.391692    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8cefe33-c85c-452e-a52c-1f9017ad72bb" path="/var/lib/kubelet/pods/a8cefe33-c85c-452e-a52c-1f9017ad72bb/volumes"
Sep 18 16:34:14.011177 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:34:14.011140    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-6888f9f64c-t7dlt"]
Sep 18 16:34:14.011610 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:34:14.011535    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8cefe33-c85c-452e-a52c-1f9017ad72bb" containerName="authorino"
Sep 18 16:34:14.011610 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:34:14.011549    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8cefe33-c85c-452e-a52c-1f9017ad72bb" containerName="authorino"
Sep 18 16:34:14.011728 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:34:14.011617    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8cefe33-c85c-452e-a52c-1f9017ad72bb" containerName="authorino"
Sep 18 16:34:14.016640 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:34:14.016616    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6888f9f64c-t7dlt"
Sep 18 16:34:14.019325 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:34:14.019303    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 16:34:14.019465 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:34:14.019302    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 18 16:34:14.020300 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:34:14.020284    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-jzpgd\""
Sep 18 16:34:14.020394 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:34:14.020287    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 16:34:14.023583 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:34:14.023563    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6888f9f64c-t7dlt"]
Sep 18 16:34:14.190327 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:34:14.190299    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f11daaa2-53bc-4a9d-b447-87a3e8a90d14-cert\") pod \"llmisvc-controller-manager-6888f9f64c-t7dlt\" (UID: \"f11daaa2-53bc-4a9d-b447-87a3e8a90d14\") " pod="kserve/llmisvc-controller-manager-6888f9f64c-t7dlt"
Sep 18 16:34:14.190504 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:34:14.190345    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nqss\" (UniqueName: \"kubernetes.io/projected/f11daaa2-53bc-4a9d-b447-87a3e8a90d14-kube-api-access-4nqss\") pod \"llmisvc-controller-manager-6888f9f64c-t7dlt\" (UID: \"f11daaa2-53bc-4a9d-b447-87a3e8a90d14\") " pod="kserve/llmisvc-controller-manager-6888f9f64c-t7dlt"
Sep 18 16:34:14.291445 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:34:14.291359    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f11daaa2-53bc-4a9d-b447-87a3e8a90d14-cert\") pod \"llmisvc-controller-manager-6888f9f64c-t7dlt\" (UID: \"f11daaa2-53bc-4a9d-b447-87a3e8a90d14\") " pod="kserve/llmisvc-controller-manager-6888f9f64c-t7dlt"
Sep 18 16:34:14.291445 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:34:14.291405    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nqss\" (UniqueName: \"kubernetes.io/projected/f11daaa2-53bc-4a9d-b447-87a3e8a90d14-kube-api-access-4nqss\") pod \"llmisvc-controller-manager-6888f9f64c-t7dlt\" (UID: \"f11daaa2-53bc-4a9d-b447-87a3e8a90d14\") " pod="kserve/llmisvc-controller-manager-6888f9f64c-t7dlt"
Sep 18 16:34:14.291675 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:34:14.291522    2585 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 18 16:34:14.291675 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:34:14.291604    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f11daaa2-53bc-4a9d-b447-87a3e8a90d14-cert podName:f11daaa2-53bc-4a9d-b447-87a3e8a90d14 nodeName:}" failed. No retries permitted until 2026-09-18 16:34:14.791583569 +0000 UTC m=+1094.574585570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f11daaa2-53bc-4a9d-b447-87a3e8a90d14-cert") pod "llmisvc-controller-manager-6888f9f64c-t7dlt" (UID: "f11daaa2-53bc-4a9d-b447-87a3e8a90d14") : secret "llmisvc-webhook-server-cert" not found
Sep 18 16:34:14.314836 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:34:14.312432    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nqss\" (UniqueName: \"kubernetes.io/projected/f11daaa2-53bc-4a9d-b447-87a3e8a90d14-kube-api-access-4nqss\") pod \"llmisvc-controller-manager-6888f9f64c-t7dlt\" (UID: \"f11daaa2-53bc-4a9d-b447-87a3e8a90d14\") " pod="kserve/llmisvc-controller-manager-6888f9f64c-t7dlt"
Sep 18 16:34:14.795997 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:34:14.795965    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f11daaa2-53bc-4a9d-b447-87a3e8a90d14-cert\") pod \"llmisvc-controller-manager-6888f9f64c-t7dlt\" (UID: \"f11daaa2-53bc-4a9d-b447-87a3e8a90d14\") " pod="kserve/llmisvc-controller-manager-6888f9f64c-t7dlt"
Sep 18 16:34:14.798463 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:34:14.798434    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f11daaa2-53bc-4a9d-b447-87a3e8a90d14-cert\") pod \"llmisvc-controller-manager-6888f9f64c-t7dlt\" (UID: \"f11daaa2-53bc-4a9d-b447-87a3e8a90d14\") " pod="kserve/llmisvc-controller-manager-6888f9f64c-t7dlt"
Sep 18 16:34:14.929499 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:34:14.929459    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6888f9f64c-t7dlt"
Sep 18 16:34:15.197773 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:34:15.197746    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6888f9f64c-t7dlt"]
Sep 18 16:34:15.200565 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:34:15.200534    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf11daaa2_53bc_4a9d_b447_87a3e8a90d14.slice/crio-d0b0ed34b3aba0f65bd18373a7c28f88814621ab988dac7a58597cae4b41a177 WatchSource:0}: Error finding container d0b0ed34b3aba0f65bd18373a7c28f88814621ab988dac7a58597cae4b41a177: Status 404 returned error can't find the container with id d0b0ed34b3aba0f65bd18373a7c28f88814621ab988dac7a58597cae4b41a177
Sep 18 16:34:15.253287 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:34:15.253264    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6888f9f64c-t7dlt" event={"ID":"f11daaa2-53bc-4a9d-b447-87a3e8a90d14","Type":"ContainerStarted","Data":"d0b0ed34b3aba0f65bd18373a7c28f88814621ab988dac7a58597cae4b41a177"}
Sep 18 16:34:18.268019 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:34:18.267988    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6888f9f64c-t7dlt" event={"ID":"f11daaa2-53bc-4a9d-b447-87a3e8a90d14","Type":"ContainerStarted","Data":"e16702c74287717de6053829a06702e236bf1551f37054b6ea5b424ca08d9c65"}
Sep 18 16:34:18.268438 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:34:18.268092    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-6888f9f64c-t7dlt"
Sep 18 16:34:18.285239 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:34:18.285191    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-6888f9f64c-t7dlt" podStartSLOduration=2.872609131 podStartE2EDuration="5.285165982s" totalImagesPullingTime="2.412556851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:34:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:34:18.283411978 +0000 UTC m=+1098.066414002" watchObservedRunningTime="2026-09-18 16:34:18.285165982 +0000 UTC m=+1098.068168038"
Sep 18 16:34:49.273902 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:34:49.273869    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-6888f9f64c-t7dlt"
Sep 18 16:35:47.462226 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:35:47.462194    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-6dcvf"]
Sep 18 16:35:47.465255 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:35:47.465238    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-6dcvf"
Sep 18 16:35:47.468141 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:35:47.468110    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 18 16:35:47.468263 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:35:47.468139    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-28qmm\""
Sep 18 16:35:47.468263 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:35:47.468139    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 18 16:35:47.473901 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:35:47.473878    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-6dcvf"]
Sep 18 16:35:47.505002 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:35:47.504974    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx9g5\" (UniqueName: \"kubernetes.io/projected/ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7-kube-api-access-tx9g5\") pod \"s3-init-6dcvf\" (UID: \"ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7\") " pod="kserve/s3-init-6dcvf"
Sep 18 16:35:47.505345 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:35:47.505317    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7-model-cache\") pod \"s3-init-6dcvf\" (UID: \"ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7\") " pod="kserve/s3-init-6dcvf"
Sep 18 16:35:47.606460 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:35:47.606422    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7-model-cache\") pod \"s3-init-6dcvf\" (UID: \"ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7\") " pod="kserve/s3-init-6dcvf"
Sep 18 16:35:47.606460 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:35:47.606466    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx9g5\" (UniqueName: \"kubernetes.io/projected/ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7-kube-api-access-tx9g5\") pod \"s3-init-6dcvf\" (UID: \"ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7\") " pod="kserve/s3-init-6dcvf"
Sep 18 16:35:47.606808 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:35:47.606788    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7-model-cache\") pod \"s3-init-6dcvf\" (UID: \"ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7\") " pod="kserve/s3-init-6dcvf"
Sep 18 16:35:47.615499 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:35:47.615469    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx9g5\" (UniqueName: \"kubernetes.io/projected/ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7-kube-api-access-tx9g5\") pod \"s3-init-6dcvf\" (UID: \"ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7\") " pod="kserve/s3-init-6dcvf"
Sep 18 16:35:47.775058 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:35:47.775026    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-6dcvf"
Sep 18 16:35:48.046066 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:35:48.045844    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-6dcvf"]
Sep 18 16:35:48.049054 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:35:48.049024    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef5df3ca_2b86_4b0e_abd2_40cc92dd62c7.slice/crio-44693ec70e02b4fb672adc84d64b95ea6088cc972d67c0ee68cce70cfb177867 WatchSource:0}: Error finding container 44693ec70e02b4fb672adc84d64b95ea6088cc972d67c0ee68cce70cfb177867: Status 404 returned error can't find the container with id 44693ec70e02b4fb672adc84d64b95ea6088cc972d67c0ee68cce70cfb177867
Sep 18 16:35:48.593136 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:35:48.593104    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6dcvf" event={"ID":"ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7","Type":"ContainerStarted","Data":"44693ec70e02b4fb672adc84d64b95ea6088cc972d67c0ee68cce70cfb177867"}
Sep 18 16:35:52.611764 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:35:52.611713    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6dcvf" event={"ID":"ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7","Type":"ContainerStarted","Data":"1efd23a4aad103ddafd94ddfc5794fc90ffec3581d23d628abdb1825d7b5da8c"}
Sep 18 16:35:54.622176 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:35:54.622136    2585 generic.go:356] "Generic (PLEG): container finished" podID="ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7" containerID="1efd23a4aad103ddafd94ddfc5794fc90ffec3581d23d628abdb1825d7b5da8c" exitCode=0
Sep 18 16:35:54.622549 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:35:54.622214    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6dcvf" event={"ID":"ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7","Type":"ContainerDied","Data":"1efd23a4aad103ddafd94ddfc5794fc90ffec3581d23d628abdb1825d7b5da8c"}
Sep 18 16:35:59.647355 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:35:59.647310    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6dcvf" event={"ID":"ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7","Type":"ContainerStarted","Data":"694b48a2c6a9c6f1f1d41179c64bca278230cb0fa84a6c23067b7385b41c7557"}
Sep 18 16:35:59.665376 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:35:59.665314    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-6dcvf" podStartSLOduration=2.12034192 podStartE2EDuration="12.665293545s" totalImagesPullingTime="8.061546537s" totalInitContainerRuntime="2.483405088s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:35:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:35:59.663457899 +0000 UTC m=+1199.446459928" watchObservedRunningTime="2026-09-18 16:35:59.665293545 +0000 UTC m=+1199.448295570"
Sep 18 16:36:02.349452 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:36:02.349426    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 16:36:02.349936 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:36:02.349595    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 16:36:02.356608 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:36:02.356587    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 16:36:02.356763 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:36:02.356700    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 16:37:04.891812 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:04.891775    2585 generic.go:356] "Generic (PLEG): container finished" podID="ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7" containerID="694b48a2c6a9c6f1f1d41179c64bca278230cb0fa84a6c23067b7385b41c7557" exitCode=0
Sep 18 16:37:04.892234 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:04.891845    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6dcvf" event={"ID":"ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7","Type":"ContainerDied","Data":"694b48a2c6a9c6f1f1d41179c64bca278230cb0fa84a6c23067b7385b41c7557"}
Sep 18 16:37:06.018834 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:06.018811    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-6dcvf"
Sep 18 16:37:06.166377 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:06.166291    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7-model-cache\") pod \"ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7\" (UID: \"ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7\") "
Sep 18 16:37:06.166547 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:06.166405    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tx9g5\" (UniqueName: \"kubernetes.io/projected/ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7-kube-api-access-tx9g5\") pod \"ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7\" (UID: \"ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7\") "
Sep 18 16:37:06.168745 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:06.168713    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7-kube-api-access-tx9g5" (OuterVolumeSpecName: "kube-api-access-tx9g5") pod "ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7" (UID: "ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7"). InnerVolumeSpecName "kube-api-access-tx9g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:37:06.185108 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:06.185077    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7-model-cache" (OuterVolumeSpecName: "model-cache") pod "ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7" (UID: "ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:37:06.267907 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:06.267870    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7-model-cache\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:37:06.267907 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:06.267903    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx9g5\" (UniqueName: \"kubernetes.io/projected/ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7-kube-api-access-tx9g5\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:37:06.901559 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:06.901520    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-6dcvf" event={"ID":"ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7","Type":"ContainerDied","Data":"44693ec70e02b4fb672adc84d64b95ea6088cc972d67c0ee68cce70cfb177867"}
Sep 18 16:37:06.901559 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:06.901537    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-6dcvf"
Sep 18 16:37:06.901559 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:06.901558    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44693ec70e02b4fb672adc84d64b95ea6088cc972d67c0ee68cce70cfb177867"
Sep 18 16:37:27.113554 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.113523    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"]
Sep 18 16:37:27.113933 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.113805    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7" containerName="s3-init"
Sep 18 16:37:27.113933 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.113816    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7" containerName="s3-init"
Sep 18 16:37:27.113933 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.113825    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7" containerName="download-hf-model"
Sep 18 16:37:27.113933 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.113831    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7" containerName="download-hf-model"
Sep 18 16:37:27.113933 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.113882    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7" containerName="s3-init"
Sep 18 16:37:27.115897 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.115874    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"
Sep 18 16:37:27.118836 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.118809    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"seaweedfs-s3-creds\""
Sep 18 16:37:27.118971 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.118819    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"openshift-service-ca.crt\""
Sep 18 16:37:27.119063 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.118976    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"default-dockercfg-lqr9f\""
Sep 18 16:37:27.119124 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.119092    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"cpu-accel-preset-kserve-self-signed-certs\""
Sep 18 16:37:27.119184 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.119166    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"kube-root-ca.crt\""
Sep 18 16:37:27.129267 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.129244    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"]
Sep 18 16:37:27.238159 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.238120    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc730c02-e57f-4d3a-8b37-5a5723607014-kserve-provision-location\") pod \"cpu-accel-preset-kserve-7b55796b88-m4x9r\" (UID: \"cc730c02-e57f-4d3a-8b37-5a5723607014\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"
Sep 18 16:37:27.238159 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.238164    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc730c02-e57f-4d3a-8b37-5a5723607014-tmp-dir\") pod \"cpu-accel-preset-kserve-7b55796b88-m4x9r\" (UID: \"cc730c02-e57f-4d3a-8b37-5a5723607014\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"
Sep 18 16:37:27.238368 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.238180    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc730c02-e57f-4d3a-8b37-5a5723607014-tls-certs\") pod \"cpu-accel-preset-kserve-7b55796b88-m4x9r\" (UID: \"cc730c02-e57f-4d3a-8b37-5a5723607014\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"
Sep 18 16:37:27.238368 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.238196    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn6wk\" (UniqueName: \"kubernetes.io/projected/cc730c02-e57f-4d3a-8b37-5a5723607014-kube-api-access-gn6wk\") pod \"cpu-accel-preset-kserve-7b55796b88-m4x9r\" (UID: \"cc730c02-e57f-4d3a-8b37-5a5723607014\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"
Sep 18 16:37:27.238368 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.238251    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc730c02-e57f-4d3a-8b37-5a5723607014-dshm\") pod \"cpu-accel-preset-kserve-7b55796b88-m4x9r\" (UID: \"cc730c02-e57f-4d3a-8b37-5a5723607014\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"
Sep 18 16:37:27.238526 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.238379    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc730c02-e57f-4d3a-8b37-5a5723607014-model-cache\") pod \"cpu-accel-preset-kserve-7b55796b88-m4x9r\" (UID: \"cc730c02-e57f-4d3a-8b37-5a5723607014\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"
Sep 18 16:37:27.238526 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.238412    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc730c02-e57f-4d3a-8b37-5a5723607014-home\") pod \"cpu-accel-preset-kserve-7b55796b88-m4x9r\" (UID: \"cc730c02-e57f-4d3a-8b37-5a5723607014\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"
Sep 18 16:37:27.339072 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.339034    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc730c02-e57f-4d3a-8b37-5a5723607014-tmp-dir\") pod \"cpu-accel-preset-kserve-7b55796b88-m4x9r\" (UID: \"cc730c02-e57f-4d3a-8b37-5a5723607014\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"
Sep 18 16:37:27.339072 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.339073    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc730c02-e57f-4d3a-8b37-5a5723607014-tls-certs\") pod \"cpu-accel-preset-kserve-7b55796b88-m4x9r\" (UID: \"cc730c02-e57f-4d3a-8b37-5a5723607014\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"
Sep 18 16:37:27.339308 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.339091    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn6wk\" (UniqueName: \"kubernetes.io/projected/cc730c02-e57f-4d3a-8b37-5a5723607014-kube-api-access-gn6wk\") pod \"cpu-accel-preset-kserve-7b55796b88-m4x9r\" (UID: \"cc730c02-e57f-4d3a-8b37-5a5723607014\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"
Sep 18 16:37:27.339308 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.339116    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc730c02-e57f-4d3a-8b37-5a5723607014-dshm\") pod \"cpu-accel-preset-kserve-7b55796b88-m4x9r\" (UID: \"cc730c02-e57f-4d3a-8b37-5a5723607014\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"
Sep 18 16:37:27.339308 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.339172    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc730c02-e57f-4d3a-8b37-5a5723607014-model-cache\") pod \"cpu-accel-preset-kserve-7b55796b88-m4x9r\" (UID: \"cc730c02-e57f-4d3a-8b37-5a5723607014\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"
Sep 18 16:37:27.339308 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.339198    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc730c02-e57f-4d3a-8b37-5a5723607014-home\") pod \"cpu-accel-preset-kserve-7b55796b88-m4x9r\" (UID: \"cc730c02-e57f-4d3a-8b37-5a5723607014\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"
Sep 18 16:37:27.339308 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.339246    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc730c02-e57f-4d3a-8b37-5a5723607014-kserve-provision-location\") pod \"cpu-accel-preset-kserve-7b55796b88-m4x9r\" (UID: \"cc730c02-e57f-4d3a-8b37-5a5723607014\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"
Sep 18 16:37:27.339563 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.339455    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc730c02-e57f-4d3a-8b37-5a5723607014-tmp-dir\") pod \"cpu-accel-preset-kserve-7b55796b88-m4x9r\" (UID: \"cc730c02-e57f-4d3a-8b37-5a5723607014\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"
Sep 18 16:37:27.339563 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.339530    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc730c02-e57f-4d3a-8b37-5a5723607014-model-cache\") pod \"cpu-accel-preset-kserve-7b55796b88-m4x9r\" (UID: \"cc730c02-e57f-4d3a-8b37-5a5723607014\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"
Sep 18 16:37:27.339628 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.339581    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc730c02-e57f-4d3a-8b37-5a5723607014-home\") pod \"cpu-accel-preset-kserve-7b55796b88-m4x9r\" (UID: \"cc730c02-e57f-4d3a-8b37-5a5723607014\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"
Sep 18 16:37:27.339661 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.339635    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc730c02-e57f-4d3a-8b37-5a5723607014-kserve-provision-location\") pod \"cpu-accel-preset-kserve-7b55796b88-m4x9r\" (UID: \"cc730c02-e57f-4d3a-8b37-5a5723607014\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"
Sep 18 16:37:27.341600 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.341580    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc730c02-e57f-4d3a-8b37-5a5723607014-dshm\") pod \"cpu-accel-preset-kserve-7b55796b88-m4x9r\" (UID: \"cc730c02-e57f-4d3a-8b37-5a5723607014\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"
Sep 18 16:37:27.341708 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.341688    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc730c02-e57f-4d3a-8b37-5a5723607014-tls-certs\") pod \"cpu-accel-preset-kserve-7b55796b88-m4x9r\" (UID: \"cc730c02-e57f-4d3a-8b37-5a5723607014\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"
Sep 18 16:37:27.347796 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.347770    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn6wk\" (UniqueName: \"kubernetes.io/projected/cc730c02-e57f-4d3a-8b37-5a5723607014-kube-api-access-gn6wk\") pod \"cpu-accel-preset-kserve-7b55796b88-m4x9r\" (UID: \"cc730c02-e57f-4d3a-8b37-5a5723607014\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"
Sep 18 16:37:27.425702 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.425625    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"
Sep 18 16:37:27.700595 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.700555    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"]
Sep 18 16:37:27.703143 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:37:27.703116    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc730c02_e57f_4d3a_8b37_5a5723607014.slice/crio-de8ef2deac6084003c63a71cc68f4c738d34721ad70cb2ebefbb5ad2ed8aea5d WatchSource:0}: Error finding container de8ef2deac6084003c63a71cc68f4c738d34721ad70cb2ebefbb5ad2ed8aea5d: Status 404 returned error can't find the container with id de8ef2deac6084003c63a71cc68f4c738d34721ad70cb2ebefbb5ad2ed8aea5d
Sep 18 16:37:27.981109 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.981074    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r" event={"ID":"cc730c02-e57f-4d3a-8b37-5a5723607014","Type":"ContainerStarted","Data":"c92794673bf42799dc50ec79891d97e1372410f75ef458b4ce15cbb35d75b1f6"}
Sep 18 16:37:27.981266 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:27.981117    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r" event={"ID":"cc730c02-e57f-4d3a-8b37-5a5723607014","Type":"ContainerStarted","Data":"de8ef2deac6084003c63a71cc68f4c738d34721ad70cb2ebefbb5ad2ed8aea5d"}
Sep 18 16:37:37.018939 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:37.018893    2585 generic.go:356] "Generic (PLEG): container finished" podID="cc730c02-e57f-4d3a-8b37-5a5723607014" containerID="c92794673bf42799dc50ec79891d97e1372410f75ef458b4ce15cbb35d75b1f6" exitCode=0
Sep 18 16:37:37.019438 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:37.018953    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r" event={"ID":"cc730c02-e57f-4d3a-8b37-5a5723607014","Type":"ContainerDied","Data":"c92794673bf42799dc50ec79891d97e1372410f75ef458b4ce15cbb35d75b1f6"}
Sep 18 16:37:50.201609 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:50.201576    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz"]
Sep 18 16:37:50.217757 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:50.217726    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz"]
Sep 18 16:37:50.217944 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:50.217871    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz"
Sep 18 16:37:50.222026 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:50.221849    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-gftvw\""
Sep 18 16:37:50.222026 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:50.221868    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-zcrtd\""
Sep 18 16:37:50.222026 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:50.221891    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 18 16:37:50.222026 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:50.221851    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 18 16:37:50.222319 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:50.222216    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 18 16:37:50.322846 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:50.322803    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rh7z\" (UniqueName: \"kubernetes.io/projected/d82864d4-5318-4a2d-ad92-121db09fd602-kube-api-access-9rh7z\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz\" (UID: \"d82864d4-5318-4a2d-ad92-121db09fd602\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz"
Sep 18 16:37:50.323027 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:50.322874    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d82864d4-5318-4a2d-ad92-121db09fd602-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz\" (UID: \"d82864d4-5318-4a2d-ad92-121db09fd602\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz"
Sep 18 16:37:50.423555 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:50.423517    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d82864d4-5318-4a2d-ad92-121db09fd602-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz\" (UID: \"d82864d4-5318-4a2d-ad92-121db09fd602\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz"
Sep 18 16:37:50.423736 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:50.423601    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rh7z\" (UniqueName: \"kubernetes.io/projected/d82864d4-5318-4a2d-ad92-121db09fd602-kube-api-access-9rh7z\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz\" (UID: \"d82864d4-5318-4a2d-ad92-121db09fd602\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz"
Sep 18 16:37:50.426419 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:50.426392    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d82864d4-5318-4a2d-ad92-121db09fd602-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz\" (UID: \"d82864d4-5318-4a2d-ad92-121db09fd602\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz"
Sep 18 16:37:50.432482 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:50.432458    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rh7z\" (UniqueName: \"kubernetes.io/projected/d82864d4-5318-4a2d-ad92-121db09fd602-kube-api-access-9rh7z\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz\" (UID: \"d82864d4-5318-4a2d-ad92-121db09fd602\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz"
Sep 18 16:37:50.531331 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:50.531293    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz"
Sep 18 16:37:50.833595 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:50.833514    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz"]
Sep 18 16:37:53.134777 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:37:53.134676    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"]
Sep 18 16:38:00.745219 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:38:00.745174    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd82864d4_5318_4a2d_ad92_121db09fd602.slice/crio-15329603843beb830079b3076ed024528206a6f1ba922c14f38ac9b94005c1dc WatchSource:0}: Error finding container 15329603843beb830079b3076ed024528206a6f1ba922c14f38ac9b94005c1dc: Status 404 returned error can't find the container with id 15329603843beb830079b3076ed024528206a6f1ba922c14f38ac9b94005c1dc
Sep 18 16:38:01.132502 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:01.132415    2585 generic.go:356] "Generic (PLEG): container finished" podID="3a522b31-c301-4372-a729-93e898489653" containerID="3076a97e832449dfa1acd9f6d48709b3cbd421c0f70926f720ba98650a558300" exitCode=0
Sep 18 16:38:01.132502 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:01.132486    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d" event={"ID":"3a522b31-c301-4372-a729-93e898489653","Type":"ContainerDied","Data":"3076a97e832449dfa1acd9f6d48709b3cbd421c0f70926f720ba98650a558300"}
Sep 18 16:38:01.132963 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:01.132938    2585 scope.go:117] "RemoveContainer" containerID="3076a97e832449dfa1acd9f6d48709b3cbd421c0f70926f720ba98650a558300"
Sep 18 16:38:01.133839 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:01.133814    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz" event={"ID":"d82864d4-5318-4a2d-ad92-121db09fd602","Type":"ContainerStarted","Data":"15329603843beb830079b3076ed024528206a6f1ba922c14f38ac9b94005c1dc"}
Sep 18 16:38:01.133984 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:01.133954    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 16:38:01.134062 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:01.134027    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 16:38:01.134108 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:01.134072    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 16:38:01.745457 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:01.745370    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:38:02.151265 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:02.150539    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d" event={"ID":"3a522b31-c301-4372-a729-93e898489653","Type":"ContainerStarted","Data":"7bb74229f5340192f2d6f72aec016d295d8c58449f0afd7dcb16dd5b45fbae87"}
Sep 18 16:38:02.872796 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:02.872719    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:38:02.874080 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:02.874042    2585 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Sep 18 16:38:02.874212 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:02.874109    2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d" podUID="3a522b31-c301-4372-a729-93e898489653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Sep 18 16:38:03.872603 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:03.872563    2585 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Sep 18 16:38:03.872804 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:03.872761    2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d" podUID="3a522b31-c301-4372-a729-93e898489653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Sep 18 16:38:04.162502 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:04.162406    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz" event={"ID":"d82864d4-5318-4a2d-ad92-121db09fd602","Type":"ContainerStarted","Data":"852a2d017826977630997cdc6c09b56bc477ae5a4b2169f6c064a40b8f5729c5"}
Sep 18 16:38:04.162502 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:04.162490    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz"
Sep 18 16:38:04.183826 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:04.183761    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz" podStartSLOduration=10.930660867 podStartE2EDuration="14.183739959s" totalImagesPullingTime="3.253079092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:38:04.181371202 +0000 UTC m=+1323.964373227" watchObservedRunningTime="2026-09-18 16:38:04.183739959 +0000 UTC m=+1323.966741982"
Sep 18 16:38:04.872400 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:04.872364    2585 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Sep 18 16:38:04.872566 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:04.872421    2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d" podUID="3a522b31-c301-4372-a729-93e898489653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Sep 18 16:38:05.872390 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:05.872352    2585 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Sep 18 16:38:05.872789 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:05.872417    2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d" podUID="3a522b31-c301-4372-a729-93e898489653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Sep 18 16:38:06.872906 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:06.872868    2585 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Sep 18 16:38:06.873354 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:06.872963    2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d" podUID="3a522b31-c301-4372-a729-93e898489653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Sep 18 16:38:07.876982 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:07.876953    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:38:07.877441 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:07.877322    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:38:07.878002 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:07.877984    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d"
Sep 18 16:38:35.170549 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:35.170456    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz"
Sep 18 16:38:37.015436 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:37.015399    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz"]
Sep 18 16:38:45.170248 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:45.170209    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz" podUID="d82864d4-5318-4a2d-ad92-121db09fd602" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:38:52.040310 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:52.040270    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz" podUID="d82864d4-5318-4a2d-ad92-121db09fd602" containerName="main" containerID="cri-o://852a2d017826977630997cdc6c09b56bc477ae5a4b2169f6c064a40b8f5729c5" gracePeriod=45
Sep 18 16:38:55.170383 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:55.170342    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz" podUID="d82864d4-5318-4a2d-ad92-121db09fd602" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:38:55.396368 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:55.396339    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r" event={"ID":"cc730c02-e57f-4d3a-8b37-5a5723607014","Type":"ContainerStarted","Data":"11a415415aa1d3df15919df6c3fcc142803169f420cb2f143b1973ba0f8a098b"}
Sep 18 16:38:55.419355 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:55.419256    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r" podStartSLOduration=2.595705824 podStartE2EDuration="1m28.419236627s" totalImagesPullingTime="1m17.361784863s" totalInitContainerRuntime="8.46174594s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:37:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:38:55.417364746 +0000 UTC m=+1375.200366769" watchObservedRunningTime="2026-09-18 16:38:55.419236627 +0000 UTC m=+1375.202238662"
Sep 18 16:38:55.426718 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:38:55.426661    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"
Sep 18 16:39:05.170606 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:05.170562    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz" podUID="d82864d4-5318-4a2d-ad92-121db09fd602" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:39:05.171192 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:05.170679    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz"
Sep 18 16:39:10.416289 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:10.416250    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r" podUID="cc730c02-e57f-4d3a-8b37-5a5723607014" containerName="main" containerID="cri-o://11a415415aa1d3df15919df6c3fcc142803169f420cb2f143b1973ba0f8a098b" gracePeriod=45
Sep 18 16:39:15.170747 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:15.170703    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz" podUID="d82864d4-5318-4a2d-ad92-121db09fd602" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:39:22.512766 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:22.512727    2585 generic.go:356] "Generic (PLEG): container finished" podID="d82864d4-5318-4a2d-ad92-121db09fd602" containerID="852a2d017826977630997cdc6c09b56bc477ae5a4b2169f6c064a40b8f5729c5" exitCode=0
Sep 18 16:39:22.513181 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:22.512806    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz" event={"ID":"d82864d4-5318-4a2d-ad92-121db09fd602","Type":"ContainerDied","Data":"852a2d017826977630997cdc6c09b56bc477ae5a4b2169f6c064a40b8f5729c5"}
Sep 18 16:39:22.810800 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:22.810767    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz"
Sep 18 16:39:22.938582 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:22.938553    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d82864d4-5318-4a2d-ad92-121db09fd602-tls-certs\") pod \"d82864d4-5318-4a2d-ad92-121db09fd602\" (UID: \"d82864d4-5318-4a2d-ad92-121db09fd602\") "
Sep 18 16:39:22.938751 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:22.938643    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rh7z\" (UniqueName: \"kubernetes.io/projected/d82864d4-5318-4a2d-ad92-121db09fd602-kube-api-access-9rh7z\") pod \"d82864d4-5318-4a2d-ad92-121db09fd602\" (UID: \"d82864d4-5318-4a2d-ad92-121db09fd602\") "
Sep 18 16:39:22.941027 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:22.940989    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d82864d4-5318-4a2d-ad92-121db09fd602-kube-api-access-9rh7z" (OuterVolumeSpecName: "kube-api-access-9rh7z") pod "d82864d4-5318-4a2d-ad92-121db09fd602" (UID: "d82864d4-5318-4a2d-ad92-121db09fd602"). InnerVolumeSpecName "kube-api-access-9rh7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:39:22.941027 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:22.941014    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d82864d4-5318-4a2d-ad92-121db09fd602-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d82864d4-5318-4a2d-ad92-121db09fd602" (UID: "d82864d4-5318-4a2d-ad92-121db09fd602"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:39:23.040137 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:23.040057    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d82864d4-5318-4a2d-ad92-121db09fd602-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:39:23.040137 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:23.040085    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rh7z\" (UniqueName: \"kubernetes.io/projected/d82864d4-5318-4a2d-ad92-121db09fd602-kube-api-access-9rh7z\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:39:23.518394 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:23.518362    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz"
Sep 18 16:39:23.518866 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:23.518369    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz" event={"ID":"d82864d4-5318-4a2d-ad92-121db09fd602","Type":"ContainerDied","Data":"15329603843beb830079b3076ed024528206a6f1ba922c14f38ac9b94005c1dc"}
Sep 18 16:39:23.518866 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:23.518486    2585 scope.go:117] "RemoveContainer" containerID="852a2d017826977630997cdc6c09b56bc477ae5a4b2169f6c064a40b8f5729c5"
Sep 18 16:39:23.540613 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:23.540586    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz"]
Sep 18 16:39:23.546184 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:23.546155    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche8hnsz"]
Sep 18 16:39:24.392816 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:24.392782    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d82864d4-5318-4a2d-ad92-121db09fd602" path="/var/lib/kubelet/pods/d82864d4-5318-4a2d-ad92-121db09fd602/volumes"
Sep 18 16:39:39.756862 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:39.756812    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr"]
Sep 18 16:39:39.757441 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:39.757296    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d82864d4-5318-4a2d-ad92-121db09fd602" containerName="main"
Sep 18 16:39:39.757441 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:39.757317    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d82864d4-5318-4a2d-ad92-121db09fd602" containerName="main"
Sep 18 16:39:39.757441 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:39.757419    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d82864d4-5318-4a2d-ad92-121db09fd602" containerName="main"
Sep 18 16:39:39.769132 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:39.769101    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr"
Sep 18 16:39:39.769661 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:39.769631    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr"]
Sep 18 16:39:39.773030 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:39.772984    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 18 16:39:39.773030 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:39.772994    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 18 16:39:39.773239 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:39.773033    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-4k22c\""
Sep 18 16:39:39.773239 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:39.772994    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 18 16:39:39.773239 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:39.772984    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-tkzlh\""
Sep 18 16:39:39.785764 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:39.785735    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffdxp\" (UniqueName: \"kubernetes.io/projected/8921d9ca-cddd-4e56-9eed-9553362b21fc-kube-api-access-ffdxp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr\" (UID: \"8921d9ca-cddd-4e56-9eed-9553362b21fc\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr"
Sep 18 16:39:39.785927 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:39.785871    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8921d9ca-cddd-4e56-9eed-9553362b21fc-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr\" (UID: \"8921d9ca-cddd-4e56-9eed-9553362b21fc\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr"
Sep 18 16:39:39.886878 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:39.886850    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8921d9ca-cddd-4e56-9eed-9553362b21fc-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr\" (UID: \"8921d9ca-cddd-4e56-9eed-9553362b21fc\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr"
Sep 18 16:39:39.887058 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:39.886900    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffdxp\" (UniqueName: \"kubernetes.io/projected/8921d9ca-cddd-4e56-9eed-9553362b21fc-kube-api-access-ffdxp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr\" (UID: \"8921d9ca-cddd-4e56-9eed-9553362b21fc\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr"
Sep 18 16:39:39.889494 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:39.889472    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8921d9ca-cddd-4e56-9eed-9553362b21fc-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr\" (UID: \"8921d9ca-cddd-4e56-9eed-9553362b21fc\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr"
Sep 18 16:39:39.895717 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:39.895695    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffdxp\" (UniqueName: \"kubernetes.io/projected/8921d9ca-cddd-4e56-9eed-9553362b21fc-kube-api-access-ffdxp\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr\" (UID: \"8921d9ca-cddd-4e56-9eed-9553362b21fc\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr"
Sep 18 16:39:40.080032 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:40.079947    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr"
Sep 18 16:39:40.365882 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:40.365836    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr"]
Sep 18 16:39:40.368705 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:39:40.368671    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8921d9ca_cddd_4e56_9eed_9553362b21fc.slice/crio-3a8645f5391ea316c29fdba851e3de38696f605d45e50d4ad96ba604fb7554b9 WatchSource:0}: Error finding container 3a8645f5391ea316c29fdba851e3de38696f605d45e50d4ad96ba604fb7554b9: Status 404 returned error can't find the container with id 3a8645f5391ea316c29fdba851e3de38696f605d45e50d4ad96ba604fb7554b9
Sep 18 16:39:40.589742 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:40.589711    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr" event={"ID":"8921d9ca-cddd-4e56-9eed-9553362b21fc","Type":"ContainerStarted","Data":"d12211ac39709efa0c7582b2dca8e07325c28617ceb16e7b2ae8c3f761157e25"}
Sep 18 16:39:40.590017 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:40.589996    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr" event={"ID":"8921d9ca-cddd-4e56-9eed-9553362b21fc","Type":"ContainerStarted","Data":"3a8645f5391ea316c29fdba851e3de38696f605d45e50d4ad96ba604fb7554b9"}
Sep 18 16:39:40.590226 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:40.590210    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr"
Sep 18 16:39:40.620014 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:40.619885    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr" podStartSLOduration=1.619864366 podStartE2EDuration="1.619864366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:39:40.617608101 +0000 UTC m=+1420.400610126" watchObservedRunningTime="2026-09-18 16:39:40.619864366 +0000 UTC m=+1420.402866391"
Sep 18 16:39:55.656027 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:55.656002    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7_cpu-accel-preset-kserve-7b55796b88-m4x9r_cc730c02-e57f-4d3a-8b37-5a5723607014/main/0.log"
Sep 18 16:39:55.656442 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:55.656418    2585 generic.go:356] "Generic (PLEG): container finished" podID="cc730c02-e57f-4d3a-8b37-5a5723607014" containerID="11a415415aa1d3df15919df6c3fcc142803169f420cb2f143b1973ba0f8a098b" exitCode=137
Sep 18 16:39:55.656509 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:55.656486    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r" event={"ID":"cc730c02-e57f-4d3a-8b37-5a5723607014","Type":"ContainerDied","Data":"11a415415aa1d3df15919df6c3fcc142803169f420cb2f143b1973ba0f8a098b"}
Sep 18 16:39:56.054858 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:56.054838    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7_cpu-accel-preset-kserve-7b55796b88-m4x9r_cc730c02-e57f-4d3a-8b37-5a5723607014/main/0.log"
Sep 18 16:39:56.056535 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:56.056519    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"
Sep 18 16:39:56.125198 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:56.125173    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc730c02-e57f-4d3a-8b37-5a5723607014-tls-certs\") pod \"cc730c02-e57f-4d3a-8b37-5a5723607014\" (UID: \"cc730c02-e57f-4d3a-8b37-5a5723607014\") "
Sep 18 16:39:56.125313 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:56.125235    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc730c02-e57f-4d3a-8b37-5a5723607014-home\") pod \"cc730c02-e57f-4d3a-8b37-5a5723607014\" (UID: \"cc730c02-e57f-4d3a-8b37-5a5723607014\") "
Sep 18 16:39:56.125313 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:56.125254    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc730c02-e57f-4d3a-8b37-5a5723607014-tmp-dir\") pod \"cc730c02-e57f-4d3a-8b37-5a5723607014\" (UID: \"cc730c02-e57f-4d3a-8b37-5a5723607014\") "
Sep 18 16:39:56.125313 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:56.125275    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gn6wk\" (UniqueName: \"kubernetes.io/projected/cc730c02-e57f-4d3a-8b37-5a5723607014-kube-api-access-gn6wk\") pod \"cc730c02-e57f-4d3a-8b37-5a5723607014\" (UID: \"cc730c02-e57f-4d3a-8b37-5a5723607014\") "
Sep 18 16:39:56.125313 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:56.125297    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc730c02-e57f-4d3a-8b37-5a5723607014-dshm\") pod \"cc730c02-e57f-4d3a-8b37-5a5723607014\" (UID: \"cc730c02-e57f-4d3a-8b37-5a5723607014\") "
Sep 18 16:39:56.125560 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:56.125319    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc730c02-e57f-4d3a-8b37-5a5723607014-kserve-provision-location\") pod \"cc730c02-e57f-4d3a-8b37-5a5723607014\" (UID: \"cc730c02-e57f-4d3a-8b37-5a5723607014\") "
Sep 18 16:39:56.125560 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:56.125381    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc730c02-e57f-4d3a-8b37-5a5723607014-model-cache\") pod \"cc730c02-e57f-4d3a-8b37-5a5723607014\" (UID: \"cc730c02-e57f-4d3a-8b37-5a5723607014\") "
Sep 18 16:39:56.125765 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:56.125720    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc730c02-e57f-4d3a-8b37-5a5723607014-model-cache" (OuterVolumeSpecName: "model-cache") pod "cc730c02-e57f-4d3a-8b37-5a5723607014" (UID: "cc730c02-e57f-4d3a-8b37-5a5723607014"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:39:56.125765 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:56.125739    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc730c02-e57f-4d3a-8b37-5a5723607014-home" (OuterVolumeSpecName: "home") pod "cc730c02-e57f-4d3a-8b37-5a5723607014" (UID: "cc730c02-e57f-4d3a-8b37-5a5723607014"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:39:56.126389 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:56.126364    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc730c02-e57f-4d3a-8b37-5a5723607014-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cc730c02-e57f-4d3a-8b37-5a5723607014" (UID: "cc730c02-e57f-4d3a-8b37-5a5723607014"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:39:56.127724 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:56.127699    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc730c02-e57f-4d3a-8b37-5a5723607014-kube-api-access-gn6wk" (OuterVolumeSpecName: "kube-api-access-gn6wk") pod "cc730c02-e57f-4d3a-8b37-5a5723607014" (UID: "cc730c02-e57f-4d3a-8b37-5a5723607014"). InnerVolumeSpecName "kube-api-access-gn6wk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:39:56.127862 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:56.127717    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc730c02-e57f-4d3a-8b37-5a5723607014-dshm" (OuterVolumeSpecName: "dshm") pod "cc730c02-e57f-4d3a-8b37-5a5723607014" (UID: "cc730c02-e57f-4d3a-8b37-5a5723607014"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:39:56.127862 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:56.127825    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc730c02-e57f-4d3a-8b37-5a5723607014-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc730c02-e57f-4d3a-8b37-5a5723607014" (UID: "cc730c02-e57f-4d3a-8b37-5a5723607014"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:39:56.183374 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:56.183303    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc730c02-e57f-4d3a-8b37-5a5723607014-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cc730c02-e57f-4d3a-8b37-5a5723607014" (UID: "cc730c02-e57f-4d3a-8b37-5a5723607014"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:39:56.226794 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:56.226767    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc730c02-e57f-4d3a-8b37-5a5723607014-dshm\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:39:56.226794 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:56.226791    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc730c02-e57f-4d3a-8b37-5a5723607014-kserve-provision-location\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:39:56.226985 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:56.226807    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc730c02-e57f-4d3a-8b37-5a5723607014-model-cache\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:39:56.226985 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:56.226821    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc730c02-e57f-4d3a-8b37-5a5723607014-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:39:56.226985 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:56.226831    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc730c02-e57f-4d3a-8b37-5a5723607014-home\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:39:56.226985 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:56.226866    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc730c02-e57f-4d3a-8b37-5a5723607014-tmp-dir\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:39:56.226985 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:56.226879    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn6wk\" (UniqueName: \"kubernetes.io/projected/cc730c02-e57f-4d3a-8b37-5a5723607014-kube-api-access-gn6wk\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:39:56.662868 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:56.662840    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7_cpu-accel-preset-kserve-7b55796b88-m4x9r_cc730c02-e57f-4d3a-8b37-5a5723607014/main/0.log"
Sep 18 16:39:56.663317 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:56.663303    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"
Sep 18 16:39:56.663393 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:56.663301    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r" event={"ID":"cc730c02-e57f-4d3a-8b37-5a5723607014","Type":"ContainerDied","Data":"de8ef2deac6084003c63a71cc68f4c738d34721ad70cb2ebefbb5ad2ed8aea5d"}
Sep 18 16:39:56.663440 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:56.663423    2585 scope.go:117] "RemoveContainer" containerID="11a415415aa1d3df15919df6c3fcc142803169f420cb2f143b1973ba0f8a098b"
Sep 18 16:39:56.672721 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:56.672701    2585 scope.go:117] "RemoveContainer" containerID="c92794673bf42799dc50ec79891d97e1372410f75ef458b4ce15cbb35d75b1f6"
Sep 18 16:39:56.684311 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:56.684291    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"]
Sep 18 16:39:56.690115 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:56.690096    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-7b55796b88-m4x9r"]
Sep 18 16:39:58.392410 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:39:58.392376    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc730c02-e57f-4d3a-8b37-5a5723607014" path="/var/lib/kubelet/pods/cc730c02-e57f-4d3a-8b37-5a5723607014/volumes"
Sep 18 16:40:11.596515 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:11.596486    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr"
Sep 18 16:40:30.395281 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.395251    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"]
Sep 18 16:40:30.395737 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.395524    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc730c02-e57f-4d3a-8b37-5a5723607014" containerName="main"
Sep 18 16:40:30.395737 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.395535    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc730c02-e57f-4d3a-8b37-5a5723607014" containerName="main"
Sep 18 16:40:30.395737 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.395544    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc730c02-e57f-4d3a-8b37-5a5723607014" containerName="storage-initializer"
Sep 18 16:40:30.395737 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.395550    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc730c02-e57f-4d3a-8b37-5a5723607014" containerName="storage-initializer"
Sep 18 16:40:30.395737 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.395599    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc730c02-e57f-4d3a-8b37-5a5723607014" containerName="main"
Sep 18 16:40:30.397306 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.397287    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"
Sep 18 16:40:30.400082 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.400008    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"default-dockercfg-dckk2\""
Sep 18 16:40:30.400082 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.400021    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"seaweedfs-s3-creds\""
Sep 18 16:40:30.400082 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.400043    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"kube-root-ca.crt\""
Sep 18 16:40:30.400320 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.400043    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"cpu-accel-precedence-kserve-self-signed-certs\""
Sep 18 16:40:30.400320 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.400107    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"openshift-service-ca.crt\""
Sep 18 16:40:30.410234 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.410212    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"]
Sep 18 16:40:30.495381 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.495351    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-tmp-dir\") pod \"cpu-accel-precedence-kserve-5477c794d4-ms274\" (UID: \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"
Sep 18 16:40:30.495546 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.495389    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-home\") pod \"cpu-accel-precedence-kserve-5477c794d4-ms274\" (UID: \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"
Sep 18 16:40:30.495546 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.495421    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-model-cache\") pod \"cpu-accel-precedence-kserve-5477c794d4-ms274\" (UID: \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"
Sep 18 16:40:30.495546 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.495449    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-kserve-provision-location\") pod \"cpu-accel-precedence-kserve-5477c794d4-ms274\" (UID: \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"
Sep 18 16:40:30.495546 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.495503    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfh4x\" (UniqueName: \"kubernetes.io/projected/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-kube-api-access-qfh4x\") pod \"cpu-accel-precedence-kserve-5477c794d4-ms274\" (UID: \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"
Sep 18 16:40:30.495546 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.495547    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-dshm\") pod \"cpu-accel-precedence-kserve-5477c794d4-ms274\" (UID: \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"
Sep 18 16:40:30.495726 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.495574    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-tls-certs\") pod \"cpu-accel-precedence-kserve-5477c794d4-ms274\" (UID: \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"
Sep 18 16:40:30.596463 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.596422    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-home\") pod \"cpu-accel-precedence-kserve-5477c794d4-ms274\" (UID: \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"
Sep 18 16:40:30.596463 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.596464    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-model-cache\") pod \"cpu-accel-precedence-kserve-5477c794d4-ms274\" (UID: \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"
Sep 18 16:40:30.596711 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.596494    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-kserve-provision-location\") pod \"cpu-accel-precedence-kserve-5477c794d4-ms274\" (UID: \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"
Sep 18 16:40:30.596711 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.596536    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfh4x\" (UniqueName: \"kubernetes.io/projected/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-kube-api-access-qfh4x\") pod \"cpu-accel-precedence-kserve-5477c794d4-ms274\" (UID: \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"
Sep 18 16:40:30.596711 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.596585    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-dshm\") pod \"cpu-accel-precedence-kserve-5477c794d4-ms274\" (UID: \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"
Sep 18 16:40:30.596711 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.596617    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-tls-certs\") pod \"cpu-accel-precedence-kserve-5477c794d4-ms274\" (UID: \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"
Sep 18 16:40:30.596711 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.596650    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-tmp-dir\") pod \"cpu-accel-precedence-kserve-5477c794d4-ms274\" (UID: \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"
Sep 18 16:40:30.596996 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.596886    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-home\") pod \"cpu-accel-precedence-kserve-5477c794d4-ms274\" (UID: \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"
Sep 18 16:40:30.596996 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.596965    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-model-cache\") pod \"cpu-accel-precedence-kserve-5477c794d4-ms274\" (UID: \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"
Sep 18 16:40:30.597098 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.597039    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-kserve-provision-location\") pod \"cpu-accel-precedence-kserve-5477c794d4-ms274\" (UID: \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"
Sep 18 16:40:30.597098 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.597079    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-tmp-dir\") pod \"cpu-accel-precedence-kserve-5477c794d4-ms274\" (UID: \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"
Sep 18 16:40:30.599028 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.599009    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-dshm\") pod \"cpu-accel-precedence-kserve-5477c794d4-ms274\" (UID: \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"
Sep 18 16:40:30.599342 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.599324    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-tls-certs\") pod \"cpu-accel-precedence-kserve-5477c794d4-ms274\" (UID: \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"
Sep 18 16:40:30.609347 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.609324    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfh4x\" (UniqueName: \"kubernetes.io/projected/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-kube-api-access-qfh4x\") pod \"cpu-accel-precedence-kserve-5477c794d4-ms274\" (UID: \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"
Sep 18 16:40:30.707623 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.707536    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"
Sep 18 16:40:30.982433 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:30.982410    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"]
Sep 18 16:40:30.984665 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:40:30.984637    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ca4596_dd90_4f62_a2e0_dd05b229f0c9.slice/crio-c65d762de101f3d2fcd16b4d1837f8ab5d85493729f9b5d90096807323891652 WatchSource:0}: Error finding container c65d762de101f3d2fcd16b4d1837f8ab5d85493729f9b5d90096807323891652: Status 404 returned error can't find the container with id c65d762de101f3d2fcd16b4d1837f8ab5d85493729f9b5d90096807323891652
Sep 18 16:40:31.800586 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:31.800545    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274" event={"ID":"57ca4596-dd90-4f62-a2e0-dd05b229f0c9","Type":"ContainerStarted","Data":"c822b1ec136f6c4af2353d0913ea00c4200836f198d9a3a1ebcf2e37938187cc"}
Sep 18 16:40:31.800586 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:31.800591    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274" event={"ID":"57ca4596-dd90-4f62-a2e0-dd05b229f0c9","Type":"ContainerStarted","Data":"c65d762de101f3d2fcd16b4d1837f8ab5d85493729f9b5d90096807323891652"}
Sep 18 16:40:35.112755 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:35.112710    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr"]
Sep 18 16:40:35.463115 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:35.463033    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"]
Sep 18 16:40:35.463308 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:35.463223    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274" podUID="57ca4596-dd90-4f62-a2e0-dd05b229f0c9" containerName="storage-initializer" containerID="cri-o://c822b1ec136f6c4af2353d0913ea00c4200836f198d9a3a1ebcf2e37938187cc" gracePeriod=60
Sep 18 16:40:37.116079 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.116054    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"
Sep 18 16:40:37.254266 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.254239    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-tls-certs\") pod \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\" (UID: \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\") "
Sep 18 16:40:37.254522 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.254286    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-home\") pod \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\" (UID: \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\") "
Sep 18 16:40:37.254522 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.254309    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-kserve-provision-location\") pod \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\" (UID: \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\") "
Sep 18 16:40:37.254522 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.254348    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfh4x\" (UniqueName: \"kubernetes.io/projected/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-kube-api-access-qfh4x\") pod \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\" (UID: \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\") "
Sep 18 16:40:37.254522 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.254366    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-dshm\") pod \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\" (UID: \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\") "
Sep 18 16:40:37.254522 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.254389    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-tmp-dir\") pod \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\" (UID: \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\") "
Sep 18 16:40:37.254522 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.254409    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-model-cache\") pod \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\" (UID: \"57ca4596-dd90-4f62-a2e0-dd05b229f0c9\") "
Sep 18 16:40:37.254797 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.254552    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-home" (OuterVolumeSpecName: "home") pod "57ca4596-dd90-4f62-a2e0-dd05b229f0c9" (UID: "57ca4596-dd90-4f62-a2e0-dd05b229f0c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:40:37.254797 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.254651    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "57ca4596-dd90-4f62-a2e0-dd05b229f0c9" (UID: "57ca4596-dd90-4f62-a2e0-dd05b229f0c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:40:37.254797 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.254777    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-model-cache" (OuterVolumeSpecName: "model-cache") pod "57ca4596-dd90-4f62-a2e0-dd05b229f0c9" (UID: "57ca4596-dd90-4f62-a2e0-dd05b229f0c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:40:37.256711 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.256685    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-dshm" (OuterVolumeSpecName: "dshm") pod "57ca4596-dd90-4f62-a2e0-dd05b229f0c9" (UID: "57ca4596-dd90-4f62-a2e0-dd05b229f0c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:40:37.256815 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.256783    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-kube-api-access-qfh4x" (OuterVolumeSpecName: "kube-api-access-qfh4x") pod "57ca4596-dd90-4f62-a2e0-dd05b229f0c9" (UID: "57ca4596-dd90-4f62-a2e0-dd05b229f0c9"). InnerVolumeSpecName "kube-api-access-qfh4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:40:37.256815 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.256789    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "57ca4596-dd90-4f62-a2e0-dd05b229f0c9" (UID: "57ca4596-dd90-4f62-a2e0-dd05b229f0c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:40:37.326293 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.326231    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "57ca4596-dd90-4f62-a2e0-dd05b229f0c9" (UID: "57ca4596-dd90-4f62-a2e0-dd05b229f0c9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:40:37.355545 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.355515    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:40:37.355545 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.355539    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-home\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:40:37.355705 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.355551    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-kserve-provision-location\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:40:37.355705 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.355588    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfh4x\" (UniqueName: \"kubernetes.io/projected/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-kube-api-access-qfh4x\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:40:37.355705 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.355601    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-dshm\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:40:37.355705 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.355614    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-tmp-dir\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:40:37.355705 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.355628    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57ca4596-dd90-4f62-a2e0-dd05b229f0c9-model-cache\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:40:37.827235 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.827193    2585 generic.go:356] "Generic (PLEG): container finished" podID="57ca4596-dd90-4f62-a2e0-dd05b229f0c9" containerID="c822b1ec136f6c4af2353d0913ea00c4200836f198d9a3a1ebcf2e37938187cc" exitCode=0
Sep 18 16:40:37.827420 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.827246    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"
Sep 18 16:40:37.827420 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.827247    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274" event={"ID":"57ca4596-dd90-4f62-a2e0-dd05b229f0c9","Type":"ContainerDied","Data":"c822b1ec136f6c4af2353d0913ea00c4200836f198d9a3a1ebcf2e37938187cc"}
Sep 18 16:40:37.827420 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.827370    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274" event={"ID":"57ca4596-dd90-4f62-a2e0-dd05b229f0c9","Type":"ContainerDied","Data":"c65d762de101f3d2fcd16b4d1837f8ab5d85493729f9b5d90096807323891652"}
Sep 18 16:40:37.827420 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.827403    2585 scope.go:117] "RemoveContainer" containerID="c822b1ec136f6c4af2353d0913ea00c4200836f198d9a3a1ebcf2e37938187cc"
Sep 18 16:40:37.836217 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.836202    2585 scope.go:117] "RemoveContainer" containerID="c822b1ec136f6c4af2353d0913ea00c4200836f198d9a3a1ebcf2e37938187cc"
Sep 18 16:40:37.836470 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:40:37.836453    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c822b1ec136f6c4af2353d0913ea00c4200836f198d9a3a1ebcf2e37938187cc\": container with ID starting with c822b1ec136f6c4af2353d0913ea00c4200836f198d9a3a1ebcf2e37938187cc not found: ID does not exist" containerID="c822b1ec136f6c4af2353d0913ea00c4200836f198d9a3a1ebcf2e37938187cc"
Sep 18 16:40:37.836519 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.836478    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c822b1ec136f6c4af2353d0913ea00c4200836f198d9a3a1ebcf2e37938187cc"} err="failed to get container status \"c822b1ec136f6c4af2353d0913ea00c4200836f198d9a3a1ebcf2e37938187cc\": rpc error: code = NotFound desc = could not find container \"c822b1ec136f6c4af2353d0913ea00c4200836f198d9a3a1ebcf2e37938187cc\": container with ID starting with c822b1ec136f6c4af2353d0913ea00c4200836f198d9a3a1ebcf2e37938187cc not found: ID does not exist"
Sep 18 16:40:37.865167 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.865146    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"]
Sep 18 16:40:37.870842 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:37.870813    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5477c794d4-ms274"]
Sep 18 16:40:38.392449 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:38.392414    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57ca4596-dd90-4f62-a2e0-dd05b229f0c9" path="/var/lib/kubelet/pods/57ca4596-dd90-4f62-a2e0-dd05b229f0c9/volumes"
Sep 18 16:40:41.596176 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:41.596141    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr" podUID="8921d9ca-cddd-4e56-9eed-9553362b21fc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:40:50.135038 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.134993    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr" podUID="8921d9ca-cddd-4e56-9eed-9553362b21fc" containerName="main" containerID="cri-o://d12211ac39709efa0c7582b2dca8e07325c28617ceb16e7b2ae8c3f761157e25" gracePeriod=45
Sep 18 16:40:50.490605 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.490573    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"]
Sep 18 16:40:50.490977 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.490951    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57ca4596-dd90-4f62-a2e0-dd05b229f0c9" containerName="storage-initializer"
Sep 18 16:40:50.491081 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.490980    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="57ca4596-dd90-4f62-a2e0-dd05b229f0c9" containerName="storage-initializer"
Sep 18 16:40:50.491142 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.491076    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="57ca4596-dd90-4f62-a2e0-dd05b229f0c9" containerName="storage-initializer"
Sep 18 16:40:50.494191 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.494168    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:40:50.496969 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.496887    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"test-cb25a14c9ba423627edeb5388343bd353-kserve-self-signed-certs\""
Sep 18 16:40:50.497162 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.497144    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"openshift-service-ca.crt\""
Sep 18 16:40:50.497264 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.497186    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"kube-root-ca.crt\""
Sep 18 16:40:50.497404 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.497386    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"seaweedfs-s3-creds\""
Sep 18 16:40:50.497512 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.497387    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 16:40:50.497512 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.497435    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"default-dockercfg-f4qgz\""
Sep 18 16:40:50.506206 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.506182    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"]
Sep 18 16:40:50.561637 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.561604    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-tmp-dir\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:40:50.561809 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.561657    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-tls-certs\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:40:50.561809 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.561716    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-kserve-provision-location\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:40:50.561809 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.561757    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-home\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:40:50.561809 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.561781    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f699p\" (UniqueName: \"kubernetes.io/projected/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-kube-api-access-f699p\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:40:50.561988 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.561822    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-cabundle-cert\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:40:50.561988 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.561869    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-dshm\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:40:50.561988 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.561895    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-model-cache\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:40:50.662927 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.662887    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-cabundle-cert\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:40:50.663074 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.662976    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-dshm\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:40:50.663074 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.663001    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-model-cache\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:40:50.663074 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.663032    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-tmp-dir\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:40:50.663231 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.663073    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-tls-certs\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:40:50.663231 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.663090    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-kserve-provision-location\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:40:50.663231 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.663119    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-home\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:40:50.663231 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.663136    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f699p\" (UniqueName: \"kubernetes.io/projected/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-kube-api-access-f699p\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:40:50.663486 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.663461    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-tmp-dir\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:40:50.663486 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.663478    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-model-cache\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:40:50.663627 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.663548    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-kserve-provision-location\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:40:50.663627 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.663599    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-home\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:40:50.663985 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.663969    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-cabundle-cert\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:40:50.665531 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.665514    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-dshm\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:40:50.665793 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.665777    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-tls-certs\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:40:50.672106 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.672083    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f699p\" (UniqueName: \"kubernetes.io/projected/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-kube-api-access-f699p\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:40:50.804857 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:50.804773    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:40:51.083810 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:51.083735    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"]
Sep 18 16:40:51.088061 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:40:51.088027    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc2277a_2021_4f02_b9fb_fb7fe9fba85f.slice/crio-1a0e1ad9a6655b0814421486435b922dceb0cbb8f13ec1d76a3f9bf73b30d497 WatchSource:0}: Error finding container 1a0e1ad9a6655b0814421486435b922dceb0cbb8f13ec1d76a3f9bf73b30d497: Status 404 returned error can't find the container with id 1a0e1ad9a6655b0814421486435b922dceb0cbb8f13ec1d76a3f9bf73b30d497
Sep 18 16:40:51.596255 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:51.596212    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr" podUID="8921d9ca-cddd-4e56-9eed-9553362b21fc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:40:51.884363 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:51.884278    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph" event={"ID":"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f","Type":"ContainerStarted","Data":"1e2d575225a276cefa85af4a9791eb24298f27daa6bea9e99d8ab1fd8b66cf24"}
Sep 18 16:40:51.884363 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:51.884319    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph" event={"ID":"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f","Type":"ContainerStarted","Data":"1a0e1ad9a6655b0814421486435b922dceb0cbb8f13ec1d76a3f9bf73b30d497"}
Sep 18 16:40:53.894586 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:53.894552    2585 generic.go:356] "Generic (PLEG): container finished" podID="ecc2277a-2021-4f02-b9fb-fb7fe9fba85f" containerID="1e2d575225a276cefa85af4a9791eb24298f27daa6bea9e99d8ab1fd8b66cf24" exitCode=0
Sep 18 16:40:53.895002 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:53.894603    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph" event={"ID":"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f","Type":"ContainerDied","Data":"1e2d575225a276cefa85af4a9791eb24298f27daa6bea9e99d8ab1fd8b66cf24"}
Sep 18 16:40:54.901177 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:54.901145    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph" event={"ID":"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f","Type":"ContainerStarted","Data":"3f9e4ef2c32c218b5996a4477b40ea2c4e13767bf1d3a1fdee2e0879a457a637"}
Sep 18 16:40:54.925086 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:54.925035    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph" podStartSLOduration=2.786073783 podStartE2EDuration="4.925019071s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.138945288s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:40:54.92277676 +0000 UTC m=+1494.705778785" watchObservedRunningTime="2026-09-18 16:40:54.925019071 +0000 UTC m=+1494.708021095"
Sep 18 16:40:55.805590 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:40:55.805552    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:41:00.805480 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:00.805433    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:41:00.807267 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:00.807225    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph" podUID="ecc2277a-2021-4f02-b9fb-fb7fe9fba85f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": dial tcp 10.133.0.30:8000: connect: connection refused"
Sep 18 16:41:01.596371 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:01.596331    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr" podUID="8921d9ca-cddd-4e56-9eed-9553362b21fc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:41:01.596564 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:01.596458    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr"
Sep 18 16:41:02.381429 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:02.381389    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 16:41:02.381893 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:02.381667    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 16:41:02.385503 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:02.385483    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 16:41:02.385631 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:02.385529    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 16:41:10.806252 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:10.806211    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph" podUID="ecc2277a-2021-4f02-b9fb-fb7fe9fba85f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": dial tcp 10.133.0.30:8000: connect: connection refused"
Sep 18 16:41:11.596574 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:11.596529    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr" podUID="8921d9ca-cddd-4e56-9eed-9553362b21fc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:41:20.385807 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:20.385779    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr"
Sep 18 16:41:20.526012 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:20.525968    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8921d9ca-cddd-4e56-9eed-9553362b21fc-tls-certs\") pod \"8921d9ca-cddd-4e56-9eed-9553362b21fc\" (UID: \"8921d9ca-cddd-4e56-9eed-9553362b21fc\") "
Sep 18 16:41:20.526182 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:20.526055    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffdxp\" (UniqueName: \"kubernetes.io/projected/8921d9ca-cddd-4e56-9eed-9553362b21fc-kube-api-access-ffdxp\") pod \"8921d9ca-cddd-4e56-9eed-9553362b21fc\" (UID: \"8921d9ca-cddd-4e56-9eed-9553362b21fc\") "
Sep 18 16:41:20.528400 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:20.528368    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8921d9ca-cddd-4e56-9eed-9553362b21fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8921d9ca-cddd-4e56-9eed-9553362b21fc" (UID: "8921d9ca-cddd-4e56-9eed-9553362b21fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:41:20.528536 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:20.528420    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8921d9ca-cddd-4e56-9eed-9553362b21fc-kube-api-access-ffdxp" (OuterVolumeSpecName: "kube-api-access-ffdxp") pod "8921d9ca-cddd-4e56-9eed-9553362b21fc" (UID: "8921d9ca-cddd-4e56-9eed-9553362b21fc"). InnerVolumeSpecName "kube-api-access-ffdxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:41:20.627330 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:20.627291    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8921d9ca-cddd-4e56-9eed-9553362b21fc-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:41:20.627330 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:20.627323    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffdxp\" (UniqueName: \"kubernetes.io/projected/8921d9ca-cddd-4e56-9eed-9553362b21fc-kube-api-access-ffdxp\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:41:20.806235 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:20.806151    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph" podUID="ecc2277a-2021-4f02-b9fb-fb7fe9fba85f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": dial tcp 10.133.0.30:8000: connect: connection refused"
Sep 18 16:41:21.011785 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:21.011748    2585 generic.go:356] "Generic (PLEG): container finished" podID="8921d9ca-cddd-4e56-9eed-9553362b21fc" containerID="d12211ac39709efa0c7582b2dca8e07325c28617ceb16e7b2ae8c3f761157e25" exitCode=0
Sep 18 16:41:21.011982 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:21.011794    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr"
Sep 18 16:41:21.011982 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:21.011831    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr" event={"ID":"8921d9ca-cddd-4e56-9eed-9553362b21fc","Type":"ContainerDied","Data":"d12211ac39709efa0c7582b2dca8e07325c28617ceb16e7b2ae8c3f761157e25"}
Sep 18 16:41:21.011982 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:21.011891    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr" event={"ID":"8921d9ca-cddd-4e56-9eed-9553362b21fc","Type":"ContainerDied","Data":"3a8645f5391ea316c29fdba851e3de38696f605d45e50d4ad96ba604fb7554b9"}
Sep 18 16:41:21.011982 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:21.011941    2585 scope.go:117] "RemoveContainer" containerID="d12211ac39709efa0c7582b2dca8e07325c28617ceb16e7b2ae8c3f761157e25"
Sep 18 16:41:21.021007 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:21.020991    2585 scope.go:117] "RemoveContainer" containerID="d12211ac39709efa0c7582b2dca8e07325c28617ceb16e7b2ae8c3f761157e25"
Sep 18 16:41:21.021288 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:41:21.021269    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d12211ac39709efa0c7582b2dca8e07325c28617ceb16e7b2ae8c3f761157e25\": container with ID starting with d12211ac39709efa0c7582b2dca8e07325c28617ceb16e7b2ae8c3f761157e25 not found: ID does not exist" containerID="d12211ac39709efa0c7582b2dca8e07325c28617ceb16e7b2ae8c3f761157e25"
Sep 18 16:41:21.021354 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:21.021294    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12211ac39709efa0c7582b2dca8e07325c28617ceb16e7b2ae8c3f761157e25"} err="failed to get container status \"d12211ac39709efa0c7582b2dca8e07325c28617ceb16e7b2ae8c3f761157e25\": rpc error: code = NotFound desc = could not find container \"d12211ac39709efa0c7582b2dca8e07325c28617ceb16e7b2ae8c3f761157e25\": container with ID starting with d12211ac39709efa0c7582b2dca8e07325c28617ceb16e7b2ae8c3f761157e25 not found: ID does not exist"
Sep 18 16:41:21.033813 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:21.033790    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr"]
Sep 18 16:41:21.037421 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:21.037400    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c962cgr"]
Sep 18 16:41:22.393225 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:22.393192    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8921d9ca-cddd-4e56-9eed-9553362b21fc" path="/var/lib/kubelet/pods/8921d9ca-cddd-4e56-9eed-9553362b21fc/volumes"
Sep 18 16:41:29.083558 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.083525    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"]
Sep 18 16:41:29.083907 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.083816    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8921d9ca-cddd-4e56-9eed-9553362b21fc" containerName="main"
Sep 18 16:41:29.083907 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.083827    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8921d9ca-cddd-4e56-9eed-9553362b21fc" containerName="main"
Sep 18 16:41:29.083907 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.083897    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8921d9ca-cddd-4e56-9eed-9553362b21fc" containerName="main"
Sep 18 16:41:29.086262 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.086245    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"
Sep 18 16:41:29.089176 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.089154    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-t9h9x\""
Sep 18 16:41:29.090738 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.090707    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 18 16:41:29.090838 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.090752    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 18 16:41:29.090838 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.090766    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 18 16:41:29.099236 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.099214    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a43c8287-91ac-46d1-bc76-32dfb7483a0a-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-26h8m\" (UID: \"a43c8287-91ac-46d1-bc76-32dfb7483a0a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"
Sep 18 16:41:29.099339 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.099257    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a43c8287-91ac-46d1-bc76-32dfb7483a0a-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-26h8m\" (UID: \"a43c8287-91ac-46d1-bc76-32dfb7483a0a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"
Sep 18 16:41:29.099339 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.099287    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxrgw\" (UniqueName: \"kubernetes.io/projected/a43c8287-91ac-46d1-bc76-32dfb7483a0a-kube-api-access-jxrgw\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-26h8m\" (UID: \"a43c8287-91ac-46d1-bc76-32dfb7483a0a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"
Sep 18 16:41:29.099339 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.099334    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a43c8287-91ac-46d1-bc76-32dfb7483a0a-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-26h8m\" (UID: \"a43c8287-91ac-46d1-bc76-32dfb7483a0a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"
Sep 18 16:41:29.099487 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.099367    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a43c8287-91ac-46d1-bc76-32dfb7483a0a-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-26h8m\" (UID: \"a43c8287-91ac-46d1-bc76-32dfb7483a0a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"
Sep 18 16:41:29.099487 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.099406    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a43c8287-91ac-46d1-bc76-32dfb7483a0a-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-26h8m\" (UID: \"a43c8287-91ac-46d1-bc76-32dfb7483a0a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"
Sep 18 16:41:29.100363 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.100342    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"]
Sep 18 16:41:29.200281 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.200230    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a43c8287-91ac-46d1-bc76-32dfb7483a0a-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-26h8m\" (UID: \"a43c8287-91ac-46d1-bc76-32dfb7483a0a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"
Sep 18 16:41:29.200492 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.200300    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a43c8287-91ac-46d1-bc76-32dfb7483a0a-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-26h8m\" (UID: \"a43c8287-91ac-46d1-bc76-32dfb7483a0a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"
Sep 18 16:41:29.200492 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.200373    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a43c8287-91ac-46d1-bc76-32dfb7483a0a-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-26h8m\" (UID: \"a43c8287-91ac-46d1-bc76-32dfb7483a0a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"
Sep 18 16:41:29.200492 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.200457    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a43c8287-91ac-46d1-bc76-32dfb7483a0a-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-26h8m\" (UID: \"a43c8287-91ac-46d1-bc76-32dfb7483a0a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"
Sep 18 16:41:29.200655 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.200493    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a43c8287-91ac-46d1-bc76-32dfb7483a0a-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-26h8m\" (UID: \"a43c8287-91ac-46d1-bc76-32dfb7483a0a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"
Sep 18 16:41:29.200655 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.200521    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxrgw\" (UniqueName: \"kubernetes.io/projected/a43c8287-91ac-46d1-bc76-32dfb7483a0a-kube-api-access-jxrgw\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-26h8m\" (UID: \"a43c8287-91ac-46d1-bc76-32dfb7483a0a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"
Sep 18 16:41:29.201059 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.201025    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a43c8287-91ac-46d1-bc76-32dfb7483a0a-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-26h8m\" (UID: \"a43c8287-91ac-46d1-bc76-32dfb7483a0a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"
Sep 18 16:41:29.201196 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.201163    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a43c8287-91ac-46d1-bc76-32dfb7483a0a-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-26h8m\" (UID: \"a43c8287-91ac-46d1-bc76-32dfb7483a0a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"
Sep 18 16:41:29.201341 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.201304    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a43c8287-91ac-46d1-bc76-32dfb7483a0a-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-26h8m\" (UID: \"a43c8287-91ac-46d1-bc76-32dfb7483a0a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"
Sep 18 16:41:29.203364 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.203339    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a43c8287-91ac-46d1-bc76-32dfb7483a0a-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-26h8m\" (UID: \"a43c8287-91ac-46d1-bc76-32dfb7483a0a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"
Sep 18 16:41:29.204051 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.204027    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a43c8287-91ac-46d1-bc76-32dfb7483a0a-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-26h8m\" (UID: \"a43c8287-91ac-46d1-bc76-32dfb7483a0a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"
Sep 18 16:41:29.208900 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.208876    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxrgw\" (UniqueName: \"kubernetes.io/projected/a43c8287-91ac-46d1-bc76-32dfb7483a0a-kube-api-access-jxrgw\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-26h8m\" (UID: \"a43c8287-91ac-46d1-bc76-32dfb7483a0a\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"
Sep 18 16:41:29.358349 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.358273    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892"]
Sep 18 16:41:29.360511 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.360488    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892"
Sep 18 16:41:29.362586 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.362558    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892"]
Sep 18 16:41:29.363370 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.363352    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-lxdjv\""
Sep 18 16:41:29.396596 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.396570    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"
Sep 18 16:41:29.403020 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.402984    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/145e2cb9-8a1f-486c-b92e-804fb7468a54-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892\" (UID: \"145e2cb9-8a1f-486c-b92e-804fb7468a54\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892"
Sep 18 16:41:29.403153 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.403063    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnmwj\" (UniqueName: \"kubernetes.io/projected/145e2cb9-8a1f-486c-b92e-804fb7468a54-kube-api-access-rnmwj\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892\" (UID: \"145e2cb9-8a1f-486c-b92e-804fb7468a54\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892"
Sep 18 16:41:29.504227 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.504191    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/145e2cb9-8a1f-486c-b92e-804fb7468a54-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892\" (UID: \"145e2cb9-8a1f-486c-b92e-804fb7468a54\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892"
Sep 18 16:41:29.504380 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.504273    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnmwj\" (UniqueName: \"kubernetes.io/projected/145e2cb9-8a1f-486c-b92e-804fb7468a54-kube-api-access-rnmwj\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892\" (UID: \"145e2cb9-8a1f-486c-b92e-804fb7468a54\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892"
Sep 18 16:41:29.506713 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.506686    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/145e2cb9-8a1f-486c-b92e-804fb7468a54-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892\" (UID: \"145e2cb9-8a1f-486c-b92e-804fb7468a54\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892"
Sep 18 16:41:29.513246 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.513222    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnmwj\" (UniqueName: \"kubernetes.io/projected/145e2cb9-8a1f-486c-b92e-804fb7468a54-kube-api-access-rnmwj\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892\" (UID: \"145e2cb9-8a1f-486c-b92e-804fb7468a54\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892"
Sep 18 16:41:29.673510 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.673421    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892"
Sep 18 16:41:29.675307 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.675282    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"]
Sep 18 16:41:29.677650 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:41:29.677619    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda43c8287_91ac_46d1_bc76_32dfb7483a0a.slice/crio-6da1116116e6279988717050d82d5ad8085f6cc3b79702f97df4d82d1be34622 WatchSource:0}: Error finding container 6da1116116e6279988717050d82d5ad8085f6cc3b79702f97df4d82d1be34622: Status 404 returned error can't find the container with id 6da1116116e6279988717050d82d5ad8085f6cc3b79702f97df4d82d1be34622
Sep 18 16:41:29.680549 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.680528    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:41:29.949507 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:29.949484    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892"]
Sep 18 16:41:29.951793 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:41:29.951766    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145e2cb9_8a1f_486c_b92e_804fb7468a54.slice/crio-8baa5e3d1a153e464ede71aad8712701eb4c29c749a73de866030870381b0ee4 WatchSource:0}: Error finding container 8baa5e3d1a153e464ede71aad8712701eb4c29c749a73de866030870381b0ee4: Status 404 returned error can't find the container with id 8baa5e3d1a153e464ede71aad8712701eb4c29c749a73de866030870381b0ee4
Sep 18 16:41:30.053802 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:30.053738    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m" event={"ID":"a43c8287-91ac-46d1-bc76-32dfb7483a0a","Type":"ContainerStarted","Data":"6da1116116e6279988717050d82d5ad8085f6cc3b79702f97df4d82d1be34622"}
Sep 18 16:41:30.055449 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:30.055421    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892" event={"ID":"145e2cb9-8a1f-486c-b92e-804fb7468a54","Type":"ContainerStarted","Data":"6e4bd31412e05b3236fe614bb05f1adbb2b3d0f56af4780f1f13199dd15d4831"}
Sep 18 16:41:30.055594 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:30.055468    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892" event={"ID":"145e2cb9-8a1f-486c-b92e-804fb7468a54","Type":"ContainerStarted","Data":"8baa5e3d1a153e464ede71aad8712701eb4c29c749a73de866030870381b0ee4"}
Sep 18 16:41:30.055594 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:30.055581    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892"
Sep 18 16:41:30.072261 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:30.072191    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892" podStartSLOduration=1.072170302 podStartE2EDuration="1.072170302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:41:30.071434454 +0000 UTC m=+1529.854436490" watchObservedRunningTime="2026-09-18 16:41:30.072170302 +0000 UTC m=+1529.855172326"
Sep 18 16:41:30.805484 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:30.805414    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph" podUID="ecc2277a-2021-4f02-b9fb-fb7fe9fba85f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": dial tcp 10.133.0.30:8000: connect: connection refused"
Sep 18 16:41:32.067591 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:32.067557    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m" event={"ID":"a43c8287-91ac-46d1-bc76-32dfb7483a0a","Type":"ContainerStarted","Data":"1f317ed5f7a30ad05346849999601483a00890cca0111e0dfedc299a622203e3"}
Sep 18 16:41:32.092478 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:32.092422    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m" podStartSLOduration=1.634810039 podStartE2EDuration="3.092407369s" totalImagesPullingTime="1.45759733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:41:32.091003521 +0000 UTC m=+1531.874005547" watchObservedRunningTime="2026-09-18 16:41:32.092407369 +0000 UTC m=+1531.875409432"
Sep 18 16:41:32.397376 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:32.397299    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"
Sep 18 16:41:39.397045 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:39.397006    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"
Sep 18 16:41:39.410204 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:39.410180    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"
Sep 18 16:41:40.115710 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:40.115679    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"
Sep 18 16:41:40.805877 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:40.805823    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph" podUID="ecc2277a-2021-4f02-b9fb-fb7fe9fba85f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": dial tcp 10.133.0.30:8000: connect: connection refused"
Sep 18 16:41:50.806279 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:41:50.806227    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph" podUID="ecc2277a-2021-4f02-b9fb-fb7fe9fba85f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": dial tcp 10.133.0.30:8000: connect: connection refused"
Sep 18 16:42:00.805851 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:00.805812    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph" podUID="ecc2277a-2021-4f02-b9fb-fb7fe9fba85f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": dial tcp 10.133.0.30:8000: connect: connection refused"
Sep 18 16:42:01.062773 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:01.062686    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892" podUID="145e2cb9-8a1f-486c-b92e-804fb7468a54" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:42:02.980809 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:02.980768    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"]
Sep 18 16:42:02.989800 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:02.989769    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892"]
Sep 18 16:42:02.991106 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:42:02.991023    2585 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 16:42:02.991106 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:42:02.991106 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 16:42:02.991106 ip-10-0-133-147 kubenswrapper[2585]:  > containerID="1f317ed5f7a30ad05346849999601483a00890cca0111e0dfedc299a622203e3" cmd=["/bin/sleep","15"]
Sep 18 16:42:02.991106 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:02.991094    2585 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m" message="" err=<
Sep 18 16:42:02.991106 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:42:02.991106 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 16:42:02.991106 ip-10-0-133-147 kubenswrapper[2585]:  >
Sep 18 16:42:02.991553 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:42:02.991111    2585 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 16:42:02.991553 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:42:02.991553 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 16:42:02.991553 ip-10-0-133-147 kubenswrapper[2585]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m" podUID="a43c8287-91ac-46d1-bc76-32dfb7483a0a" containerName="main" containerID="cri-o://1f317ed5f7a30ad05346849999601483a00890cca0111e0dfedc299a622203e3"
Sep 18 16:42:02.991553 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:02.991166    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m" podUID="a43c8287-91ac-46d1-bc76-32dfb7483a0a" containerName="main" containerID="cri-o://1f317ed5f7a30ad05346849999601483a00890cca0111e0dfedc299a622203e3" gracePeriod=60
Sep 18 16:42:02.991867 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:02.991756    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892" podUID="145e2cb9-8a1f-486c-b92e-804fb7468a54" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:42:03.104626 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:03.104591    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m" podUID="a43c8287-91ac-46d1-bc76-32dfb7483a0a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/health\": dial tcp 10.133.0.31:8000: connect: connection refused"
Sep 18 16:42:03.330477 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:03.330449    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"
Sep 18 16:42:03.518291 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:03.518208    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a43c8287-91ac-46d1-bc76-32dfb7483a0a-model-cache\") pod \"a43c8287-91ac-46d1-bc76-32dfb7483a0a\" (UID: \"a43c8287-91ac-46d1-bc76-32dfb7483a0a\") "
Sep 18 16:42:03.518452 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:03.518300    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxrgw\" (UniqueName: \"kubernetes.io/projected/a43c8287-91ac-46d1-bc76-32dfb7483a0a-kube-api-access-jxrgw\") pod \"a43c8287-91ac-46d1-bc76-32dfb7483a0a\" (UID: \"a43c8287-91ac-46d1-bc76-32dfb7483a0a\") "
Sep 18 16:42:03.518452 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:03.518326    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a43c8287-91ac-46d1-bc76-32dfb7483a0a-tls-certs\") pod \"a43c8287-91ac-46d1-bc76-32dfb7483a0a\" (UID: \"a43c8287-91ac-46d1-bc76-32dfb7483a0a\") "
Sep 18 16:42:03.518452 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:03.518372    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a43c8287-91ac-46d1-bc76-32dfb7483a0a-tmp-dir\") pod \"a43c8287-91ac-46d1-bc76-32dfb7483a0a\" (UID: \"a43c8287-91ac-46d1-bc76-32dfb7483a0a\") "
Sep 18 16:42:03.518452 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:03.518396    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a43c8287-91ac-46d1-bc76-32dfb7483a0a-home\") pod \"a43c8287-91ac-46d1-bc76-32dfb7483a0a\" (UID: \"a43c8287-91ac-46d1-bc76-32dfb7483a0a\") "
Sep 18 16:42:03.518452 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:03.518417    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a43c8287-91ac-46d1-bc76-32dfb7483a0a-dshm\") pod \"a43c8287-91ac-46d1-bc76-32dfb7483a0a\" (UID: \"a43c8287-91ac-46d1-bc76-32dfb7483a0a\") "
Sep 18 16:42:03.518710 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:03.518538    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43c8287-91ac-46d1-bc76-32dfb7483a0a-model-cache" (OuterVolumeSpecName: "model-cache") pod "a43c8287-91ac-46d1-bc76-32dfb7483a0a" (UID: "a43c8287-91ac-46d1-bc76-32dfb7483a0a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:42:03.518710 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:03.518618    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43c8287-91ac-46d1-bc76-32dfb7483a0a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a43c8287-91ac-46d1-bc76-32dfb7483a0a" (UID: "a43c8287-91ac-46d1-bc76-32dfb7483a0a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:42:03.518710 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:03.518673    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a43c8287-91ac-46d1-bc76-32dfb7483a0a-model-cache\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:42:03.518903 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:03.518774    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43c8287-91ac-46d1-bc76-32dfb7483a0a-home" (OuterVolumeSpecName: "home") pod "a43c8287-91ac-46d1-bc76-32dfb7483a0a" (UID: "a43c8287-91ac-46d1-bc76-32dfb7483a0a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:42:03.520882 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:03.520857    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43c8287-91ac-46d1-bc76-32dfb7483a0a-dshm" (OuterVolumeSpecName: "dshm") pod "a43c8287-91ac-46d1-bc76-32dfb7483a0a" (UID: "a43c8287-91ac-46d1-bc76-32dfb7483a0a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:42:03.521008 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:03.520879    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a43c8287-91ac-46d1-bc76-32dfb7483a0a-kube-api-access-jxrgw" (OuterVolumeSpecName: "kube-api-access-jxrgw") pod "a43c8287-91ac-46d1-bc76-32dfb7483a0a" (UID: "a43c8287-91ac-46d1-bc76-32dfb7483a0a"). InnerVolumeSpecName "kube-api-access-jxrgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:42:03.521008 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:03.520899    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a43c8287-91ac-46d1-bc76-32dfb7483a0a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a43c8287-91ac-46d1-bc76-32dfb7483a0a" (UID: "a43c8287-91ac-46d1-bc76-32dfb7483a0a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:42:03.619854 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:03.619822    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxrgw\" (UniqueName: \"kubernetes.io/projected/a43c8287-91ac-46d1-bc76-32dfb7483a0a-kube-api-access-jxrgw\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:42:03.619854 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:03.619854    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a43c8287-91ac-46d1-bc76-32dfb7483a0a-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:42:03.620101 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:03.619897    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a43c8287-91ac-46d1-bc76-32dfb7483a0a-tmp-dir\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:42:03.620101 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:03.619937    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a43c8287-91ac-46d1-bc76-32dfb7483a0a-home\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:42:03.620101 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:03.619951    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a43c8287-91ac-46d1-bc76-32dfb7483a0a-dshm\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:42:04.210232 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:04.210194    2585 generic.go:356] "Generic (PLEG): container finished" podID="a43c8287-91ac-46d1-bc76-32dfb7483a0a" containerID="1f317ed5f7a30ad05346849999601483a00890cca0111e0dfedc299a622203e3" exitCode=0
Sep 18 16:42:04.210615 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:04.210239    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"
Sep 18 16:42:04.210615 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:04.210265    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m" event={"ID":"a43c8287-91ac-46d1-bc76-32dfb7483a0a","Type":"ContainerDied","Data":"1f317ed5f7a30ad05346849999601483a00890cca0111e0dfedc299a622203e3"}
Sep 18 16:42:04.210615 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:04.210310    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m" event={"ID":"a43c8287-91ac-46d1-bc76-32dfb7483a0a","Type":"ContainerDied","Data":"6da1116116e6279988717050d82d5ad8085f6cc3b79702f97df4d82d1be34622"}
Sep 18 16:42:04.210615 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:04.210334    2585 scope.go:117] "RemoveContainer" containerID="1f317ed5f7a30ad05346849999601483a00890cca0111e0dfedc299a622203e3"
Sep 18 16:42:04.220154 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:04.220132    2585 scope.go:117] "RemoveContainer" containerID="1f317ed5f7a30ad05346849999601483a00890cca0111e0dfedc299a622203e3"
Sep 18 16:42:04.220444 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:42:04.220428    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f317ed5f7a30ad05346849999601483a00890cca0111e0dfedc299a622203e3\": container with ID starting with 1f317ed5f7a30ad05346849999601483a00890cca0111e0dfedc299a622203e3 not found: ID does not exist" containerID="1f317ed5f7a30ad05346849999601483a00890cca0111e0dfedc299a622203e3"
Sep 18 16:42:04.220498 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:04.220450    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f317ed5f7a30ad05346849999601483a00890cca0111e0dfedc299a622203e3"} err="failed to get container status \"1f317ed5f7a30ad05346849999601483a00890cca0111e0dfedc299a622203e3\": rpc error: code = NotFound desc = could not find container \"1f317ed5f7a30ad05346849999601483a00890cca0111e0dfedc299a622203e3\": container with ID starting with 1f317ed5f7a30ad05346849999601483a00890cca0111e0dfedc299a622203e3 not found: ID does not exist"
Sep 18 16:42:04.234808 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:04.234771    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"]
Sep 18 16:42:04.241906 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:04.241885    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-26h8m"]
Sep 18 16:42:04.392460 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:04.392427    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a43c8287-91ac-46d1-bc76-32dfb7483a0a" path="/var/lib/kubelet/pods/a43c8287-91ac-46d1-bc76-32dfb7483a0a/volumes"
Sep 18 16:42:10.814822 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:10.814790    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:42:10.822643 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:10.822619    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:42:12.992309 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:12.992265    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892" podUID="145e2cb9-8a1f-486c-b92e-804fb7468a54" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:42:15.240251 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:15.240217    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"]
Sep 18 16:42:18.010259 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:18.010207    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892" podUID="145e2cb9-8a1f-486c-b92e-804fb7468a54" containerName="main" containerID="cri-o://6e4bd31412e05b3236fe614bb05f1adbb2b3d0f56af4780f1f13199dd15d4831" gracePeriod=45
Sep 18 16:42:22.991848 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:22.991810    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892" podUID="145e2cb9-8a1f-486c-b92e-804fb7468a54" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:42:30.260272 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:30.260230    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph" podUID="ecc2277a-2021-4f02-b9fb-fb7fe9fba85f" containerName="main" containerID="cri-o://3f9e4ef2c32c218b5996a4477b40ea2c4e13767bf1d3a1fdee2e0879a457a637" gracePeriod=45
Sep 18 16:42:30.816263 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:30.816220    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph" podUID="ecc2277a-2021-4f02-b9fb-fb7fe9fba85f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/health\": dial tcp 10.133.0.30:8000: connect: connection refused"
Sep 18 16:42:31.037468 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.037441    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:42:31.138242 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.138152    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-home\") pod \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") "
Sep 18 16:42:31.138242 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.138200    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f699p\" (UniqueName: \"kubernetes.io/projected/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-kube-api-access-f699p\") pod \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") "
Sep 18 16:42:31.138242 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.138226    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-cabundle-cert\") pod \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") "
Sep 18 16:42:31.138528 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.138260    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-kserve-provision-location\") pod \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") "
Sep 18 16:42:31.138528 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.138293    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-tmp-dir\") pod \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") "
Sep 18 16:42:31.138528 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.138326    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-model-cache\") pod \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") "
Sep 18 16:42:31.138528 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.138348    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-tls-certs\") pod \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") "
Sep 18 16:42:31.138528 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.138375    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-dshm\") pod \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\" (UID: \"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f\") "
Sep 18 16:42:31.138763 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.138659    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-model-cache" (OuterVolumeSpecName: "model-cache") pod "ecc2277a-2021-4f02-b9fb-fb7fe9fba85f" (UID: "ecc2277a-2021-4f02-b9fb-fb7fe9fba85f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:42:31.138763 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.138665    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ecc2277a-2021-4f02-b9fb-fb7fe9fba85f" (UID: "ecc2277a-2021-4f02-b9fb-fb7fe9fba85f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:42:31.138763 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.138675    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-home" (OuterVolumeSpecName: "home") pod "ecc2277a-2021-4f02-b9fb-fb7fe9fba85f" (UID: "ecc2277a-2021-4f02-b9fb-fb7fe9fba85f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:42:31.140956 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.140929    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-dshm" (OuterVolumeSpecName: "dshm") pod "ecc2277a-2021-4f02-b9fb-fb7fe9fba85f" (UID: "ecc2277a-2021-4f02-b9fb-fb7fe9fba85f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:42:31.141089 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.140989    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ecc2277a-2021-4f02-b9fb-fb7fe9fba85f" (UID: "ecc2277a-2021-4f02-b9fb-fb7fe9fba85f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:42:31.141194 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.141174    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-kube-api-access-f699p" (OuterVolumeSpecName: "kube-api-access-f699p") pod "ecc2277a-2021-4f02-b9fb-fb7fe9fba85f" (UID: "ecc2277a-2021-4f02-b9fb-fb7fe9fba85f"). InnerVolumeSpecName "kube-api-access-f699p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:42:31.156293 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.156237    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ecc2277a-2021-4f02-b9fb-fb7fe9fba85f" (UID: "ecc2277a-2021-4f02-b9fb-fb7fe9fba85f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:42:31.161884 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.161846    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ecc2277a-2021-4f02-b9fb-fb7fe9fba85f" (UID: "ecc2277a-2021-4f02-b9fb-fb7fe9fba85f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:42:31.239861 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.239823    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f699p\" (UniqueName: \"kubernetes.io/projected/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-kube-api-access-f699p\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:42:31.239861 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.239856    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-cabundle-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:42:31.239861 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.239866    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-kserve-provision-location\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:42:31.240098 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.239876    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-tmp-dir\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:42:31.240098 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.239886    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-model-cache\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:42:31.240098 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.239893    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:42:31.240098 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.239900    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-dshm\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:42:31.240098 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.239907    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f-home\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:42:31.324375 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.324335    2585 generic.go:356] "Generic (PLEG): container finished" podID="ecc2277a-2021-4f02-b9fb-fb7fe9fba85f" containerID="3f9e4ef2c32c218b5996a4477b40ea2c4e13767bf1d3a1fdee2e0879a457a637" exitCode=0
Sep 18 16:42:31.324771 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.324399    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"
Sep 18 16:42:31.324771 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.324430    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph" event={"ID":"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f","Type":"ContainerDied","Data":"3f9e4ef2c32c218b5996a4477b40ea2c4e13767bf1d3a1fdee2e0879a457a637"}
Sep 18 16:42:31.324771 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.324467    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph" event={"ID":"ecc2277a-2021-4f02-b9fb-fb7fe9fba85f","Type":"ContainerDied","Data":"1a0e1ad9a6655b0814421486435b922dceb0cbb8f13ec1d76a3f9bf73b30d497"}
Sep 18 16:42:31.324771 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.324485    2585 scope.go:117] "RemoveContainer" containerID="3f9e4ef2c32c218b5996a4477b40ea2c4e13767bf1d3a1fdee2e0879a457a637"
Sep 18 16:42:31.333645 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.333623    2585 scope.go:117] "RemoveContainer" containerID="1e2d575225a276cefa85af4a9791eb24298f27daa6bea9e99d8ab1fd8b66cf24"
Sep 18 16:42:31.341289 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.341272    2585 scope.go:117] "RemoveContainer" containerID="3f9e4ef2c32c218b5996a4477b40ea2c4e13767bf1d3a1fdee2e0879a457a637"
Sep 18 16:42:31.341542 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:42:31.341526    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f9e4ef2c32c218b5996a4477b40ea2c4e13767bf1d3a1fdee2e0879a457a637\": container with ID starting with 3f9e4ef2c32c218b5996a4477b40ea2c4e13767bf1d3a1fdee2e0879a457a637 not found: ID does not exist" containerID="3f9e4ef2c32c218b5996a4477b40ea2c4e13767bf1d3a1fdee2e0879a457a637"
Sep 18 16:42:31.341585 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.341550    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f9e4ef2c32c218b5996a4477b40ea2c4e13767bf1d3a1fdee2e0879a457a637"} err="failed to get container status \"3f9e4ef2c32c218b5996a4477b40ea2c4e13767bf1d3a1fdee2e0879a457a637\": rpc error: code = NotFound desc = could not find container \"3f9e4ef2c32c218b5996a4477b40ea2c4e13767bf1d3a1fdee2e0879a457a637\": container with ID starting with 3f9e4ef2c32c218b5996a4477b40ea2c4e13767bf1d3a1fdee2e0879a457a637 not found: ID does not exist"
Sep 18 16:42:31.341585 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.341567    2585 scope.go:117] "RemoveContainer" containerID="1e2d575225a276cefa85af4a9791eb24298f27daa6bea9e99d8ab1fd8b66cf24"
Sep 18 16:42:31.341829 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:42:31.341810    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e2d575225a276cefa85af4a9791eb24298f27daa6bea9e99d8ab1fd8b66cf24\": container with ID starting with 1e2d575225a276cefa85af4a9791eb24298f27daa6bea9e99d8ab1fd8b66cf24 not found: ID does not exist" containerID="1e2d575225a276cefa85af4a9791eb24298f27daa6bea9e99d8ab1fd8b66cf24"
Sep 18 16:42:31.341890 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.341834    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e2d575225a276cefa85af4a9791eb24298f27daa6bea9e99d8ab1fd8b66cf24"} err="failed to get container status \"1e2d575225a276cefa85af4a9791eb24298f27daa6bea9e99d8ab1fd8b66cf24\": rpc error: code = NotFound desc = could not find container \"1e2d575225a276cefa85af4a9791eb24298f27daa6bea9e99d8ab1fd8b66cf24\": container with ID starting with 1e2d575225a276cefa85af4a9791eb24298f27daa6bea9e99d8ab1fd8b66cf24 not found: ID does not exist"
Sep 18 16:42:31.350043 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.350016    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"]
Sep 18 16:42:31.355276 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:31.355252    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-7957f85b4-nlmph"]
Sep 18 16:42:32.392502 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:32.392468    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecc2277a-2021-4f02-b9fb-fb7fe9fba85f" path="/var/lib/kubelet/pods/ecc2277a-2021-4f02-b9fb-fb7fe9fba85f/volumes"
Sep 18 16:42:32.991896 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:32.991856    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892" podUID="145e2cb9-8a1f-486c-b92e-804fb7468a54" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:42:42.992414 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:42.992330    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892" podUID="145e2cb9-8a1f-486c-b92e-804fb7468a54" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:42:48.251072 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:48.251042    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892"
Sep 18 16:42:48.384109 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:48.384032    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnmwj\" (UniqueName: \"kubernetes.io/projected/145e2cb9-8a1f-486c-b92e-804fb7468a54-kube-api-access-rnmwj\") pod \"145e2cb9-8a1f-486c-b92e-804fb7468a54\" (UID: \"145e2cb9-8a1f-486c-b92e-804fb7468a54\") "
Sep 18 16:42:48.384109 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:48.384075    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/145e2cb9-8a1f-486c-b92e-804fb7468a54-tls-certs\") pod \"145e2cb9-8a1f-486c-b92e-804fb7468a54\" (UID: \"145e2cb9-8a1f-486c-b92e-804fb7468a54\") "
Sep 18 16:42:48.386413 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:48.386384    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/145e2cb9-8a1f-486c-b92e-804fb7468a54-kube-api-access-rnmwj" (OuterVolumeSpecName: "kube-api-access-rnmwj") pod "145e2cb9-8a1f-486c-b92e-804fb7468a54" (UID: "145e2cb9-8a1f-486c-b92e-804fb7468a54"). InnerVolumeSpecName "kube-api-access-rnmwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:42:48.386536 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:48.386384    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/145e2cb9-8a1f-486c-b92e-804fb7468a54-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "145e2cb9-8a1f-486c-b92e-804fb7468a54" (UID: "145e2cb9-8a1f-486c-b92e-804fb7468a54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:42:48.394045 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:48.394018    2585 generic.go:356] "Generic (PLEG): container finished" podID="145e2cb9-8a1f-486c-b92e-804fb7468a54" containerID="6e4bd31412e05b3236fe614bb05f1adbb2b3d0f56af4780f1f13199dd15d4831" exitCode=0
Sep 18 16:42:48.394163 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:48.394061    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892" event={"ID":"145e2cb9-8a1f-486c-b92e-804fb7468a54","Type":"ContainerDied","Data":"6e4bd31412e05b3236fe614bb05f1adbb2b3d0f56af4780f1f13199dd15d4831"}
Sep 18 16:42:48.394163 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:48.394074    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892"
Sep 18 16:42:48.394163 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:48.394101    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892" event={"ID":"145e2cb9-8a1f-486c-b92e-804fb7468a54","Type":"ContainerDied","Data":"8baa5e3d1a153e464ede71aad8712701eb4c29c749a73de866030870381b0ee4"}
Sep 18 16:42:48.394163 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:48.394123    2585 scope.go:117] "RemoveContainer" containerID="6e4bd31412e05b3236fe614bb05f1adbb2b3d0f56af4780f1f13199dd15d4831"
Sep 18 16:42:48.404954 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:48.404935    2585 scope.go:117] "RemoveContainer" containerID="6e4bd31412e05b3236fe614bb05f1adbb2b3d0f56af4780f1f13199dd15d4831"
Sep 18 16:42:48.405257 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:42:48.405237    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e4bd31412e05b3236fe614bb05f1adbb2b3d0f56af4780f1f13199dd15d4831\": container with ID starting with 6e4bd31412e05b3236fe614bb05f1adbb2b3d0f56af4780f1f13199dd15d4831 not found: ID does not exist" containerID="6e4bd31412e05b3236fe614bb05f1adbb2b3d0f56af4780f1f13199dd15d4831"
Sep 18 16:42:48.405325 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:48.405270    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e4bd31412e05b3236fe614bb05f1adbb2b3d0f56af4780f1f13199dd15d4831"} err="failed to get container status \"6e4bd31412e05b3236fe614bb05f1adbb2b3d0f56af4780f1f13199dd15d4831\": rpc error: code = NotFound desc = could not find container \"6e4bd31412e05b3236fe614bb05f1adbb2b3d0f56af4780f1f13199dd15d4831\": container with ID starting with 6e4bd31412e05b3236fe614bb05f1adbb2b3d0f56af4780f1f13199dd15d4831 not found: ID does not exist"
Sep 18 16:42:48.417536 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:48.417513    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892"]
Sep 18 16:42:48.421511 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:48.421488    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc562892"]
Sep 18 16:42:48.485625 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:48.485604    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnmwj\" (UniqueName: \"kubernetes.io/projected/145e2cb9-8a1f-486c-b92e-804fb7468a54-kube-api-access-rnmwj\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:42:48.485727 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:48.485628    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/145e2cb9-8a1f-486c-b92e-804fb7468a54-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:42:50.394185 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:42:50.394152    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="145e2cb9-8a1f-486c-b92e-804fb7468a54" path="/var/lib/kubelet/pods/145e2cb9-8a1f-486c-b92e-804fb7468a54/volumes"
Sep 18 16:43:03.187015 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.186981    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"]
Sep 18 16:43:03.187392 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.187269    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecc2277a-2021-4f02-b9fb-fb7fe9fba85f" containerName="storage-initializer"
Sep 18 16:43:03.187392 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.187279    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecc2277a-2021-4f02-b9fb-fb7fe9fba85f" containerName="storage-initializer"
Sep 18 16:43:03.187392 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.187288    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a43c8287-91ac-46d1-bc76-32dfb7483a0a" containerName="main"
Sep 18 16:43:03.187392 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.187293    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a43c8287-91ac-46d1-bc76-32dfb7483a0a" containerName="main"
Sep 18 16:43:03.187392 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.187300    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="145e2cb9-8a1f-486c-b92e-804fb7468a54" containerName="main"
Sep 18 16:43:03.187392 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.187305    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="145e2cb9-8a1f-486c-b92e-804fb7468a54" containerName="main"
Sep 18 16:43:03.187392 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.187320    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecc2277a-2021-4f02-b9fb-fb7fe9fba85f" containerName="main"
Sep 18 16:43:03.187392 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.187325    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecc2277a-2021-4f02-b9fb-fb7fe9fba85f" containerName="main"
Sep 18 16:43:03.187392 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.187369    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a43c8287-91ac-46d1-bc76-32dfb7483a0a" containerName="main"
Sep 18 16:43:03.187659 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.187403    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="145e2cb9-8a1f-486c-b92e-804fb7468a54" containerName="main"
Sep 18 16:43:03.187659 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.187412    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecc2277a-2021-4f02-b9fb-fb7fe9fba85f" containerName="main"
Sep 18 16:43:03.189941 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.189905    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"
Sep 18 16:43:03.192559 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.192541    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 18 16:43:03.193794 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.193777    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 18 16:43:03.193896 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.193778    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-rqldl\""
Sep 18 16:43:03.193896 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.193778    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 18 16:43:03.198811 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.198786    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"]
Sep 18 16:43:03.306989 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.306954    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzr4p\" (UniqueName: \"kubernetes.io/projected/486940b1-cab9-4d61-b7bb-3f5f110561e6-kube-api-access-wzr4p\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-n77g6\" (UID: \"486940b1-cab9-4d61-b7bb-3f5f110561e6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"
Sep 18 16:43:03.307145 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.306993    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/486940b1-cab9-4d61-b7bb-3f5f110561e6-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-n77g6\" (UID: \"486940b1-cab9-4d61-b7bb-3f5f110561e6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"
Sep 18 16:43:03.307145 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.307028    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/486940b1-cab9-4d61-b7bb-3f5f110561e6-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-n77g6\" (UID: \"486940b1-cab9-4d61-b7bb-3f5f110561e6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"
Sep 18 16:43:03.307145 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.307083    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/486940b1-cab9-4d61-b7bb-3f5f110561e6-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-n77g6\" (UID: \"486940b1-cab9-4d61-b7bb-3f5f110561e6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"
Sep 18 16:43:03.307145 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.307107    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/486940b1-cab9-4d61-b7bb-3f5f110561e6-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-n77g6\" (UID: \"486940b1-cab9-4d61-b7bb-3f5f110561e6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"
Sep 18 16:43:03.307145 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.307126    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/486940b1-cab9-4d61-b7bb-3f5f110561e6-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-n77g6\" (UID: \"486940b1-cab9-4d61-b7bb-3f5f110561e6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"
Sep 18 16:43:03.408126 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.408095    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/486940b1-cab9-4d61-b7bb-3f5f110561e6-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-n77g6\" (UID: \"486940b1-cab9-4d61-b7bb-3f5f110561e6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"
Sep 18 16:43:03.408126 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.408129    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/486940b1-cab9-4d61-b7bb-3f5f110561e6-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-n77g6\" (UID: \"486940b1-cab9-4d61-b7bb-3f5f110561e6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"
Sep 18 16:43:03.408314 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.408154    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/486940b1-cab9-4d61-b7bb-3f5f110561e6-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-n77g6\" (UID: \"486940b1-cab9-4d61-b7bb-3f5f110561e6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"
Sep 18 16:43:03.408314 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.408208    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzr4p\" (UniqueName: \"kubernetes.io/projected/486940b1-cab9-4d61-b7bb-3f5f110561e6-kube-api-access-wzr4p\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-n77g6\" (UID: \"486940b1-cab9-4d61-b7bb-3f5f110561e6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"
Sep 18 16:43:03.408314 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.408244    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/486940b1-cab9-4d61-b7bb-3f5f110561e6-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-n77g6\" (UID: \"486940b1-cab9-4d61-b7bb-3f5f110561e6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"
Sep 18 16:43:03.408314 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.408270    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/486940b1-cab9-4d61-b7bb-3f5f110561e6-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-n77g6\" (UID: \"486940b1-cab9-4d61-b7bb-3f5f110561e6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"
Sep 18 16:43:03.408574 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.408552    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/486940b1-cab9-4d61-b7bb-3f5f110561e6-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-n77g6\" (UID: \"486940b1-cab9-4d61-b7bb-3f5f110561e6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"
Sep 18 16:43:03.408613 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.408580    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/486940b1-cab9-4d61-b7bb-3f5f110561e6-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-n77g6\" (UID: \"486940b1-cab9-4d61-b7bb-3f5f110561e6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"
Sep 18 16:43:03.408681 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.408655    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/486940b1-cab9-4d61-b7bb-3f5f110561e6-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-n77g6\" (UID: \"486940b1-cab9-4d61-b7bb-3f5f110561e6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"
Sep 18 16:43:03.410646 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.410624    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/486940b1-cab9-4d61-b7bb-3f5f110561e6-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-n77g6\" (UID: \"486940b1-cab9-4d61-b7bb-3f5f110561e6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"
Sep 18 16:43:03.410792 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.410776    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/486940b1-cab9-4d61-b7bb-3f5f110561e6-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-n77g6\" (UID: \"486940b1-cab9-4d61-b7bb-3f5f110561e6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"
Sep 18 16:43:03.416493 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.416473    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzr4p\" (UniqueName: \"kubernetes.io/projected/486940b1-cab9-4d61-b7bb-3f5f110561e6-kube-api-access-wzr4p\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-n77g6\" (UID: \"486940b1-cab9-4d61-b7bb-3f5f110561e6\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"
Sep 18 16:43:03.500067 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.500038    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"
Sep 18 16:43:03.770364 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:03.770269    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"]
Sep 18 16:43:03.773223 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:43:03.773195    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486940b1_cab9_4d61_b7bb_3f5f110561e6.slice/crio-7f7057c54ebd4ef2da5bdcf535107dea5ccb4c1c4735560118b064f6612e9860 WatchSource:0}: Error finding container 7f7057c54ebd4ef2da5bdcf535107dea5ccb4c1c4735560118b064f6612e9860: Status 404 returned error can't find the container with id 7f7057c54ebd4ef2da5bdcf535107dea5ccb4c1c4735560118b064f6612e9860
Sep 18 16:43:04.461466 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:04.461429    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6" event={"ID":"486940b1-cab9-4d61-b7bb-3f5f110561e6","Type":"ContainerStarted","Data":"c3dec983408452d7666cf171b8fa187bd716b531e7ad7717cad0bc2f7e8c6d6b"}
Sep 18 16:43:04.461850 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:04.461501    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6" event={"ID":"486940b1-cab9-4d61-b7bb-3f5f110561e6","Type":"ContainerStarted","Data":"7f7057c54ebd4ef2da5bdcf535107dea5ccb4c1c4735560118b064f6612e9860"}
Sep 18 16:43:04.481540 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:04.481488    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6" podStartSLOduration=1.481472948 podStartE2EDuration="1.481472948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:43:04.480350256 +0000 UTC m=+1624.263352285" watchObservedRunningTime="2026-09-18 16:43:04.481472948 +0000 UTC m=+1624.264474969"
Sep 18 16:43:04.501070 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:04.501047    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"
Sep 18 16:43:13.501099 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:13.501066    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"
Sep 18 16:43:13.513246 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:13.513220    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"
Sep 18 16:43:14.511586 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:14.511558    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"
Sep 18 16:43:23.445839 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:23.445808    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd"]
Sep 18 16:43:23.448112 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:23.448095    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd"
Sep 18 16:43:23.450963 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:23.450941    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 18 16:43:23.452091 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:23.452075    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-xp2ct\""
Sep 18 16:43:23.452176 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:23.452149    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 18 16:43:23.452176 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:23.452156    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-k9vpl\""
Sep 18 16:43:23.452339 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:23.452319    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 18 16:43:23.459211 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:23.459188    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd"]
Sep 18 16:43:23.479211 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:23.479190    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl4zc\" (UniqueName: \"kubernetes.io/projected/a549ce2e-37d6-496b-bca4-6856bcfcfd78-kube-api-access-dl4zc\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd\" (UID: \"a549ce2e-37d6-496b-bca4-6856bcfcfd78\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd"
Sep 18 16:43:23.479331 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:23.479309    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a549ce2e-37d6-496b-bca4-6856bcfcfd78-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd\" (UID: \"a549ce2e-37d6-496b-bca4-6856bcfcfd78\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd"
Sep 18 16:43:23.580431 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:23.580393    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a549ce2e-37d6-496b-bca4-6856bcfcfd78-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd\" (UID: \"a549ce2e-37d6-496b-bca4-6856bcfcfd78\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd"
Sep 18 16:43:23.580585 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:23.580453    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl4zc\" (UniqueName: \"kubernetes.io/projected/a549ce2e-37d6-496b-bca4-6856bcfcfd78-kube-api-access-dl4zc\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd\" (UID: \"a549ce2e-37d6-496b-bca4-6856bcfcfd78\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd"
Sep 18 16:43:23.583100 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:23.583078    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a549ce2e-37d6-496b-bca4-6856bcfcfd78-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd\" (UID: \"a549ce2e-37d6-496b-bca4-6856bcfcfd78\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd"
Sep 18 16:43:23.589387 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:23.589368    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl4zc\" (UniqueName: \"kubernetes.io/projected/a549ce2e-37d6-496b-bca4-6856bcfcfd78-kube-api-access-dl4zc\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd\" (UID: \"a549ce2e-37d6-496b-bca4-6856bcfcfd78\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd"
Sep 18 16:43:23.758528 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:23.758494    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd"
Sep 18 16:43:24.028287 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:24.028215    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd"]
Sep 18 16:43:24.031300 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:43:24.031275    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda549ce2e_37d6_496b_bca4_6856bcfcfd78.slice/crio-00f75250260c2434b10e30650d782b308ac48357299a0356d071792040f689a1 WatchSource:0}: Error finding container 00f75250260c2434b10e30650d782b308ac48357299a0356d071792040f689a1: Status 404 returned error can't find the container with id 00f75250260c2434b10e30650d782b308ac48357299a0356d071792040f689a1
Sep 18 16:43:24.543074 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:24.543041    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd" event={"ID":"a549ce2e-37d6-496b-bca4-6856bcfcfd78","Type":"ContainerStarted","Data":"e0e30a4de4e7fee3dd29a025b3928be37d128202023ce968b7c5fcfe686d1348"}
Sep 18 16:43:24.543536 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:24.543101    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd" event={"ID":"a549ce2e-37d6-496b-bca4-6856bcfcfd78","Type":"ContainerStarted","Data":"00f75250260c2434b10e30650d782b308ac48357299a0356d071792040f689a1"}
Sep 18 16:43:24.543536 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:24.543137    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd"
Sep 18 16:43:24.561668 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:24.561618    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd" podStartSLOduration=1.5616054799999999 podStartE2EDuration="1.56160548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:43:24.559777837 +0000 UTC m=+1644.342779863" watchObservedRunningTime="2026-09-18 16:43:24.56160548 +0000 UTC m=+1644.344607504"
Sep 18 16:43:37.340020 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:37.339980    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"]
Sep 18 16:43:37.350296 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:43:37.350263    2585 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 16:43:37.350296 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:43:37.350296 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 16:43:37.350296 ip-10-0-133-147 kubenswrapper[2585]:  > containerID="c3dec983408452d7666cf171b8fa187bd716b531e7ad7717cad0bc2f7e8c6d6b" cmd=["/bin/sleep","15"]
Sep 18 16:43:37.350539 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:37.350314    2585 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6" message="" err=<
Sep 18 16:43:37.350539 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:43:37.350539 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 16:43:37.350539 ip-10-0-133-147 kubenswrapper[2585]:  >
Sep 18 16:43:37.350539 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:43:37.350328    2585 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 16:43:37.350539 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:43:37.350539 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 16:43:37.350539 ip-10-0-133-147 kubenswrapper[2585]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6" podUID="486940b1-cab9-4d61-b7bb-3f5f110561e6" containerName="main" containerID="cri-o://c3dec983408452d7666cf171b8fa187bd716b531e7ad7717cad0bc2f7e8c6d6b"
Sep 18 16:43:37.350539 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:37.350365    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6" podUID="486940b1-cab9-4d61-b7bb-3f5f110561e6" containerName="main" containerID="cri-o://c3dec983408452d7666cf171b8fa187bd716b531e7ad7717cad0bc2f7e8c6d6b" gracePeriod=60
Sep 18 16:43:37.501190 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:37.501153    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6" podUID="486940b1-cab9-4d61-b7bb-3f5f110561e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Sep 18 16:43:37.598508 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:37.598435    2585 generic.go:356] "Generic (PLEG): container finished" podID="486940b1-cab9-4d61-b7bb-3f5f110561e6" containerID="c3dec983408452d7666cf171b8fa187bd716b531e7ad7717cad0bc2f7e8c6d6b" exitCode=0
Sep 18 16:43:37.598625 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:37.598507    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6" event={"ID":"486940b1-cab9-4d61-b7bb-3f5f110561e6","Type":"ContainerDied","Data":"c3dec983408452d7666cf171b8fa187bd716b531e7ad7717cad0bc2f7e8c6d6b"}
Sep 18 16:43:37.690452 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:37.690427    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"
Sep 18 16:43:37.794801 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:37.794773    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/486940b1-cab9-4d61-b7bb-3f5f110561e6-dshm\") pod \"486940b1-cab9-4d61-b7bb-3f5f110561e6\" (UID: \"486940b1-cab9-4d61-b7bb-3f5f110561e6\") "
Sep 18 16:43:37.794987 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:37.794833    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/486940b1-cab9-4d61-b7bb-3f5f110561e6-home\") pod \"486940b1-cab9-4d61-b7bb-3f5f110561e6\" (UID: \"486940b1-cab9-4d61-b7bb-3f5f110561e6\") "
Sep 18 16:43:37.794987 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:37.794857    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/486940b1-cab9-4d61-b7bb-3f5f110561e6-tmp-dir\") pod \"486940b1-cab9-4d61-b7bb-3f5f110561e6\" (UID: \"486940b1-cab9-4d61-b7bb-3f5f110561e6\") "
Sep 18 16:43:37.794987 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:37.794885    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/486940b1-cab9-4d61-b7bb-3f5f110561e6-model-cache\") pod \"486940b1-cab9-4d61-b7bb-3f5f110561e6\" (UID: \"486940b1-cab9-4d61-b7bb-3f5f110561e6\") "
Sep 18 16:43:37.794987 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:37.794907    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/486940b1-cab9-4d61-b7bb-3f5f110561e6-tls-certs\") pod \"486940b1-cab9-4d61-b7bb-3f5f110561e6\" (UID: \"486940b1-cab9-4d61-b7bb-3f5f110561e6\") "
Sep 18 16:43:37.794987 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:37.794953    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzr4p\" (UniqueName: \"kubernetes.io/projected/486940b1-cab9-4d61-b7bb-3f5f110561e6-kube-api-access-wzr4p\") pod \"486940b1-cab9-4d61-b7bb-3f5f110561e6\" (UID: \"486940b1-cab9-4d61-b7bb-3f5f110561e6\") "
Sep 18 16:43:37.795267 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:37.795072    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486940b1-cab9-4d61-b7bb-3f5f110561e6-home" (OuterVolumeSpecName: "home") pod "486940b1-cab9-4d61-b7bb-3f5f110561e6" (UID: "486940b1-cab9-4d61-b7bb-3f5f110561e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:43:37.795267 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:37.795166    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/486940b1-cab9-4d61-b7bb-3f5f110561e6-home\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:43:37.795267 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:37.795186    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486940b1-cab9-4d61-b7bb-3f5f110561e6-model-cache" (OuterVolumeSpecName: "model-cache") pod "486940b1-cab9-4d61-b7bb-3f5f110561e6" (UID: "486940b1-cab9-4d61-b7bb-3f5f110561e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:43:37.795267 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:37.795202    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486940b1-cab9-4d61-b7bb-3f5f110561e6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "486940b1-cab9-4d61-b7bb-3f5f110561e6" (UID: "486940b1-cab9-4d61-b7bb-3f5f110561e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:43:37.797269 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:37.797236    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/486940b1-cab9-4d61-b7bb-3f5f110561e6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "486940b1-cab9-4d61-b7bb-3f5f110561e6" (UID: "486940b1-cab9-4d61-b7bb-3f5f110561e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:43:37.797351 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:37.797274    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486940b1-cab9-4d61-b7bb-3f5f110561e6-dshm" (OuterVolumeSpecName: "dshm") pod "486940b1-cab9-4d61-b7bb-3f5f110561e6" (UID: "486940b1-cab9-4d61-b7bb-3f5f110561e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:43:37.797351 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:37.797283    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486940b1-cab9-4d61-b7bb-3f5f110561e6-kube-api-access-wzr4p" (OuterVolumeSpecName: "kube-api-access-wzr4p") pod "486940b1-cab9-4d61-b7bb-3f5f110561e6" (UID: "486940b1-cab9-4d61-b7bb-3f5f110561e6"). InnerVolumeSpecName "kube-api-access-wzr4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:43:37.895762 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:37.895694    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzr4p\" (UniqueName: \"kubernetes.io/projected/486940b1-cab9-4d61-b7bb-3f5f110561e6-kube-api-access-wzr4p\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:43:37.895762 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:37.895719    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/486940b1-cab9-4d61-b7bb-3f5f110561e6-dshm\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:43:37.895762 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:37.895732    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/486940b1-cab9-4d61-b7bb-3f5f110561e6-tmp-dir\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:43:37.895762 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:37.895743    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/486940b1-cab9-4d61-b7bb-3f5f110561e6-model-cache\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:43:37.895762 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:37.895754    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/486940b1-cab9-4d61-b7bb-3f5f110561e6-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:43:38.603952 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:38.603888    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6" event={"ID":"486940b1-cab9-4d61-b7bb-3f5f110561e6","Type":"ContainerDied","Data":"7f7057c54ebd4ef2da5bdcf535107dea5ccb4c1c4735560118b064f6612e9860"}
Sep 18 16:43:38.604364 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:38.603985    2585 scope.go:117] "RemoveContainer" containerID="c3dec983408452d7666cf171b8fa187bd716b531e7ad7717cad0bc2f7e8c6d6b"
Sep 18 16:43:38.604364 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:38.603903    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"
Sep 18 16:43:38.621019 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:38.620996    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"]
Sep 18 16:43:38.626662 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:38.626642    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-n77g6"]
Sep 18 16:43:40.395935 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:40.393811    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="486940b1-cab9-4d61-b7bb-3f5f110561e6" path="/var/lib/kubelet/pods/486940b1-cab9-4d61-b7bb-3f5f110561e6/volumes"
Sep 18 16:43:55.548826 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:43:55.548796    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd"
Sep 18 16:44:02.784360 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:44:02.783643    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd"]
Sep 18 16:44:05.548275 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:44:05.548233    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd" podUID="a549ce2e-37d6-496b-bca4-6856bcfcfd78" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:44:15.549044 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:44:15.548963    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd" podUID="a549ce2e-37d6-496b-bca4-6856bcfcfd78" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:44:17.815881 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:44:17.815823    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd" podUID="a549ce2e-37d6-496b-bca4-6856bcfcfd78" containerName="main" containerID="cri-o://e0e30a4de4e7fee3dd29a025b3928be37d128202023ce968b7c5fcfe686d1348" gracePeriod=45
Sep 18 16:44:25.548465 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:44:25.548427    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd" podUID="a549ce2e-37d6-496b-bca4-6856bcfcfd78" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:44:25.548869 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:44:25.548533    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd"
Sep 18 16:44:35.548636 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:44:35.548593    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd" podUID="a549ce2e-37d6-496b-bca4-6856bcfcfd78" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:44:45.548508 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:44:45.548462    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd" podUID="a549ce2e-37d6-496b-bca4-6856bcfcfd78" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:44:48.062254 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:44:48.062231    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd"
Sep 18 16:44:48.107423 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:44:48.107358    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a549ce2e-37d6-496b-bca4-6856bcfcfd78-tls-certs\") pod \"a549ce2e-37d6-496b-bca4-6856bcfcfd78\" (UID: \"a549ce2e-37d6-496b-bca4-6856bcfcfd78\") "
Sep 18 16:44:48.107423 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:44:48.107391    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dl4zc\" (UniqueName: \"kubernetes.io/projected/a549ce2e-37d6-496b-bca4-6856bcfcfd78-kube-api-access-dl4zc\") pod \"a549ce2e-37d6-496b-bca4-6856bcfcfd78\" (UID: \"a549ce2e-37d6-496b-bca4-6856bcfcfd78\") "
Sep 18 16:44:48.109753 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:44:48.109723    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a549ce2e-37d6-496b-bca4-6856bcfcfd78-kube-api-access-dl4zc" (OuterVolumeSpecName: "kube-api-access-dl4zc") pod "a549ce2e-37d6-496b-bca4-6856bcfcfd78" (UID: "a549ce2e-37d6-496b-bca4-6856bcfcfd78"). InnerVolumeSpecName "kube-api-access-dl4zc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:44:48.109856 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:44:48.109769    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a549ce2e-37d6-496b-bca4-6856bcfcfd78-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a549ce2e-37d6-496b-bca4-6856bcfcfd78" (UID: "a549ce2e-37d6-496b-bca4-6856bcfcfd78"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:44:48.208299 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:44:48.208264    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a549ce2e-37d6-496b-bca4-6856bcfcfd78-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:44:48.208425 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:44:48.208306    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl4zc\" (UniqueName: \"kubernetes.io/projected/a549ce2e-37d6-496b-bca4-6856bcfcfd78-kube-api-access-dl4zc\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:44:48.865111 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:44:48.865074    2585 generic.go:356] "Generic (PLEG): container finished" podID="a549ce2e-37d6-496b-bca4-6856bcfcfd78" containerID="e0e30a4de4e7fee3dd29a025b3928be37d128202023ce968b7c5fcfe686d1348" exitCode=0
Sep 18 16:44:48.865278 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:44:48.865119    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd"
Sep 18 16:44:48.865278 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:44:48.865138    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd" event={"ID":"a549ce2e-37d6-496b-bca4-6856bcfcfd78","Type":"ContainerDied","Data":"e0e30a4de4e7fee3dd29a025b3928be37d128202023ce968b7c5fcfe686d1348"}
Sep 18 16:44:48.865278 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:44:48.865180    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd" event={"ID":"a549ce2e-37d6-496b-bca4-6856bcfcfd78","Type":"ContainerDied","Data":"00f75250260c2434b10e30650d782b308ac48357299a0356d071792040f689a1"}
Sep 18 16:44:48.865278 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:44:48.865200    2585 scope.go:117] "RemoveContainer" containerID="e0e30a4de4e7fee3dd29a025b3928be37d128202023ce968b7c5fcfe686d1348"
Sep 18 16:44:48.873333 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:44:48.873317    2585 scope.go:117] "RemoveContainer" containerID="e0e30a4de4e7fee3dd29a025b3928be37d128202023ce968b7c5fcfe686d1348"
Sep 18 16:44:48.873603 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:44:48.873587    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0e30a4de4e7fee3dd29a025b3928be37d128202023ce968b7c5fcfe686d1348\": container with ID starting with e0e30a4de4e7fee3dd29a025b3928be37d128202023ce968b7c5fcfe686d1348 not found: ID does not exist" containerID="e0e30a4de4e7fee3dd29a025b3928be37d128202023ce968b7c5fcfe686d1348"
Sep 18 16:44:48.873644 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:44:48.873610    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0e30a4de4e7fee3dd29a025b3928be37d128202023ce968b7c5fcfe686d1348"} err="failed to get container status \"e0e30a4de4e7fee3dd29a025b3928be37d128202023ce968b7c5fcfe686d1348\": rpc error: code = NotFound desc = could not find container \"e0e30a4de4e7fee3dd29a025b3928be37d128202023ce968b7c5fcfe686d1348\": container with ID starting with e0e30a4de4e7fee3dd29a025b3928be37d128202023ce968b7c5fcfe686d1348 not found: ID does not exist"
Sep 18 16:44:48.881527 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:44:48.881501    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd"]
Sep 18 16:44:48.884987 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:44:48.884965    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-bq8zd"]
Sep 18 16:44:50.391365 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:44:50.391338    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a549ce2e-37d6-496b-bca4-6856bcfcfd78" path="/var/lib/kubelet/pods/a549ce2e-37d6-496b-bca4-6856bcfcfd78/volumes"
Sep 18 16:45:09.474812 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.474779    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"]
Sep 18 16:45:09.475181 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.475079    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="486940b1-cab9-4d61-b7bb-3f5f110561e6" containerName="main"
Sep 18 16:45:09.475181 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.475094    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="486940b1-cab9-4d61-b7bb-3f5f110561e6" containerName="main"
Sep 18 16:45:09.475181 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.475132    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a549ce2e-37d6-496b-bca4-6856bcfcfd78" containerName="main"
Sep 18 16:45:09.475181 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.475138    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a549ce2e-37d6-496b-bca4-6856bcfcfd78" containerName="main"
Sep 18 16:45:09.475313 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.475206    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a549ce2e-37d6-496b-bca4-6856bcfcfd78" containerName="main"
Sep 18 16:45:09.475313 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.475214    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="486940b1-cab9-4d61-b7bb-3f5f110561e6" containerName="main"
Sep 18 16:45:09.479174 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.479158    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"
Sep 18 16:45:09.483119 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.483084    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 18 16:45:09.483254 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.483086    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 18 16:45:09.483254 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.483127    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 18 16:45:09.483254 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.483137    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-jhc6z\""
Sep 18 16:45:09.487205 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.487184    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"]
Sep 18 16:45:09.562753 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.562724    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-d6wgq\" (UID: \"b0de8954-d625-44b2-8bdd-d7942fe0bc4b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"
Sep 18 16:45:09.562884 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.562755    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-d6wgq\" (UID: \"b0de8954-d625-44b2-8bdd-d7942fe0bc4b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"
Sep 18 16:45:09.562884 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.562773    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-home\") pod \"fc-concurrency-test-kserve-b5d44467c-d6wgq\" (UID: \"b0de8954-d625-44b2-8bdd-d7942fe0bc4b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"
Sep 18 16:45:09.562884 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.562801    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-d6wgq\" (UID: \"b0de8954-d625-44b2-8bdd-d7942fe0bc4b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"
Sep 18 16:45:09.562884 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.562842    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-d6wgq\" (UID: \"b0de8954-d625-44b2-8bdd-d7942fe0bc4b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"
Sep 18 16:45:09.562884 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.562867    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc7k4\" (UniqueName: \"kubernetes.io/projected/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-kube-api-access-dc7k4\") pod \"fc-concurrency-test-kserve-b5d44467c-d6wgq\" (UID: \"b0de8954-d625-44b2-8bdd-d7942fe0bc4b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"
Sep 18 16:45:09.664067 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.664035    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-d6wgq\" (UID: \"b0de8954-d625-44b2-8bdd-d7942fe0bc4b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"
Sep 18 16:45:09.664334 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.664069    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-d6wgq\" (UID: \"b0de8954-d625-44b2-8bdd-d7942fe0bc4b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"
Sep 18 16:45:09.664334 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.664100    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc7k4\" (UniqueName: \"kubernetes.io/projected/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-kube-api-access-dc7k4\") pod \"fc-concurrency-test-kserve-b5d44467c-d6wgq\" (UID: \"b0de8954-d625-44b2-8bdd-d7942fe0bc4b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"
Sep 18 16:45:09.664334 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.664151    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-d6wgq\" (UID: \"b0de8954-d625-44b2-8bdd-d7942fe0bc4b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"
Sep 18 16:45:09.664334 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.664171    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-d6wgq\" (UID: \"b0de8954-d625-44b2-8bdd-d7942fe0bc4b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"
Sep 18 16:45:09.664334 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.664191    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-home\") pod \"fc-concurrency-test-kserve-b5d44467c-d6wgq\" (UID: \"b0de8954-d625-44b2-8bdd-d7942fe0bc4b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"
Sep 18 16:45:09.664654 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.664630    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-home\") pod \"fc-concurrency-test-kserve-b5d44467c-d6wgq\" (UID: \"b0de8954-d625-44b2-8bdd-d7942fe0bc4b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"
Sep 18 16:45:09.664840 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.664669    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-d6wgq\" (UID: \"b0de8954-d625-44b2-8bdd-d7942fe0bc4b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"
Sep 18 16:45:09.665146 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.665123    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-d6wgq\" (UID: \"b0de8954-d625-44b2-8bdd-d7942fe0bc4b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"
Sep 18 16:45:09.666603 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.666587    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-d6wgq\" (UID: \"b0de8954-d625-44b2-8bdd-d7942fe0bc4b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"
Sep 18 16:45:09.666785 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.666769    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-d6wgq\" (UID: \"b0de8954-d625-44b2-8bdd-d7942fe0bc4b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"
Sep 18 16:45:09.672658 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.672636    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc7k4\" (UniqueName: \"kubernetes.io/projected/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-kube-api-access-dc7k4\") pod \"fc-concurrency-test-kserve-b5d44467c-d6wgq\" (UID: \"b0de8954-d625-44b2-8bdd-d7942fe0bc4b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"
Sep 18 16:45:09.789866 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.789840    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"
Sep 18 16:45:09.820945 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.820898    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz"]
Sep 18 16:45:09.824780 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.824757    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz"
Sep 18 16:45:09.827526 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.827507    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-jvn2b\""
Sep 18 16:45:09.836336 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.836311    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz"]
Sep 18 16:45:09.866798 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.866630    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4nk6\" (UniqueName: \"kubernetes.io/projected/03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb-kube-api-access-z4nk6\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz\" (UID: \"03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz"
Sep 18 16:45:09.866798 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.866724    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz\" (UID: \"03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz"
Sep 18 16:45:09.967248 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.967214    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4nk6\" (UniqueName: \"kubernetes.io/projected/03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb-kube-api-access-z4nk6\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz\" (UID: \"03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz"
Sep 18 16:45:09.967395 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.967289    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz\" (UID: \"03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz"
Sep 18 16:45:09.970031 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.970007    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz\" (UID: \"03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz"
Sep 18 16:45:09.981030 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:09.981008    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4nk6\" (UniqueName: \"kubernetes.io/projected/03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb-kube-api-access-z4nk6\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz\" (UID: \"03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz"
Sep 18 16:45:10.086174 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:10.086114    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"]
Sep 18 16:45:10.088965 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:45:10.088940    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0de8954_d625_44b2_8bdd_d7942fe0bc4b.slice/crio-0fe6f6e9cbe8aa4e7e52c33ebd084860393586266d76c0d6087063e889a4f478 WatchSource:0}: Error finding container 0fe6f6e9cbe8aa4e7e52c33ebd084860393586266d76c0d6087063e889a4f478: Status 404 returned error can't find the container with id 0fe6f6e9cbe8aa4e7e52c33ebd084860393586266d76c0d6087063e889a4f478
Sep 18 16:45:10.136625 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:10.136601    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz"
Sep 18 16:45:10.414826 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:10.414747    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz"]
Sep 18 16:45:10.420278 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:45:10.420247    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03bcfc1b_b7ea_4bec_ac06_ee2e8d9ecbbb.slice/crio-d6b75ca2e47c68dedb97887797f5efb9d775a9d017eca525793647ed38f98a6d WatchSource:0}: Error finding container d6b75ca2e47c68dedb97887797f5efb9d775a9d017eca525793647ed38f98a6d: Status 404 returned error can't find the container with id d6b75ca2e47c68dedb97887797f5efb9d775a9d017eca525793647ed38f98a6d
Sep 18 16:45:10.955411 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:10.954626    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz" event={"ID":"03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb","Type":"ContainerStarted","Data":"7585f41809ac3442efe735b4c4c117650e2245d4f8146b176732b102895d7588"}
Sep 18 16:45:10.955411 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:10.954732    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz" event={"ID":"03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb","Type":"ContainerStarted","Data":"d6b75ca2e47c68dedb97887797f5efb9d775a9d017eca525793647ed38f98a6d"}
Sep 18 16:45:10.955411 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:10.955344    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz"
Sep 18 16:45:10.957602 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:10.957573    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq" event={"ID":"b0de8954-d625-44b2-8bdd-d7942fe0bc4b","Type":"ContainerStarted","Data":"4250b89c1c20c04ceecb4fb549369ba51db749a88c5966543ea79978ebceb9ad"}
Sep 18 16:45:10.957733 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:10.957615    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq" event={"ID":"b0de8954-d625-44b2-8bdd-d7942fe0bc4b","Type":"ContainerStarted","Data":"0fe6f6e9cbe8aa4e7e52c33ebd084860393586266d76c0d6087063e889a4f478"}
Sep 18 16:45:10.978799 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:10.978757    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz" podStartSLOduration=1.978744978 podStartE2EDuration="1.978744978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:45:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:45:10.977479899 +0000 UTC m=+1750.760481925" watchObservedRunningTime="2026-09-18 16:45:10.978744978 +0000 UTC m=+1750.761747001"
Sep 18 16:45:10.996559 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:10.996504    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq" podStartSLOduration=1.9964866570000002 podStartE2EDuration="1.996486657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:45:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:45:10.995848601 +0000 UTC m=+1750.778850646" watchObservedRunningTime="2026-09-18 16:45:10.996486657 +0000 UTC m=+1750.779488682"
Sep 18 16:45:11.790010 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:11.789974    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"
Sep 18 16:45:19.790331 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:19.790295    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"
Sep 18 16:45:19.802987 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:19.802960    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"
Sep 18 16:45:20.004197 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:20.004167    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"
Sep 18 16:45:41.964196 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:41.964126    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz"
Sep 18 16:45:48.698004 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:48.697969    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz"]
Sep 18 16:45:48.706355 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:48.706329    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"]
Sep 18 16:45:48.714533 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:45:48.714495    2585 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 16:45:48.714533 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:45:48.714533 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 16:45:48.714533 ip-10-0-133-147 kubenswrapper[2585]:  > containerID="4250b89c1c20c04ceecb4fb549369ba51db749a88c5966543ea79978ebceb9ad" cmd=["/bin/sleep","15"]
Sep 18 16:45:48.714832 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:48.714560    2585 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq" message="" err=<
Sep 18 16:45:48.714832 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:45:48.714832 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 16:45:48.714832 ip-10-0-133-147 kubenswrapper[2585]:  >
Sep 18 16:45:48.714832 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:45:48.714645    2585 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 16:45:48.714832 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:45:48.714832 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 16:45:48.714832 ip-10-0-133-147 kubenswrapper[2585]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq" podUID="b0de8954-d625-44b2-8bdd-d7942fe0bc4b" containerName="main" containerID="cri-o://4250b89c1c20c04ceecb4fb549369ba51db749a88c5966543ea79978ebceb9ad"
Sep 18 16:45:48.714832 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:48.714683    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq" podUID="b0de8954-d625-44b2-8bdd-d7942fe0bc4b" containerName="main" containerID="cri-o://4250b89c1c20c04ceecb4fb549369ba51db749a88c5966543ea79978ebceb9ad" gracePeriod=60
Sep 18 16:45:49.063179 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:49.063154    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"
Sep 18 16:45:49.072620 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:49.072592    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-model-cache\") pod \"b0de8954-d625-44b2-8bdd-d7942fe0bc4b\" (UID: \"b0de8954-d625-44b2-8bdd-d7942fe0bc4b\") "
Sep 18 16:45:49.072734 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:49.072639    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-dshm\") pod \"b0de8954-d625-44b2-8bdd-d7942fe0bc4b\" (UID: \"b0de8954-d625-44b2-8bdd-d7942fe0bc4b\") "
Sep 18 16:45:49.072734 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:49.072683    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-tmp-dir\") pod \"b0de8954-d625-44b2-8bdd-d7942fe0bc4b\" (UID: \"b0de8954-d625-44b2-8bdd-d7942fe0bc4b\") "
Sep 18 16:45:49.072734 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:49.072708    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-home\") pod \"b0de8954-d625-44b2-8bdd-d7942fe0bc4b\" (UID: \"b0de8954-d625-44b2-8bdd-d7942fe0bc4b\") "
Sep 18 16:45:49.072902 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:49.072748    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-tls-certs\") pod \"b0de8954-d625-44b2-8bdd-d7942fe0bc4b\" (UID: \"b0de8954-d625-44b2-8bdd-d7942fe0bc4b\") "
Sep 18 16:45:49.072902 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:49.072786    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dc7k4\" (UniqueName: \"kubernetes.io/projected/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-kube-api-access-dc7k4\") pod \"b0de8954-d625-44b2-8bdd-d7942fe0bc4b\" (UID: \"b0de8954-d625-44b2-8bdd-d7942fe0bc4b\") "
Sep 18 16:45:49.072902 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:49.072829    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-model-cache" (OuterVolumeSpecName: "model-cache") pod "b0de8954-d625-44b2-8bdd-d7942fe0bc4b" (UID: "b0de8954-d625-44b2-8bdd-d7942fe0bc4b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:45:49.073068 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:49.072953    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b0de8954-d625-44b2-8bdd-d7942fe0bc4b" (UID: "b0de8954-d625-44b2-8bdd-d7942fe0bc4b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:45:49.073068 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:49.073002    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-home" (OuterVolumeSpecName: "home") pod "b0de8954-d625-44b2-8bdd-d7942fe0bc4b" (UID: "b0de8954-d625-44b2-8bdd-d7942fe0bc4b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:45:49.073068 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:49.073062    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-model-cache\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:45:49.073259 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:49.073078    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-tmp-dir\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:45:49.075142 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:49.075112    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b0de8954-d625-44b2-8bdd-d7942fe0bc4b" (UID: "b0de8954-d625-44b2-8bdd-d7942fe0bc4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:45:49.075244 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:49.075175    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-kube-api-access-dc7k4" (OuterVolumeSpecName: "kube-api-access-dc7k4") pod "b0de8954-d625-44b2-8bdd-d7942fe0bc4b" (UID: "b0de8954-d625-44b2-8bdd-d7942fe0bc4b"). InnerVolumeSpecName "kube-api-access-dc7k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:45:49.075244 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:49.075186    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-dshm" (OuterVolumeSpecName: "dshm") pod "b0de8954-d625-44b2-8bdd-d7942fe0bc4b" (UID: "b0de8954-d625-44b2-8bdd-d7942fe0bc4b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:45:49.110663 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:49.110619    2585 generic.go:356] "Generic (PLEG): container finished" podID="b0de8954-d625-44b2-8bdd-d7942fe0bc4b" containerID="4250b89c1c20c04ceecb4fb549369ba51db749a88c5966543ea79978ebceb9ad" exitCode=0
Sep 18 16:45:49.110846 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:49.110668    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq" event={"ID":"b0de8954-d625-44b2-8bdd-d7942fe0bc4b","Type":"ContainerDied","Data":"4250b89c1c20c04ceecb4fb549369ba51db749a88c5966543ea79978ebceb9ad"}
Sep 18 16:45:49.110846 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:49.110684    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"
Sep 18 16:45:49.110846 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:49.110724    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq" event={"ID":"b0de8954-d625-44b2-8bdd-d7942fe0bc4b","Type":"ContainerDied","Data":"0fe6f6e9cbe8aa4e7e52c33ebd084860393586266d76c0d6087063e889a4f478"}
Sep 18 16:45:49.110846 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:49.110780    2585 scope.go:117] "RemoveContainer" containerID="4250b89c1c20c04ceecb4fb549369ba51db749a88c5966543ea79978ebceb9ad"
Sep 18 16:45:49.121398 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:49.121379    2585 scope.go:117] "RemoveContainer" containerID="4250b89c1c20c04ceecb4fb549369ba51db749a88c5966543ea79978ebceb9ad"
Sep 18 16:45:49.121687 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:45:49.121668    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4250b89c1c20c04ceecb4fb549369ba51db749a88c5966543ea79978ebceb9ad\": container with ID starting with 4250b89c1c20c04ceecb4fb549369ba51db749a88c5966543ea79978ebceb9ad not found: ID does not exist" containerID="4250b89c1c20c04ceecb4fb549369ba51db749a88c5966543ea79978ebceb9ad"
Sep 18 16:45:49.121744 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:49.121693    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4250b89c1c20c04ceecb4fb549369ba51db749a88c5966543ea79978ebceb9ad"} err="failed to get container status \"4250b89c1c20c04ceecb4fb549369ba51db749a88c5966543ea79978ebceb9ad\": rpc error: code = NotFound desc = could not find container \"4250b89c1c20c04ceecb4fb549369ba51db749a88c5966543ea79978ebceb9ad\": container with ID starting with 4250b89c1c20c04ceecb4fb549369ba51db749a88c5966543ea79978ebceb9ad not found: ID does not exist"
Sep 18 16:45:49.135959 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:49.135937    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"]
Sep 18 16:45:49.139935 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:49.139900    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq"]
Sep 18 16:45:49.173850 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:49.173829    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-dshm\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:45:49.173850 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:49.173848    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-home\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:45:49.173992 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:49.173857    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:45:49.173992 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:49.173865    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc7k4\" (UniqueName: \"kubernetes.io/projected/b0de8954-d625-44b2-8bdd-d7942fe0bc4b-kube-api-access-dc7k4\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:45:49.994319 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:49.994262    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-d6wgq" podUID="b0de8954-d625-44b2-8bdd-d7942fe0bc4b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": context deadline exceeded"
Sep 18 16:45:50.392884 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:50.392808    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0de8954-d625-44b2-8bdd-d7942fe0bc4b" path="/var/lib/kubelet/pods/b0de8954-d625-44b2-8bdd-d7942fe0bc4b/volumes"
Sep 18 16:45:51.963844 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:45:51.963809    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz" podUID="03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:46:01.963718 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:01.963680    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz" podUID="03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:46:02.415600 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:02.415570    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 16:46:02.417469 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:02.417446    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 16:46:02.419357 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:02.419338    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 16:46:02.421191 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:02.421170    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 16:46:03.720073 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:03.720029    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz" podUID="03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb" containerName="main" containerID="cri-o://7585f41809ac3442efe735b4c4c117650e2245d4f8146b176732b102895d7588" gracePeriod=45
Sep 18 16:46:11.963516 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:11.963478    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz" podUID="03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:46:11.963949 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:11.963596    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz"
Sep 18 16:46:21.963880 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:21.963839    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz" podUID="03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:46:31.963683 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:31.963644    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz" podUID="03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:46:33.962520 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:33.962492    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz"
Sep 18 16:46:34.135165 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:34.135124    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4nk6\" (UniqueName: \"kubernetes.io/projected/03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb-kube-api-access-z4nk6\") pod \"03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb\" (UID: \"03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb\") "
Sep 18 16:46:34.135374 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:34.135184    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb-tls-certs\") pod \"03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb\" (UID: \"03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb\") "
Sep 18 16:46:34.137428 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:34.137399    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb" (UID: "03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:46:34.137538 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:34.137425    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb-kube-api-access-z4nk6" (OuterVolumeSpecName: "kube-api-access-z4nk6") pod "03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb" (UID: "03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb"). InnerVolumeSpecName "kube-api-access-z4nk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:46:34.236793 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:34.236755    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4nk6\" (UniqueName: \"kubernetes.io/projected/03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb-kube-api-access-z4nk6\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:46:34.236793 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:34.236787    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:46:34.287286 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:34.287253    2585 generic.go:356] "Generic (PLEG): container finished" podID="03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb" containerID="7585f41809ac3442efe735b4c4c117650e2245d4f8146b176732b102895d7588" exitCode=0
Sep 18 16:46:34.287464 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:34.287301    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz"
Sep 18 16:46:34.287464 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:34.287334    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz" event={"ID":"03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb","Type":"ContainerDied","Data":"7585f41809ac3442efe735b4c4c117650e2245d4f8146b176732b102895d7588"}
Sep 18 16:46:34.287464 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:34.287387    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz" event={"ID":"03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb","Type":"ContainerDied","Data":"d6b75ca2e47c68dedb97887797f5efb9d775a9d017eca525793647ed38f98a6d"}
Sep 18 16:46:34.287464 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:34.287409    2585 scope.go:117] "RemoveContainer" containerID="7585f41809ac3442efe735b4c4c117650e2245d4f8146b176732b102895d7588"
Sep 18 16:46:34.296389 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:34.296374    2585 scope.go:117] "RemoveContainer" containerID="7585f41809ac3442efe735b4c4c117650e2245d4f8146b176732b102895d7588"
Sep 18 16:46:34.296659 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:46:34.296642    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7585f41809ac3442efe735b4c4c117650e2245d4f8146b176732b102895d7588\": container with ID starting with 7585f41809ac3442efe735b4c4c117650e2245d4f8146b176732b102895d7588 not found: ID does not exist" containerID="7585f41809ac3442efe735b4c4c117650e2245d4f8146b176732b102895d7588"
Sep 18 16:46:34.296701 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:34.296666    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7585f41809ac3442efe735b4c4c117650e2245d4f8146b176732b102895d7588"} err="failed to get container status \"7585f41809ac3442efe735b4c4c117650e2245d4f8146b176732b102895d7588\": rpc error: code = NotFound desc = could not find container \"7585f41809ac3442efe735b4c4c117650e2245d4f8146b176732b102895d7588\": container with ID starting with 7585f41809ac3442efe735b4c4c117650e2245d4f8146b176732b102895d7588 not found: ID does not exist"
Sep 18 16:46:34.309169 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:34.309142    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz"]
Sep 18 16:46:34.314482 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:34.314461    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-jphsz"]
Sep 18 16:46:34.391941 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:34.391838    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb" path="/var/lib/kubelet/pods/03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb/volumes"
Sep 18 16:46:39.016005 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.015970    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"]
Sep 18 16:46:39.016503 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.016263    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb" containerName="main"
Sep 18 16:46:39.016503 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.016277    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb" containerName="main"
Sep 18 16:46:39.016503 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.016289    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0de8954-d625-44b2-8bdd-d7942fe0bc4b" containerName="main"
Sep 18 16:46:39.016503 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.016299    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0de8954-d625-44b2-8bdd-d7942fe0bc4b" containerName="main"
Sep 18 16:46:39.016503 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.016360    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0de8954-d625-44b2-8bdd-d7942fe0bc4b" containerName="main"
Sep 18 16:46:39.016503 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.016378    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="03bcfc1b-b7ea-4bec-ac06-ee2e8d9ecbbb" containerName="main"
Sep 18 16:46:39.020860 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.020837    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:39.023904 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.023872    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-5jtkh\""
Sep 18 16:46:39.024034 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.023937    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 18 16:46:39.024103 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.024060    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 18 16:46:39.024163 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.024101    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 18 16:46:39.033830 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.033802    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"]
Sep 18 16:46:39.176935 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.176886    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-credential-socket\") pod \"router-gateway-1-openshift-default-6fd8d584bd-92zp2\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:39.177109 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.176959    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-istio-data\") pod \"router-gateway-1-openshift-default-6fd8d584bd-92zp2\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:39.177109 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.176989    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-workload-certs\") pod \"router-gateway-1-openshift-default-6fd8d584bd-92zp2\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:39.177109 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.177009    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-istio-envoy\") pod \"router-gateway-1-openshift-default-6fd8d584bd-92zp2\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:39.177109 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.177045    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxgxj\" (UniqueName: \"kubernetes.io/projected/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-kube-api-access-rxgxj\") pod \"router-gateway-1-openshift-default-6fd8d584bd-92zp2\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:39.177109 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.177069    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-workload-socket\") pod \"router-gateway-1-openshift-default-6fd8d584bd-92zp2\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:39.180932 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.177085    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-istio-podinfo\") pod \"router-gateway-1-openshift-default-6fd8d584bd-92zp2\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:39.180932 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.177591    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6fd8d584bd-92zp2\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:39.180932 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.177640    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-istio-token\") pod \"router-gateway-1-openshift-default-6fd8d584bd-92zp2\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:39.278826 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.278734    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6fd8d584bd-92zp2\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:39.278826 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.278778    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-istio-token\") pod \"router-gateway-1-openshift-default-6fd8d584bd-92zp2\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:39.278826 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.278799    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-credential-socket\") pod \"router-gateway-1-openshift-default-6fd8d584bd-92zp2\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:39.278826 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.278827    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-istio-data\") pod \"router-gateway-1-openshift-default-6fd8d584bd-92zp2\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:39.279197 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.278845    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-workload-certs\") pod \"router-gateway-1-openshift-default-6fd8d584bd-92zp2\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:39.279197 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.278864    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-istio-envoy\") pod \"router-gateway-1-openshift-default-6fd8d584bd-92zp2\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:39.279197 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.278983    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxgxj\" (UniqueName: \"kubernetes.io/projected/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-kube-api-access-rxgxj\") pod \"router-gateway-1-openshift-default-6fd8d584bd-92zp2\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:39.279197 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.279037    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-workload-socket\") pod \"router-gateway-1-openshift-default-6fd8d584bd-92zp2\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:39.279197 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.279065    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-istio-podinfo\") pod \"router-gateway-1-openshift-default-6fd8d584bd-92zp2\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:39.279471 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.279445    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-istio-data\") pod \"router-gateway-1-openshift-default-6fd8d584bd-92zp2\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:39.279529 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.279482    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-workload-certs\") pod \"router-gateway-1-openshift-default-6fd8d584bd-92zp2\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:39.279529 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.279509    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-credential-socket\") pod \"router-gateway-1-openshift-default-6fd8d584bd-92zp2\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:39.279610 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.279539    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-workload-socket\") pod \"router-gateway-1-openshift-default-6fd8d584bd-92zp2\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:39.279643 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.279628    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6fd8d584bd-92zp2\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:39.281431 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.281410    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-istio-envoy\") pod \"router-gateway-1-openshift-default-6fd8d584bd-92zp2\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:39.281614 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.281596    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-istio-podinfo\") pod \"router-gateway-1-openshift-default-6fd8d584bd-92zp2\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:39.287090 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.287065    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-istio-token\") pod \"router-gateway-1-openshift-default-6fd8d584bd-92zp2\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:39.287213 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.287193    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxgxj\" (UniqueName: \"kubernetes.io/projected/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-kube-api-access-rxgxj\") pod \"router-gateway-1-openshift-default-6fd8d584bd-92zp2\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:39.331605 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.331570    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:39.608361 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.608335    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"]
Sep 18 16:46:39.611046 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:46:39.611018    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe9d0bf9_a3ce_4102_83fe_9f9cfeb0f14b.slice/crio-eec3b3b89bc264ac72891a22e8d21462878cd03ce81155902bc3566df2eaa23b WatchSource:0}: Error finding container eec3b3b89bc264ac72891a22e8d21462878cd03ce81155902bc3566df2eaa23b: Status 404 returned error can't find the container with id eec3b3b89bc264ac72891a22e8d21462878cd03ce81155902bc3566df2eaa23b
Sep 18 16:46:39.613131 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.613099    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 16:46:39.613205 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.613167    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 16:46:39.613205 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:39.613196    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 16:46:40.313264 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:40.313227    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2" event={"ID":"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b","Type":"ContainerStarted","Data":"bf1e1e8cff8662c4e4726f77d91d47f865590bc9f35bbee90a5f7accd2417cd9"}
Sep 18 16:46:40.313639 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:40.313281    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2" event={"ID":"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b","Type":"ContainerStarted","Data":"eec3b3b89bc264ac72891a22e8d21462878cd03ce81155902bc3566df2eaa23b"}
Sep 18 16:46:40.331772 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:40.331742    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:40.333195 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:40.333169    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2" podUID="fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 18 16:46:40.335336 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:40.335287    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2" podStartSLOduration=2.335275526 podStartE2EDuration="2.335275526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:46:40.33386341 +0000 UTC m=+1840.116865450" watchObservedRunningTime="2026-09-18 16:46:40.335275526 +0000 UTC m=+1840.118277549"
Sep 18 16:46:41.336322 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:41.336293    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:42.321865 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:42.321825    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:42.322867 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:42.322848    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:46:55.982524 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:55.982489    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9"]
Sep 18 16:46:55.985802 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:55.985779    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9"
Sep 18 16:46:55.989593 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:55.989572    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-fl5jx\""
Sep 18 16:46:55.989715 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:55.989630    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Sep 18 16:46:55.995063 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:55.995041    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9"]
Sep 18 16:46:56.010783 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:56.010757    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a08cfbe-c4a2-49eb-887e-40177db95cf9-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9\" (UID: \"0a08cfbe-c4a2-49eb-887e-40177db95cf9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9"
Sep 18 16:46:56.010906 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:56.010811    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a08cfbe-c4a2-49eb-887e-40177db95cf9-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9\" (UID: \"0a08cfbe-c4a2-49eb-887e-40177db95cf9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9"
Sep 18 16:46:56.010906 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:56.010829    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a08cfbe-c4a2-49eb-887e-40177db95cf9-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9\" (UID: \"0a08cfbe-c4a2-49eb-887e-40177db95cf9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9"
Sep 18 16:46:56.011066 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:56.010908    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a08cfbe-c4a2-49eb-887e-40177db95cf9-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9\" (UID: \"0a08cfbe-c4a2-49eb-887e-40177db95cf9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9"
Sep 18 16:46:56.011066 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:56.010995    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a08cfbe-c4a2-49eb-887e-40177db95cf9-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9\" (UID: \"0a08cfbe-c4a2-49eb-887e-40177db95cf9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9"
Sep 18 16:46:56.011066 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:56.011023    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bhfw\" (UniqueName: \"kubernetes.io/projected/0a08cfbe-c4a2-49eb-887e-40177db95cf9-kube-api-access-5bhfw\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9\" (UID: \"0a08cfbe-c4a2-49eb-887e-40177db95cf9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9"
Sep 18 16:46:56.111623 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:56.111584    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a08cfbe-c4a2-49eb-887e-40177db95cf9-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9\" (UID: \"0a08cfbe-c4a2-49eb-887e-40177db95cf9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9"
Sep 18 16:46:56.111821 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:56.111632    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a08cfbe-c4a2-49eb-887e-40177db95cf9-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9\" (UID: \"0a08cfbe-c4a2-49eb-887e-40177db95cf9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9"
Sep 18 16:46:56.111821 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:56.111650    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bhfw\" (UniqueName: \"kubernetes.io/projected/0a08cfbe-c4a2-49eb-887e-40177db95cf9-kube-api-access-5bhfw\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9\" (UID: \"0a08cfbe-c4a2-49eb-887e-40177db95cf9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9"
Sep 18 16:46:56.111821 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:56.111678    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a08cfbe-c4a2-49eb-887e-40177db95cf9-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9\" (UID: \"0a08cfbe-c4a2-49eb-887e-40177db95cf9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9"
Sep 18 16:46:56.111821 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:56.111718    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a08cfbe-c4a2-49eb-887e-40177db95cf9-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9\" (UID: \"0a08cfbe-c4a2-49eb-887e-40177db95cf9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9"
Sep 18 16:46:56.111821 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:56.111732    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a08cfbe-c4a2-49eb-887e-40177db95cf9-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9\" (UID: \"0a08cfbe-c4a2-49eb-887e-40177db95cf9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9"
Sep 18 16:46:56.112144 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:56.112096    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a08cfbe-c4a2-49eb-887e-40177db95cf9-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9\" (UID: \"0a08cfbe-c4a2-49eb-887e-40177db95cf9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9"
Sep 18 16:46:56.112198 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:56.112142    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a08cfbe-c4a2-49eb-887e-40177db95cf9-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9\" (UID: \"0a08cfbe-c4a2-49eb-887e-40177db95cf9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9"
Sep 18 16:46:56.112253 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:56.112234    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a08cfbe-c4a2-49eb-887e-40177db95cf9-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9\" (UID: \"0a08cfbe-c4a2-49eb-887e-40177db95cf9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9"
Sep 18 16:46:56.114229 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:56.114211    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a08cfbe-c4a2-49eb-887e-40177db95cf9-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9\" (UID: \"0a08cfbe-c4a2-49eb-887e-40177db95cf9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9"
Sep 18 16:46:56.114516 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:56.114495    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a08cfbe-c4a2-49eb-887e-40177db95cf9-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9\" (UID: \"0a08cfbe-c4a2-49eb-887e-40177db95cf9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9"
Sep 18 16:46:56.120260 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:56.120229    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bhfw\" (UniqueName: \"kubernetes.io/projected/0a08cfbe-c4a2-49eb-887e-40177db95cf9-kube-api-access-5bhfw\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9\" (UID: \"0a08cfbe-c4a2-49eb-887e-40177db95cf9\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9"
Sep 18 16:46:56.297632 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:56.297541    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9"
Sep 18 16:46:56.576059 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:56.575939    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9"]
Sep 18 16:46:56.579222 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:46:56.579195    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a08cfbe_c4a2_49eb_887e_40177db95cf9.slice/crio-e54e0eb0182bcbd1a3bf8a6a6fc55e59b3f2341e661f984fa713fe4eeaf3ad46 WatchSource:0}: Error finding container e54e0eb0182bcbd1a3bf8a6a6fc55e59b3f2341e661f984fa713fe4eeaf3ad46: Status 404 returned error can't find the container with id e54e0eb0182bcbd1a3bf8a6a6fc55e59b3f2341e661f984fa713fe4eeaf3ad46
Sep 18 16:46:57.381046 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:57.380999    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9" event={"ID":"0a08cfbe-c4a2-49eb-887e-40177db95cf9","Type":"ContainerStarted","Data":"fc0765f64d90bda82a0920c992accfc389987b743dffb4028bb9b160331f3f54"}
Sep 18 16:46:57.381416 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:57.381063    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9" event={"ID":"0a08cfbe-c4a2-49eb-887e-40177db95cf9","Type":"ContainerStarted","Data":"e54e0eb0182bcbd1a3bf8a6a6fc55e59b3f2341e661f984fa713fe4eeaf3ad46"}
Sep 18 16:46:57.400726 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:57.400678    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9" podStartSLOduration=2.400663812 podStartE2EDuration="2.400663812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:46:57.399228786 +0000 UTC m=+1857.182230811" watchObservedRunningTime="2026-09-18 16:46:57.400663812 +0000 UTC m=+1857.183665835"
Sep 18 16:46:58.297868 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:46:58.297841    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9"
Sep 18 16:47:01.403137 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:01.403099    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9"]
Sep 18 16:47:01.412000 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:47:01.411965    2585 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 16:47:01.412000 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:47:01.412000 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 16:47:01.412000 ip-10-0-133-147 kubenswrapper[2585]:  > containerID="fc0765f64d90bda82a0920c992accfc389987b743dffb4028bb9b160331f3f54" cmd=["/bin/sleep","15"]
Sep 18 16:47:01.412240 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:01.412010    2585 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9" message="" err=<
Sep 18 16:47:01.412240 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:47:01.412240 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 16:47:01.412240 ip-10-0-133-147 kubenswrapper[2585]:  >
Sep 18 16:47:01.412240 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:47:01.412022    2585 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 16:47:01.412240 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:47:01.412240 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 16:47:01.412240 ip-10-0-133-147 kubenswrapper[2585]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9" podUID="0a08cfbe-c4a2-49eb-887e-40177db95cf9" containerName="main" containerID="cri-o://fc0765f64d90bda82a0920c992accfc389987b743dffb4028bb9b160331f3f54"
Sep 18 16:47:01.412240 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:01.412056    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9" podUID="0a08cfbe-c4a2-49eb-887e-40177db95cf9" containerName="main" containerID="cri-o://fc0765f64d90bda82a0920c992accfc389987b743dffb4028bb9b160331f3f54" gracePeriod=60
Sep 18 16:47:01.689406 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:01.689378    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9"
Sep 18 16:47:01.764361 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:01.764326    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a08cfbe-c4a2-49eb-887e-40177db95cf9-tls-certs\") pod \"0a08cfbe-c4a2-49eb-887e-40177db95cf9\" (UID: \"0a08cfbe-c4a2-49eb-887e-40177db95cf9\") "
Sep 18 16:47:01.764530 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:01.764370    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bhfw\" (UniqueName: \"kubernetes.io/projected/0a08cfbe-c4a2-49eb-887e-40177db95cf9-kube-api-access-5bhfw\") pod \"0a08cfbe-c4a2-49eb-887e-40177db95cf9\" (UID: \"0a08cfbe-c4a2-49eb-887e-40177db95cf9\") "
Sep 18 16:47:01.764530 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:01.764400    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a08cfbe-c4a2-49eb-887e-40177db95cf9-tmp-dir\") pod \"0a08cfbe-c4a2-49eb-887e-40177db95cf9\" (UID: \"0a08cfbe-c4a2-49eb-887e-40177db95cf9\") "
Sep 18 16:47:01.764530 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:01.764421    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a08cfbe-c4a2-49eb-887e-40177db95cf9-model-cache\") pod \"0a08cfbe-c4a2-49eb-887e-40177db95cf9\" (UID: \"0a08cfbe-c4a2-49eb-887e-40177db95cf9\") "
Sep 18 16:47:01.764530 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:01.764469    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a08cfbe-c4a2-49eb-887e-40177db95cf9-home\") pod \"0a08cfbe-c4a2-49eb-887e-40177db95cf9\" (UID: \"0a08cfbe-c4a2-49eb-887e-40177db95cf9\") "
Sep 18 16:47:01.764530 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:01.764527    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a08cfbe-c4a2-49eb-887e-40177db95cf9-dshm\") pod \"0a08cfbe-c4a2-49eb-887e-40177db95cf9\" (UID: \"0a08cfbe-c4a2-49eb-887e-40177db95cf9\") "
Sep 18 16:47:01.764826 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:01.764692    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a08cfbe-c4a2-49eb-887e-40177db95cf9-model-cache" (OuterVolumeSpecName: "model-cache") pod "0a08cfbe-c4a2-49eb-887e-40177db95cf9" (UID: "0a08cfbe-c4a2-49eb-887e-40177db95cf9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:47:01.764826 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:01.764723    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a08cfbe-c4a2-49eb-887e-40177db95cf9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0a08cfbe-c4a2-49eb-887e-40177db95cf9" (UID: "0a08cfbe-c4a2-49eb-887e-40177db95cf9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:47:01.764826 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:01.764793    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a08cfbe-c4a2-49eb-887e-40177db95cf9-home" (OuterVolumeSpecName: "home") pod "0a08cfbe-c4a2-49eb-887e-40177db95cf9" (UID: "0a08cfbe-c4a2-49eb-887e-40177db95cf9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:47:01.765193 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:01.765161    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a08cfbe-c4a2-49eb-887e-40177db95cf9-tmp-dir\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:47:01.765193 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:01.765188    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a08cfbe-c4a2-49eb-887e-40177db95cf9-model-cache\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:47:01.765295 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:01.765203    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a08cfbe-c4a2-49eb-887e-40177db95cf9-home\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:47:01.766886 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:01.766861    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a08cfbe-c4a2-49eb-887e-40177db95cf9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0a08cfbe-c4a2-49eb-887e-40177db95cf9" (UID: "0a08cfbe-c4a2-49eb-887e-40177db95cf9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:47:01.766886 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:01.766876    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a08cfbe-c4a2-49eb-887e-40177db95cf9-kube-api-access-5bhfw" (OuterVolumeSpecName: "kube-api-access-5bhfw") pod "0a08cfbe-c4a2-49eb-887e-40177db95cf9" (UID: "0a08cfbe-c4a2-49eb-887e-40177db95cf9"). InnerVolumeSpecName "kube-api-access-5bhfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:47:01.767039 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:01.766929    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a08cfbe-c4a2-49eb-887e-40177db95cf9-dshm" (OuterVolumeSpecName: "dshm") pod "0a08cfbe-c4a2-49eb-887e-40177db95cf9" (UID: "0a08cfbe-c4a2-49eb-887e-40177db95cf9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:47:01.866312 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:01.866286    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a08cfbe-c4a2-49eb-887e-40177db95cf9-dshm\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:47:01.866312 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:01.866313    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a08cfbe-c4a2-49eb-887e-40177db95cf9-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:47:01.866488 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:01.866324    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bhfw\" (UniqueName: \"kubernetes.io/projected/0a08cfbe-c4a2-49eb-887e-40177db95cf9-kube-api-access-5bhfw\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:47:02.401566 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:02.401536    2585 generic.go:356] "Generic (PLEG): container finished" podID="0a08cfbe-c4a2-49eb-887e-40177db95cf9" containerID="fc0765f64d90bda82a0920c992accfc389987b743dffb4028bb9b160331f3f54" exitCode=0
Sep 18 16:47:02.401680 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:02.401581    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9"
Sep 18 16:47:02.401680 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:02.401597    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9" event={"ID":"0a08cfbe-c4a2-49eb-887e-40177db95cf9","Type":"ContainerDied","Data":"fc0765f64d90bda82a0920c992accfc389987b743dffb4028bb9b160331f3f54"}
Sep 18 16:47:02.401680 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:02.401639    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9" event={"ID":"0a08cfbe-c4a2-49eb-887e-40177db95cf9","Type":"ContainerDied","Data":"e54e0eb0182bcbd1a3bf8a6a6fc55e59b3f2341e661f984fa713fe4eeaf3ad46"}
Sep 18 16:47:02.401680 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:02.401661    2585 scope.go:117] "RemoveContainer" containerID="fc0765f64d90bda82a0920c992accfc389987b743dffb4028bb9b160331f3f54"
Sep 18 16:47:02.410739 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:02.410722    2585 scope.go:117] "RemoveContainer" containerID="fc0765f64d90bda82a0920c992accfc389987b743dffb4028bb9b160331f3f54"
Sep 18 16:47:02.411133 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:47:02.410992    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc0765f64d90bda82a0920c992accfc389987b743dffb4028bb9b160331f3f54\": container with ID starting with fc0765f64d90bda82a0920c992accfc389987b743dffb4028bb9b160331f3f54 not found: ID does not exist" containerID="fc0765f64d90bda82a0920c992accfc389987b743dffb4028bb9b160331f3f54"
Sep 18 16:47:02.411133 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:02.411016    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc0765f64d90bda82a0920c992accfc389987b743dffb4028bb9b160331f3f54"} err="failed to get container status \"fc0765f64d90bda82a0920c992accfc389987b743dffb4028bb9b160331f3f54\": rpc error: code = NotFound desc = could not find container \"fc0765f64d90bda82a0920c992accfc389987b743dffb4028bb9b160331f3f54\": container with ID starting with fc0765f64d90bda82a0920c992accfc389987b743dffb4028bb9b160331f3f54 not found: ID does not exist"
Sep 18 16:47:02.420057 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:02.420030    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9"]
Sep 18 16:47:02.422460 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:02.422439    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-hffr9"]
Sep 18 16:47:03.126986 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:03.126949    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"]
Sep 18 16:47:03.127264 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:03.127156    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2" podUID="fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b" containerName="istio-proxy" containerID="cri-o://bf1e1e8cff8662c4e4726f77d91d47f865590bc9f35bbee90a5f7accd2417cd9" gracePeriod=30
Sep 18 16:47:04.391433 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:04.391398    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a08cfbe-c4a2-49eb-887e-40177db95cf9" path="/var/lib/kubelet/pods/0a08cfbe-c4a2-49eb-887e-40177db95cf9/volumes"
Sep 18 16:47:05.415720 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:05.415692    2585 generic.go:356] "Generic (PLEG): container finished" podID="fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b" containerID="bf1e1e8cff8662c4e4726f77d91d47f865590bc9f35bbee90a5f7accd2417cd9" exitCode=0
Sep 18 16:47:05.416130 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:05.415772    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2" event={"ID":"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b","Type":"ContainerDied","Data":"bf1e1e8cff8662c4e4726f77d91d47f865590bc9f35bbee90a5f7accd2417cd9"}
Sep 18 16:47:05.465272 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:05.465251    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:47:05.502008 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:05.501983    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-istio-envoy\") pod \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") "
Sep 18 16:47:05.502148 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:05.502019    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-workload-socket\") pod \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") "
Sep 18 16:47:05.502148 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:05.502045    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-istiod-ca-cert\") pod \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") "
Sep 18 16:47:05.502148 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:05.502076    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-istio-token\") pod \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") "
Sep 18 16:47:05.502148 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:05.502107    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-istio-podinfo\") pod \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") "
Sep 18 16:47:05.502148 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:05.502144    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-istio-data\") pod \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") "
Sep 18 16:47:05.502413 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:05.502169    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-workload-certs\") pod \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") "
Sep 18 16:47:05.502413 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:05.502220    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxgxj\" (UniqueName: \"kubernetes.io/projected/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-kube-api-access-rxgxj\") pod \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") "
Sep 18 16:47:05.502413 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:05.502267    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-credential-socket\") pod \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\" (UID: \"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b\") "
Sep 18 16:47:05.502413 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:05.502291    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b" (UID: "fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:47:05.503744 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:05.503014    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b" (UID: "fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:47:05.503744 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:05.503306    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b" (UID: "fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:47:05.503744 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:05.503458    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b" (UID: "fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:47:05.503744 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:05.503502    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-istio-data" (OuterVolumeSpecName: "istio-data") pod "fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b" (UID: "fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:47:05.503744 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:05.502511    2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-workload-socket\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:47:05.507968 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:05.505435    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b" (UID: "fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:47:05.512791 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:05.512762    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-istio-token" (OuterVolumeSpecName: "istio-token") pod "fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b" (UID: "fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:47:05.514502 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:05.514477    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-kube-api-access-rxgxj" (OuterVolumeSpecName: "kube-api-access-rxgxj") pod "fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b" (UID: "fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b"). InnerVolumeSpecName "kube-api-access-rxgxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:47:05.514856 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:05.514831    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b" (UID: "fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 16:47:05.604044 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:05.603959    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxgxj\" (UniqueName: \"kubernetes.io/projected/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-kube-api-access-rxgxj\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:47:05.604044 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:05.603996    2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-credential-socket\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:47:05.604044 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:05.604006    2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-istio-envoy\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:47:05.604044 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:05.604015    2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-istiod-ca-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:47:05.604044 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:05.604023    2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-istio-token\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:47:05.604044 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:05.604031    2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-istio-podinfo\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:47:05.604044 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:05.604038    2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-istio-data\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:47:05.604044 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:05.604046    2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b-workload-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:47:06.421376 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:06.421353    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"
Sep 18 16:47:06.421761 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:06.421344    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2" event={"ID":"fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b","Type":"ContainerDied","Data":"eec3b3b89bc264ac72891a22e8d21462878cd03ce81155902bc3566df2eaa23b"}
Sep 18 16:47:06.421761 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:06.421482    2585 scope.go:117] "RemoveContainer" containerID="bf1e1e8cff8662c4e4726f77d91d47f865590bc9f35bbee90a5f7accd2417cd9"
Sep 18 16:47:06.439838 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:06.439810    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"]
Sep 18 16:47:06.444765 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:06.444741    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-6fd8d584bd-92zp2"]
Sep 18 16:47:08.392968 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:08.392926    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b" path="/var/lib/kubelet/pods/fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b/volumes"
Sep 18 16:47:12.267622 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.267545    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz"]
Sep 18 16:47:12.268012 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.267995    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b" containerName="istio-proxy"
Sep 18 16:47:12.268055 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.268016    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b" containerName="istio-proxy"
Sep 18 16:47:12.268055 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.268028    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a08cfbe-c4a2-49eb-887e-40177db95cf9" containerName="main"
Sep 18 16:47:12.268055 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.268036    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a08cfbe-c4a2-49eb-887e-40177db95cf9" containerName="main"
Sep 18 16:47:12.268150 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.268119    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a08cfbe-c4a2-49eb-887e-40177db95cf9" containerName="main"
Sep 18 16:47:12.268150 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.268132    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe9d0bf9-a3ce-4102-83fe-9f9cfeb0f14b" containerName="istio-proxy"
Sep 18 16:47:12.271160 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.271140    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz"
Sep 18 16:47:12.273811 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.273786    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 18 16:47:12.274980 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.274959    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 18 16:47:12.275105 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.275026    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-btsww\""
Sep 18 16:47:12.275177 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.275122    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Sep 18 16:47:12.283205 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.283183    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz"]
Sep 18 16:47:12.355705 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.355670    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz\" (UID: \"5aa519e6-6c80-4577-8bc6-cd7bfda25f45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz"
Sep 18 16:47:12.355705 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.355704    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz\" (UID: \"5aa519e6-6c80-4577-8bc6-cd7bfda25f45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz"
Sep 18 16:47:12.355900 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.355724    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsrf6\" (UniqueName: \"kubernetes.io/projected/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-kube-api-access-gsrf6\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz\" (UID: \"5aa519e6-6c80-4577-8bc6-cd7bfda25f45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz"
Sep 18 16:47:12.355900 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.355764    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz\" (UID: \"5aa519e6-6c80-4577-8bc6-cd7bfda25f45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz"
Sep 18 16:47:12.355900 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.355791    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz\" (UID: \"5aa519e6-6c80-4577-8bc6-cd7bfda25f45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz"
Sep 18 16:47:12.355900 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.355812    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz\" (UID: \"5aa519e6-6c80-4577-8bc6-cd7bfda25f45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz"
Sep 18 16:47:12.456695 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.456662    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz\" (UID: \"5aa519e6-6c80-4577-8bc6-cd7bfda25f45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz"
Sep 18 16:47:12.456886 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.456704    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz\" (UID: \"5aa519e6-6c80-4577-8bc6-cd7bfda25f45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz"
Sep 18 16:47:12.456886 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.456722    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsrf6\" (UniqueName: \"kubernetes.io/projected/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-kube-api-access-gsrf6\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz\" (UID: \"5aa519e6-6c80-4577-8bc6-cd7bfda25f45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz"
Sep 18 16:47:12.456886 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.456762    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz\" (UID: \"5aa519e6-6c80-4577-8bc6-cd7bfda25f45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz"
Sep 18 16:47:12.456886 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.456819    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz\" (UID: \"5aa519e6-6c80-4577-8bc6-cd7bfda25f45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz"
Sep 18 16:47:12.456886 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.456853    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz\" (UID: \"5aa519e6-6c80-4577-8bc6-cd7bfda25f45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz"
Sep 18 16:47:12.457183 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.457158    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz\" (UID: \"5aa519e6-6c80-4577-8bc6-cd7bfda25f45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz"
Sep 18 16:47:12.457255 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.457222    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz\" (UID: \"5aa519e6-6c80-4577-8bc6-cd7bfda25f45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz"
Sep 18 16:47:12.457297 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.457265    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz\" (UID: \"5aa519e6-6c80-4577-8bc6-cd7bfda25f45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz"
Sep 18 16:47:12.459339 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.459320    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz\" (UID: \"5aa519e6-6c80-4577-8bc6-cd7bfda25f45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz"
Sep 18 16:47:12.459507 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.459491    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz\" (UID: \"5aa519e6-6c80-4577-8bc6-cd7bfda25f45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz"
Sep 18 16:47:12.464985 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.464961    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsrf6\" (UniqueName: \"kubernetes.io/projected/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-kube-api-access-gsrf6\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz\" (UID: \"5aa519e6-6c80-4577-8bc6-cd7bfda25f45\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz"
Sep 18 16:47:12.581698 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.581623    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz"
Sep 18 16:47:12.856000 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:12.855904    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz"]
Sep 18 16:47:12.858750 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:47:12.858722    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa519e6_6c80_4577_8bc6_cd7bfda25f45.slice/crio-a8619cbcc240f6bee9891347c0495962ba25c4a4d016e253c38104d982a8d994 WatchSource:0}: Error finding container a8619cbcc240f6bee9891347c0495962ba25c4a4d016e253c38104d982a8d994: Status 404 returned error can't find the container with id a8619cbcc240f6bee9891347c0495962ba25c4a4d016e253c38104d982a8d994
Sep 18 16:47:13.456017 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:13.455981    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz" event={"ID":"5aa519e6-6c80-4577-8bc6-cd7bfda25f45","Type":"ContainerStarted","Data":"2b0552ba9544052e2116f7390e69aa4f3cfa2e7fd34fd351375fef2a9639efa5"}
Sep 18 16:47:13.456487 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:13.456031    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz" event={"ID":"5aa519e6-6c80-4577-8bc6-cd7bfda25f45","Type":"ContainerStarted","Data":"a8619cbcc240f6bee9891347c0495962ba25c4a4d016e253c38104d982a8d994"}
Sep 18 16:47:13.474843 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:13.474796    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz" podStartSLOduration=1.47478132 podStartE2EDuration="1.47478132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:47:13.473043418 +0000 UTC m=+1873.256045442" watchObservedRunningTime="2026-09-18 16:47:13.47478132 +0000 UTC m=+1873.257783344"
Sep 18 16:47:13.581741 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:13.581709    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz"
Sep 18 16:47:20.374887 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:20.374847    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz"]
Sep 18 16:47:20.384582 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:47:20.384544    2585 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 16:47:20.384582 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:47:20.384582 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 16:47:20.384582 ip-10-0-133-147 kubenswrapper[2585]:  > containerID="2b0552ba9544052e2116f7390e69aa4f3cfa2e7fd34fd351375fef2a9639efa5" cmd=["/bin/sleep","15"]
Sep 18 16:47:20.384858 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:20.384595    2585 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz" message="" err=<
Sep 18 16:47:20.384858 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:47:20.384858 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 16:47:20.384858 ip-10-0-133-147 kubenswrapper[2585]:  >
Sep 18 16:47:20.384858 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:47:20.384609    2585 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 16:47:20.384858 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:47:20.384858 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 16:47:20.384858 ip-10-0-133-147 kubenswrapper[2585]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz" podUID="5aa519e6-6c80-4577-8bc6-cd7bfda25f45" containerName="main" containerID="cri-o://2b0552ba9544052e2116f7390e69aa4f3cfa2e7fd34fd351375fef2a9639efa5"
Sep 18 16:47:20.384858 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:20.384645    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz" podUID="5aa519e6-6c80-4577-8bc6-cd7bfda25f45" containerName="main" containerID="cri-o://2b0552ba9544052e2116f7390e69aa4f3cfa2e7fd34fd351375fef2a9639efa5" gracePeriod=60
Sep 18 16:47:20.629453 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:20.629391    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz"
Sep 18 16:47:20.722421 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:20.722378    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-dshm\") pod \"5aa519e6-6c80-4577-8bc6-cd7bfda25f45\" (UID: \"5aa519e6-6c80-4577-8bc6-cd7bfda25f45\") "
Sep 18 16:47:20.722597 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:20.722435    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-tmp-dir\") pod \"5aa519e6-6c80-4577-8bc6-cd7bfda25f45\" (UID: \"5aa519e6-6c80-4577-8bc6-cd7bfda25f45\") "
Sep 18 16:47:20.722597 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:20.722474    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-tls-certs\") pod \"5aa519e6-6c80-4577-8bc6-cd7bfda25f45\" (UID: \"5aa519e6-6c80-4577-8bc6-cd7bfda25f45\") "
Sep 18 16:47:20.722597 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:20.722497    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-home\") pod \"5aa519e6-6c80-4577-8bc6-cd7bfda25f45\" (UID: \"5aa519e6-6c80-4577-8bc6-cd7bfda25f45\") "
Sep 18 16:47:20.722751 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:20.722654    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-model-cache\") pod \"5aa519e6-6c80-4577-8bc6-cd7bfda25f45\" (UID: \"5aa519e6-6c80-4577-8bc6-cd7bfda25f45\") "
Sep 18 16:47:20.722751 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:20.722723    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsrf6\" (UniqueName: \"kubernetes.io/projected/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-kube-api-access-gsrf6\") pod \"5aa519e6-6c80-4577-8bc6-cd7bfda25f45\" (UID: \"5aa519e6-6c80-4577-8bc6-cd7bfda25f45\") "
Sep 18 16:47:20.722890 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:20.722751    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5aa519e6-6c80-4577-8bc6-cd7bfda25f45" (UID: "5aa519e6-6c80-4577-8bc6-cd7bfda25f45"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:47:20.722890 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:20.722761    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-home" (OuterVolumeSpecName: "home") pod "5aa519e6-6c80-4577-8bc6-cd7bfda25f45" (UID: "5aa519e6-6c80-4577-8bc6-cd7bfda25f45"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:47:20.723045 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:20.722907    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-model-cache" (OuterVolumeSpecName: "model-cache") pod "5aa519e6-6c80-4577-8bc6-cd7bfda25f45" (UID: "5aa519e6-6c80-4577-8bc6-cd7bfda25f45"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:47:20.723045 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:20.723012    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-model-cache\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:47:20.723045 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:20.723030    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-tmp-dir\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:47:20.723045 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:20.723043    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-home\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:47:20.725216 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:20.725187    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-kube-api-access-gsrf6" (OuterVolumeSpecName: "kube-api-access-gsrf6") pod "5aa519e6-6c80-4577-8bc6-cd7bfda25f45" (UID: "5aa519e6-6c80-4577-8bc6-cd7bfda25f45"). InnerVolumeSpecName "kube-api-access-gsrf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:47:20.725355 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:20.725309    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5aa519e6-6c80-4577-8bc6-cd7bfda25f45" (UID: "5aa519e6-6c80-4577-8bc6-cd7bfda25f45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:47:20.725355 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:20.725325    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-dshm" (OuterVolumeSpecName: "dshm") pod "5aa519e6-6c80-4577-8bc6-cd7bfda25f45" (UID: "5aa519e6-6c80-4577-8bc6-cd7bfda25f45"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:47:20.824122 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:20.824075    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsrf6\" (UniqueName: \"kubernetes.io/projected/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-kube-api-access-gsrf6\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:47:20.824122 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:20.824119    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-dshm\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:47:20.824327 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:20.824133    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aa519e6-6c80-4577-8bc6-cd7bfda25f45-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:47:21.489003 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:21.488962    2585 generic.go:356] "Generic (PLEG): container finished" podID="5aa519e6-6c80-4577-8bc6-cd7bfda25f45" containerID="2b0552ba9544052e2116f7390e69aa4f3cfa2e7fd34fd351375fef2a9639efa5" exitCode=0
Sep 18 16:47:21.489390 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:21.489023    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz"
Sep 18 16:47:21.489390 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:21.489030    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz" event={"ID":"5aa519e6-6c80-4577-8bc6-cd7bfda25f45","Type":"ContainerDied","Data":"2b0552ba9544052e2116f7390e69aa4f3cfa2e7fd34fd351375fef2a9639efa5"}
Sep 18 16:47:21.489390 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:21.489080    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz" event={"ID":"5aa519e6-6c80-4577-8bc6-cd7bfda25f45","Type":"ContainerDied","Data":"a8619cbcc240f6bee9891347c0495962ba25c4a4d016e253c38104d982a8d994"}
Sep 18 16:47:21.489390 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:21.489111    2585 scope.go:117] "RemoveContainer" containerID="2b0552ba9544052e2116f7390e69aa4f3cfa2e7fd34fd351375fef2a9639efa5"
Sep 18 16:47:21.498423 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:21.498409    2585 scope.go:117] "RemoveContainer" containerID="2b0552ba9544052e2116f7390e69aa4f3cfa2e7fd34fd351375fef2a9639efa5"
Sep 18 16:47:21.498671 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:47:21.498649    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b0552ba9544052e2116f7390e69aa4f3cfa2e7fd34fd351375fef2a9639efa5\": container with ID starting with 2b0552ba9544052e2116f7390e69aa4f3cfa2e7fd34fd351375fef2a9639efa5 not found: ID does not exist" containerID="2b0552ba9544052e2116f7390e69aa4f3cfa2e7fd34fd351375fef2a9639efa5"
Sep 18 16:47:21.498765 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:21.498678    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b0552ba9544052e2116f7390e69aa4f3cfa2e7fd34fd351375fef2a9639efa5"} err="failed to get container status \"2b0552ba9544052e2116f7390e69aa4f3cfa2e7fd34fd351375fef2a9639efa5\": rpc error: code = NotFound desc = could not find container \"2b0552ba9544052e2116f7390e69aa4f3cfa2e7fd34fd351375fef2a9639efa5\": container with ID starting with 2b0552ba9544052e2116f7390e69aa4f3cfa2e7fd34fd351375fef2a9639efa5 not found: ID does not exist"
Sep 18 16:47:21.511767 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:21.511741    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz"]
Sep 18 16:47:21.515493 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:21.515472    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf2vzzz"]
Sep 18 16:47:22.391670 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:22.391636    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5aa519e6-6c80-4577-8bc6-cd7bfda25f45" path="/var/lib/kubelet/pods/5aa519e6-6c80-4577-8bc6-cd7bfda25f45/volumes"
Sep 18 16:47:34.936743 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:34.936711    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"]
Sep 18 16:47:34.937202 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:34.937026    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aa519e6-6c80-4577-8bc6-cd7bfda25f45" containerName="main"
Sep 18 16:47:34.937202 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:34.937039    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aa519e6-6c80-4577-8bc6-cd7bfda25f45" containerName="main"
Sep 18 16:47:34.937202 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:34.937114    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="5aa519e6-6c80-4577-8bc6-cd7bfda25f45" containerName="main"
Sep 18 16:47:34.939224 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:34.939190    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"
Sep 18 16:47:34.943441 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:34.943368    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-bd5ws\""
Sep 18 16:47:34.943441 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:34.943381    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 18 16:47:34.943441 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:34.943366    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 18 16:47:34.943705 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:34.943413    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 18 16:47:34.951006 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:34.950984    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"]
Sep 18 16:47:35.035066 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.035031    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvjdd\" (UniqueName: \"kubernetes.io/projected/95c75f4c-c162-4670-95b1-7689316bfe87-kube-api-access-fvjdd\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg\" (UID: \"95c75f4c-c162-4670-95b1-7689316bfe87\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"
Sep 18 16:47:35.035278 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.035079    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95c75f4c-c162-4670-95b1-7689316bfe87-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg\" (UID: \"95c75f4c-c162-4670-95b1-7689316bfe87\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"
Sep 18 16:47:35.035278 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.035134    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95c75f4c-c162-4670-95b1-7689316bfe87-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg\" (UID: \"95c75f4c-c162-4670-95b1-7689316bfe87\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"
Sep 18 16:47:35.035278 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.035174    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95c75f4c-c162-4670-95b1-7689316bfe87-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg\" (UID: \"95c75f4c-c162-4670-95b1-7689316bfe87\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"
Sep 18 16:47:35.035278 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.035260    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95c75f4c-c162-4670-95b1-7689316bfe87-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg\" (UID: \"95c75f4c-c162-4670-95b1-7689316bfe87\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"
Sep 18 16:47:35.035464 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.035313    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95c75f4c-c162-4670-95b1-7689316bfe87-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg\" (UID: \"95c75f4c-c162-4670-95b1-7689316bfe87\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"
Sep 18 16:47:35.136025 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.135998    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95c75f4c-c162-4670-95b1-7689316bfe87-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg\" (UID: \"95c75f4c-c162-4670-95b1-7689316bfe87\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"
Sep 18 16:47:35.136190 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.136042    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95c75f4c-c162-4670-95b1-7689316bfe87-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg\" (UID: \"95c75f4c-c162-4670-95b1-7689316bfe87\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"
Sep 18 16:47:35.136190 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.136072    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvjdd\" (UniqueName: \"kubernetes.io/projected/95c75f4c-c162-4670-95b1-7689316bfe87-kube-api-access-fvjdd\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg\" (UID: \"95c75f4c-c162-4670-95b1-7689316bfe87\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"
Sep 18 16:47:35.136190 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.136105    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95c75f4c-c162-4670-95b1-7689316bfe87-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg\" (UID: \"95c75f4c-c162-4670-95b1-7689316bfe87\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"
Sep 18 16:47:35.136190 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.136146    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95c75f4c-c162-4670-95b1-7689316bfe87-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg\" (UID: \"95c75f4c-c162-4670-95b1-7689316bfe87\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"
Sep 18 16:47:35.136190 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.136176    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95c75f4c-c162-4670-95b1-7689316bfe87-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg\" (UID: \"95c75f4c-c162-4670-95b1-7689316bfe87\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"
Sep 18 16:47:35.136540 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.136517    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95c75f4c-c162-4670-95b1-7689316bfe87-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg\" (UID: \"95c75f4c-c162-4670-95b1-7689316bfe87\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"
Sep 18 16:47:35.136629 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.136543    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95c75f4c-c162-4670-95b1-7689316bfe87-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg\" (UID: \"95c75f4c-c162-4670-95b1-7689316bfe87\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"
Sep 18 16:47:35.136629 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.136583    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95c75f4c-c162-4670-95b1-7689316bfe87-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg\" (UID: \"95c75f4c-c162-4670-95b1-7689316bfe87\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"
Sep 18 16:47:35.138565 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.138548    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95c75f4c-c162-4670-95b1-7689316bfe87-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg\" (UID: \"95c75f4c-c162-4670-95b1-7689316bfe87\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"
Sep 18 16:47:35.138791 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.138773    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95c75f4c-c162-4670-95b1-7689316bfe87-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg\" (UID: \"95c75f4c-c162-4670-95b1-7689316bfe87\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"
Sep 18 16:47:35.150857 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.150834    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvjdd\" (UniqueName: \"kubernetes.io/projected/95c75f4c-c162-4670-95b1-7689316bfe87-kube-api-access-fvjdd\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg\" (UID: \"95c75f4c-c162-4670-95b1-7689316bfe87\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"
Sep 18 16:47:35.161825 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.161797    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq"]
Sep 18 16:47:35.164425 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.164405    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq"
Sep 18 16:47:35.166967 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.166942    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-ww7jq\""
Sep 18 16:47:35.173576 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.173555    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq"]
Sep 18 16:47:35.236757 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.236733    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b7922c0-6961-4bdd-8b40-047382a72c0f-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq\" (UID: \"9b7922c0-6961-4bdd-8b40-047382a72c0f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq"
Sep 18 16:47:35.236872 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.236779    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7tcw\" (UniqueName: \"kubernetes.io/projected/9b7922c0-6961-4bdd-8b40-047382a72c0f-kube-api-access-r7tcw\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq\" (UID: \"9b7922c0-6961-4bdd-8b40-047382a72c0f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq"
Sep 18 16:47:35.249871 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.249839    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"
Sep 18 16:47:35.337481 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.337441    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7tcw\" (UniqueName: \"kubernetes.io/projected/9b7922c0-6961-4bdd-8b40-047382a72c0f-kube-api-access-r7tcw\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq\" (UID: \"9b7922c0-6961-4bdd-8b40-047382a72c0f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq"
Sep 18 16:47:35.337611 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.337523    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b7922c0-6961-4bdd-8b40-047382a72c0f-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq\" (UID: \"9b7922c0-6961-4bdd-8b40-047382a72c0f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq"
Sep 18 16:47:35.340221 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.340204    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b7922c0-6961-4bdd-8b40-047382a72c0f-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq\" (UID: \"9b7922c0-6961-4bdd-8b40-047382a72c0f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq"
Sep 18 16:47:35.346642 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.346621    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7tcw\" (UniqueName: \"kubernetes.io/projected/9b7922c0-6961-4bdd-8b40-047382a72c0f-kube-api-access-r7tcw\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq\" (UID: \"9b7922c0-6961-4bdd-8b40-047382a72c0f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq"
Sep 18 16:47:35.476611 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.476577    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq"
Sep 18 16:47:35.520503 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.520476    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"]
Sep 18 16:47:35.523127 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:47:35.523094    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c75f4c_c162_4670_95b1_7689316bfe87.slice/crio-bf862a90aa46f906c012a4a282a99056acbf20b3ce2f1932225c83ed6b799d69 WatchSource:0}: Error finding container bf862a90aa46f906c012a4a282a99056acbf20b3ce2f1932225c83ed6b799d69: Status 404 returned error can't find the container with id bf862a90aa46f906c012a4a282a99056acbf20b3ce2f1932225c83ed6b799d69
Sep 18 16:47:35.547043 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.547013    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg" event={"ID":"95c75f4c-c162-4670-95b1-7689316bfe87","Type":"ContainerStarted","Data":"bf862a90aa46f906c012a4a282a99056acbf20b3ce2f1932225c83ed6b799d69"}
Sep 18 16:47:35.754052 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:35.754027    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq"]
Sep 18 16:47:35.756237 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:47:35.756212    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b7922c0_6961_4bdd_8b40_047382a72c0f.slice/crio-77817f46b937753e9748c7efd90e96ba333777579c113b918b305a1cb5ce5b55 WatchSource:0}: Error finding container 77817f46b937753e9748c7efd90e96ba333777579c113b918b305a1cb5ce5b55: Status 404 returned error can't find the container with id 77817f46b937753e9748c7efd90e96ba333777579c113b918b305a1cb5ce5b55
Sep 18 16:47:36.552503 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:36.552471    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq" event={"ID":"9b7922c0-6961-4bdd-8b40-047382a72c0f","Type":"ContainerStarted","Data":"3a449a7b95031c9a99a753e034011dbd4a4fe785b436b0628e05c4d1ef50cf0c"}
Sep 18 16:47:36.552864 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:36.552519    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq" event={"ID":"9b7922c0-6961-4bdd-8b40-047382a72c0f","Type":"ContainerStarted","Data":"77817f46b937753e9748c7efd90e96ba333777579c113b918b305a1cb5ce5b55"}
Sep 18 16:47:36.552864 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:36.552637    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq"
Sep 18 16:47:36.554129 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:36.554109    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg" event={"ID":"95c75f4c-c162-4670-95b1-7689316bfe87","Type":"ContainerStarted","Data":"2e3f0097751b8f160881df439f69eb7d13ebb7b27868dac928c798c6079d310c"}
Sep 18 16:47:36.569604 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:36.569562    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq" podStartSLOduration=1.569549418 podStartE2EDuration="1.569549418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:47:36.568168125 +0000 UTC m=+1896.351170170" watchObservedRunningTime="2026-09-18 16:47:36.569549418 +0000 UTC m=+1896.352551441"
Sep 18 16:47:36.585867 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:36.585809    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg" podStartSLOduration=2.5857987380000003 podStartE2EDuration="2.585798738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:47:36.584905055 +0000 UTC m=+1896.367907074" watchObservedRunningTime="2026-09-18 16:47:36.585798738 +0000 UTC m=+1896.368800763"
Sep 18 16:47:37.250948 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:37.250887    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"
Sep 18 16:47:45.250441 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:45.250410    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"
Sep 18 16:47:45.263247 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:45.263218    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"
Sep 18 16:47:45.600687 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:47:45.600609    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"
Sep 18 16:48:07.560160 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:07.560130    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq"
Sep 18 16:48:17.028722 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.028688    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq"]
Sep 18 16:48:17.059635 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.059603    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"]
Sep 18 16:48:17.068007 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:48:17.067963    2585 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 16:48:17.068007 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:48:17.068007 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 16:48:17.068007 ip-10-0-133-147 kubenswrapper[2585]:  > containerID="2e3f0097751b8f160881df439f69eb7d13ebb7b27868dac928c798c6079d310c" cmd=["/bin/sleep","15"]
Sep 18 16:48:17.068249 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.068025    2585 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg" message="" err=<
Sep 18 16:48:17.068249 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:48:17.068249 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 16:48:17.068249 ip-10-0-133-147 kubenswrapper[2585]:  >
Sep 18 16:48:17.068249 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:48:17.068040    2585 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 16:48:17.068249 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:48:17.068249 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 16:48:17.068249 ip-10-0-133-147 kubenswrapper[2585]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg" podUID="95c75f4c-c162-4670-95b1-7689316bfe87" containerName="main" containerID="cri-o://2e3f0097751b8f160881df439f69eb7d13ebb7b27868dac928c798c6079d310c"
Sep 18 16:48:17.068249 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.068092    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg" podUID="95c75f4c-c162-4670-95b1-7689316bfe87" containerName="main" containerID="cri-o://2e3f0097751b8f160881df439f69eb7d13ebb7b27868dac928c798c6079d310c" gracePeriod=60
Sep 18 16:48:17.409572 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.409548    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"
Sep 18 16:48:17.471135 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.471103    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95c75f4c-c162-4670-95b1-7689316bfe87-model-cache\") pod \"95c75f4c-c162-4670-95b1-7689316bfe87\" (UID: \"95c75f4c-c162-4670-95b1-7689316bfe87\") "
Sep 18 16:48:17.471299 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.471147    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95c75f4c-c162-4670-95b1-7689316bfe87-tmp-dir\") pod \"95c75f4c-c162-4670-95b1-7689316bfe87\" (UID: \"95c75f4c-c162-4670-95b1-7689316bfe87\") "
Sep 18 16:48:17.471299 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.471170    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95c75f4c-c162-4670-95b1-7689316bfe87-dshm\") pod \"95c75f4c-c162-4670-95b1-7689316bfe87\" (UID: \"95c75f4c-c162-4670-95b1-7689316bfe87\") "
Sep 18 16:48:17.471299 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.471209    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95c75f4c-c162-4670-95b1-7689316bfe87-home\") pod \"95c75f4c-c162-4670-95b1-7689316bfe87\" (UID: \"95c75f4c-c162-4670-95b1-7689316bfe87\") "
Sep 18 16:48:17.471299 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.471264    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvjdd\" (UniqueName: \"kubernetes.io/projected/95c75f4c-c162-4670-95b1-7689316bfe87-kube-api-access-fvjdd\") pod \"95c75f4c-c162-4670-95b1-7689316bfe87\" (UID: \"95c75f4c-c162-4670-95b1-7689316bfe87\") "
Sep 18 16:48:17.471299 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.471297    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95c75f4c-c162-4670-95b1-7689316bfe87-tls-certs\") pod \"95c75f4c-c162-4670-95b1-7689316bfe87\" (UID: \"95c75f4c-c162-4670-95b1-7689316bfe87\") "
Sep 18 16:48:17.471569 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.471419    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c75f4c-c162-4670-95b1-7689316bfe87-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "95c75f4c-c162-4670-95b1-7689316bfe87" (UID: "95c75f4c-c162-4670-95b1-7689316bfe87"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:48:17.471569 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.471451    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c75f4c-c162-4670-95b1-7689316bfe87-model-cache" (OuterVolumeSpecName: "model-cache") pod "95c75f4c-c162-4670-95b1-7689316bfe87" (UID: "95c75f4c-c162-4670-95b1-7689316bfe87"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:48:17.471569 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.471491    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c75f4c-c162-4670-95b1-7689316bfe87-home" (OuterVolumeSpecName: "home") pod "95c75f4c-c162-4670-95b1-7689316bfe87" (UID: "95c75f4c-c162-4670-95b1-7689316bfe87"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:48:17.471744 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.471606    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95c75f4c-c162-4670-95b1-7689316bfe87-model-cache\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:48:17.471744 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.471625    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95c75f4c-c162-4670-95b1-7689316bfe87-tmp-dir\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:48:17.471744 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.471638    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95c75f4c-c162-4670-95b1-7689316bfe87-home\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:48:17.473604 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.473580    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c75f4c-c162-4670-95b1-7689316bfe87-kube-api-access-fvjdd" (OuterVolumeSpecName: "kube-api-access-fvjdd") pod "95c75f4c-c162-4670-95b1-7689316bfe87" (UID: "95c75f4c-c162-4670-95b1-7689316bfe87"). InnerVolumeSpecName "kube-api-access-fvjdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:48:17.473718 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.473620    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c75f4c-c162-4670-95b1-7689316bfe87-dshm" (OuterVolumeSpecName: "dshm") pod "95c75f4c-c162-4670-95b1-7689316bfe87" (UID: "95c75f4c-c162-4670-95b1-7689316bfe87"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:48:17.473718 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.473654    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95c75f4c-c162-4670-95b1-7689316bfe87-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "95c75f4c-c162-4670-95b1-7689316bfe87" (UID: "95c75f4c-c162-4670-95b1-7689316bfe87"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:48:17.560209 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.560144    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq" podUID="9b7922c0-6961-4bdd-8b40-047382a72c0f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:48:17.572800 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.572779    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95c75f4c-c162-4670-95b1-7689316bfe87-dshm\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:48:17.572800 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.572799    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvjdd\" (UniqueName: \"kubernetes.io/projected/95c75f4c-c162-4670-95b1-7689316bfe87-kube-api-access-fvjdd\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:48:17.573019 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.572809    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95c75f4c-c162-4670-95b1-7689316bfe87-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:48:17.725555 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.725524    2585 generic.go:356] "Generic (PLEG): container finished" podID="95c75f4c-c162-4670-95b1-7689316bfe87" containerID="2e3f0097751b8f160881df439f69eb7d13ebb7b27868dac928c798c6079d310c" exitCode=0
Sep 18 16:48:17.725717 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.725578    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"
Sep 18 16:48:17.725717 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.725615    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg" event={"ID":"95c75f4c-c162-4670-95b1-7689316bfe87","Type":"ContainerDied","Data":"2e3f0097751b8f160881df439f69eb7d13ebb7b27868dac928c798c6079d310c"}
Sep 18 16:48:17.725717 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.725662    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg" event={"ID":"95c75f4c-c162-4670-95b1-7689316bfe87","Type":"ContainerDied","Data":"bf862a90aa46f906c012a4a282a99056acbf20b3ce2f1932225c83ed6b799d69"}
Sep 18 16:48:17.725717 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.725682    2585 scope.go:117] "RemoveContainer" containerID="2e3f0097751b8f160881df439f69eb7d13ebb7b27868dac928c798c6079d310c"
Sep 18 16:48:17.735106 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.735084    2585 scope.go:117] "RemoveContainer" containerID="2e3f0097751b8f160881df439f69eb7d13ebb7b27868dac928c798c6079d310c"
Sep 18 16:48:17.735373 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:48:17.735348    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e3f0097751b8f160881df439f69eb7d13ebb7b27868dac928c798c6079d310c\": container with ID starting with 2e3f0097751b8f160881df439f69eb7d13ebb7b27868dac928c798c6079d310c not found: ID does not exist" containerID="2e3f0097751b8f160881df439f69eb7d13ebb7b27868dac928c798c6079d310c"
Sep 18 16:48:17.735482 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.735378    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e3f0097751b8f160881df439f69eb7d13ebb7b27868dac928c798c6079d310c"} err="failed to get container status \"2e3f0097751b8f160881df439f69eb7d13ebb7b27868dac928c798c6079d310c\": rpc error: code = NotFound desc = could not find container \"2e3f0097751b8f160881df439f69eb7d13ebb7b27868dac928c798c6079d310c\": container with ID starting with 2e3f0097751b8f160881df439f69eb7d13ebb7b27868dac928c798c6079d310c not found: ID does not exist"
Sep 18 16:48:17.749843 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.749813    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"]
Sep 18 16:48:17.753882 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:17.753847    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444gqgdg"]
Sep 18 16:48:18.391616 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:18.391584    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95c75f4c-c162-4670-95b1-7689316bfe87" path="/var/lib/kubelet/pods/95c75f4c-c162-4670-95b1-7689316bfe87/volumes"
Sep 18 16:48:27.560553 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:27.560513    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq" podUID="9b7922c0-6961-4bdd-8b40-047382a72c0f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:48:32.047829 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:32.047788    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq" podUID="9b7922c0-6961-4bdd-8b40-047382a72c0f" containerName="main" containerID="cri-o://3a449a7b95031c9a99a753e034011dbd4a4fe785b436b0628e05c4d1ef50cf0c" gracePeriod=45
Sep 18 16:48:37.560577 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:37.560534    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq" podUID="9b7922c0-6961-4bdd-8b40-047382a72c0f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:48:37.560998 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:37.560669    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq"
Sep 18 16:48:47.560208 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:47.560113    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq" podUID="9b7922c0-6961-4bdd-8b40-047382a72c0f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:48:57.559711 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:48:57.559671    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq" podUID="9b7922c0-6961-4bdd-8b40-047382a72c0f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:49:02.291776 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:02.291746    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq"
Sep 18 16:49:02.394609 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:02.394539    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7tcw\" (UniqueName: \"kubernetes.io/projected/9b7922c0-6961-4bdd-8b40-047382a72c0f-kube-api-access-r7tcw\") pod \"9b7922c0-6961-4bdd-8b40-047382a72c0f\" (UID: \"9b7922c0-6961-4bdd-8b40-047382a72c0f\") "
Sep 18 16:49:02.394609 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:02.394577    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b7922c0-6961-4bdd-8b40-047382a72c0f-tls-certs\") pod \"9b7922c0-6961-4bdd-8b40-047382a72c0f\" (UID: \"9b7922c0-6961-4bdd-8b40-047382a72c0f\") "
Sep 18 16:49:02.396539 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:02.396513    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b7922c0-6961-4bdd-8b40-047382a72c0f-kube-api-access-r7tcw" (OuterVolumeSpecName: "kube-api-access-r7tcw") pod "9b7922c0-6961-4bdd-8b40-047382a72c0f" (UID: "9b7922c0-6961-4bdd-8b40-047382a72c0f"). InnerVolumeSpecName "kube-api-access-r7tcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:49:02.396640 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:02.396582    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b7922c0-6961-4bdd-8b40-047382a72c0f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b7922c0-6961-4bdd-8b40-047382a72c0f" (UID: "9b7922c0-6961-4bdd-8b40-047382a72c0f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:49:02.495922 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:02.495884    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7tcw\" (UniqueName: \"kubernetes.io/projected/9b7922c0-6961-4bdd-8b40-047382a72c0f-kube-api-access-r7tcw\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:49:02.496038 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:02.495929    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b7922c0-6961-4bdd-8b40-047382a72c0f-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:49:02.895829 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:02.895800    2585 generic.go:356] "Generic (PLEG): container finished" podID="9b7922c0-6961-4bdd-8b40-047382a72c0f" containerID="3a449a7b95031c9a99a753e034011dbd4a4fe785b436b0628e05c4d1ef50cf0c" exitCode=0
Sep 18 16:49:02.896015 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:02.895841    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq"
Sep 18 16:49:02.896015 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:02.895861    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq" event={"ID":"9b7922c0-6961-4bdd-8b40-047382a72c0f","Type":"ContainerDied","Data":"3a449a7b95031c9a99a753e034011dbd4a4fe785b436b0628e05c4d1ef50cf0c"}
Sep 18 16:49:02.896015 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:02.895904    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq" event={"ID":"9b7922c0-6961-4bdd-8b40-047382a72c0f","Type":"ContainerDied","Data":"77817f46b937753e9748c7efd90e96ba333777579c113b918b305a1cb5ce5b55"}
Sep 18 16:49:02.896015 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:02.895941    2585 scope.go:117] "RemoveContainer" containerID="3a449a7b95031c9a99a753e034011dbd4a4fe785b436b0628e05c4d1ef50cf0c"
Sep 18 16:49:02.904549 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:02.904531    2585 scope.go:117] "RemoveContainer" containerID="3a449a7b95031c9a99a753e034011dbd4a4fe785b436b0628e05c4d1ef50cf0c"
Sep 18 16:49:02.904798 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:49:02.904780    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a449a7b95031c9a99a753e034011dbd4a4fe785b436b0628e05c4d1ef50cf0c\": container with ID starting with 3a449a7b95031c9a99a753e034011dbd4a4fe785b436b0628e05c4d1ef50cf0c not found: ID does not exist" containerID="3a449a7b95031c9a99a753e034011dbd4a4fe785b436b0628e05c4d1ef50cf0c"
Sep 18 16:49:02.904861 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:02.904806    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a449a7b95031c9a99a753e034011dbd4a4fe785b436b0628e05c4d1ef50cf0c"} err="failed to get container status \"3a449a7b95031c9a99a753e034011dbd4a4fe785b436b0628e05c4d1ef50cf0c\": rpc error: code = NotFound desc = could not find container \"3a449a7b95031c9a99a753e034011dbd4a4fe785b436b0628e05c4d1ef50cf0c\": container with ID starting with 3a449a7b95031c9a99a753e034011dbd4a4fe785b436b0628e05c4d1ef50cf0c not found: ID does not exist"
Sep 18 16:49:02.916557 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:02.916531    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq"]
Sep 18 16:49:02.920260 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:02.920242    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedkhqrq"]
Sep 18 16:49:04.391339 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:04.391298    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b7922c0-6961-4bdd-8b40-047382a72c0f" path="/var/lib/kubelet/pods/9b7922c0-6961-4bdd-8b40-047382a72c0f/volumes"
Sep 18 16:49:15.771113 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.771073    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"]
Sep 18 16:49:15.771568 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.771369    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95c75f4c-c162-4670-95b1-7689316bfe87" containerName="main"
Sep 18 16:49:15.771568 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.771384    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="95c75f4c-c162-4670-95b1-7689316bfe87" containerName="main"
Sep 18 16:49:15.771568 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.771395    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b7922c0-6961-4bdd-8b40-047382a72c0f" containerName="main"
Sep 18 16:49:15.771568 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.771400    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b7922c0-6961-4bdd-8b40-047382a72c0f" containerName="main"
Sep 18 16:49:15.771568 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.771468    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="95c75f4c-c162-4670-95b1-7689316bfe87" containerName="main"
Sep 18 16:49:15.771568 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.771481    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b7922c0-6961-4bdd-8b40-047382a72c0f" containerName="main"
Sep 18 16:49:15.775840 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.775822    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:15.780191 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.780169    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 18 16:49:15.780410 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.780392    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-bptcr\""
Sep 18 16:49:15.780488 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.780392    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 16:49:15.780488 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.780439    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 18 16:49:15.780488 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.780437    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 18 16:49:15.780680 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.780666    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 18 16:49:15.787200 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.787179    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"]
Sep 18 16:49:15.891724 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.891699    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/703c8671-9127-4654-8578-92b0aca4f6dc-home\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-5jjcp\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:15.891828 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.891736    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fc5t\" (UniqueName: \"kubernetes.io/projected/703c8671-9127-4654-8578-92b0aca4f6dc-kube-api-access-7fc5t\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-5jjcp\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:15.891828 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.891755    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/703c8671-9127-4654-8578-92b0aca4f6dc-tls-certs\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-5jjcp\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:15.891828 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.891816    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/703c8671-9127-4654-8578-92b0aca4f6dc-model-cache\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-5jjcp\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:15.891965 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.891835    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/703c8671-9127-4654-8578-92b0aca4f6dc-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-5jjcp\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:15.891965 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.891931    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/703c8671-9127-4654-8578-92b0aca4f6dc-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-5jjcp\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:15.891965 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.891959    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/703c8671-9127-4654-8578-92b0aca4f6dc-dshm\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-5jjcp\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:15.892065 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.892020    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/703c8671-9127-4654-8578-92b0aca4f6dc-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-5jjcp\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:15.992970 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.992936    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/703c8671-9127-4654-8578-92b0aca4f6dc-home\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-5jjcp\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:15.993128 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.992990    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fc5t\" (UniqueName: \"kubernetes.io/projected/703c8671-9127-4654-8578-92b0aca4f6dc-kube-api-access-7fc5t\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-5jjcp\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:15.993128 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.993020    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/703c8671-9127-4654-8578-92b0aca4f6dc-tls-certs\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-5jjcp\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:15.993128 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.993048    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/703c8671-9127-4654-8578-92b0aca4f6dc-model-cache\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-5jjcp\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:15.993128 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.993072    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/703c8671-9127-4654-8578-92b0aca4f6dc-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-5jjcp\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:15.993128 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.993101    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/703c8671-9127-4654-8578-92b0aca4f6dc-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-5jjcp\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:15.993128 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.993126    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/703c8671-9127-4654-8578-92b0aca4f6dc-dshm\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-5jjcp\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:15.993441 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.993189    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/703c8671-9127-4654-8578-92b0aca4f6dc-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-5jjcp\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:15.993502 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.993439    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/703c8671-9127-4654-8578-92b0aca4f6dc-home\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-5jjcp\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:15.993502 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.993459    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/703c8671-9127-4654-8578-92b0aca4f6dc-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-5jjcp\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:15.993595 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.993552    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/703c8671-9127-4654-8578-92b0aca4f6dc-model-cache\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-5jjcp\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:15.993691 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.993669    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/703c8671-9127-4654-8578-92b0aca4f6dc-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-5jjcp\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:15.994021 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.994001    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/703c8671-9127-4654-8578-92b0aca4f6dc-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-5jjcp\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:15.995777 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.995756    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/703c8671-9127-4654-8578-92b0aca4f6dc-tls-certs\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-5jjcp\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:15.995874 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:15.995792    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/703c8671-9127-4654-8578-92b0aca4f6dc-dshm\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-5jjcp\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:16.001404 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:16.001384    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fc5t\" (UniqueName: \"kubernetes.io/projected/703c8671-9127-4654-8578-92b0aca4f6dc-kube-api-access-7fc5t\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-5jjcp\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:16.038598 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:16.038528    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt"]
Sep 18 16:49:16.042978 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:16.042958    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt"
Sep 18 16:49:16.054253 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:16.054233    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt"]
Sep 18 16:49:16.086873 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:16.086852    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:16.194629 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:16.194598    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e84a7350-e841-4af5-9de0-e11cd2405c54-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt\" (UID: \"e84a7350-e841-4af5-9de0-e11cd2405c54\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt"
Sep 18 16:49:16.194767 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:16.194663    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e84a7350-e841-4af5-9de0-e11cd2405c54-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt\" (UID: \"e84a7350-e841-4af5-9de0-e11cd2405c54\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt"
Sep 18 16:49:16.194767 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:16.194690    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e84a7350-e841-4af5-9de0-e11cd2405c54-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt\" (UID: \"e84a7350-e841-4af5-9de0-e11cd2405c54\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt"
Sep 18 16:49:16.194869 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:16.194801    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e84a7350-e841-4af5-9de0-e11cd2405c54-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt\" (UID: \"e84a7350-e841-4af5-9de0-e11cd2405c54\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt"
Sep 18 16:49:16.295404 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:16.295333    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e84a7350-e841-4af5-9de0-e11cd2405c54-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt\" (UID: \"e84a7350-e841-4af5-9de0-e11cd2405c54\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt"
Sep 18 16:49:16.295404 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:16.295387    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e84a7350-e841-4af5-9de0-e11cd2405c54-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt\" (UID: \"e84a7350-e841-4af5-9de0-e11cd2405c54\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt"
Sep 18 16:49:16.295564 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:16.295415    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e84a7350-e841-4af5-9de0-e11cd2405c54-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt\" (UID: \"e84a7350-e841-4af5-9de0-e11cd2405c54\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt"
Sep 18 16:49:16.295564 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:16.295436    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e84a7350-e841-4af5-9de0-e11cd2405c54-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt\" (UID: \"e84a7350-e841-4af5-9de0-e11cd2405c54\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt"
Sep 18 16:49:16.295817 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:16.295777    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e84a7350-e841-4af5-9de0-e11cd2405c54-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt\" (UID: \"e84a7350-e841-4af5-9de0-e11cd2405c54\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt"
Sep 18 16:49:16.295817 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:16.295796    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e84a7350-e841-4af5-9de0-e11cd2405c54-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt\" (UID: \"e84a7350-e841-4af5-9de0-e11cd2405c54\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt"
Sep 18 16:49:16.296152 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:16.296134    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e84a7350-e841-4af5-9de0-e11cd2405c54-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt\" (UID: \"e84a7350-e841-4af5-9de0-e11cd2405c54\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt"
Sep 18 16:49:16.298064 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:16.298045    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e84a7350-e841-4af5-9de0-e11cd2405c54-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt\" (UID: \"e84a7350-e841-4af5-9de0-e11cd2405c54\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt"
Sep 18 16:49:16.353926 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:16.353889    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt"
Sep 18 16:49:16.364503 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:16.364482    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"]
Sep 18 16:49:16.632138 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:16.632116    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt"]
Sep 18 16:49:16.634790 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:49:16.634764    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode84a7350_e841_4af5_9de0_e11cd2405c54.slice/crio-be94cbf2c03e25a6a1965e11940078d36373709cd25ee23bedbeddf0798156ef WatchSource:0}: Error finding container be94cbf2c03e25a6a1965e11940078d36373709cd25ee23bedbeddf0798156ef: Status 404 returned error can't find the container with id be94cbf2c03e25a6a1965e11940078d36373709cd25ee23bedbeddf0798156ef
Sep 18 16:49:16.955161 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:16.955070    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt" event={"ID":"e84a7350-e841-4af5-9de0-e11cd2405c54","Type":"ContainerStarted","Data":"18702861d9b3e08274356edf64e2ef49aa60a2a5acb3feacbb8e6b3cc6596dd8"}
Sep 18 16:49:16.955161 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:16.955132    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt" event={"ID":"e84a7350-e841-4af5-9de0-e11cd2405c54","Type":"ContainerStarted","Data":"be94cbf2c03e25a6a1965e11940078d36373709cd25ee23bedbeddf0798156ef"}
Sep 18 16:49:16.956843 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:16.956816    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp" event={"ID":"703c8671-9127-4654-8578-92b0aca4f6dc","Type":"ContainerStarted","Data":"dfc83dcfcd34e48d9aafa015c5418c3d31ec0a63795d45870eb9830d08aeea98"}
Sep 18 16:49:16.956969 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:16.956856    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp" event={"ID":"703c8671-9127-4654-8578-92b0aca4f6dc","Type":"ContainerStarted","Data":"15131577ac16b917bd1d502cc094c7610c6d47356499a6b6da060b273a601fa5"}
Sep 18 16:49:18.967900 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:18.967856    2585 generic.go:356] "Generic (PLEG): container finished" podID="703c8671-9127-4654-8578-92b0aca4f6dc" containerID="dfc83dcfcd34e48d9aafa015c5418c3d31ec0a63795d45870eb9830d08aeea98" exitCode=0
Sep 18 16:49:18.968353 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:18.967947    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp" event={"ID":"703c8671-9127-4654-8578-92b0aca4f6dc","Type":"ContainerDied","Data":"dfc83dcfcd34e48d9aafa015c5418c3d31ec0a63795d45870eb9830d08aeea98"}
Sep 18 16:49:19.975445 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:19.975412    2585 generic.go:356] "Generic (PLEG): container finished" podID="e84a7350-e841-4af5-9de0-e11cd2405c54" containerID="18702861d9b3e08274356edf64e2ef49aa60a2a5acb3feacbb8e6b3cc6596dd8" exitCode=0
Sep 18 16:49:19.975935 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:19.975449    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt" event={"ID":"e84a7350-e841-4af5-9de0-e11cd2405c54","Type":"ContainerDied","Data":"18702861d9b3e08274356edf64e2ef49aa60a2a5acb3feacbb8e6b3cc6596dd8"}
Sep 18 16:49:19.979768 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:19.979745    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp" event={"ID":"703c8671-9127-4654-8578-92b0aca4f6dc","Type":"ContainerStarted","Data":"000b84ea4f5d814c9d90b8b07311d5b4c580ccae39db03f85cd3c68f1895dbb4"}
Sep 18 16:49:20.015886 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:20.015838    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp" podStartSLOduration=2.54404502 podStartE2EDuration="5.015822668s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.471777648s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:49:20.014056414 +0000 UTC m=+1999.797058439" watchObservedRunningTime="2026-09-18 16:49:20.015822668 +0000 UTC m=+1999.798824692"
Sep 18 16:49:20.087873 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:20.087844    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:20.985968 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:20.985934    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt" event={"ID":"e84a7350-e841-4af5-9de0-e11cd2405c54","Type":"ContainerStarted","Data":"973238a58b5b5ad90c2b0c269444c8d4328a7a120a0859bf666a82e2d496605e"}
Sep 18 16:49:20.986487 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:20.986255    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt"
Sep 18 16:49:21.003852 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:21.003802    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt" podStartSLOduration=2.523370285 podStartE2EDuration="5.003784838s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.480414553s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:49:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:49:21.003470084 +0000 UTC m=+2000.786472135" watchObservedRunningTime="2026-09-18 16:49:21.003784838 +0000 UTC m=+2000.786786894"
Sep 18 16:49:26.087780 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:26.087738    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:26.100149 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:26.100120    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:27.027858 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:27.027831    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:52.002118 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:52.002081    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt"
Sep 18 16:49:53.468647 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:53.468617    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt_e84a7350-e841-4af5-9de0-e11cd2405c54/main/0.log"
Sep 18 16:49:53.648554 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:53.648520    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt"]
Sep 18 16:49:53.648775 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:53.648750    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt" podUID="e84a7350-e841-4af5-9de0-e11cd2405c54" containerName="main" containerID="cri-o://973238a58b5b5ad90c2b0c269444c8d4328a7a120a0859bf666a82e2d496605e" gracePeriod=30
Sep 18 16:49:53.665797 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:53.665763    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"]
Sep 18 16:49:53.674702 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:49:53.674663    2585 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 16:49:53.674702 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:49:53.674702 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 16:49:53.674702 ip-10-0-133-147 kubenswrapper[2585]:  > containerID="000b84ea4f5d814c9d90b8b07311d5b4c580ccae39db03f85cd3c68f1895dbb4" cmd=["/bin/sleep","15"]
Sep 18 16:49:53.674989 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:53.674720    2585 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp" message="" err=<
Sep 18 16:49:53.674989 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:49:53.674989 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 16:49:53.674989 ip-10-0-133-147 kubenswrapper[2585]:  >
Sep 18 16:49:53.674989 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:49:53.674737    2585 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 16:49:53.674989 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:49:53.674989 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 16:49:53.674989 ip-10-0-133-147 kubenswrapper[2585]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp" podUID="703c8671-9127-4654-8578-92b0aca4f6dc" containerName="main" containerID="cri-o://000b84ea4f5d814c9d90b8b07311d5b4c580ccae39db03f85cd3c68f1895dbb4"
Sep 18 16:49:53.674989 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:53.674790    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp" podUID="703c8671-9127-4654-8578-92b0aca4f6dc" containerName="main" containerID="cri-o://000b84ea4f5d814c9d90b8b07311d5b4c580ccae39db03f85cd3c68f1895dbb4" gracePeriod=60
Sep 18 16:49:54.014091 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.014065    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:54.093855 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.093755    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/703c8671-9127-4654-8578-92b0aca4f6dc-home\") pod \"703c8671-9127-4654-8578-92b0aca4f6dc\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") "
Sep 18 16:49:54.094046 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.093866    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/703c8671-9127-4654-8578-92b0aca4f6dc-tmp-dir\") pod \"703c8671-9127-4654-8578-92b0aca4f6dc\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") "
Sep 18 16:49:54.094046 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.093897    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/703c8671-9127-4654-8578-92b0aca4f6dc-model-cache\") pod \"703c8671-9127-4654-8578-92b0aca4f6dc\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") "
Sep 18 16:49:54.094046 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.093940    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/703c8671-9127-4654-8578-92b0aca4f6dc-cabundle-cert\") pod \"703c8671-9127-4654-8578-92b0aca4f6dc\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") "
Sep 18 16:49:54.094046 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.093967    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/703c8671-9127-4654-8578-92b0aca4f6dc-dshm\") pod \"703c8671-9127-4654-8578-92b0aca4f6dc\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") "
Sep 18 16:49:54.094046 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.093994    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fc5t\" (UniqueName: \"kubernetes.io/projected/703c8671-9127-4654-8578-92b0aca4f6dc-kube-api-access-7fc5t\") pod \"703c8671-9127-4654-8578-92b0aca4f6dc\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") "
Sep 18 16:49:54.094325 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.094061    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/703c8671-9127-4654-8578-92b0aca4f6dc-home" (OuterVolumeSpecName: "home") pod "703c8671-9127-4654-8578-92b0aca4f6dc" (UID: "703c8671-9127-4654-8578-92b0aca4f6dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:49:54.094325 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.094126    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/703c8671-9127-4654-8578-92b0aca4f6dc-tls-certs\") pod \"703c8671-9127-4654-8578-92b0aca4f6dc\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") "
Sep 18 16:49:54.094325 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.094139    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/703c8671-9127-4654-8578-92b0aca4f6dc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "703c8671-9127-4654-8578-92b0aca4f6dc" (UID: "703c8671-9127-4654-8578-92b0aca4f6dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:49:54.094325 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.094183    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/703c8671-9127-4654-8578-92b0aca4f6dc-kserve-provision-location\") pod \"703c8671-9127-4654-8578-92b0aca4f6dc\" (UID: \"703c8671-9127-4654-8578-92b0aca4f6dc\") "
Sep 18 16:49:54.094325 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.094189    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/703c8671-9127-4654-8578-92b0aca4f6dc-model-cache" (OuterVolumeSpecName: "model-cache") pod "703c8671-9127-4654-8578-92b0aca4f6dc" (UID: "703c8671-9127-4654-8578-92b0aca4f6dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:49:54.094588 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.094450    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/703c8671-9127-4654-8578-92b0aca4f6dc-home\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:49:54.094588 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.094466    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/703c8671-9127-4654-8578-92b0aca4f6dc-tmp-dir\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:49:54.094588 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.094479    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/703c8671-9127-4654-8578-92b0aca4f6dc-model-cache\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:49:54.094588 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.094445    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/703c8671-9127-4654-8578-92b0aca4f6dc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "703c8671-9127-4654-8578-92b0aca4f6dc" (UID: "703c8671-9127-4654-8578-92b0aca4f6dc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:49:54.096521 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.096497    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/703c8671-9127-4654-8578-92b0aca4f6dc-dshm" (OuterVolumeSpecName: "dshm") pod "703c8671-9127-4654-8578-92b0aca4f6dc" (UID: "703c8671-9127-4654-8578-92b0aca4f6dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:49:54.096638 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.096626    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/703c8671-9127-4654-8578-92b0aca4f6dc-kube-api-access-7fc5t" (OuterVolumeSpecName: "kube-api-access-7fc5t") pod "703c8671-9127-4654-8578-92b0aca4f6dc" (UID: "703c8671-9127-4654-8578-92b0aca4f6dc"). InnerVolumeSpecName "kube-api-access-7fc5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:49:54.096717 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.096701    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/703c8671-9127-4654-8578-92b0aca4f6dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "703c8671-9127-4654-8578-92b0aca4f6dc" (UID: "703c8671-9127-4654-8578-92b0aca4f6dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:49:54.115782 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.115752    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/703c8671-9127-4654-8578-92b0aca4f6dc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "703c8671-9127-4654-8578-92b0aca4f6dc" (UID: "703c8671-9127-4654-8578-92b0aca4f6dc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:49:54.133176 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.133141    2585 generic.go:356] "Generic (PLEG): container finished" podID="703c8671-9127-4654-8578-92b0aca4f6dc" containerID="000b84ea4f5d814c9d90b8b07311d5b4c580ccae39db03f85cd3c68f1895dbb4" exitCode=0
Sep 18 16:49:54.133289 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.133202    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"
Sep 18 16:49:54.133289 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.133218    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp" event={"ID":"703c8671-9127-4654-8578-92b0aca4f6dc","Type":"ContainerDied","Data":"000b84ea4f5d814c9d90b8b07311d5b4c580ccae39db03f85cd3c68f1895dbb4"}
Sep 18 16:49:54.133289 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.133276    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp" event={"ID":"703c8671-9127-4654-8578-92b0aca4f6dc","Type":"ContainerDied","Data":"15131577ac16b917bd1d502cc094c7610c6d47356499a6b6da060b273a601fa5"}
Sep 18 16:49:54.133441 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.133299    2585 scope.go:117] "RemoveContainer" containerID="000b84ea4f5d814c9d90b8b07311d5b4c580ccae39db03f85cd3c68f1895dbb4"
Sep 18 16:49:54.183785 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.181790    2585 scope.go:117] "RemoveContainer" containerID="dfc83dcfcd34e48d9aafa015c5418c3d31ec0a63795d45870eb9830d08aeea98"
Sep 18 16:49:54.192183 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.192155    2585 scope.go:117] "RemoveContainer" containerID="000b84ea4f5d814c9d90b8b07311d5b4c580ccae39db03f85cd3c68f1895dbb4"
Sep 18 16:49:54.192518 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:49:54.192496    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"000b84ea4f5d814c9d90b8b07311d5b4c580ccae39db03f85cd3c68f1895dbb4\": container with ID starting with 000b84ea4f5d814c9d90b8b07311d5b4c580ccae39db03f85cd3c68f1895dbb4 not found: ID does not exist" containerID="000b84ea4f5d814c9d90b8b07311d5b4c580ccae39db03f85cd3c68f1895dbb4"
Sep 18 16:49:54.192603 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.192530    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"000b84ea4f5d814c9d90b8b07311d5b4c580ccae39db03f85cd3c68f1895dbb4"} err="failed to get container status \"000b84ea4f5d814c9d90b8b07311d5b4c580ccae39db03f85cd3c68f1895dbb4\": rpc error: code = NotFound desc = could not find container \"000b84ea4f5d814c9d90b8b07311d5b4c580ccae39db03f85cd3c68f1895dbb4\": container with ID starting with 000b84ea4f5d814c9d90b8b07311d5b4c580ccae39db03f85cd3c68f1895dbb4 not found: ID does not exist"
Sep 18 16:49:54.192603 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.192555    2585 scope.go:117] "RemoveContainer" containerID="dfc83dcfcd34e48d9aafa015c5418c3d31ec0a63795d45870eb9830d08aeea98"
Sep 18 16:49:54.192878 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:49:54.192860    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfc83dcfcd34e48d9aafa015c5418c3d31ec0a63795d45870eb9830d08aeea98\": container with ID starting with dfc83dcfcd34e48d9aafa015c5418c3d31ec0a63795d45870eb9830d08aeea98 not found: ID does not exist" containerID="dfc83dcfcd34e48d9aafa015c5418c3d31ec0a63795d45870eb9830d08aeea98"
Sep 18 16:49:54.193006 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.192885    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfc83dcfcd34e48d9aafa015c5418c3d31ec0a63795d45870eb9830d08aeea98"} err="failed to get container status \"dfc83dcfcd34e48d9aafa015c5418c3d31ec0a63795d45870eb9830d08aeea98\": rpc error: code = NotFound desc = could not find container \"dfc83dcfcd34e48d9aafa015c5418c3d31ec0a63795d45870eb9830d08aeea98\": container with ID starting with dfc83dcfcd34e48d9aafa015c5418c3d31ec0a63795d45870eb9830d08aeea98 not found: ID does not exist"
Sep 18 16:49:54.195238 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.195211    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/703c8671-9127-4654-8578-92b0aca4f6dc-kserve-provision-location\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:49:54.195344 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.195240    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/703c8671-9127-4654-8578-92b0aca4f6dc-cabundle-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:49:54.195344 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.195248    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"]
Sep 18 16:49:54.195344 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.195257    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/703c8671-9127-4654-8578-92b0aca4f6dc-dshm\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:49:54.195344 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.195308    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fc5t\" (UniqueName: \"kubernetes.io/projected/703c8671-9127-4654-8578-92b0aca4f6dc-kube-api-access-7fc5t\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:49:54.195344 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.195324    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/703c8671-9127-4654-8578-92b0aca4f6dc-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:49:54.197040 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.196906    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-5jjcp"]
Sep 18 16:49:54.281704 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.281680    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt"
Sep 18 16:49:54.391844 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.391767    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="703c8671-9127-4654-8578-92b0aca4f6dc" path="/var/lib/kubelet/pods/703c8671-9127-4654-8578-92b0aca4f6dc/volumes"
Sep 18 16:49:54.396795 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.396775    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e84a7350-e841-4af5-9de0-e11cd2405c54-tls-certs\") pod \"e84a7350-e841-4af5-9de0-e11cd2405c54\" (UID: \"e84a7350-e841-4af5-9de0-e11cd2405c54\") "
Sep 18 16:49:54.396943 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.396826    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e84a7350-e841-4af5-9de0-e11cd2405c54-kserve-provision-location\") pod \"e84a7350-e841-4af5-9de0-e11cd2405c54\" (UID: \"e84a7350-e841-4af5-9de0-e11cd2405c54\") "
Sep 18 16:49:54.396943 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.396852    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e84a7350-e841-4af5-9de0-e11cd2405c54-cabundle-cert\") pod \"e84a7350-e841-4af5-9de0-e11cd2405c54\" (UID: \"e84a7350-e841-4af5-9de0-e11cd2405c54\") "
Sep 18 16:49:54.396943 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.396939    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e84a7350-e841-4af5-9de0-e11cd2405c54-tmp-dir\") pod \"e84a7350-e841-4af5-9de0-e11cd2405c54\" (UID: \"e84a7350-e841-4af5-9de0-e11cd2405c54\") "
Sep 18 16:49:54.397225 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.397202    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e84a7350-e841-4af5-9de0-e11cd2405c54-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e84a7350-e841-4af5-9de0-e11cd2405c54" (UID: "e84a7350-e841-4af5-9de0-e11cd2405c54"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:49:54.397314 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.397292    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84a7350-e841-4af5-9de0-e11cd2405c54-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e84a7350-e841-4af5-9de0-e11cd2405c54" (UID: "e84a7350-e841-4af5-9de0-e11cd2405c54"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:49:54.399013 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.398995    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e84a7350-e841-4af5-9de0-e11cd2405c54-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e84a7350-e841-4af5-9de0-e11cd2405c54" (UID: "e84a7350-e841-4af5-9de0-e11cd2405c54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:49:54.417711 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.417689    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84a7350-e841-4af5-9de0-e11cd2405c54-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e84a7350-e841-4af5-9de0-e11cd2405c54" (UID: "e84a7350-e841-4af5-9de0-e11cd2405c54"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:49:54.497803 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.497772    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e84a7350-e841-4af5-9de0-e11cd2405c54-tmp-dir\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:49:54.497803 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.497803    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e84a7350-e841-4af5-9de0-e11cd2405c54-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:49:54.498196 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.497819    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e84a7350-e841-4af5-9de0-e11cd2405c54-kserve-provision-location\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:49:54.498196 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:54.497831    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e84a7350-e841-4af5-9de0-e11cd2405c54-cabundle-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:49:55.138634 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:55.138541    2585 generic.go:356] "Generic (PLEG): container finished" podID="e84a7350-e841-4af5-9de0-e11cd2405c54" containerID="973238a58b5b5ad90c2b0c269444c8d4328a7a120a0859bf666a82e2d496605e" exitCode=0
Sep 18 16:49:55.138808 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:55.138635    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt" event={"ID":"e84a7350-e841-4af5-9de0-e11cd2405c54","Type":"ContainerDied","Data":"973238a58b5b5ad90c2b0c269444c8d4328a7a120a0859bf666a82e2d496605e"}
Sep 18 16:49:55.138808 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:55.138657    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt"
Sep 18 16:49:55.138808 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:55.138724    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt" event={"ID":"e84a7350-e841-4af5-9de0-e11cd2405c54","Type":"ContainerDied","Data":"be94cbf2c03e25a6a1965e11940078d36373709cd25ee23bedbeddf0798156ef"}
Sep 18 16:49:55.138808 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:55.138755    2585 scope.go:117] "RemoveContainer" containerID="973238a58b5b5ad90c2b0c269444c8d4328a7a120a0859bf666a82e2d496605e"
Sep 18 16:49:55.148254 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:55.148240    2585 scope.go:117] "RemoveContainer" containerID="18702861d9b3e08274356edf64e2ef49aa60a2a5acb3feacbb8e6b3cc6596dd8"
Sep 18 16:49:55.155761 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:55.155741    2585 scope.go:117] "RemoveContainer" containerID="973238a58b5b5ad90c2b0c269444c8d4328a7a120a0859bf666a82e2d496605e"
Sep 18 16:49:55.156037 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:49:55.156017    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"973238a58b5b5ad90c2b0c269444c8d4328a7a120a0859bf666a82e2d496605e\": container with ID starting with 973238a58b5b5ad90c2b0c269444c8d4328a7a120a0859bf666a82e2d496605e not found: ID does not exist" containerID="973238a58b5b5ad90c2b0c269444c8d4328a7a120a0859bf666a82e2d496605e"
Sep 18 16:49:55.156109 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:55.156044    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"973238a58b5b5ad90c2b0c269444c8d4328a7a120a0859bf666a82e2d496605e"} err="failed to get container status \"973238a58b5b5ad90c2b0c269444c8d4328a7a120a0859bf666a82e2d496605e\": rpc error: code = NotFound desc = could not find container \"973238a58b5b5ad90c2b0c269444c8d4328a7a120a0859bf666a82e2d496605e\": container with ID starting with 973238a58b5b5ad90c2b0c269444c8d4328a7a120a0859bf666a82e2d496605e not found: ID does not exist"
Sep 18 16:49:55.156109 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:55.156061    2585 scope.go:117] "RemoveContainer" containerID="18702861d9b3e08274356edf64e2ef49aa60a2a5acb3feacbb8e6b3cc6596dd8"
Sep 18 16:49:55.156313 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:49:55.156294    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18702861d9b3e08274356edf64e2ef49aa60a2a5acb3feacbb8e6b3cc6596dd8\": container with ID starting with 18702861d9b3e08274356edf64e2ef49aa60a2a5acb3feacbb8e6b3cc6596dd8 not found: ID does not exist" containerID="18702861d9b3e08274356edf64e2ef49aa60a2a5acb3feacbb8e6b3cc6596dd8"
Sep 18 16:49:55.156361 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:55.156318    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18702861d9b3e08274356edf64e2ef49aa60a2a5acb3feacbb8e6b3cc6596dd8"} err="failed to get container status \"18702861d9b3e08274356edf64e2ef49aa60a2a5acb3feacbb8e6b3cc6596dd8\": rpc error: code = NotFound desc = could not find container \"18702861d9b3e08274356edf64e2ef49aa60a2a5acb3feacbb8e6b3cc6596dd8\": container with ID starting with 18702861d9b3e08274356edf64e2ef49aa60a2a5acb3feacbb8e6b3cc6596dd8 not found: ID does not exist"
Sep 18 16:49:55.161651 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:55.161620    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt"]
Sep 18 16:49:55.167391 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:55.167370    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-54549c4cb9-rg5gt"]
Sep 18 16:49:56.392199 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:49:56.392164    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e84a7350-e841-4af5-9de0-e11cd2405c54" path="/var/lib/kubelet/pods/e84a7350-e841-4af5-9de0-e11cd2405c54/volumes"
Sep 18 16:50:50.319555 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.319521    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"]
Sep 18 16:50:50.320047 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.319797    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e84a7350-e841-4af5-9de0-e11cd2405c54" containerName="storage-initializer"
Sep 18 16:50:50.320047 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.319809    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e84a7350-e841-4af5-9de0-e11cd2405c54" containerName="storage-initializer"
Sep 18 16:50:50.320047 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.319817    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="703c8671-9127-4654-8578-92b0aca4f6dc" containerName="main"
Sep 18 16:50:50.320047 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.319823    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="703c8671-9127-4654-8578-92b0aca4f6dc" containerName="main"
Sep 18 16:50:50.320047 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.319852    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="703c8671-9127-4654-8578-92b0aca4f6dc" containerName="storage-initializer"
Sep 18 16:50:50.320047 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.319858    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="703c8671-9127-4654-8578-92b0aca4f6dc" containerName="storage-initializer"
Sep 18 16:50:50.320047 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.319865    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e84a7350-e841-4af5-9de0-e11cd2405c54" containerName="main"
Sep 18 16:50:50.320047 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.319870    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e84a7350-e841-4af5-9de0-e11cd2405c54" containerName="main"
Sep 18 16:50:50.320047 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.319940    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="703c8671-9127-4654-8578-92b0aca4f6dc" containerName="main"
Sep 18 16:50:50.320047 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.319948    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e84a7350-e841-4af5-9de0-e11cd2405c54" containerName="main"
Sep 18 16:50:50.321656 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.321640    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:50:50.324804 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.324779    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 18 16:50:50.325986 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.325962    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 18 16:50:50.325986 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.325981    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 18 16:50:50.326173 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.325997    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 18 16:50:50.326680 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.326664    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 16:50:50.326945 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.326901    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-xcr42\""
Sep 18 16:50:50.338194 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.338175    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"]
Sep 18 16:50:50.401615 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.401580    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-tls-certs\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-bg92n\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:50:50.401786 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.401740    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-bg92n\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:50:50.401856 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.401787    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-model-cache\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-bg92n\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:50:50.401856 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.401816    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-bg92n\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:50:50.401856 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.401838    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-home\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-bg92n\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:50:50.402028 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.401872    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-dshm\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-bg92n\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:50:50.402028 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.401907    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgp6z\" (UniqueName: \"kubernetes.io/projected/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-kube-api-access-fgp6z\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-bg92n\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:50:50.402135 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.402083    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-bg92n\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:50:50.502608 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.502569    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-bg92n\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:50:50.503964 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.503007    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-tls-certs\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-bg92n\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:50:50.503964 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.503103    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-bg92n\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:50:50.503964 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.503147    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-model-cache\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-bg92n\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:50:50.503964 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.503153    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-bg92n\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:50:50.503964 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.503179    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-bg92n\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:50:50.503964 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.503207    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-home\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-bg92n\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:50:50.503964 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.503237    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-dshm\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-bg92n\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:50:50.503964 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.503279    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgp6z\" (UniqueName: \"kubernetes.io/projected/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-kube-api-access-fgp6z\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-bg92n\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:50:50.503964 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.503638    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-home\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-bg92n\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:50:50.503964 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.503657    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-model-cache\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-bg92n\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:50:50.503964 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.503906    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-bg92n\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:50:50.504890 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.504862    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-bg92n\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:50:50.507347 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.506943    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-dshm\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-bg92n\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:50:50.508211 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.507965    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-tls-certs\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-bg92n\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:50:50.516720 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.516695    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgp6z\" (UniqueName: \"kubernetes.io/projected/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-kube-api-access-fgp6z\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-bg92n\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:50:50.523837 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.523815    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j"]
Sep 18 16:50:50.526391 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.526373    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j"
Sep 18 16:50:50.537873 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.537853    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j"]
Sep 18 16:50:50.604498 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.604433    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af9427d-ba8f-4484-a520-d82bf578b8e1-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j\" (UID: \"0af9427d-ba8f-4484-a520-d82bf578b8e1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j"
Sep 18 16:50:50.604498 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.604464    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0af9427d-ba8f-4484-a520-d82bf578b8e1-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j\" (UID: \"0af9427d-ba8f-4484-a520-d82bf578b8e1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j"
Sep 18 16:50:50.604638 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.604502    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0af9427d-ba8f-4484-a520-d82bf578b8e1-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j\" (UID: \"0af9427d-ba8f-4484-a520-d82bf578b8e1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j"
Sep 18 16:50:50.604638 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.604565    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af9427d-ba8f-4484-a520-d82bf578b8e1-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j\" (UID: \"0af9427d-ba8f-4484-a520-d82bf578b8e1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j"
Sep 18 16:50:50.632191 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.632166    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:50:50.705959 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.705830    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0af9427d-ba8f-4484-a520-d82bf578b8e1-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j\" (UID: \"0af9427d-ba8f-4484-a520-d82bf578b8e1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j"
Sep 18 16:50:50.705959 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.705882    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af9427d-ba8f-4484-a520-d82bf578b8e1-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j\" (UID: \"0af9427d-ba8f-4484-a520-d82bf578b8e1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j"
Sep 18 16:50:50.706186 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.705975    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af9427d-ba8f-4484-a520-d82bf578b8e1-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j\" (UID: \"0af9427d-ba8f-4484-a520-d82bf578b8e1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j"
Sep 18 16:50:50.706186 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.706000    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0af9427d-ba8f-4484-a520-d82bf578b8e1-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j\" (UID: \"0af9427d-ba8f-4484-a520-d82bf578b8e1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j"
Sep 18 16:50:50.706367 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.706347    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0af9427d-ba8f-4484-a520-d82bf578b8e1-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j\" (UID: \"0af9427d-ba8f-4484-a520-d82bf578b8e1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j"
Sep 18 16:50:50.706474 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.706442    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af9427d-ba8f-4484-a520-d82bf578b8e1-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j\" (UID: \"0af9427d-ba8f-4484-a520-d82bf578b8e1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j"
Sep 18 16:50:50.706637 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.706611    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0af9427d-ba8f-4484-a520-d82bf578b8e1-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j\" (UID: \"0af9427d-ba8f-4484-a520-d82bf578b8e1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j"
Sep 18 16:50:50.709033 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.709011    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af9427d-ba8f-4484-a520-d82bf578b8e1-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j\" (UID: \"0af9427d-ba8f-4484-a520-d82bf578b8e1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j"
Sep 18 16:50:50.836775 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.836742    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j"
Sep 18 16:50:50.908236 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:50.908209    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"]
Sep 18 16:50:50.910084 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:50:50.910052    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe67ad75_2476_4c83_a9ba_cd2cbc5751e6.slice/crio-9f609ad1d5adf4b202f7b99d6b23c541d8668940a421c199739486ffe44ffccc WatchSource:0}: Error finding container 9f609ad1d5adf4b202f7b99d6b23c541d8668940a421c199739486ffe44ffccc: Status 404 returned error can't find the container with id 9f609ad1d5adf4b202f7b99d6b23c541d8668940a421c199739486ffe44ffccc
Sep 18 16:50:51.106871 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:51.106846    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j"]
Sep 18 16:50:51.109287 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:50:51.109255    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af9427d_ba8f_4484_a520_d82bf578b8e1.slice/crio-cf047d6c3ac1caca69cb5cb86d9989751fa24b86f9df5174d55d74024c1e455d WatchSource:0}: Error finding container cf047d6c3ac1caca69cb5cb86d9989751fa24b86f9df5174d55d74024c1e455d: Status 404 returned error can't find the container with id cf047d6c3ac1caca69cb5cb86d9989751fa24b86f9df5174d55d74024c1e455d
Sep 18 16:50:51.375750 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:51.375707    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j" event={"ID":"0af9427d-ba8f-4484-a520-d82bf578b8e1","Type":"ContainerStarted","Data":"da64cb06c6b916f29da9296d7fb6b18d609a9b28c3f4c148034a72efc4d7bb9d"}
Sep 18 16:50:51.376224 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:51.375776    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j" event={"ID":"0af9427d-ba8f-4484-a520-d82bf578b8e1","Type":"ContainerStarted","Data":"cf047d6c3ac1caca69cb5cb86d9989751fa24b86f9df5174d55d74024c1e455d"}
Sep 18 16:50:51.377745 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:51.377716    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n" event={"ID":"be67ad75-2476-4c83-a9ba-cd2cbc5751e6","Type":"ContainerStarted","Data":"13b72c956de44809f33e8a0124f559538d070b261030b0614d330d76951dcc38"}
Sep 18 16:50:51.377854 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:51.377773    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n" event={"ID":"be67ad75-2476-4c83-a9ba-cd2cbc5751e6","Type":"ContainerStarted","Data":"9f609ad1d5adf4b202f7b99d6b23c541d8668940a421c199739486ffe44ffccc"}
Sep 18 16:50:54.392692 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:54.392660    2585 generic.go:356] "Generic (PLEG): container finished" podID="0af9427d-ba8f-4484-a520-d82bf578b8e1" containerID="da64cb06c6b916f29da9296d7fb6b18d609a9b28c3f4c148034a72efc4d7bb9d" exitCode=0
Sep 18 16:50:54.393118 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:54.392726    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j" event={"ID":"0af9427d-ba8f-4484-a520-d82bf578b8e1","Type":"ContainerDied","Data":"da64cb06c6b916f29da9296d7fb6b18d609a9b28c3f4c148034a72efc4d7bb9d"}
Sep 18 16:50:54.394190 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:54.394169    2585 generic.go:356] "Generic (PLEG): container finished" podID="be67ad75-2476-4c83-a9ba-cd2cbc5751e6" containerID="13b72c956de44809f33e8a0124f559538d070b261030b0614d330d76951dcc38" exitCode=0
Sep 18 16:50:54.394283 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:54.394242    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n" event={"ID":"be67ad75-2476-4c83-a9ba-cd2cbc5751e6","Type":"ContainerDied","Data":"13b72c956de44809f33e8a0124f559538d070b261030b0614d330d76951dcc38"}
Sep 18 16:50:55.400650 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:55.400616    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j" event={"ID":"0af9427d-ba8f-4484-a520-d82bf578b8e1","Type":"ContainerStarted","Data":"5474a88251ceb86a1a41759442998dd331d241f10031a0f9a8767b2ef796406d"}
Sep 18 16:50:55.401064 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:55.400865    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j"
Sep 18 16:50:55.402818 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:55.402792    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n" event={"ID":"be67ad75-2476-4c83-a9ba-cd2cbc5751e6","Type":"ContainerStarted","Data":"942f68386a13aae539ba378d71b6aa099d77f91580fba991816a32f409cc7786"}
Sep 18 16:50:55.419323 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:55.419277    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j" podStartSLOduration=2.9241555999999997 podStartE2EDuration="5.419263771s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.495108171s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:50:55.417658981 +0000 UTC m=+2095.200661031" watchObservedRunningTime="2026-09-18 16:50:55.419263771 +0000 UTC m=+2095.202265795"
Sep 18 16:50:55.437588 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:55.437540    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n" podStartSLOduration=2.9590093729999998 podStartE2EDuration="5.437522868s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.478513495s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:50:55.435553501 +0000 UTC m=+2095.218555524" watchObservedRunningTime="2026-09-18 16:50:55.437522868 +0000 UTC m=+2095.220524895"
Sep 18 16:50:55.632807 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:50:55.632779    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:51:00.633340 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:00.633305    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:51:00.646230 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:00.646203    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:51:01.438013 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:01.437983    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:51:02.444425 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:02.444396    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 16:51:02.445286 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:02.445269    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 16:51:02.448433 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:02.448413    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 16:51:02.449193 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:02.449169    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 16:51:26.417353 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:26.417323    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j"
Sep 18 16:51:42.212006 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:42.211903    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j_0af9427d-ba8f-4484-a520-d82bf578b8e1/main/0.log"
Sep 18 16:51:43.847015 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:43.846972    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j"]
Sep 18 16:51:43.847453 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:43.847251    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j" podUID="0af9427d-ba8f-4484-a520-d82bf578b8e1" containerName="main" containerID="cri-o://5474a88251ceb86a1a41759442998dd331d241f10031a0f9a8767b2ef796406d" gracePeriod=30
Sep 18 16:51:43.855799 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:43.855771    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"]
Sep 18 16:51:43.864791 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:51:43.864759    2585 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 16:51:43.864791 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:51:43.864791 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 16:51:43.864791 ip-10-0-133-147 kubenswrapper[2585]:  > containerID="942f68386a13aae539ba378d71b6aa099d77f91580fba991816a32f409cc7786" cmd=["/bin/sleep","15"]
Sep 18 16:51:43.865102 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:43.864809    2585 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n" message="" err=<
Sep 18 16:51:43.865102 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:51:43.865102 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 16:51:43.865102 ip-10-0-133-147 kubenswrapper[2585]:  >
Sep 18 16:51:43.865102 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:51:43.864822    2585 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 16:51:43.865102 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:51:43.865102 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 16:51:43.865102 ip-10-0-133-147 kubenswrapper[2585]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n" podUID="be67ad75-2476-4c83-a9ba-cd2cbc5751e6" containerName="main" containerID="cri-o://942f68386a13aae539ba378d71b6aa099d77f91580fba991816a32f409cc7786"
Sep 18 16:51:43.865102 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:43.864855    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n" podUID="be67ad75-2476-4c83-a9ba-cd2cbc5751e6" containerName="main" containerID="cri-o://942f68386a13aae539ba378d71b6aa099d77f91580fba991816a32f409cc7786" gracePeriod=60
Sep 18 16:51:44.213444 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.213413    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:51:44.258593 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.258561    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-tls-certs\") pod \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") "
Sep 18 16:51:44.258593 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.258596    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-tmp-dir\") pod \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") "
Sep 18 16:51:44.258782 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.258643    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-kserve-provision-location\") pod \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") "
Sep 18 16:51:44.258782 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.258746    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgp6z\" (UniqueName: \"kubernetes.io/projected/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-kube-api-access-fgp6z\") pod \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") "
Sep 18 16:51:44.258866 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.258796    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-dshm\") pod \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") "
Sep 18 16:51:44.258866 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.258836    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-cabundle-cert\") pod \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") "
Sep 18 16:51:44.258866 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.258840    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "be67ad75-2476-4c83-a9ba-cd2cbc5751e6" (UID: "be67ad75-2476-4c83-a9ba-cd2cbc5751e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:51:44.259040 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.258879    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-model-cache\") pod \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") "
Sep 18 16:51:44.259040 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.258908    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-home\") pod \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\" (UID: \"be67ad75-2476-4c83-a9ba-cd2cbc5751e6\") "
Sep 18 16:51:44.259170 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.259154    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-tmp-dir\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:51:44.259318 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.259280    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-model-cache" (OuterVolumeSpecName: "model-cache") pod "be67ad75-2476-4c83-a9ba-cd2cbc5751e6" (UID: "be67ad75-2476-4c83-a9ba-cd2cbc5751e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:51:44.259388 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.259354    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "be67ad75-2476-4c83-a9ba-cd2cbc5751e6" (UID: "be67ad75-2476-4c83-a9ba-cd2cbc5751e6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:51:44.259579 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.259560    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-home" (OuterVolumeSpecName: "home") pod "be67ad75-2476-4c83-a9ba-cd2cbc5751e6" (UID: "be67ad75-2476-4c83-a9ba-cd2cbc5751e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:51:44.261187 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.261155    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "be67ad75-2476-4c83-a9ba-cd2cbc5751e6" (UID: "be67ad75-2476-4c83-a9ba-cd2cbc5751e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:51:44.261395 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.261371    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-dshm" (OuterVolumeSpecName: "dshm") pod "be67ad75-2476-4c83-a9ba-cd2cbc5751e6" (UID: "be67ad75-2476-4c83-a9ba-cd2cbc5751e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:51:44.261474 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.261392    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-kube-api-access-fgp6z" (OuterVolumeSpecName: "kube-api-access-fgp6z") pod "be67ad75-2476-4c83-a9ba-cd2cbc5751e6" (UID: "be67ad75-2476-4c83-a9ba-cd2cbc5751e6"). InnerVolumeSpecName "kube-api-access-fgp6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:51:44.281110 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.281070    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "be67ad75-2476-4c83-a9ba-cd2cbc5751e6" (UID: "be67ad75-2476-4c83-a9ba-cd2cbc5751e6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:51:44.360409 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.360366    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:51:44.360409 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.360409    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-kserve-provision-location\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:51:44.360595 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.360425    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgp6z\" (UniqueName: \"kubernetes.io/projected/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-kube-api-access-fgp6z\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:51:44.360595 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.360439    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-dshm\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:51:44.360595 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.360453    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-cabundle-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:51:44.360595 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.360466    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-model-cache\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:51:44.360595 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.360479    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be67ad75-2476-4c83-a9ba-cd2cbc5751e6-home\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:51:44.490459 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.490434    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j"
Sep 18 16:51:44.561437 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.561395    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0af9427d-ba8f-4484-a520-d82bf578b8e1-kserve-provision-location\") pod \"0af9427d-ba8f-4484-a520-d82bf578b8e1\" (UID: \"0af9427d-ba8f-4484-a520-d82bf578b8e1\") "
Sep 18 16:51:44.561437 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.561430    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af9427d-ba8f-4484-a520-d82bf578b8e1-tls-certs\") pod \"0af9427d-ba8f-4484-a520-d82bf578b8e1\" (UID: \"0af9427d-ba8f-4484-a520-d82bf578b8e1\") "
Sep 18 16:51:44.561697 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.561450    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af9427d-ba8f-4484-a520-d82bf578b8e1-tmp-dir\") pod \"0af9427d-ba8f-4484-a520-d82bf578b8e1\" (UID: \"0af9427d-ba8f-4484-a520-d82bf578b8e1\") "
Sep 18 16:51:44.561697 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.561525    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0af9427d-ba8f-4484-a520-d82bf578b8e1-cabundle-cert\") pod \"0af9427d-ba8f-4484-a520-d82bf578b8e1\" (UID: \"0af9427d-ba8f-4484-a520-d82bf578b8e1\") "
Sep 18 16:51:44.561999 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.561973    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af9427d-ba8f-4484-a520-d82bf578b8e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0af9427d-ba8f-4484-a520-d82bf578b8e1" (UID: "0af9427d-ba8f-4484-a520-d82bf578b8e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:51:44.562123 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.561973    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af9427d-ba8f-4484-a520-d82bf578b8e1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0af9427d-ba8f-4484-a520-d82bf578b8e1" (UID: "0af9427d-ba8f-4484-a520-d82bf578b8e1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:51:44.564444 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.564422    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0af9427d-ba8f-4484-a520-d82bf578b8e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0af9427d-ba8f-4484-a520-d82bf578b8e1" (UID: "0af9427d-ba8f-4484-a520-d82bf578b8e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:51:44.595430 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.595393    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af9427d-ba8f-4484-a520-d82bf578b8e1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0af9427d-ba8f-4484-a520-d82bf578b8e1" (UID: "0af9427d-ba8f-4484-a520-d82bf578b8e1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:51:44.600623 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.600597    2585 generic.go:356] "Generic (PLEG): container finished" podID="0af9427d-ba8f-4484-a520-d82bf578b8e1" containerID="5474a88251ceb86a1a41759442998dd331d241f10031a0f9a8767b2ef796406d" exitCode=0
Sep 18 16:51:44.600721 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.600651    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j"
Sep 18 16:51:44.600721 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.600672    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j" event={"ID":"0af9427d-ba8f-4484-a520-d82bf578b8e1","Type":"ContainerDied","Data":"5474a88251ceb86a1a41759442998dd331d241f10031a0f9a8767b2ef796406d"}
Sep 18 16:51:44.600825 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.600742    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j" event={"ID":"0af9427d-ba8f-4484-a520-d82bf578b8e1","Type":"ContainerDied","Data":"cf047d6c3ac1caca69cb5cb86d9989751fa24b86f9df5174d55d74024c1e455d"}
Sep 18 16:51:44.600825 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.600777    2585 scope.go:117] "RemoveContainer" containerID="5474a88251ceb86a1a41759442998dd331d241f10031a0f9a8767b2ef796406d"
Sep 18 16:51:44.602607 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.602584    2585 generic.go:356] "Generic (PLEG): container finished" podID="be67ad75-2476-4c83-a9ba-cd2cbc5751e6" containerID="942f68386a13aae539ba378d71b6aa099d77f91580fba991816a32f409cc7786" exitCode=0
Sep 18 16:51:44.602708 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.602638    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n" event={"ID":"be67ad75-2476-4c83-a9ba-cd2cbc5751e6","Type":"ContainerDied","Data":"942f68386a13aae539ba378d71b6aa099d77f91580fba991816a32f409cc7786"}
Sep 18 16:51:44.602708 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.602670    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n" event={"ID":"be67ad75-2476-4c83-a9ba-cd2cbc5751e6","Type":"ContainerDied","Data":"9f609ad1d5adf4b202f7b99d6b23c541d8668940a421c199739486ffe44ffccc"}
Sep 18 16:51:44.602708 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.602640    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"
Sep 18 16:51:44.612964 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.612949    2585 scope.go:117] "RemoveContainer" containerID="da64cb06c6b916f29da9296d7fb6b18d609a9b28c3f4c148034a72efc4d7bb9d"
Sep 18 16:51:44.619742 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.619720    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"]
Sep 18 16:51:44.622003 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.621981    2585 scope.go:117] "RemoveContainer" containerID="5474a88251ceb86a1a41759442998dd331d241f10031a0f9a8767b2ef796406d"
Sep 18 16:51:44.622265 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:51:44.622248    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5474a88251ceb86a1a41759442998dd331d241f10031a0f9a8767b2ef796406d\": container with ID starting with 5474a88251ceb86a1a41759442998dd331d241f10031a0f9a8767b2ef796406d not found: ID does not exist" containerID="5474a88251ceb86a1a41759442998dd331d241f10031a0f9a8767b2ef796406d"
Sep 18 16:51:44.622315 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.622275    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5474a88251ceb86a1a41759442998dd331d241f10031a0f9a8767b2ef796406d"} err="failed to get container status \"5474a88251ceb86a1a41759442998dd331d241f10031a0f9a8767b2ef796406d\": rpc error: code = NotFound desc = could not find container \"5474a88251ceb86a1a41759442998dd331d241f10031a0f9a8767b2ef796406d\": container with ID starting with 5474a88251ceb86a1a41759442998dd331d241f10031a0f9a8767b2ef796406d not found: ID does not exist"
Sep 18 16:51:44.622315 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.622292    2585 scope.go:117] "RemoveContainer" containerID="da64cb06c6b916f29da9296d7fb6b18d609a9b28c3f4c148034a72efc4d7bb9d"
Sep 18 16:51:44.622537 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:51:44.622517    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da64cb06c6b916f29da9296d7fb6b18d609a9b28c3f4c148034a72efc4d7bb9d\": container with ID starting with da64cb06c6b916f29da9296d7fb6b18d609a9b28c3f4c148034a72efc4d7bb9d not found: ID does not exist" containerID="da64cb06c6b916f29da9296d7fb6b18d609a9b28c3f4c148034a72efc4d7bb9d"
Sep 18 16:51:44.622603 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.622547    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da64cb06c6b916f29da9296d7fb6b18d609a9b28c3f4c148034a72efc4d7bb9d"} err="failed to get container status \"da64cb06c6b916f29da9296d7fb6b18d609a9b28c3f4c148034a72efc4d7bb9d\": rpc error: code = NotFound desc = could not find container \"da64cb06c6b916f29da9296d7fb6b18d609a9b28c3f4c148034a72efc4d7bb9d\": container with ID starting with da64cb06c6b916f29da9296d7fb6b18d609a9b28c3f4c148034a72efc4d7bb9d not found: ID does not exist"
Sep 18 16:51:44.622603 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.622570    2585 scope.go:117] "RemoveContainer" containerID="942f68386a13aae539ba378d71b6aa099d77f91580fba991816a32f409cc7786"
Sep 18 16:51:44.627961 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.627939    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-bg92n"]
Sep 18 16:51:44.630926 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.630888    2585 scope.go:117] "RemoveContainer" containerID="13b72c956de44809f33e8a0124f559538d070b261030b0614d330d76951dcc38"
Sep 18 16:51:44.638506 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.638488    2585 scope.go:117] "RemoveContainer" containerID="942f68386a13aae539ba378d71b6aa099d77f91580fba991816a32f409cc7786"
Sep 18 16:51:44.638718 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.638703    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j"]
Sep 18 16:51:44.638781 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:51:44.638749    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"942f68386a13aae539ba378d71b6aa099d77f91580fba991816a32f409cc7786\": container with ID starting with 942f68386a13aae539ba378d71b6aa099d77f91580fba991816a32f409cc7786 not found: ID does not exist" containerID="942f68386a13aae539ba378d71b6aa099d77f91580fba991816a32f409cc7786"
Sep 18 16:51:44.638781 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.638767    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"942f68386a13aae539ba378d71b6aa099d77f91580fba991816a32f409cc7786"} err="failed to get container status \"942f68386a13aae539ba378d71b6aa099d77f91580fba991816a32f409cc7786\": rpc error: code = NotFound desc = could not find container \"942f68386a13aae539ba378d71b6aa099d77f91580fba991816a32f409cc7786\": container with ID starting with 942f68386a13aae539ba378d71b6aa099d77f91580fba991816a32f409cc7786 not found: ID does not exist"
Sep 18 16:51:44.638859 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.638783    2585 scope.go:117] "RemoveContainer" containerID="13b72c956de44809f33e8a0124f559538d070b261030b0614d330d76951dcc38"
Sep 18 16:51:44.639037 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:51:44.639019    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13b72c956de44809f33e8a0124f559538d070b261030b0614d330d76951dcc38\": container with ID starting with 13b72c956de44809f33e8a0124f559538d070b261030b0614d330d76951dcc38 not found: ID does not exist" containerID="13b72c956de44809f33e8a0124f559538d070b261030b0614d330d76951dcc38"
Sep 18 16:51:44.639097 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.639042    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13b72c956de44809f33e8a0124f559538d070b261030b0614d330d76951dcc38"} err="failed to get container status \"13b72c956de44809f33e8a0124f559538d070b261030b0614d330d76951dcc38\": rpc error: code = NotFound desc = could not find container \"13b72c956de44809f33e8a0124f559538d070b261030b0614d330d76951dcc38\": container with ID starting with 13b72c956de44809f33e8a0124f559538d070b261030b0614d330d76951dcc38 not found: ID does not exist"
Sep 18 16:51:44.644496 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.644475    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-84ff9bbb9f-dq29j"]
Sep 18 16:51:44.662954 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.662884    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0af9427d-ba8f-4484-a520-d82bf578b8e1-kserve-provision-location\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:51:44.662954 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.662902    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af9427d-ba8f-4484-a520-d82bf578b8e1-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:51:44.662954 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.662929    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af9427d-ba8f-4484-a520-d82bf578b8e1-tmp-dir\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:51:44.662954 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:44.662939    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0af9427d-ba8f-4484-a520-d82bf578b8e1-cabundle-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:51:46.396774 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:46.396734    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0af9427d-ba8f-4484-a520-d82bf578b8e1" path="/var/lib/kubelet/pods/0af9427d-ba8f-4484-a520-d82bf578b8e1/volumes"
Sep 18 16:51:46.397411 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:51:46.397395    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be67ad75-2476-4c83-a9ba-cd2cbc5751e6" path="/var/lib/kubelet/pods/be67ad75-2476-4c83-a9ba-cd2cbc5751e6/volumes"
Sep 18 16:52:43.265823 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.265781    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"]
Sep 18 16:52:43.266305 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.266240    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0af9427d-ba8f-4484-a520-d82bf578b8e1" containerName="storage-initializer"
Sep 18 16:52:43.266305 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.266260    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0af9427d-ba8f-4484-a520-d82bf578b8e1" containerName="storage-initializer"
Sep 18 16:52:43.266305 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.266286    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be67ad75-2476-4c83-a9ba-cd2cbc5751e6" containerName="main"
Sep 18 16:52:43.266305 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.266297    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="be67ad75-2476-4c83-a9ba-cd2cbc5751e6" containerName="main"
Sep 18 16:52:43.266437 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.266312    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be67ad75-2476-4c83-a9ba-cd2cbc5751e6" containerName="storage-initializer"
Sep 18 16:52:43.266437 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.266321    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="be67ad75-2476-4c83-a9ba-cd2cbc5751e6" containerName="storage-initializer"
Sep 18 16:52:43.266437 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.266335    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0af9427d-ba8f-4484-a520-d82bf578b8e1" containerName="main"
Sep 18 16:52:43.266437 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.266343    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0af9427d-ba8f-4484-a520-d82bf578b8e1" containerName="main"
Sep 18 16:52:43.266437 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.266415    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0af9427d-ba8f-4484-a520-d82bf578b8e1" containerName="main"
Sep 18 16:52:43.266437 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.266425    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="be67ad75-2476-4c83-a9ba-cd2cbc5751e6" containerName="main"
Sep 18 16:52:43.269670 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.269653    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:52:43.272287 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.272266    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 16:52:43.272445 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.272374    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 18 16:52:43.272582 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.272565    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 18 16:52:43.273613 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.273594    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-h9l5s\""
Sep 18 16:52:43.273834 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.273602    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 18 16:52:43.273949 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.273613    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 18 16:52:43.280303 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.280279    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"]
Sep 18 16:52:43.312830 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.312803    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da70e390-eb1e-4a6b-a605-e0ad48beec01-home\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-dvtnx\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:52:43.312975 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.312851    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lpdk\" (UniqueName: \"kubernetes.io/projected/da70e390-eb1e-4a6b-a605-e0ad48beec01-kube-api-access-4lpdk\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-dvtnx\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:52:43.312975 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.312948    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da70e390-eb1e-4a6b-a605-e0ad48beec01-dshm\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-dvtnx\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:52:43.313066 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.312986    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da70e390-eb1e-4a6b-a605-e0ad48beec01-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-dvtnx\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:52:43.313066 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.313018    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da70e390-eb1e-4a6b-a605-e0ad48beec01-model-cache\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-dvtnx\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:52:43.313066 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.313033    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da70e390-eb1e-4a6b-a605-e0ad48beec01-tls-certs\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-dvtnx\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:52:43.313207 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.313097    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da70e390-eb1e-4a6b-a605-e0ad48beec01-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-dvtnx\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:52:43.313207 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.313139    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da70e390-eb1e-4a6b-a605-e0ad48beec01-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-dvtnx\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:52:43.414246 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.414221    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da70e390-eb1e-4a6b-a605-e0ad48beec01-model-cache\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-dvtnx\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:52:43.414360 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.414249    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da70e390-eb1e-4a6b-a605-e0ad48beec01-tls-certs\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-dvtnx\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:52:43.414360 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.414276    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da70e390-eb1e-4a6b-a605-e0ad48beec01-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-dvtnx\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:52:43.414360 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.414305    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da70e390-eb1e-4a6b-a605-e0ad48beec01-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-dvtnx\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:52:43.414360 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.414338    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da70e390-eb1e-4a6b-a605-e0ad48beec01-home\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-dvtnx\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:52:43.414548 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.414385    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lpdk\" (UniqueName: \"kubernetes.io/projected/da70e390-eb1e-4a6b-a605-e0ad48beec01-kube-api-access-4lpdk\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-dvtnx\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:52:43.414548 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.414426    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da70e390-eb1e-4a6b-a605-e0ad48beec01-dshm\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-dvtnx\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:52:43.414702 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.414673    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da70e390-eb1e-4a6b-a605-e0ad48beec01-model-cache\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-dvtnx\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:52:43.414831 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.414713    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da70e390-eb1e-4a6b-a605-e0ad48beec01-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-dvtnx\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:52:43.414831 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.414727    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da70e390-eb1e-4a6b-a605-e0ad48beec01-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-dvtnx\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:52:43.414831 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.414745    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da70e390-eb1e-4a6b-a605-e0ad48beec01-home\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-dvtnx\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:52:43.415064 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.415044    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da70e390-eb1e-4a6b-a605-e0ad48beec01-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-dvtnx\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:52:43.415285 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.415266    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da70e390-eb1e-4a6b-a605-e0ad48beec01-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-dvtnx\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:52:43.416797 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.416777    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da70e390-eb1e-4a6b-a605-e0ad48beec01-dshm\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-dvtnx\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:52:43.417017 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.416998    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da70e390-eb1e-4a6b-a605-e0ad48beec01-tls-certs\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-dvtnx\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:52:43.422566 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.422536    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lpdk\" (UniqueName: \"kubernetes.io/projected/da70e390-eb1e-4a6b-a605-e0ad48beec01-kube-api-access-4lpdk\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-dvtnx\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:52:43.490342 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.490313    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9"]
Sep 18 16:52:43.495334 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.495315    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9"
Sep 18 16:52:43.505687 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.505664    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9"]
Sep 18 16:52:43.515904 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.515853    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2cfedea-f20b-42ba-9264-df054e1cc4aa-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9\" (UID: \"d2cfedea-f20b-42ba-9264-df054e1cc4aa\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9"
Sep 18 16:52:43.515904 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.515889    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2cfedea-f20b-42ba-9264-df054e1cc4aa-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9\" (UID: \"d2cfedea-f20b-42ba-9264-df054e1cc4aa\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9"
Sep 18 16:52:43.516049 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.515956    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2cfedea-f20b-42ba-9264-df054e1cc4aa-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9\" (UID: \"d2cfedea-f20b-42ba-9264-df054e1cc4aa\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9"
Sep 18 16:52:43.516049 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.515995    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2cfedea-f20b-42ba-9264-df054e1cc4aa-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9\" (UID: \"d2cfedea-f20b-42ba-9264-df054e1cc4aa\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9"
Sep 18 16:52:43.582341 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.582307    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:52:43.617343 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.617308    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2cfedea-f20b-42ba-9264-df054e1cc4aa-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9\" (UID: \"d2cfedea-f20b-42ba-9264-df054e1cc4aa\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9"
Sep 18 16:52:43.617509 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.617364    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2cfedea-f20b-42ba-9264-df054e1cc4aa-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9\" (UID: \"d2cfedea-f20b-42ba-9264-df054e1cc4aa\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9"
Sep 18 16:52:43.617509 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.617467    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2cfedea-f20b-42ba-9264-df054e1cc4aa-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9\" (UID: \"d2cfedea-f20b-42ba-9264-df054e1cc4aa\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9"
Sep 18 16:52:43.617509 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.617499    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2cfedea-f20b-42ba-9264-df054e1cc4aa-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9\" (UID: \"d2cfedea-f20b-42ba-9264-df054e1cc4aa\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9"
Sep 18 16:52:43.617788 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.617762    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2cfedea-f20b-42ba-9264-df054e1cc4aa-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9\" (UID: \"d2cfedea-f20b-42ba-9264-df054e1cc4aa\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9"
Sep 18 16:52:43.618214 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.618179    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2cfedea-f20b-42ba-9264-df054e1cc4aa-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9\" (UID: \"d2cfedea-f20b-42ba-9264-df054e1cc4aa\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9"
Sep 18 16:52:43.618214 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.618201    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2cfedea-f20b-42ba-9264-df054e1cc4aa-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9\" (UID: \"d2cfedea-f20b-42ba-9264-df054e1cc4aa\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9"
Sep 18 16:52:43.621726 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.621677    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2cfedea-f20b-42ba-9264-df054e1cc4aa-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9\" (UID: \"d2cfedea-f20b-42ba-9264-df054e1cc4aa\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9"
Sep 18 16:52:43.806631 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.806546    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9"
Sep 18 16:52:43.863614 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:43.861395    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"]
Sep 18 16:52:43.867134 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:52:43.867109    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda70e390_eb1e_4a6b_a605_e0ad48beec01.slice/crio-cf0812fc6805bb85163b8a80e0bfaa75c4e977fb48f602fe87241e0c405c9ae2 WatchSource:0}: Error finding container cf0812fc6805bb85163b8a80e0bfaa75c4e977fb48f602fe87241e0c405c9ae2: Status 404 returned error can't find the container with id cf0812fc6805bb85163b8a80e0bfaa75c4e977fb48f602fe87241e0c405c9ae2
Sep 18 16:52:44.078296 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:44.078269    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9"]
Sep 18 16:52:44.081536 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:52:44.081503    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2cfedea_f20b_42ba_9264_df054e1cc4aa.slice/crio-b22ab53edabc519c2015da63bbfded1089157b5bcb95418ba8b4f8457e48aeae WatchSource:0}: Error finding container b22ab53edabc519c2015da63bbfded1089157b5bcb95418ba8b4f8457e48aeae: Status 404 returned error can't find the container with id b22ab53edabc519c2015da63bbfded1089157b5bcb95418ba8b4f8457e48aeae
Sep 18 16:52:44.841732 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:44.841691    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9" event={"ID":"d2cfedea-f20b-42ba-9264-df054e1cc4aa","Type":"ContainerStarted","Data":"9dbcd5c46390c76c8c1600705640c7cbe9a9693ae80fb3b2a9864d46e3a8df4e"}
Sep 18 16:52:44.842260 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:44.841758    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9" event={"ID":"d2cfedea-f20b-42ba-9264-df054e1cc4aa","Type":"ContainerStarted","Data":"b22ab53edabc519c2015da63bbfded1089157b5bcb95418ba8b4f8457e48aeae"}
Sep 18 16:52:44.844031 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:44.844003    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx" event={"ID":"da70e390-eb1e-4a6b-a605-e0ad48beec01","Type":"ContainerStarted","Data":"88d3efd8568375357dc5b453da7ad5d3ed7fe400b851e6d3e92a3eca8f1bc860"}
Sep 18 16:52:44.844155 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:44.844054    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx" event={"ID":"da70e390-eb1e-4a6b-a605-e0ad48beec01","Type":"ContainerStarted","Data":"cf0812fc6805bb85163b8a80e0bfaa75c4e977fb48f602fe87241e0c405c9ae2"}
Sep 18 16:52:46.854665 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:46.854633    2585 generic.go:356] "Generic (PLEG): container finished" podID="d2cfedea-f20b-42ba-9264-df054e1cc4aa" containerID="9dbcd5c46390c76c8c1600705640c7cbe9a9693ae80fb3b2a9864d46e3a8df4e" exitCode=0
Sep 18 16:52:46.855148 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:46.854715    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9" event={"ID":"d2cfedea-f20b-42ba-9264-df054e1cc4aa","Type":"ContainerDied","Data":"9dbcd5c46390c76c8c1600705640c7cbe9a9693ae80fb3b2a9864d46e3a8df4e"}
Sep 18 16:52:46.856331 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:46.856308    2585 generic.go:356] "Generic (PLEG): container finished" podID="da70e390-eb1e-4a6b-a605-e0ad48beec01" containerID="88d3efd8568375357dc5b453da7ad5d3ed7fe400b851e6d3e92a3eca8f1bc860" exitCode=0
Sep 18 16:52:46.856415 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:46.856342    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx" event={"ID":"da70e390-eb1e-4a6b-a605-e0ad48beec01","Type":"ContainerDied","Data":"88d3efd8568375357dc5b453da7ad5d3ed7fe400b851e6d3e92a3eca8f1bc860"}
Sep 18 16:52:47.862232 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:47.862192    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9" event={"ID":"d2cfedea-f20b-42ba-9264-df054e1cc4aa","Type":"ContainerStarted","Data":"03bf3f8cf1334cc4a187eb07e998f9d9ac0122cfb5a0c0b3cb75b6cee25e2591"}
Sep 18 16:52:47.862664 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:47.862415    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9"
Sep 18 16:52:47.864240 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:47.864210    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx" event={"ID":"da70e390-eb1e-4a6b-a605-e0ad48beec01","Type":"ContainerStarted","Data":"3b29d8960720556a2df88efe8bed33283526b909b894db8c7b8b8a29d43cd5b8"}
Sep 18 16:52:47.884614 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:47.884569    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9" podStartSLOduration=2.453034603 podStartE2EDuration="4.884555663s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.43152106s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:52:47.882138886 +0000 UTC m=+2207.665140904" watchObservedRunningTime="2026-09-18 16:52:47.884555663 +0000 UTC m=+2207.667557686"
Sep 18 16:52:47.902657 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:47.902605    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx" podStartSLOduration=2.407540665 podStartE2EDuration="4.902594633s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.495053968s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:52:47.902291585 +0000 UTC m=+2207.685293611" watchObservedRunningTime="2026-09-18 16:52:47.902594633 +0000 UTC m=+2207.685596658"
Sep 18 16:52:48.582848 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:48.582808    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:52:53.582642 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:53.582605    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:52:53.601650 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:53.601612    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:52:53.899604 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:52:53.899528    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:53:18.878978 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:18.878890    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9"
Sep 18 16:53:20.011889 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:20.011855    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9_d2cfedea-f20b-42ba-9264-df054e1cc4aa/main/0.log"
Sep 18 16:53:20.729751 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:20.729716    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"]
Sep 18 16:53:20.731331 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:20.731283    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9"]
Sep 18 16:53:20.731537 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:20.731474    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9" podUID="d2cfedea-f20b-42ba-9264-df054e1cc4aa" containerName="main" containerID="cri-o://03bf3f8cf1334cc4a187eb07e998f9d9ac0122cfb5a0c0b3cb75b6cee25e2591" gracePeriod=30
Sep 18 16:53:20.739434 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:53:20.739395    2585 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 16:53:20.739434 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:53:20.739434 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 16:53:20.739434 ip-10-0-133-147 kubenswrapper[2585]:  > containerID="3b29d8960720556a2df88efe8bed33283526b909b894db8c7b8b8a29d43cd5b8" cmd=["/bin/sleep","15"]
Sep 18 16:53:20.739634 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:20.739455    2585 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx" message="" err=<
Sep 18 16:53:20.739634 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:53:20.739634 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 16:53:20.739634 ip-10-0-133-147 kubenswrapper[2585]:  >
Sep 18 16:53:20.739634 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:53:20.739467    2585 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 16:53:20.739634 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:53:20.739634 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 16:53:20.739634 ip-10-0-133-147 kubenswrapper[2585]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx" podUID="da70e390-eb1e-4a6b-a605-e0ad48beec01" containerName="main" containerID="cri-o://3b29d8960720556a2df88efe8bed33283526b909b894db8c7b8b8a29d43cd5b8"
Sep 18 16:53:20.740038 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:20.739858    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx" podUID="da70e390-eb1e-4a6b-a605-e0ad48beec01" containerName="main" containerID="cri-o://3b29d8960720556a2df88efe8bed33283526b909b894db8c7b8b8a29d43cd5b8" gracePeriod=60
Sep 18 16:53:20.888220 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:20.888185    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx" podUID="da70e390-eb1e-4a6b-a605-e0ad48beec01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 18 16:53:21.006395 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.006309    2585 generic.go:356] "Generic (PLEG): container finished" podID="da70e390-eb1e-4a6b-a605-e0ad48beec01" containerID="3b29d8960720556a2df88efe8bed33283526b909b894db8c7b8b8a29d43cd5b8" exitCode=0
Sep 18 16:53:21.006395 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.006356    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx" event={"ID":"da70e390-eb1e-4a6b-a605-e0ad48beec01","Type":"ContainerDied","Data":"3b29d8960720556a2df88efe8bed33283526b909b894db8c7b8b8a29d43cd5b8"}
Sep 18 16:53:21.094556 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.094533    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:53:21.233401 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.233374    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da70e390-eb1e-4a6b-a605-e0ad48beec01-tls-certs\") pod \"da70e390-eb1e-4a6b-a605-e0ad48beec01\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") "
Sep 18 16:53:21.233528 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.233420    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da70e390-eb1e-4a6b-a605-e0ad48beec01-home\") pod \"da70e390-eb1e-4a6b-a605-e0ad48beec01\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") "
Sep 18 16:53:21.233528 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.233457    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da70e390-eb1e-4a6b-a605-e0ad48beec01-tmp-dir\") pod \"da70e390-eb1e-4a6b-a605-e0ad48beec01\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") "
Sep 18 16:53:21.233654 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.233633    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da70e390-eb1e-4a6b-a605-e0ad48beec01-dshm\") pod \"da70e390-eb1e-4a6b-a605-e0ad48beec01\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") "
Sep 18 16:53:21.233713 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.233669    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da70e390-eb1e-4a6b-a605-e0ad48beec01-kserve-provision-location\") pod \"da70e390-eb1e-4a6b-a605-e0ad48beec01\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") "
Sep 18 16:53:21.233713 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.233675    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da70e390-eb1e-4a6b-a605-e0ad48beec01-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "da70e390-eb1e-4a6b-a605-e0ad48beec01" (UID: "da70e390-eb1e-4a6b-a605-e0ad48beec01"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:53:21.233819 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.233703    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da70e390-eb1e-4a6b-a605-e0ad48beec01-home" (OuterVolumeSpecName: "home") pod "da70e390-eb1e-4a6b-a605-e0ad48beec01" (UID: "da70e390-eb1e-4a6b-a605-e0ad48beec01"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:53:21.233819 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.233726    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da70e390-eb1e-4a6b-a605-e0ad48beec01-cabundle-cert\") pod \"da70e390-eb1e-4a6b-a605-e0ad48beec01\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") "
Sep 18 16:53:21.233819 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.233807    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lpdk\" (UniqueName: \"kubernetes.io/projected/da70e390-eb1e-4a6b-a605-e0ad48beec01-kube-api-access-4lpdk\") pod \"da70e390-eb1e-4a6b-a605-e0ad48beec01\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") "
Sep 18 16:53:21.233995 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.233872    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da70e390-eb1e-4a6b-a605-e0ad48beec01-model-cache\") pod \"da70e390-eb1e-4a6b-a605-e0ad48beec01\" (UID: \"da70e390-eb1e-4a6b-a605-e0ad48beec01\") "
Sep 18 16:53:21.234139 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.234115    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da70e390-eb1e-4a6b-a605-e0ad48beec01-model-cache" (OuterVolumeSpecName: "model-cache") pod "da70e390-eb1e-4a6b-a605-e0ad48beec01" (UID: "da70e390-eb1e-4a6b-a605-e0ad48beec01"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:53:21.234244 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.234145    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da70e390-eb1e-4a6b-a605-e0ad48beec01-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "da70e390-eb1e-4a6b-a605-e0ad48beec01" (UID: "da70e390-eb1e-4a6b-a605-e0ad48beec01"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:53:21.234244 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.234177    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da70e390-eb1e-4a6b-a605-e0ad48beec01-home\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:53:21.234244 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.234221    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da70e390-eb1e-4a6b-a605-e0ad48beec01-tmp-dir\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:53:21.237900 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.237873    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da70e390-eb1e-4a6b-a605-e0ad48beec01-dshm" (OuterVolumeSpecName: "dshm") pod "da70e390-eb1e-4a6b-a605-e0ad48beec01" (UID: "da70e390-eb1e-4a6b-a605-e0ad48beec01"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:53:21.238059 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.238037    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da70e390-eb1e-4a6b-a605-e0ad48beec01-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "da70e390-eb1e-4a6b-a605-e0ad48beec01" (UID: "da70e390-eb1e-4a6b-a605-e0ad48beec01"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:53:21.238059 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.238045    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da70e390-eb1e-4a6b-a605-e0ad48beec01-kube-api-access-4lpdk" (OuterVolumeSpecName: "kube-api-access-4lpdk") pod "da70e390-eb1e-4a6b-a605-e0ad48beec01" (UID: "da70e390-eb1e-4a6b-a605-e0ad48beec01"). InnerVolumeSpecName "kube-api-access-4lpdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:53:21.257291 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.257224    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da70e390-eb1e-4a6b-a605-e0ad48beec01-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "da70e390-eb1e-4a6b-a605-e0ad48beec01" (UID: "da70e390-eb1e-4a6b-a605-e0ad48beec01"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:53:21.335483 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.335447    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da70e390-eb1e-4a6b-a605-e0ad48beec01-dshm\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:53:21.335483 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.335479    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da70e390-eb1e-4a6b-a605-e0ad48beec01-kserve-provision-location\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:53:21.335656 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.335490    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da70e390-eb1e-4a6b-a605-e0ad48beec01-cabundle-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:53:21.335656 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.335500    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lpdk\" (UniqueName: \"kubernetes.io/projected/da70e390-eb1e-4a6b-a605-e0ad48beec01-kube-api-access-4lpdk\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:53:21.335656 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.335509    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da70e390-eb1e-4a6b-a605-e0ad48beec01-model-cache\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:53:21.335656 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.335517    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da70e390-eb1e-4a6b-a605-e0ad48beec01-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:53:21.362948 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.362906    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9"
Sep 18 16:53:21.435788 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.435759    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2cfedea-f20b-42ba-9264-df054e1cc4aa-tmp-dir\") pod \"d2cfedea-f20b-42ba-9264-df054e1cc4aa\" (UID: \"d2cfedea-f20b-42ba-9264-df054e1cc4aa\") "
Sep 18 16:53:21.435954 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.435802    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2cfedea-f20b-42ba-9264-df054e1cc4aa-kserve-provision-location\") pod \"d2cfedea-f20b-42ba-9264-df054e1cc4aa\" (UID: \"d2cfedea-f20b-42ba-9264-df054e1cc4aa\") "
Sep 18 16:53:21.435954 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.435840    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2cfedea-f20b-42ba-9264-df054e1cc4aa-tls-certs\") pod \"d2cfedea-f20b-42ba-9264-df054e1cc4aa\" (UID: \"d2cfedea-f20b-42ba-9264-df054e1cc4aa\") "
Sep 18 16:53:21.435954 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.435938    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2cfedea-f20b-42ba-9264-df054e1cc4aa-cabundle-cert\") pod \"d2cfedea-f20b-42ba-9264-df054e1cc4aa\" (UID: \"d2cfedea-f20b-42ba-9264-df054e1cc4aa\") "
Sep 18 16:53:21.436261 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.436229    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2cfedea-f20b-42ba-9264-df054e1cc4aa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d2cfedea-f20b-42ba-9264-df054e1cc4aa" (UID: "d2cfedea-f20b-42ba-9264-df054e1cc4aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:53:21.436386 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.436290    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2cfedea-f20b-42ba-9264-df054e1cc4aa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d2cfedea-f20b-42ba-9264-df054e1cc4aa" (UID: "d2cfedea-f20b-42ba-9264-df054e1cc4aa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:53:21.438208 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.438185    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2cfedea-f20b-42ba-9264-df054e1cc4aa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d2cfedea-f20b-42ba-9264-df054e1cc4aa" (UID: "d2cfedea-f20b-42ba-9264-df054e1cc4aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:53:21.457904 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.457868    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2cfedea-f20b-42ba-9264-df054e1cc4aa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d2cfedea-f20b-42ba-9264-df054e1cc4aa" (UID: "d2cfedea-f20b-42ba-9264-df054e1cc4aa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:53:21.537535 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.537467    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2cfedea-f20b-42ba-9264-df054e1cc4aa-kserve-provision-location\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:53:21.537535 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.537490    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2cfedea-f20b-42ba-9264-df054e1cc4aa-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:53:21.537535 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.537499    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2cfedea-f20b-42ba-9264-df054e1cc4aa-cabundle-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:53:21.537535 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:21.537507    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2cfedea-f20b-42ba-9264-df054e1cc4aa-tmp-dir\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:53:22.012583 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:22.012541    2585 generic.go:356] "Generic (PLEG): container finished" podID="d2cfedea-f20b-42ba-9264-df054e1cc4aa" containerID="03bf3f8cf1334cc4a187eb07e998f9d9ac0122cfb5a0c0b3cb75b6cee25e2591" exitCode=0
Sep 18 16:53:22.012757 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:22.012598    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9"
Sep 18 16:53:22.012757 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:22.012623    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9" event={"ID":"d2cfedea-f20b-42ba-9264-df054e1cc4aa","Type":"ContainerDied","Data":"03bf3f8cf1334cc4a187eb07e998f9d9ac0122cfb5a0c0b3cb75b6cee25e2591"}
Sep 18 16:53:22.012757 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:22.012688    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9" event={"ID":"d2cfedea-f20b-42ba-9264-df054e1cc4aa","Type":"ContainerDied","Data":"b22ab53edabc519c2015da63bbfded1089157b5bcb95418ba8b4f8457e48aeae"}
Sep 18 16:53:22.012757 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:22.012719    2585 scope.go:117] "RemoveContainer" containerID="03bf3f8cf1334cc4a187eb07e998f9d9ac0122cfb5a0c0b3cb75b6cee25e2591"
Sep 18 16:53:22.014415 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:22.014386    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx" event={"ID":"da70e390-eb1e-4a6b-a605-e0ad48beec01","Type":"ContainerDied","Data":"cf0812fc6805bb85163b8a80e0bfaa75c4e977fb48f602fe87241e0c405c9ae2"}
Sep 18 16:53:22.014415 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:22.014399    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"
Sep 18 16:53:22.021747 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:22.021724    2585 scope.go:117] "RemoveContainer" containerID="9dbcd5c46390c76c8c1600705640c7cbe9a9693ae80fb3b2a9864d46e3a8df4e"
Sep 18 16:53:22.031511 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:22.031493    2585 scope.go:117] "RemoveContainer" containerID="03bf3f8cf1334cc4a187eb07e998f9d9ac0122cfb5a0c0b3cb75b6cee25e2591"
Sep 18 16:53:22.031800 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:53:22.031784    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03bf3f8cf1334cc4a187eb07e998f9d9ac0122cfb5a0c0b3cb75b6cee25e2591\": container with ID starting with 03bf3f8cf1334cc4a187eb07e998f9d9ac0122cfb5a0c0b3cb75b6cee25e2591 not found: ID does not exist" containerID="03bf3f8cf1334cc4a187eb07e998f9d9ac0122cfb5a0c0b3cb75b6cee25e2591"
Sep 18 16:53:22.031847 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:22.031808    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03bf3f8cf1334cc4a187eb07e998f9d9ac0122cfb5a0c0b3cb75b6cee25e2591"} err="failed to get container status \"03bf3f8cf1334cc4a187eb07e998f9d9ac0122cfb5a0c0b3cb75b6cee25e2591\": rpc error: code = NotFound desc = could not find container \"03bf3f8cf1334cc4a187eb07e998f9d9ac0122cfb5a0c0b3cb75b6cee25e2591\": container with ID starting with 03bf3f8cf1334cc4a187eb07e998f9d9ac0122cfb5a0c0b3cb75b6cee25e2591 not found: ID does not exist"
Sep 18 16:53:22.031847 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:22.031823    2585 scope.go:117] "RemoveContainer" containerID="9dbcd5c46390c76c8c1600705640c7cbe9a9693ae80fb3b2a9864d46e3a8df4e"
Sep 18 16:53:22.032406 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:53:22.032388    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dbcd5c46390c76c8c1600705640c7cbe9a9693ae80fb3b2a9864d46e3a8df4e\": container with ID starting with 9dbcd5c46390c76c8c1600705640c7cbe9a9693ae80fb3b2a9864d46e3a8df4e not found: ID does not exist" containerID="9dbcd5c46390c76c8c1600705640c7cbe9a9693ae80fb3b2a9864d46e3a8df4e"
Sep 18 16:53:22.032449 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:22.032412    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dbcd5c46390c76c8c1600705640c7cbe9a9693ae80fb3b2a9864d46e3a8df4e"} err="failed to get container status \"9dbcd5c46390c76c8c1600705640c7cbe9a9693ae80fb3b2a9864d46e3a8df4e\": rpc error: code = NotFound desc = could not find container \"9dbcd5c46390c76c8c1600705640c7cbe9a9693ae80fb3b2a9864d46e3a8df4e\": container with ID starting with 9dbcd5c46390c76c8c1600705640c7cbe9a9693ae80fb3b2a9864d46e3a8df4e not found: ID does not exist"
Sep 18 16:53:22.032449 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:22.032429    2585 scope.go:117] "RemoveContainer" containerID="3b29d8960720556a2df88efe8bed33283526b909b894db8c7b8b8a29d43cd5b8"
Sep 18 16:53:22.040712 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:22.040692    2585 scope.go:117] "RemoveContainer" containerID="88d3efd8568375357dc5b453da7ad5d3ed7fe400b851e6d3e92a3eca8f1bc860"
Sep 18 16:53:22.041360 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:22.041342    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"]
Sep 18 16:53:22.045926 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:22.045891    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-dvtnx"]
Sep 18 16:53:22.056832 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:22.056812    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9"]
Sep 18 16:53:22.060630 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:22.060596    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-69cb8d959-sf5t9"]
Sep 18 16:53:22.392248 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:22.392215    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2cfedea-f20b-42ba-9264-df054e1cc4aa" path="/var/lib/kubelet/pods/d2cfedea-f20b-42ba-9264-df054e1cc4aa/volumes"
Sep 18 16:53:22.392662 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:53:22.392648    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da70e390-eb1e-4a6b-a605-e0ad48beec01" path="/var/lib/kubelet/pods/da70e390-eb1e-4a6b-a605-e0ad48beec01/volumes"
Sep 18 16:54:21.586853 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.586811    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"]
Sep 18 16:54:21.587401 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.587123    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da70e390-eb1e-4a6b-a605-e0ad48beec01" containerName="storage-initializer"
Sep 18 16:54:21.587401 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.587136    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="da70e390-eb1e-4a6b-a605-e0ad48beec01" containerName="storage-initializer"
Sep 18 16:54:21.587401 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.587154    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da70e390-eb1e-4a6b-a605-e0ad48beec01" containerName="main"
Sep 18 16:54:21.587401 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.587164    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="da70e390-eb1e-4a6b-a605-e0ad48beec01" containerName="main"
Sep 18 16:54:21.587401 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.587177    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2cfedea-f20b-42ba-9264-df054e1cc4aa" containerName="storage-initializer"
Sep 18 16:54:21.587401 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.587183    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2cfedea-f20b-42ba-9264-df054e1cc4aa" containerName="storage-initializer"
Sep 18 16:54:21.587401 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.587192    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2cfedea-f20b-42ba-9264-df054e1cc4aa" containerName="main"
Sep 18 16:54:21.587401 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.587197    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2cfedea-f20b-42ba-9264-df054e1cc4aa" containerName="main"
Sep 18 16:54:21.587401 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.587248    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2cfedea-f20b-42ba-9264-df054e1cc4aa" containerName="main"
Sep 18 16:54:21.587401 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.587258    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="da70e390-eb1e-4a6b-a605-e0ad48beec01" containerName="main"
Sep 18 16:54:21.589858 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.589836    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"
Sep 18 16:54:21.594594 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.594571    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 18 16:54:21.594748 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.594599    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-rr6vv\""
Sep 18 16:54:21.594748 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.594603    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 18 16:54:21.594748 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.594728    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 18 16:54:21.606609 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.606588    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"]
Sep 18 16:54:21.669409 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.669373    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p\" (UID: \"64bff7b3-d944-4359-8e51-c61f0e4ea5bd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"
Sep 18 16:54:21.669409 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.669410    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p\" (UID: \"64bff7b3-d944-4359-8e51-c61f0e4ea5bd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"
Sep 18 16:54:21.669609 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.669435    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p\" (UID: \"64bff7b3-d944-4359-8e51-c61f0e4ea5bd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"
Sep 18 16:54:21.669609 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.669511    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p\" (UID: \"64bff7b3-d944-4359-8e51-c61f0e4ea5bd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"
Sep 18 16:54:21.669609 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.669547    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl98v\" (UniqueName: \"kubernetes.io/projected/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-kube-api-access-vl98v\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p\" (UID: \"64bff7b3-d944-4359-8e51-c61f0e4ea5bd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"
Sep 18 16:54:21.669609 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.669568    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p\" (UID: \"64bff7b3-d944-4359-8e51-c61f0e4ea5bd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"
Sep 18 16:54:21.769977 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.769943    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p\" (UID: \"64bff7b3-d944-4359-8e51-c61f0e4ea5bd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"
Sep 18 16:54:21.769977 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.769981    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p\" (UID: \"64bff7b3-d944-4359-8e51-c61f0e4ea5bd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"
Sep 18 16:54:21.770205 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.770014    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p\" (UID: \"64bff7b3-d944-4359-8e51-c61f0e4ea5bd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"
Sep 18 16:54:21.770205 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.770050    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p\" (UID: \"64bff7b3-d944-4359-8e51-c61f0e4ea5bd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"
Sep 18 16:54:21.770205 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.770075    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl98v\" (UniqueName: \"kubernetes.io/projected/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-kube-api-access-vl98v\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p\" (UID: \"64bff7b3-d944-4359-8e51-c61f0e4ea5bd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"
Sep 18 16:54:21.770205 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.770092    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p\" (UID: \"64bff7b3-d944-4359-8e51-c61f0e4ea5bd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"
Sep 18 16:54:21.770417 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.770388    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p\" (UID: \"64bff7b3-d944-4359-8e51-c61f0e4ea5bd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"
Sep 18 16:54:21.770501 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.770483    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p\" (UID: \"64bff7b3-d944-4359-8e51-c61f0e4ea5bd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"
Sep 18 16:54:21.770555 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.770505    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p\" (UID: \"64bff7b3-d944-4359-8e51-c61f0e4ea5bd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"
Sep 18 16:54:21.772479 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.772458    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p\" (UID: \"64bff7b3-d944-4359-8e51-c61f0e4ea5bd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"
Sep 18 16:54:21.772783 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.772762    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p\" (UID: \"64bff7b3-d944-4359-8e51-c61f0e4ea5bd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"
Sep 18 16:54:21.779654 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.779629    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl98v\" (UniqueName: \"kubernetes.io/projected/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-kube-api-access-vl98v\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p\" (UID: \"64bff7b3-d944-4359-8e51-c61f0e4ea5bd\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"
Sep 18 16:54:21.900366 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:21.900289    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"
Sep 18 16:54:22.173611 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:22.173586    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"]
Sep 18 16:54:22.175881 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:54:22.175849    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64bff7b3_d944_4359_8e51_c61f0e4ea5bd.slice/crio-20b524cd7347df1caf914ead9820bceeae13341b069ece78b47818f55e302a29 WatchSource:0}: Error finding container 20b524cd7347df1caf914ead9820bceeae13341b069ece78b47818f55e302a29: Status 404 returned error can't find the container with id 20b524cd7347df1caf914ead9820bceeae13341b069ece78b47818f55e302a29
Sep 18 16:54:22.257076 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:22.257045    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p" event={"ID":"64bff7b3-d944-4359-8e51-c61f0e4ea5bd","Type":"ContainerStarted","Data":"e6b02c6bd5022a867a51f2a66ff62298f4db06854a773c6e397ffb2f4ed9439e"}
Sep 18 16:54:22.257192 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:22.257130    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p" event={"ID":"64bff7b3-d944-4359-8e51-c61f0e4ea5bd","Type":"ContainerStarted","Data":"20b524cd7347df1caf914ead9820bceeae13341b069ece78b47818f55e302a29"}
Sep 18 16:54:22.276741 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:22.276699    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p" podStartSLOduration=1.2766837930000001 podStartE2EDuration="1.276683793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:54:22.274813764 +0000 UTC m=+2302.057815793" watchObservedRunningTime="2026-09-18 16:54:22.276683793 +0000 UTC m=+2302.059685817"
Sep 18 16:54:22.900370 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:22.900344    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"
Sep 18 16:54:31.901316 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:31.901277    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"
Sep 18 16:54:31.913392 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:31.913367    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"
Sep 18 16:54:32.310699 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:32.310671    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"
Sep 18 16:54:45.407292 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:45.407197    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx"]
Sep 18 16:54:45.410111 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:45.410095    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx"
Sep 18 16:54:45.415234 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:45.415210    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 18 16:54:45.415699 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:45.415269    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 18 16:54:45.415699 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:45.415304    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 18 16:54:45.415699 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:45.415457    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-hzp4w\""
Sep 18 16:54:45.415699 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:45.415320    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-9njrp\""
Sep 18 16:54:45.420972 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:45.420952    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx"]
Sep 18 16:54:45.479209 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:45.479186    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8747c04b-a381-4e88-bed6-7c8ea1204031-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx\" (UID: \"8747c04b-a381-4e88-bed6-7c8ea1204031\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx"
Sep 18 16:54:45.479345 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:45.479218    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8hwj\" (UniqueName: \"kubernetes.io/projected/8747c04b-a381-4e88-bed6-7c8ea1204031-kube-api-access-t8hwj\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx\" (UID: \"8747c04b-a381-4e88-bed6-7c8ea1204031\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx"
Sep 18 16:54:45.580311 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:45.580276    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8747c04b-a381-4e88-bed6-7c8ea1204031-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx\" (UID: \"8747c04b-a381-4e88-bed6-7c8ea1204031\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx"
Sep 18 16:54:45.580473 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:45.580317    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8hwj\" (UniqueName: \"kubernetes.io/projected/8747c04b-a381-4e88-bed6-7c8ea1204031-kube-api-access-t8hwj\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx\" (UID: \"8747c04b-a381-4e88-bed6-7c8ea1204031\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx"
Sep 18 16:54:45.583027 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:45.583004    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8747c04b-a381-4e88-bed6-7c8ea1204031-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx\" (UID: \"8747c04b-a381-4e88-bed6-7c8ea1204031\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx"
Sep 18 16:54:45.617752 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:45.617723    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8hwj\" (UniqueName: \"kubernetes.io/projected/8747c04b-a381-4e88-bed6-7c8ea1204031-kube-api-access-t8hwj\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx\" (UID: \"8747c04b-a381-4e88-bed6-7c8ea1204031\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx"
Sep 18 16:54:45.722119 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:45.722048    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx"
Sep 18 16:54:45.992964 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:45.992939    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx"]
Sep 18 16:54:45.994992 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:54:45.994965    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8747c04b_a381_4e88_bed6_7c8ea1204031.slice/crio-4c3bab47622284fda8060ebe2fede643d5226d1c377a7551c38ef9d5bee101c2 WatchSource:0}: Error finding container 4c3bab47622284fda8060ebe2fede643d5226d1c377a7551c38ef9d5bee101c2: Status 404 returned error can't find the container with id 4c3bab47622284fda8060ebe2fede643d5226d1c377a7551c38ef9d5bee101c2
Sep 18 16:54:46.358118 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:46.358028    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx" event={"ID":"8747c04b-a381-4e88-bed6-7c8ea1204031","Type":"ContainerStarted","Data":"8ed4ca1e051444c836c4be05ac860dfe50b956ce080a91e2798853990b882111"}
Sep 18 16:54:46.358118 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:46.358082    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx" event={"ID":"8747c04b-a381-4e88-bed6-7c8ea1204031","Type":"ContainerStarted","Data":"4c3bab47622284fda8060ebe2fede643d5226d1c377a7551c38ef9d5bee101c2"}
Sep 18 16:54:46.358320 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:46.358198    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx"
Sep 18 16:54:46.376669 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:46.376623    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx" podStartSLOduration=1.376608083 podStartE2EDuration="1.376608083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:54:46.375513581 +0000 UTC m=+2326.158515605" watchObservedRunningTime="2026-09-18 16:54:46.376608083 +0000 UTC m=+2326.159610107"
Sep 18 16:54:56.010226 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.010195    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"]
Sep 18 16:54:56.021291 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:54:56.021234    2585 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 16:54:56.021291 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:54:56.021291 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 16:54:56.021291 ip-10-0-133-147 kubenswrapper[2585]:  > containerID="e6b02c6bd5022a867a51f2a66ff62298f4db06854a773c6e397ffb2f4ed9439e" cmd=["/bin/sleep","15"]
Sep 18 16:54:56.021632 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.021313    2585 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p" message="" err=<
Sep 18 16:54:56.021632 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:54:56.021632 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 16:54:56.021632 ip-10-0-133-147 kubenswrapper[2585]:  >
Sep 18 16:54:56.021632 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:54:56.021330    2585 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 16:54:56.021632 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:54:56.021632 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 16:54:56.021632 ip-10-0-133-147 kubenswrapper[2585]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p" podUID="64bff7b3-d944-4359-8e51-c61f0e4ea5bd" containerName="main" containerID="cri-o://e6b02c6bd5022a867a51f2a66ff62298f4db06854a773c6e397ffb2f4ed9439e"
Sep 18 16:54:56.021632 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.021376    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p" podUID="64bff7b3-d944-4359-8e51-c61f0e4ea5bd" containerName="main" containerID="cri-o://e6b02c6bd5022a867a51f2a66ff62298f4db06854a773c6e397ffb2f4ed9439e" gracePeriod=60
Sep 18 16:54:56.372013 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.371992    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"
Sep 18 16:54:56.403961 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.403906    2585 generic.go:356] "Generic (PLEG): container finished" podID="64bff7b3-d944-4359-8e51-c61f0e4ea5bd" containerID="e6b02c6bd5022a867a51f2a66ff62298f4db06854a773c6e397ffb2f4ed9439e" exitCode=0
Sep 18 16:54:56.404117 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.403978    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"
Sep 18 16:54:56.404117 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.404017    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p" event={"ID":"64bff7b3-d944-4359-8e51-c61f0e4ea5bd","Type":"ContainerDied","Data":"e6b02c6bd5022a867a51f2a66ff62298f4db06854a773c6e397ffb2f4ed9439e"}
Sep 18 16:54:56.404117 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.404063    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p" event={"ID":"64bff7b3-d944-4359-8e51-c61f0e4ea5bd","Type":"ContainerDied","Data":"20b524cd7347df1caf914ead9820bceeae13341b069ece78b47818f55e302a29"}
Sep 18 16:54:56.404117 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.404085    2585 scope.go:117] "RemoveContainer" containerID="e6b02c6bd5022a867a51f2a66ff62298f4db06854a773c6e397ffb2f4ed9439e"
Sep 18 16:54:56.412395 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.412377    2585 scope.go:117] "RemoveContainer" containerID="e6b02c6bd5022a867a51f2a66ff62298f4db06854a773c6e397ffb2f4ed9439e"
Sep 18 16:54:56.412643 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:54:56.412617    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6b02c6bd5022a867a51f2a66ff62298f4db06854a773c6e397ffb2f4ed9439e\": container with ID starting with e6b02c6bd5022a867a51f2a66ff62298f4db06854a773c6e397ffb2f4ed9439e not found: ID does not exist" containerID="e6b02c6bd5022a867a51f2a66ff62298f4db06854a773c6e397ffb2f4ed9439e"
Sep 18 16:54:56.412735 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.412654    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6b02c6bd5022a867a51f2a66ff62298f4db06854a773c6e397ffb2f4ed9439e"} err="failed to get container status \"e6b02c6bd5022a867a51f2a66ff62298f4db06854a773c6e397ffb2f4ed9439e\": rpc error: code = NotFound desc = could not find container \"e6b02c6bd5022a867a51f2a66ff62298f4db06854a773c6e397ffb2f4ed9439e\": container with ID starting with e6b02c6bd5022a867a51f2a66ff62298f4db06854a773c6e397ffb2f4ed9439e not found: ID does not exist"
Sep 18 16:54:56.466568 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.466550    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-tmp-dir\") pod \"64bff7b3-d944-4359-8e51-c61f0e4ea5bd\" (UID: \"64bff7b3-d944-4359-8e51-c61f0e4ea5bd\") "
Sep 18 16:54:56.466735 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.466574    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-dshm\") pod \"64bff7b3-d944-4359-8e51-c61f0e4ea5bd\" (UID: \"64bff7b3-d944-4359-8e51-c61f0e4ea5bd\") "
Sep 18 16:54:56.466735 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.466632    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-model-cache\") pod \"64bff7b3-d944-4359-8e51-c61f0e4ea5bd\" (UID: \"64bff7b3-d944-4359-8e51-c61f0e4ea5bd\") "
Sep 18 16:54:56.466735 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.466656    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vl98v\" (UniqueName: \"kubernetes.io/projected/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-kube-api-access-vl98v\") pod \"64bff7b3-d944-4359-8e51-c61f0e4ea5bd\" (UID: \"64bff7b3-d944-4359-8e51-c61f0e4ea5bd\") "
Sep 18 16:54:56.466735 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.466671    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-tls-certs\") pod \"64bff7b3-d944-4359-8e51-c61f0e4ea5bd\" (UID: \"64bff7b3-d944-4359-8e51-c61f0e4ea5bd\") "
Sep 18 16:54:56.466735 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.466698    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-home\") pod \"64bff7b3-d944-4359-8e51-c61f0e4ea5bd\" (UID: \"64bff7b3-d944-4359-8e51-c61f0e4ea5bd\") "
Sep 18 16:54:56.467051 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.466811    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "64bff7b3-d944-4359-8e51-c61f0e4ea5bd" (UID: "64bff7b3-d944-4359-8e51-c61f0e4ea5bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:54:56.467051 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.467012    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "64bff7b3-d944-4359-8e51-c61f0e4ea5bd" (UID: "64bff7b3-d944-4359-8e51-c61f0e4ea5bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:54:56.467146 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.467052    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-tmp-dir\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:54:56.467146 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.467050    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-home" (OuterVolumeSpecName: "home") pod "64bff7b3-d944-4359-8e51-c61f0e4ea5bd" (UID: "64bff7b3-d944-4359-8e51-c61f0e4ea5bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:54:56.468997 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.468978    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-dshm" (OuterVolumeSpecName: "dshm") pod "64bff7b3-d944-4359-8e51-c61f0e4ea5bd" (UID: "64bff7b3-d944-4359-8e51-c61f0e4ea5bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:54:56.469090 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.469025    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "64bff7b3-d944-4359-8e51-c61f0e4ea5bd" (UID: "64bff7b3-d944-4359-8e51-c61f0e4ea5bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:54:56.469090 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.469051    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-kube-api-access-vl98v" (OuterVolumeSpecName: "kube-api-access-vl98v") pod "64bff7b3-d944-4359-8e51-c61f0e4ea5bd" (UID: "64bff7b3-d944-4359-8e51-c61f0e4ea5bd"). InnerVolumeSpecName "kube-api-access-vl98v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:54:56.567770 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.567716    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-dshm\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:54:56.567770 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.567735    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-model-cache\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:54:56.567770 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.567746    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl98v\" (UniqueName: \"kubernetes.io/projected/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-kube-api-access-vl98v\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:54:56.567770 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.567755    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:54:56.567770 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.567763    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64bff7b3-d944-4359-8e51-c61f0e4ea5bd-home\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:54:56.726731 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.726692    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"]
Sep 18 16:54:56.733652 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:56.733630    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p"]
Sep 18 16:54:57.300528 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:57.300486    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-tnr8p" podUID="64bff7b3-d944-4359-8e51-c61f0e4ea5bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:54:58.392218 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:54:58.392186    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64bff7b3-d944-4359-8e51-c61f0e4ea5bd" path="/var/lib/kubelet/pods/64bff7b3-d944-4359-8e51-c61f0e4ea5bd/volumes"
Sep 18 16:55:17.363687 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:55:17.363658    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx"
Sep 18 16:56:02.471262 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:56:02.471234    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 16:56:02.473108 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:56:02.473083    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 16:56:02.475400 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:56:02.475380    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 16:56:02.476946 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:56:02.476901    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 16:57:22.405666 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:57:22.405632    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx"]
Sep 18 16:57:27.364034 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:57:27.363997    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx" podUID="8747c04b-a381-4e88-bed6-7c8ea1204031" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:57:37.364161 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:57:37.364115    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx" podUID="8747c04b-a381-4e88-bed6-7c8ea1204031" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:57:37.426618 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:57:37.426588    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx" podUID="8747c04b-a381-4e88-bed6-7c8ea1204031" containerName="main" containerID="cri-o://8ed4ca1e051444c836c4be05ac860dfe50b956ce080a91e2798853990b882111" gracePeriod=45
Sep 18 16:57:47.364007 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:57:47.363892    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx" podUID="8747c04b-a381-4e88-bed6-7c8ea1204031" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:57:47.364448 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:57:47.364044    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx"
Sep 18 16:57:57.363736 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:57:57.363700    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx" podUID="8747c04b-a381-4e88-bed6-7c8ea1204031" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:58:07.364312 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:07.364271    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx" podUID="8747c04b-a381-4e88-bed6-7c8ea1204031" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 16:58:07.666930 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:07.666888    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx"
Sep 18 16:58:07.723335 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:07.723302    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8hwj\" (UniqueName: \"kubernetes.io/projected/8747c04b-a381-4e88-bed6-7c8ea1204031-kube-api-access-t8hwj\") pod \"8747c04b-a381-4e88-bed6-7c8ea1204031\" (UID: \"8747c04b-a381-4e88-bed6-7c8ea1204031\") "
Sep 18 16:58:07.723489 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:07.723342    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8747c04b-a381-4e88-bed6-7c8ea1204031-tls-certs\") pod \"8747c04b-a381-4e88-bed6-7c8ea1204031\" (UID: \"8747c04b-a381-4e88-bed6-7c8ea1204031\") "
Sep 18 16:58:07.725750 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:07.725714    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8747c04b-a381-4e88-bed6-7c8ea1204031-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8747c04b-a381-4e88-bed6-7c8ea1204031" (UID: "8747c04b-a381-4e88-bed6-7c8ea1204031"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:58:07.725750 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:07.725725    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8747c04b-a381-4e88-bed6-7c8ea1204031-kube-api-access-t8hwj" (OuterVolumeSpecName: "kube-api-access-t8hwj") pod "8747c04b-a381-4e88-bed6-7c8ea1204031" (UID: "8747c04b-a381-4e88-bed6-7c8ea1204031"). InnerVolumeSpecName "kube-api-access-t8hwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:58:07.823952 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:07.823900    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8747c04b-a381-4e88-bed6-7c8ea1204031-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:58:07.823952 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:07.823946    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8hwj\" (UniqueName: \"kubernetes.io/projected/8747c04b-a381-4e88-bed6-7c8ea1204031-kube-api-access-t8hwj\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 16:58:08.146547 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:08.146510    2585 generic.go:356] "Generic (PLEG): container finished" podID="8747c04b-a381-4e88-bed6-7c8ea1204031" containerID="8ed4ca1e051444c836c4be05ac860dfe50b956ce080a91e2798853990b882111" exitCode=0
Sep 18 16:58:08.146718 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:08.146558    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx" event={"ID":"8747c04b-a381-4e88-bed6-7c8ea1204031","Type":"ContainerDied","Data":"8ed4ca1e051444c836c4be05ac860dfe50b956ce080a91e2798853990b882111"}
Sep 18 16:58:08.146718 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:08.146599    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx" event={"ID":"8747c04b-a381-4e88-bed6-7c8ea1204031","Type":"ContainerDied","Data":"4c3bab47622284fda8060ebe2fede643d5226d1c377a7551c38ef9d5bee101c2"}
Sep 18 16:58:08.146718 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:08.146619    2585 scope.go:117] "RemoveContainer" containerID="8ed4ca1e051444c836c4be05ac860dfe50b956ce080a91e2798853990b882111"
Sep 18 16:58:08.146718 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:08.146560    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx"
Sep 18 16:58:08.155389 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:08.155368    2585 scope.go:117] "RemoveContainer" containerID="8ed4ca1e051444c836c4be05ac860dfe50b956ce080a91e2798853990b882111"
Sep 18 16:58:08.155649 ip-10-0-133-147 kubenswrapper[2585]: E0918 16:58:08.155628    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ed4ca1e051444c836c4be05ac860dfe50b956ce080a91e2798853990b882111\": container with ID starting with 8ed4ca1e051444c836c4be05ac860dfe50b956ce080a91e2798853990b882111 not found: ID does not exist" containerID="8ed4ca1e051444c836c4be05ac860dfe50b956ce080a91e2798853990b882111"
Sep 18 16:58:08.155713 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:08.155663    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ed4ca1e051444c836c4be05ac860dfe50b956ce080a91e2798853990b882111"} err="failed to get container status \"8ed4ca1e051444c836c4be05ac860dfe50b956ce080a91e2798853990b882111\": rpc error: code = NotFound desc = could not find container \"8ed4ca1e051444c836c4be05ac860dfe50b956ce080a91e2798853990b882111\": container with ID starting with 8ed4ca1e051444c836c4be05ac860dfe50b956ce080a91e2798853990b882111 not found: ID does not exist"
Sep 18 16:58:08.168484 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:08.168461    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx"]
Sep 18 16:58:08.172400 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:08.172376    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-d4jmx"]
Sep 18 16:58:08.391416 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:08.391386    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8747c04b-a381-4e88-bed6-7c8ea1204031" path="/var/lib/kubelet/pods/8747c04b-a381-4e88-bed6-7c8ea1204031/volumes"
Sep 18 16:58:12.265253 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.265221    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"]
Sep 18 16:58:12.265593 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.265543    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8747c04b-a381-4e88-bed6-7c8ea1204031" containerName="main"
Sep 18 16:58:12.265593 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.265556    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8747c04b-a381-4e88-bed6-7c8ea1204031" containerName="main"
Sep 18 16:58:12.265593 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.265565    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64bff7b3-d944-4359-8e51-c61f0e4ea5bd" containerName="main"
Sep 18 16:58:12.265593 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.265571    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="64bff7b3-d944-4359-8e51-c61f0e4ea5bd" containerName="main"
Sep 18 16:58:12.265740 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.265618    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="64bff7b3-d944-4359-8e51-c61f0e4ea5bd" containerName="main"
Sep 18 16:58:12.265740 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.265627    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8747c04b-a381-4e88-bed6-7c8ea1204031" containerName="main"
Sep 18 16:58:12.269716 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.269700    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:12.273223 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.273199    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Sep 18 16:58:12.273405 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.273199    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 18 16:58:12.273405 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.273242    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-5bw5t\""
Sep 18 16:58:12.273526 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.273512    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 18 16:58:12.292832 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.292807    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"]
Sep 18 16:58:12.358605 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.358565    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12da816c-e8fd-4ddb-a2e5-afcb91519d36-workload-certs\") pod \"router-gateway-1-openshift-default-6f9866b8dc-nkztd\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:12.358793 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.358621    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4z8h\" (UniqueName: \"kubernetes.io/projected/12da816c-e8fd-4ddb-a2e5-afcb91519d36-kube-api-access-s4z8h\") pod \"router-gateway-1-openshift-default-6f9866b8dc-nkztd\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:12.358793 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.358657    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12da816c-e8fd-4ddb-a2e5-afcb91519d36-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6f9866b8dc-nkztd\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:12.358793 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.358690    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12da816c-e8fd-4ddb-a2e5-afcb91519d36-istio-podinfo\") pod \"router-gateway-1-openshift-default-6f9866b8dc-nkztd\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:12.358793 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.358740    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12da816c-e8fd-4ddb-a2e5-afcb91519d36-credential-socket\") pod \"router-gateway-1-openshift-default-6f9866b8dc-nkztd\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:12.358984 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.358818    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12da816c-e8fd-4ddb-a2e5-afcb91519d36-istio-data\") pod \"router-gateway-1-openshift-default-6f9866b8dc-nkztd\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:12.358984 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.358883    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12da816c-e8fd-4ddb-a2e5-afcb91519d36-istio-envoy\") pod \"router-gateway-1-openshift-default-6f9866b8dc-nkztd\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:12.358984 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.358941    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12da816c-e8fd-4ddb-a2e5-afcb91519d36-istio-token\") pod \"router-gateway-1-openshift-default-6f9866b8dc-nkztd\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:12.359081 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.358990    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12da816c-e8fd-4ddb-a2e5-afcb91519d36-workload-socket\") pod \"router-gateway-1-openshift-default-6f9866b8dc-nkztd\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:12.459805 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.459768    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12da816c-e8fd-4ddb-a2e5-afcb91519d36-workload-socket\") pod \"router-gateway-1-openshift-default-6f9866b8dc-nkztd\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:12.459805 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.459797    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12da816c-e8fd-4ddb-a2e5-afcb91519d36-workload-certs\") pod \"router-gateway-1-openshift-default-6f9866b8dc-nkztd\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:12.460052 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.459827    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4z8h\" (UniqueName: \"kubernetes.io/projected/12da816c-e8fd-4ddb-a2e5-afcb91519d36-kube-api-access-s4z8h\") pod \"router-gateway-1-openshift-default-6f9866b8dc-nkztd\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:12.460052 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.459859    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12da816c-e8fd-4ddb-a2e5-afcb91519d36-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6f9866b8dc-nkztd\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:12.460052 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.459887    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12da816c-e8fd-4ddb-a2e5-afcb91519d36-istio-podinfo\") pod \"router-gateway-1-openshift-default-6f9866b8dc-nkztd\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:12.460052 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.459928    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12da816c-e8fd-4ddb-a2e5-afcb91519d36-credential-socket\") pod \"router-gateway-1-openshift-default-6f9866b8dc-nkztd\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:12.460052 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.459977    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12da816c-e8fd-4ddb-a2e5-afcb91519d36-istio-data\") pod \"router-gateway-1-openshift-default-6f9866b8dc-nkztd\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:12.460052 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.460025    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12da816c-e8fd-4ddb-a2e5-afcb91519d36-istio-envoy\") pod \"router-gateway-1-openshift-default-6f9866b8dc-nkztd\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:12.460052 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.460048    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12da816c-e8fd-4ddb-a2e5-afcb91519d36-istio-token\") pod \"router-gateway-1-openshift-default-6f9866b8dc-nkztd\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:12.460325 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.460260    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12da816c-e8fd-4ddb-a2e5-afcb91519d36-workload-certs\") pod \"router-gateway-1-openshift-default-6f9866b8dc-nkztd\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:12.460325 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.460312    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12da816c-e8fd-4ddb-a2e5-afcb91519d36-workload-socket\") pod \"router-gateway-1-openshift-default-6f9866b8dc-nkztd\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:12.460409 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.460381    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12da816c-e8fd-4ddb-a2e5-afcb91519d36-istio-data\") pod \"router-gateway-1-openshift-default-6f9866b8dc-nkztd\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:12.460549 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.460528    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12da816c-e8fd-4ddb-a2e5-afcb91519d36-credential-socket\") pod \"router-gateway-1-openshift-default-6f9866b8dc-nkztd\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:12.460886 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.460863    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12da816c-e8fd-4ddb-a2e5-afcb91519d36-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6f9866b8dc-nkztd\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:12.462564 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.462543    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12da816c-e8fd-4ddb-a2e5-afcb91519d36-istio-envoy\") pod \"router-gateway-1-openshift-default-6f9866b8dc-nkztd\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:12.462646 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.462591    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12da816c-e8fd-4ddb-a2e5-afcb91519d36-istio-podinfo\") pod \"router-gateway-1-openshift-default-6f9866b8dc-nkztd\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:12.468301 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.468277    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12da816c-e8fd-4ddb-a2e5-afcb91519d36-istio-token\") pod \"router-gateway-1-openshift-default-6f9866b8dc-nkztd\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:12.468371 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.468298    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4z8h\" (UniqueName: \"kubernetes.io/projected/12da816c-e8fd-4ddb-a2e5-afcb91519d36-kube-api-access-s4z8h\") pod \"router-gateway-1-openshift-default-6f9866b8dc-nkztd\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:12.579546 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.579472    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:12.858593 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.858510    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"]
Sep 18 16:58:12.861229 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:58:12.861194    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12da816c_e8fd_4ddb_a2e5_afcb91519d36.slice/crio-fe382ef8c086c3f115d18f92d117bfd9c58ef3afe48e6e49d37dfe354a24cf08 WatchSource:0}: Error finding container fe382ef8c086c3f115d18f92d117bfd9c58ef3afe48e6e49d37dfe354a24cf08: Status 404 returned error can't find the container with id fe382ef8c086c3f115d18f92d117bfd9c58ef3afe48e6e49d37dfe354a24cf08
Sep 18 16:58:12.863191 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.863154    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 16:58:12.863322 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.863256    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 16:58:12.863322 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:12.863303    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 16:58:13.170603 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:13.170508    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd" event={"ID":"12da816c-e8fd-4ddb-a2e5-afcb91519d36","Type":"ContainerStarted","Data":"572efe14cc1c61872ccc5b69c724f3bc0ea385bf9d285e7caaaefc539993f8e7"}
Sep 18 16:58:13.170603 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:13.170570    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd" event={"ID":"12da816c-e8fd-4ddb-a2e5-afcb91519d36","Type":"ContainerStarted","Data":"fe382ef8c086c3f115d18f92d117bfd9c58ef3afe48e6e49d37dfe354a24cf08"}
Sep 18 16:58:13.191559 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:13.191502    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd" podStartSLOduration=1.19148179 podStartE2EDuration="1.19148179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:13.190873337 +0000 UTC m=+2532.973875377" watchObservedRunningTime="2026-09-18 16:58:13.19148179 +0000 UTC m=+2532.974483816"
Sep 18 16:58:13.580003 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:13.579960    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:13.584884 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:13.584859    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:14.174761 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:14.174723    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:14.175646 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:14.175627    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:29.215282 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:29.215233    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd" podUID="12da816c-e8fd-4ddb-a2e5-afcb91519d36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": read tcp 10.133.0.2:48172->10.133.0.50:15021: read: connection reset by peer"
Sep 18 16:58:29.236488 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:29.236459    2585 generic.go:356] "Generic (PLEG): container finished" podID="12da816c-e8fd-4ddb-a2e5-afcb91519d36" containerID="572efe14cc1c61872ccc5b69c724f3bc0ea385bf9d285e7caaaefc539993f8e7" exitCode=0
Sep 18 16:58:29.236601 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:29.236528    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd" event={"ID":"12da816c-e8fd-4ddb-a2e5-afcb91519d36","Type":"ContainerDied","Data":"572efe14cc1c61872ccc5b69c724f3bc0ea385bf9d285e7caaaefc539993f8e7"}
Sep 18 16:58:29.236971 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:29.236945    2585 scope.go:117] "RemoveContainer" containerID="572efe14cc1c61872ccc5b69c724f3bc0ea385bf9d285e7caaaefc539993f8e7"
Sep 18 16:58:29.238025 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:29.237998    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 16:58:29.238115 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:29.238067    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 16:58:29.238115 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:29.238097    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 16:58:30.245053 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:30.245019    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd" event={"ID":"12da816c-e8fd-4ddb-a2e5-afcb91519d36","Type":"ContainerStarted","Data":"713c809026732a0f750e9cc22fb1881aa15acc641f6cc2c984424a1a1eaedc95"}
Sep 18 16:58:30.579724 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:30.579634    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:30.579865 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:30.579841    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd" podUID="12da816c-e8fd-4ddb-a2e5-afcb91519d36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 18 16:58:31.580655 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:31.580607    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd" podUID="12da816c-e8fd-4ddb-a2e5-afcb91519d36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 18 16:58:32.580017 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:32.579973    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd" podUID="12da816c-e8fd-4ddb-a2e5-afcb91519d36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 18 16:58:33.580950 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:33.580882    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd" podUID="12da816c-e8fd-4ddb-a2e5-afcb91519d36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 18 16:58:34.580764 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:34.580723    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd" podUID="12da816c-e8fd-4ddb-a2e5-afcb91519d36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 18 16:58:35.584093 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:35.584059    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:35.584560 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:35.584339    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:58:35.585170 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:58:35.585154    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 16:59:11.300336 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:59:11.300295    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl"]
Sep 18 16:59:11.303214 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:59:11.303189    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl"
Sep 18 16:59:11.307359 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:59:11.307337    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-mmq9x\""
Sep 18 16:59:11.307512 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:59:11.307409    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 18 16:59:11.307512 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:59:11.307414    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 18 16:59:11.307512 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:59:11.307477    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-v27sr\""
Sep 18 16:59:11.307512 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:59:11.307337    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 18 16:59:11.312542 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:59:11.312521    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl"]
Sep 18 16:59:11.351602 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:59:11.351566    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7747bd2c-ab4d-45f6-87d7-a460912d08fe-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl\" (UID: \"7747bd2c-ab4d-45f6-87d7-a460912d08fe\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl"
Sep 18 16:59:11.351759 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:59:11.351646    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk2sq\" (UniqueName: \"kubernetes.io/projected/7747bd2c-ab4d-45f6-87d7-a460912d08fe-kube-api-access-fk2sq\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl\" (UID: \"7747bd2c-ab4d-45f6-87d7-a460912d08fe\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl"
Sep 18 16:59:11.452775 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:59:11.452741    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7747bd2c-ab4d-45f6-87d7-a460912d08fe-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl\" (UID: \"7747bd2c-ab4d-45f6-87d7-a460912d08fe\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl"
Sep 18 16:59:11.452976 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:59:11.452803    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk2sq\" (UniqueName: \"kubernetes.io/projected/7747bd2c-ab4d-45f6-87d7-a460912d08fe-kube-api-access-fk2sq\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl\" (UID: \"7747bd2c-ab4d-45f6-87d7-a460912d08fe\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl"
Sep 18 16:59:11.455666 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:59:11.455632    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7747bd2c-ab4d-45f6-87d7-a460912d08fe-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl\" (UID: \"7747bd2c-ab4d-45f6-87d7-a460912d08fe\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl"
Sep 18 16:59:11.461822 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:59:11.461793    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk2sq\" (UniqueName: \"kubernetes.io/projected/7747bd2c-ab4d-45f6-87d7-a460912d08fe-kube-api-access-fk2sq\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl\" (UID: \"7747bd2c-ab4d-45f6-87d7-a460912d08fe\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl"
Sep 18 16:59:11.614087 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:59:11.614005    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl"
Sep 18 16:59:11.892964 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:59:11.892863    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl"]
Sep 18 16:59:11.896164 ip-10-0-133-147 kubenswrapper[2585]: W0918 16:59:11.896136    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7747bd2c_ab4d_45f6_87d7_a460912d08fe.slice/crio-9ded0f3c1021c4760c90ade660d944738a104b7c9d85d9f1dae1c15b5557c48f WatchSource:0}: Error finding container 9ded0f3c1021c4760c90ade660d944738a104b7c9d85d9f1dae1c15b5557c48f: Status 404 returned error can't find the container with id 9ded0f3c1021c4760c90ade660d944738a104b7c9d85d9f1dae1c15b5557c48f
Sep 18 16:59:12.410870 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:59:12.410834    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl" event={"ID":"7747bd2c-ab4d-45f6-87d7-a460912d08fe","Type":"ContainerStarted","Data":"28666f89a4898f82d7fa922da1a962dcacd391a8630a641b5d40fe7b9c493578"}
Sep 18 16:59:12.411282 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:59:12.410885    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl" event={"ID":"7747bd2c-ab4d-45f6-87d7-a460912d08fe","Type":"ContainerStarted","Data":"9ded0f3c1021c4760c90ade660d944738a104b7c9d85d9f1dae1c15b5557c48f"}
Sep 18 16:59:12.411282 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:59:12.410947    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl"
Sep 18 16:59:12.430210 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:59:12.430137    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl" podStartSLOduration=1.430113556 podStartE2EDuration="1.430113556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:59:12.427698484 +0000 UTC m=+2592.210700511" watchObservedRunningTime="2026-09-18 16:59:12.430113556 +0000 UTC m=+2592.213115581"
Sep 18 16:59:43.415811 ip-10-0-133-147 kubenswrapper[2585]: I0918 16:59:43.415780    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl"
Sep 18 17:00:01.190036 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:01.190006    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"]
Sep 18 17:00:01.192379 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:01.192358    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"
Sep 18 17:00:01.195234 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:01.195212    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 18 17:00:01.206376 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:01.206352    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"]
Sep 18 17:00:01.258946 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:01.258906    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cxhp\" (UniqueName: \"kubernetes.io/projected/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-kube-api-access-4cxhp\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx\" (UID: \"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"
Sep 18 17:00:01.258946 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:01.258949    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx\" (UID: \"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"
Sep 18 17:00:01.259114 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:01.258980    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx\" (UID: \"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"
Sep 18 17:00:01.259114 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:01.259048    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx\" (UID: \"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"
Sep 18 17:00:01.259114 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:01.259087    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx\" (UID: \"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"
Sep 18 17:00:01.259212 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:01.259112    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx\" (UID: \"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"
Sep 18 17:00:01.359648 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:01.359614    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx\" (UID: \"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"
Sep 18 17:00:01.359838 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:01.359679    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cxhp\" (UniqueName: \"kubernetes.io/projected/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-kube-api-access-4cxhp\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx\" (UID: \"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"
Sep 18 17:00:01.359838 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:01.359702    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx\" (UID: \"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"
Sep 18 17:00:01.359838 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:01.359736    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx\" (UID: \"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"
Sep 18 17:00:01.359838 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:01.359779    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx\" (UID: \"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"
Sep 18 17:00:01.360096 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:01.359938    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx\" (UID: \"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"
Sep 18 17:00:01.360192 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:01.360170    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx\" (UID: \"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"
Sep 18 17:00:01.360257 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:01.360228    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx\" (UID: \"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"
Sep 18 17:00:01.360324 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:01.360303    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx\" (UID: \"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"
Sep 18 17:00:01.362686 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:01.362655    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx\" (UID: \"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"
Sep 18 17:00:01.363240 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:01.363214    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx\" (UID: \"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"
Sep 18 17:00:01.370660 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:01.370633    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cxhp\" (UniqueName: \"kubernetes.io/projected/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-kube-api-access-4cxhp\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx\" (UID: \"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"
Sep 18 17:00:01.502951 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:01.502895    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"
Sep 18 17:00:01.781834 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:01.781744    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"]
Sep 18 17:00:01.784306 ip-10-0-133-147 kubenswrapper[2585]: W0918 17:00:01.784274    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode45cc3ff_c0e1_4881_a5c3_da1053cbbc60.slice/crio-4779cd0672d0c266c5bfc36e5e03c0ac8d9db8354aea5f202fbf51a353dcc728 WatchSource:0}: Error finding container 4779cd0672d0c266c5bfc36e5e03c0ac8d9db8354aea5f202fbf51a353dcc728: Status 404 returned error can't find the container with id 4779cd0672d0c266c5bfc36e5e03c0ac8d9db8354aea5f202fbf51a353dcc728
Sep 18 17:00:02.621894 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:02.621845    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx" event={"ID":"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60","Type":"ContainerStarted","Data":"f8c00cc09ae6785aad92451e42cebb55acb84044ab56bd27301f4887f291cdaa"}
Sep 18 17:00:02.622458 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:02.621934    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx" event={"ID":"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60","Type":"ContainerStarted","Data":"4779cd0672d0c266c5bfc36e5e03c0ac8d9db8354aea5f202fbf51a353dcc728"}
Sep 18 17:00:02.647459 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:02.647409    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx" podStartSLOduration=1.647393749 podStartE2EDuration="1.647393749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:00:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:00:02.645837575 +0000 UTC m=+2642.428839607" watchObservedRunningTime="2026-09-18 17:00:02.647393749 +0000 UTC m=+2642.430395773"
Sep 18 17:00:03.503516 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:03.503479    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"
Sep 18 17:00:11.503702 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:11.503667    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"
Sep 18 17:00:11.516631 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:11.516610    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"
Sep 18 17:00:11.670103 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:11.670077    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"
Sep 18 17:00:35.747228 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:35.747190    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"]
Sep 18 17:00:35.756718 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:00:35.756686    2585 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 17:00:35.756718 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 17:00:35.756718 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 17:00:35.756718 ip-10-0-133-147 kubenswrapper[2585]:  > containerID="f8c00cc09ae6785aad92451e42cebb55acb84044ab56bd27301f4887f291cdaa" cmd=["/bin/sleep","15"]
Sep 18 17:00:35.756993 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:35.756732    2585 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx" message="" err=<
Sep 18 17:00:35.756993 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 17:00:35.756993 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 17:00:35.756993 ip-10-0-133-147 kubenswrapper[2585]:  >
Sep 18 17:00:35.756993 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:00:35.756746    2585 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 17:00:35.756993 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 17:00:35.756993 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 17:00:35.756993 ip-10-0-133-147 kubenswrapper[2585]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx" podUID="e45cc3ff-c0e1-4881-a5c3-da1053cbbc60" containerName="main" containerID="cri-o://f8c00cc09ae6785aad92451e42cebb55acb84044ab56bd27301f4887f291cdaa"
Sep 18 17:00:35.756993 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:35.756780    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx" podUID="e45cc3ff-c0e1-4881-a5c3-da1053cbbc60" containerName="main" containerID="cri-o://f8c00cc09ae6785aad92451e42cebb55acb84044ab56bd27301f4887f291cdaa" gracePeriod=60
Sep 18 17:00:36.101147 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:36.101110    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"
Sep 18 17:00:36.257733 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:36.257696    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-home\") pod \"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60\" (UID: \"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60\") "
Sep 18 17:00:36.257968 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:36.257747    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cxhp\" (UniqueName: \"kubernetes.io/projected/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-kube-api-access-4cxhp\") pod \"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60\" (UID: \"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60\") "
Sep 18 17:00:36.257968 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:36.257774    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-dshm\") pod \"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60\" (UID: \"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60\") "
Sep 18 17:00:36.257968 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:36.257827    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-model-cache\") pod \"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60\" (UID: \"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60\") "
Sep 18 17:00:36.257968 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:36.257884    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-tmp-dir\") pod \"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60\" (UID: \"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60\") "
Sep 18 17:00:36.257968 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:36.257942    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-tls-certs\") pod \"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60\" (UID: \"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60\") "
Sep 18 17:00:36.258238 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:36.258003    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-home" (OuterVolumeSpecName: "home") pod "e45cc3ff-c0e1-4881-a5c3-da1053cbbc60" (UID: "e45cc3ff-c0e1-4881-a5c3-da1053cbbc60"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:00:36.258238 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:36.258147    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-model-cache" (OuterVolumeSpecName: "model-cache") pod "e45cc3ff-c0e1-4881-a5c3-da1053cbbc60" (UID: "e45cc3ff-c0e1-4881-a5c3-da1053cbbc60"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:00:36.258238 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:36.258218    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e45cc3ff-c0e1-4881-a5c3-da1053cbbc60" (UID: "e45cc3ff-c0e1-4881-a5c3-da1053cbbc60"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:00:36.258398 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:36.258244    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-home\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:00:36.258398 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:36.258258    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-model-cache\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:00:36.260367 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:36.260339    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-dshm" (OuterVolumeSpecName: "dshm") pod "e45cc3ff-c0e1-4881-a5c3-da1053cbbc60" (UID: "e45cc3ff-c0e1-4881-a5c3-da1053cbbc60"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:00:36.260367 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:36.260335    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e45cc3ff-c0e1-4881-a5c3-da1053cbbc60" (UID: "e45cc3ff-c0e1-4881-a5c3-da1053cbbc60"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:00:36.260367 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:36.260360    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-kube-api-access-4cxhp" (OuterVolumeSpecName: "kube-api-access-4cxhp") pod "e45cc3ff-c0e1-4881-a5c3-da1053cbbc60" (UID: "e45cc3ff-c0e1-4881-a5c3-da1053cbbc60"). InnerVolumeSpecName "kube-api-access-4cxhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:00:36.359538 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:36.359459    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-tmp-dir\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:00:36.359538 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:36.359487    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:00:36.359538 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:36.359500    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cxhp\" (UniqueName: \"kubernetes.io/projected/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-kube-api-access-4cxhp\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:00:36.359538 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:36.359511    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60-dshm\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:00:36.762300 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:36.762259    2585 generic.go:356] "Generic (PLEG): container finished" podID="e45cc3ff-c0e1-4881-a5c3-da1053cbbc60" containerID="f8c00cc09ae6785aad92451e42cebb55acb84044ab56bd27301f4887f291cdaa" exitCode=0
Sep 18 17:00:36.762748 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:36.762304    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"
Sep 18 17:00:36.762748 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:36.762327    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx" event={"ID":"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60","Type":"ContainerDied","Data":"f8c00cc09ae6785aad92451e42cebb55acb84044ab56bd27301f4887f291cdaa"}
Sep 18 17:00:36.762748 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:36.762384    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx" event={"ID":"e45cc3ff-c0e1-4881-a5c3-da1053cbbc60","Type":"ContainerDied","Data":"4779cd0672d0c266c5bfc36e5e03c0ac8d9db8354aea5f202fbf51a353dcc728"}
Sep 18 17:00:36.762748 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:36.762417    2585 scope.go:117] "RemoveContainer" containerID="f8c00cc09ae6785aad92451e42cebb55acb84044ab56bd27301f4887f291cdaa"
Sep 18 17:00:36.773110 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:36.773090    2585 scope.go:117] "RemoveContainer" containerID="f8c00cc09ae6785aad92451e42cebb55acb84044ab56bd27301f4887f291cdaa"
Sep 18 17:00:36.773430 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:00:36.773407    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8c00cc09ae6785aad92451e42cebb55acb84044ab56bd27301f4887f291cdaa\": container with ID starting with f8c00cc09ae6785aad92451e42cebb55acb84044ab56bd27301f4887f291cdaa not found: ID does not exist" containerID="f8c00cc09ae6785aad92451e42cebb55acb84044ab56bd27301f4887f291cdaa"
Sep 18 17:00:36.773528 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:36.773435    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c00cc09ae6785aad92451e42cebb55acb84044ab56bd27301f4887f291cdaa"} err="failed to get container status \"f8c00cc09ae6785aad92451e42cebb55acb84044ab56bd27301f4887f291cdaa\": rpc error: code = NotFound desc = could not find container \"f8c00cc09ae6785aad92451e42cebb55acb84044ab56bd27301f4887f291cdaa\": container with ID starting with f8c00cc09ae6785aad92451e42cebb55acb84044ab56bd27301f4887f291cdaa not found: ID does not exist"
Sep 18 17:00:36.781740 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:36.781687    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"]
Sep 18 17:00:36.783793 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:36.783774    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-d76rx"]
Sep 18 17:00:38.393177 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:38.393135    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e45cc3ff-c0e1-4881-a5c3-da1053cbbc60" path="/var/lib/kubelet/pods/e45cc3ff-c0e1-4881-a5c3-da1053cbbc60/volumes"
Sep 18 17:00:53.119862 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:53.119773    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"]
Sep 18 17:00:53.120394 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:53.120027    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd" podUID="12da816c-e8fd-4ddb-a2e5-afcb91519d36" containerName="istio-proxy" containerID="cri-o://713c809026732a0f750e9cc22fb1881aa15acc641f6cc2c984424a1a1eaedc95" gracePeriod=30
Sep 18 17:00:58.370065 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.370039    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 17:00:58.441747 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.441715    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12da816c-e8fd-4ddb-a2e5-afcb91519d36-istiod-ca-cert\") pod \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") "
Sep 18 17:00:58.441965 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.441762    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12da816c-e8fd-4ddb-a2e5-afcb91519d36-istio-envoy\") pod \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") "
Sep 18 17:00:58.441965 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.441802    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12da816c-e8fd-4ddb-a2e5-afcb91519d36-istio-data\") pod \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") "
Sep 18 17:00:58.441965 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.441829    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12da816c-e8fd-4ddb-a2e5-afcb91519d36-workload-socket\") pod \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") "
Sep 18 17:00:58.441965 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.441855    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12da816c-e8fd-4ddb-a2e5-afcb91519d36-istio-token\") pod \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") "
Sep 18 17:00:58.441965 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.441891    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12da816c-e8fd-4ddb-a2e5-afcb91519d36-workload-certs\") pod \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") "
Sep 18 17:00:58.441965 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.441935    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12da816c-e8fd-4ddb-a2e5-afcb91519d36-credential-socket\") pod \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") "
Sep 18 17:00:58.442297 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.441975    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12da816c-e8fd-4ddb-a2e5-afcb91519d36-istio-podinfo\") pod \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") "
Sep 18 17:00:58.442297 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.442009    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4z8h\" (UniqueName: \"kubernetes.io/projected/12da816c-e8fd-4ddb-a2e5-afcb91519d36-kube-api-access-s4z8h\") pod \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\" (UID: \"12da816c-e8fd-4ddb-a2e5-afcb91519d36\") "
Sep 18 17:00:58.442297 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.442208    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12da816c-e8fd-4ddb-a2e5-afcb91519d36-istio-data" (OuterVolumeSpecName: "istio-data") pod "12da816c-e8fd-4ddb-a2e5-afcb91519d36" (UID: "12da816c-e8fd-4ddb-a2e5-afcb91519d36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:00:58.442466 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.442252    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12da816c-e8fd-4ddb-a2e5-afcb91519d36-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "12da816c-e8fd-4ddb-a2e5-afcb91519d36" (UID: "12da816c-e8fd-4ddb-a2e5-afcb91519d36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:00:58.442466 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.442295    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12da816c-e8fd-4ddb-a2e5-afcb91519d36-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "12da816c-e8fd-4ddb-a2e5-afcb91519d36" (UID: "12da816c-e8fd-4ddb-a2e5-afcb91519d36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:00:58.442466 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.442346    2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12da816c-e8fd-4ddb-a2e5-afcb91519d36-istio-data\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:00:58.442466 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.442397    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12da816c-e8fd-4ddb-a2e5-afcb91519d36-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "12da816c-e8fd-4ddb-a2e5-afcb91519d36" (UID: "12da816c-e8fd-4ddb-a2e5-afcb91519d36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:00:58.442466 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.442408    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12da816c-e8fd-4ddb-a2e5-afcb91519d36-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "12da816c-e8fd-4ddb-a2e5-afcb91519d36" (UID: "12da816c-e8fd-4ddb-a2e5-afcb91519d36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:00:58.444557 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.444527    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12da816c-e8fd-4ddb-a2e5-afcb91519d36-kube-api-access-s4z8h" (OuterVolumeSpecName: "kube-api-access-s4z8h") pod "12da816c-e8fd-4ddb-a2e5-afcb91519d36" (UID: "12da816c-e8fd-4ddb-a2e5-afcb91519d36"). InnerVolumeSpecName "kube-api-access-s4z8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:00:58.444677 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.444629    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12da816c-e8fd-4ddb-a2e5-afcb91519d36-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "12da816c-e8fd-4ddb-a2e5-afcb91519d36" (UID: "12da816c-e8fd-4ddb-a2e5-afcb91519d36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:00:58.444907 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.444889    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12da816c-e8fd-4ddb-a2e5-afcb91519d36-istio-token" (OuterVolumeSpecName: "istio-token") pod "12da816c-e8fd-4ddb-a2e5-afcb91519d36" (UID: "12da816c-e8fd-4ddb-a2e5-afcb91519d36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:00:58.444907 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.444889    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12da816c-e8fd-4ddb-a2e5-afcb91519d36-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "12da816c-e8fd-4ddb-a2e5-afcb91519d36" (UID: "12da816c-e8fd-4ddb-a2e5-afcb91519d36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 17:00:58.543147 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.543099    2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12da816c-e8fd-4ddb-a2e5-afcb91519d36-workload-socket\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:00:58.543147 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.543138    2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12da816c-e8fd-4ddb-a2e5-afcb91519d36-istio-token\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:00:58.543147 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.543148    2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12da816c-e8fd-4ddb-a2e5-afcb91519d36-workload-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:00:58.543147 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.543156    2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12da816c-e8fd-4ddb-a2e5-afcb91519d36-credential-socket\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:00:58.543147 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.543165    2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12da816c-e8fd-4ddb-a2e5-afcb91519d36-istio-podinfo\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:00:58.543452 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.543176    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4z8h\" (UniqueName: \"kubernetes.io/projected/12da816c-e8fd-4ddb-a2e5-afcb91519d36-kube-api-access-s4z8h\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:00:58.543452 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.543187    2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12da816c-e8fd-4ddb-a2e5-afcb91519d36-istiod-ca-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:00:58.543452 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.543195    2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12da816c-e8fd-4ddb-a2e5-afcb91519d36-istio-envoy\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:00:58.850333 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.850291    2585 generic.go:356] "Generic (PLEG): container finished" podID="12da816c-e8fd-4ddb-a2e5-afcb91519d36" containerID="713c809026732a0f750e9cc22fb1881aa15acc641f6cc2c984424a1a1eaedc95" exitCode=0
Sep 18 17:00:58.850511 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.850346    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"
Sep 18 17:00:58.850511 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.850365    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd" event={"ID":"12da816c-e8fd-4ddb-a2e5-afcb91519d36","Type":"ContainerDied","Data":"713c809026732a0f750e9cc22fb1881aa15acc641f6cc2c984424a1a1eaedc95"}
Sep 18 17:00:58.850511 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.850421    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd" event={"ID":"12da816c-e8fd-4ddb-a2e5-afcb91519d36","Type":"ContainerDied","Data":"fe382ef8c086c3f115d18f92d117bfd9c58ef3afe48e6e49d37dfe354a24cf08"}
Sep 18 17:00:58.850511 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.850445    2585 scope.go:117] "RemoveContainer" containerID="713c809026732a0f750e9cc22fb1881aa15acc641f6cc2c984424a1a1eaedc95"
Sep 18 17:00:58.859491 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.859474    2585 scope.go:117] "RemoveContainer" containerID="572efe14cc1c61872ccc5b69c724f3bc0ea385bf9d285e7caaaefc539993f8e7"
Sep 18 17:00:58.867373 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.867355    2585 scope.go:117] "RemoveContainer" containerID="713c809026732a0f750e9cc22fb1881aa15acc641f6cc2c984424a1a1eaedc95"
Sep 18 17:00:58.867659 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:00:58.867640    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"713c809026732a0f750e9cc22fb1881aa15acc641f6cc2c984424a1a1eaedc95\": container with ID starting with 713c809026732a0f750e9cc22fb1881aa15acc641f6cc2c984424a1a1eaedc95 not found: ID does not exist" containerID="713c809026732a0f750e9cc22fb1881aa15acc641f6cc2c984424a1a1eaedc95"
Sep 18 17:00:58.867703 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.867670    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713c809026732a0f750e9cc22fb1881aa15acc641f6cc2c984424a1a1eaedc95"} err="failed to get container status \"713c809026732a0f750e9cc22fb1881aa15acc641f6cc2c984424a1a1eaedc95\": rpc error: code = NotFound desc = could not find container \"713c809026732a0f750e9cc22fb1881aa15acc641f6cc2c984424a1a1eaedc95\": container with ID starting with 713c809026732a0f750e9cc22fb1881aa15acc641f6cc2c984424a1a1eaedc95 not found: ID does not exist"
Sep 18 17:00:58.867703 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.867687    2585 scope.go:117] "RemoveContainer" containerID="572efe14cc1c61872ccc5b69c724f3bc0ea385bf9d285e7caaaefc539993f8e7"
Sep 18 17:00:58.867967 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:00:58.867945    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"572efe14cc1c61872ccc5b69c724f3bc0ea385bf9d285e7caaaefc539993f8e7\": container with ID starting with 572efe14cc1c61872ccc5b69c724f3bc0ea385bf9d285e7caaaefc539993f8e7 not found: ID does not exist" containerID="572efe14cc1c61872ccc5b69c724f3bc0ea385bf9d285e7caaaefc539993f8e7"
Sep 18 17:00:58.868035 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.867975    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"572efe14cc1c61872ccc5b69c724f3bc0ea385bf9d285e7caaaefc539993f8e7"} err="failed to get container status \"572efe14cc1c61872ccc5b69c724f3bc0ea385bf9d285e7caaaefc539993f8e7\": rpc error: code = NotFound desc = could not find container \"572efe14cc1c61872ccc5b69c724f3bc0ea385bf9d285e7caaaefc539993f8e7\": container with ID starting with 572efe14cc1c61872ccc5b69c724f3bc0ea385bf9d285e7caaaefc539993f8e7 not found: ID does not exist"
Sep 18 17:00:58.872648 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.872628    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"]
Sep 18 17:00:58.876769 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:00:58.876747    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-nkztd"]
Sep 18 17:01:00.393078 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:00.393044    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12da816c-e8fd-4ddb-a2e5-afcb91519d36" path="/var/lib/kubelet/pods/12da816c-e8fd-4ddb-a2e5-afcb91519d36/volumes"
Sep 18 17:01:02.495676 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:02.495646    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 17:01:02.499570 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:02.499548    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 17:01:02.499739 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:02.499689    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 17:01:02.503396 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:02.503375    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 17:01:04.863180 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:04.863143    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh"]
Sep 18 17:01:04.863601 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:04.863563    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12da816c-e8fd-4ddb-a2e5-afcb91519d36" containerName="istio-proxy"
Sep 18 17:01:04.863601 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:04.863579    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="12da816c-e8fd-4ddb-a2e5-afcb91519d36" containerName="istio-proxy"
Sep 18 17:01:04.863601 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:04.863597    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12da816c-e8fd-4ddb-a2e5-afcb91519d36" containerName="istio-proxy"
Sep 18 17:01:04.863730 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:04.863606    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="12da816c-e8fd-4ddb-a2e5-afcb91519d36" containerName="istio-proxy"
Sep 18 17:01:04.863730 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:04.863633    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e45cc3ff-c0e1-4881-a5c3-da1053cbbc60" containerName="main"
Sep 18 17:01:04.863730 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:04.863642    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e45cc3ff-c0e1-4881-a5c3-da1053cbbc60" containerName="main"
Sep 18 17:01:04.863730 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:04.863713    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e45cc3ff-c0e1-4881-a5c3-da1053cbbc60" containerName="main"
Sep 18 17:01:04.863730 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:04.863727    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="12da816c-e8fd-4ddb-a2e5-afcb91519d36" containerName="istio-proxy"
Sep 18 17:01:04.863896 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:04.863737    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="12da816c-e8fd-4ddb-a2e5-afcb91519d36" containerName="istio-proxy"
Sep 18 17:01:04.868309 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:04.868288    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh"
Sep 18 17:01:04.871095 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:04.871061    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 18 17:01:04.871230 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:04.871109    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 18 17:01:04.872224 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:04.872200    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-f6q6c\""
Sep 18 17:01:04.872224 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:04.872219    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-88fl5\""
Sep 18 17:01:04.872411 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:04.872291    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 18 17:01:04.877700 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:04.877665    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh"]
Sep 18 17:01:04.900529 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:04.900495    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f670e915-f703-417f-aad7-ebec5bedc07b-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh\" (UID: \"f670e915-f703-417f-aad7-ebec5bedc07b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh"
Sep 18 17:01:04.900529 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:04.900530    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgvmb\" (UniqueName: \"kubernetes.io/projected/f670e915-f703-417f-aad7-ebec5bedc07b-kube-api-access-tgvmb\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh\" (UID: \"f670e915-f703-417f-aad7-ebec5bedc07b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh"
Sep 18 17:01:05.001838 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:05.001796    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f670e915-f703-417f-aad7-ebec5bedc07b-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh\" (UID: \"f670e915-f703-417f-aad7-ebec5bedc07b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh"
Sep 18 17:01:05.001838 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:05.001838    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgvmb\" (UniqueName: \"kubernetes.io/projected/f670e915-f703-417f-aad7-ebec5bedc07b-kube-api-access-tgvmb\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh\" (UID: \"f670e915-f703-417f-aad7-ebec5bedc07b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh"
Sep 18 17:01:05.004617 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:05.004592    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f670e915-f703-417f-aad7-ebec5bedc07b-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh\" (UID: \"f670e915-f703-417f-aad7-ebec5bedc07b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh"
Sep 18 17:01:05.010835 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:05.010810    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgvmb\" (UniqueName: \"kubernetes.io/projected/f670e915-f703-417f-aad7-ebec5bedc07b-kube-api-access-tgvmb\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh\" (UID: \"f670e915-f703-417f-aad7-ebec5bedc07b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh"
Sep 18 17:01:05.180283 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:05.180188    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh"
Sep 18 17:01:05.454216 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:05.454190    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh"]
Sep 18 17:01:05.456792 ip-10-0-133-147 kubenswrapper[2585]: W0918 17:01:05.456762    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf670e915_f703_417f_aad7_ebec5bedc07b.slice/crio-2df9a35fd2fa0210767b984c049ec85ddd54cf47edc20fcca8a299730d801d68 WatchSource:0}: Error finding container 2df9a35fd2fa0210767b984c049ec85ddd54cf47edc20fcca8a299730d801d68: Status 404 returned error can't find the container with id 2df9a35fd2fa0210767b984c049ec85ddd54cf47edc20fcca8a299730d801d68
Sep 18 17:01:05.887495 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:05.887453    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh" event={"ID":"f670e915-f703-417f-aad7-ebec5bedc07b","Type":"ContainerStarted","Data":"8dc11d6926f4ba4a231af59bd1e13df4dedddad8ddcacee02dea8c80040b0be0"}
Sep 18 17:01:05.888105 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:05.887513    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh" event={"ID":"f670e915-f703-417f-aad7-ebec5bedc07b","Type":"ContainerStarted","Data":"2df9a35fd2fa0210767b984c049ec85ddd54cf47edc20fcca8a299730d801d68"}
Sep 18 17:01:05.888105 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:05.887541    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh"
Sep 18 17:01:05.907737 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:05.907682    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh" podStartSLOduration=1.907665992 podStartE2EDuration="1.907665992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:01:05.906857068 +0000 UTC m=+2705.689859105" watchObservedRunningTime="2026-09-18 17:01:05.907665992 +0000 UTC m=+2705.690668017"
Sep 18 17:01:25.822130 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:25.822096    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl"]
Sep 18 17:01:33.416557 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:33.416511    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl" podUID="7747bd2c-ab4d-45f6-87d7-a460912d08fe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:01:36.893329 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:36.893293    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh"
Sep 18 17:01:40.842314 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:40.842253    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl" podUID="7747bd2c-ab4d-45f6-87d7-a460912d08fe" containerName="main" containerID="cri-o://28666f89a4898f82d7fa922da1a962dcacd391a8630a641b5d40fe7b9c493578" gracePeriod=45
Sep 18 17:01:43.416679 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:43.416635    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl" podUID="7747bd2c-ab4d-45f6-87d7-a460912d08fe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:01:53.416335 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:53.416294    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl" podUID="7747bd2c-ab4d-45f6-87d7-a460912d08fe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:01:53.416757 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:01:53.416408    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl"
Sep 18 17:02:03.415836 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:03.415793    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl" podUID="7747bd2c-ab4d-45f6-87d7-a460912d08fe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:02:11.089680 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:11.089656    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl"
Sep 18 17:02:11.132988 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:11.132889    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7747bd2c-ab4d-45f6-87d7-a460912d08fe-tls-certs\") pod \"7747bd2c-ab4d-45f6-87d7-a460912d08fe\" (UID: \"7747bd2c-ab4d-45f6-87d7-a460912d08fe\") "
Sep 18 17:02:11.133145 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:11.132990    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fk2sq\" (UniqueName: \"kubernetes.io/projected/7747bd2c-ab4d-45f6-87d7-a460912d08fe-kube-api-access-fk2sq\") pod \"7747bd2c-ab4d-45f6-87d7-a460912d08fe\" (UID: \"7747bd2c-ab4d-45f6-87d7-a460912d08fe\") "
Sep 18 17:02:11.135161 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:11.135130    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7747bd2c-ab4d-45f6-87d7-a460912d08fe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7747bd2c-ab4d-45f6-87d7-a460912d08fe" (UID: "7747bd2c-ab4d-45f6-87d7-a460912d08fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:02:11.135301 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:11.135285    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7747bd2c-ab4d-45f6-87d7-a460912d08fe-kube-api-access-fk2sq" (OuterVolumeSpecName: "kube-api-access-fk2sq") pod "7747bd2c-ab4d-45f6-87d7-a460912d08fe" (UID: "7747bd2c-ab4d-45f6-87d7-a460912d08fe"). InnerVolumeSpecName "kube-api-access-fk2sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:02:11.148109 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:11.148083    2585 generic.go:356] "Generic (PLEG): container finished" podID="7747bd2c-ab4d-45f6-87d7-a460912d08fe" containerID="28666f89a4898f82d7fa922da1a962dcacd391a8630a641b5d40fe7b9c493578" exitCode=0
Sep 18 17:02:11.148227 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:11.148127    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl"
Sep 18 17:02:11.148227 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:11.148138    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl" event={"ID":"7747bd2c-ab4d-45f6-87d7-a460912d08fe","Type":"ContainerDied","Data":"28666f89a4898f82d7fa922da1a962dcacd391a8630a641b5d40fe7b9c493578"}
Sep 18 17:02:11.148227 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:11.148180    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl" event={"ID":"7747bd2c-ab4d-45f6-87d7-a460912d08fe","Type":"ContainerDied","Data":"9ded0f3c1021c4760c90ade660d944738a104b7c9d85d9f1dae1c15b5557c48f"}
Sep 18 17:02:11.148227 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:11.148202    2585 scope.go:117] "RemoveContainer" containerID="28666f89a4898f82d7fa922da1a962dcacd391a8630a641b5d40fe7b9c493578"
Sep 18 17:02:11.157475 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:11.157456    2585 scope.go:117] "RemoveContainer" containerID="28666f89a4898f82d7fa922da1a962dcacd391a8630a641b5d40fe7b9c493578"
Sep 18 17:02:11.157724 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:02:11.157705    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28666f89a4898f82d7fa922da1a962dcacd391a8630a641b5d40fe7b9c493578\": container with ID starting with 28666f89a4898f82d7fa922da1a962dcacd391a8630a641b5d40fe7b9c493578 not found: ID does not exist" containerID="28666f89a4898f82d7fa922da1a962dcacd391a8630a641b5d40fe7b9c493578"
Sep 18 17:02:11.157786 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:11.157738    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28666f89a4898f82d7fa922da1a962dcacd391a8630a641b5d40fe7b9c493578"} err="failed to get container status \"28666f89a4898f82d7fa922da1a962dcacd391a8630a641b5d40fe7b9c493578\": rpc error: code = NotFound desc = could not find container \"28666f89a4898f82d7fa922da1a962dcacd391a8630a641b5d40fe7b9c493578\": container with ID starting with 28666f89a4898f82d7fa922da1a962dcacd391a8630a641b5d40fe7b9c493578 not found: ID does not exist"
Sep 18 17:02:11.172923 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:11.172885    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl"]
Sep 18 17:02:11.176635 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:11.176615    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schepwqxl"]
Sep 18 17:02:11.233755 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:11.233730    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk2sq\" (UniqueName: \"kubernetes.io/projected/7747bd2c-ab4d-45f6-87d7-a460912d08fe-kube-api-access-fk2sq\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:02:11.233755 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:11.233756    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7747bd2c-ab4d-45f6-87d7-a460912d08fe-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:02:12.392271 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:12.392186    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7747bd2c-ab4d-45f6-87d7-a460912d08fe" path="/var/lib/kubelet/pods/7747bd2c-ab4d-45f6-87d7-a460912d08fe/volumes"
Sep 18 17:02:27.172935 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.172881    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"]
Sep 18 17:02:27.173306 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.173223    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7747bd2c-ab4d-45f6-87d7-a460912d08fe" containerName="main"
Sep 18 17:02:27.173306 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.173237    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7747bd2c-ab4d-45f6-87d7-a460912d08fe" containerName="main"
Sep 18 17:02:27.173306 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.173290    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="7747bd2c-ab4d-45f6-87d7-a460912d08fe" containerName="main"
Sep 18 17:02:27.177419 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.177398    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:02:27.180322 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.180295    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 18 17:02:27.180458 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.180302    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 17:02:27.180580 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.180564    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 18 17:02:27.180654 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.180597    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 18 17:02:27.180654 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.180642    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-h4cw4\""
Sep 18 17:02:27.180770 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.180683    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 18 17:02:27.192894 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.192864    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"]
Sep 18 17:02:27.275852 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.275823    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfmds\" (UniqueName: \"kubernetes.io/projected/a47b2735-b04f-4626-874b-7bd05fbdfa64-kube-api-access-hfmds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:02:27.276049 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.275868    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a47b2735-b04f-4626-874b-7bd05fbdfa64-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:02:27.276049 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.275896    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a47b2735-b04f-4626-874b-7bd05fbdfa64-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:02:27.276049 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.275979    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a47b2735-b04f-4626-874b-7bd05fbdfa64-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:02:27.276049 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.276014    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a47b2735-b04f-4626-874b-7bd05fbdfa64-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:02:27.276240 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.276060    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a47b2735-b04f-4626-874b-7bd05fbdfa64-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:02:27.276240 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.276088    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a47b2735-b04f-4626-874b-7bd05fbdfa64-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:02:27.276240 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.276117    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a47b2735-b04f-4626-874b-7bd05fbdfa64-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:02:27.377271 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.377235    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a47b2735-b04f-4626-874b-7bd05fbdfa64-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:02:27.377271 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.377279    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a47b2735-b04f-4626-874b-7bd05fbdfa64-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:02:27.377525 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.377299    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a47b2735-b04f-4626-874b-7bd05fbdfa64-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:02:27.377525 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.377317    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a47b2735-b04f-4626-874b-7bd05fbdfa64-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:02:27.377525 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.377363    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a47b2735-b04f-4626-874b-7bd05fbdfa64-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:02:27.377525 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.377386    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a47b2735-b04f-4626-874b-7bd05fbdfa64-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:02:27.377525 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.377400    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a47b2735-b04f-4626-874b-7bd05fbdfa64-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:02:27.377525 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.377450    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfmds\" (UniqueName: \"kubernetes.io/projected/a47b2735-b04f-4626-874b-7bd05fbdfa64-kube-api-access-hfmds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:02:27.377843 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.377722    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a47b2735-b04f-4626-874b-7bd05fbdfa64-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:02:27.377843 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.377795    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a47b2735-b04f-4626-874b-7bd05fbdfa64-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:02:27.378142 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.378106    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a47b2735-b04f-4626-874b-7bd05fbdfa64-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:02:27.378260 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.378158    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a47b2735-b04f-4626-874b-7bd05fbdfa64-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:02:27.378260 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.378159    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a47b2735-b04f-4626-874b-7bd05fbdfa64-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:02:27.380052 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.380024    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a47b2735-b04f-4626-874b-7bd05fbdfa64-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:02:27.380135 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.380063    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a47b2735-b04f-4626-874b-7bd05fbdfa64-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:02:27.386201 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.386172    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfmds\" (UniqueName: \"kubernetes.io/projected/a47b2735-b04f-4626-874b-7bd05fbdfa64-kube-api-access-hfmds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:02:27.487779 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.487746    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:02:27.767720 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:27.767636    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"]
Sep 18 17:02:27.770636 ip-10-0-133-147 kubenswrapper[2585]: W0918 17:02:27.770606    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47b2735_b04f_4626_874b_7bd05fbdfa64.slice/crio-8c35f080439eec42edf3fd8b91993c312736263c8d8a6759e80a04dc88de9164 WatchSource:0}: Error finding container 8c35f080439eec42edf3fd8b91993c312736263c8d8a6759e80a04dc88de9164: Status 404 returned error can't find the container with id 8c35f080439eec42edf3fd8b91993c312736263c8d8a6759e80a04dc88de9164
Sep 18 17:02:28.225348 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:28.225235    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc" event={"ID":"a47b2735-b04f-4626-874b-7bd05fbdfa64","Type":"ContainerStarted","Data":"0fef7c16cbb253cf1d886a89f1e1486e1328e990416e4f1c6709d26cdaa704fc"}
Sep 18 17:02:28.225348 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:28.225320    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc" event={"ID":"a47b2735-b04f-4626-874b-7bd05fbdfa64","Type":"ContainerStarted","Data":"8c35f080439eec42edf3fd8b91993c312736263c8d8a6759e80a04dc88de9164"}
Sep 18 17:02:31.239853 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:31.239814    2585 generic.go:356] "Generic (PLEG): container finished" podID="a47b2735-b04f-4626-874b-7bd05fbdfa64" containerID="0fef7c16cbb253cf1d886a89f1e1486e1328e990416e4f1c6709d26cdaa704fc" exitCode=0
Sep 18 17:02:31.240271 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:31.239889    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc" event={"ID":"a47b2735-b04f-4626-874b-7bd05fbdfa64","Type":"ContainerDied","Data":"0fef7c16cbb253cf1d886a89f1e1486e1328e990416e4f1c6709d26cdaa704fc"}
Sep 18 17:02:31.241450 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:31.241435    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 17:02:59.384857 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:59.384757    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc" event={"ID":"a47b2735-b04f-4626-874b-7bd05fbdfa64","Type":"ContainerStarted","Data":"35b1e7556932b8fdbe7ac1729740cb1a6a9f9b4aafa0714a906bc147dadc34b8"}
Sep 18 17:02:59.409373 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:59.409320    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc" podStartSLOduration=1.922467346 podStartE2EDuration="32.409305197s" totalImagesPullingTime="27.845930914s" totalInitContainerRuntime="2.640906937s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:02:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:02:59.406841966 +0000 UTC m=+2819.189844014" watchObservedRunningTime="2026-09-18 17:02:59.409305197 +0000 UTC m=+2819.192307221"
Sep 18 17:02:59.488740 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:02:59.488708    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:03:07.488631 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:03:07.488587    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:03:07.490486 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:03:07.490453    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc" podUID="a47b2735-b04f-4626-874b-7bd05fbdfa64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 18 17:03:17.488893 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:03:17.488843    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc" podUID="a47b2735-b04f-4626-874b-7bd05fbdfa64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 18 17:03:19.713786 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:03:19.713748    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh"]
Sep 18 17:03:26.893751 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:03:26.893709    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh" podUID="f670e915-f703-417f-aad7-ebec5bedc07b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:03:27.488475 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:03:27.488427    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc" podUID="a47b2735-b04f-4626-874b-7bd05fbdfa64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 18 17:03:34.737498 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:03:34.737451    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh" podUID="f670e915-f703-417f-aad7-ebec5bedc07b" containerName="main" containerID="cri-o://8dc11d6926f4ba4a231af59bd1e13df4dedddad8ddcacee02dea8c80040b0be0" gracePeriod=45
Sep 18 17:03:36.894197 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:03:36.894159    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh" podUID="f670e915-f703-417f-aad7-ebec5bedc07b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:03:37.488725 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:03:37.488682    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc" podUID="a47b2735-b04f-4626-874b-7bd05fbdfa64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 18 17:03:46.893684 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:03:46.893594    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh" podUID="f670e915-f703-417f-aad7-ebec5bedc07b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:03:46.894090 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:03:46.893743    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh"
Sep 18 17:03:47.488426 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:03:47.488387    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc" podUID="a47b2735-b04f-4626-874b-7bd05fbdfa64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 18 17:03:56.893492 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:03:56.893440    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh" podUID="f670e915-f703-417f-aad7-ebec5bedc07b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:03:57.488322 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:03:57.488279    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc" podUID="a47b2735-b04f-4626-874b-7bd05fbdfa64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 18 17:04:04.976575 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:04:04.976540    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh"
Sep 18 17:04:05.157792 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:04:05.157690    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgvmb\" (UniqueName: \"kubernetes.io/projected/f670e915-f703-417f-aad7-ebec5bedc07b-kube-api-access-tgvmb\") pod \"f670e915-f703-417f-aad7-ebec5bedc07b\" (UID: \"f670e915-f703-417f-aad7-ebec5bedc07b\") "
Sep 18 17:04:05.158004 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:04:05.157815    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f670e915-f703-417f-aad7-ebec5bedc07b-tls-certs\") pod \"f670e915-f703-417f-aad7-ebec5bedc07b\" (UID: \"f670e915-f703-417f-aad7-ebec5bedc07b\") "
Sep 18 17:04:05.160250 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:04:05.160221    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f670e915-f703-417f-aad7-ebec5bedc07b-kube-api-access-tgvmb" (OuterVolumeSpecName: "kube-api-access-tgvmb") pod "f670e915-f703-417f-aad7-ebec5bedc07b" (UID: "f670e915-f703-417f-aad7-ebec5bedc07b"). InnerVolumeSpecName "kube-api-access-tgvmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:04:05.160358 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:04:05.160243    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f670e915-f703-417f-aad7-ebec5bedc07b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f670e915-f703-417f-aad7-ebec5bedc07b" (UID: "f670e915-f703-417f-aad7-ebec5bedc07b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:04:05.258784 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:04:05.258748    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f670e915-f703-417f-aad7-ebec5bedc07b-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:04:05.258784 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:04:05.258782    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgvmb\" (UniqueName: \"kubernetes.io/projected/f670e915-f703-417f-aad7-ebec5bedc07b-kube-api-access-tgvmb\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:04:05.668654 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:04:05.668617    2585 generic.go:356] "Generic (PLEG): container finished" podID="f670e915-f703-417f-aad7-ebec5bedc07b" containerID="8dc11d6926f4ba4a231af59bd1e13df4dedddad8ddcacee02dea8c80040b0be0" exitCode=0
Sep 18 17:04:05.668833 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:04:05.668673    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh"
Sep 18 17:04:05.668833 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:04:05.668703    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh" event={"ID":"f670e915-f703-417f-aad7-ebec5bedc07b","Type":"ContainerDied","Data":"8dc11d6926f4ba4a231af59bd1e13df4dedddad8ddcacee02dea8c80040b0be0"}
Sep 18 17:04:05.668833 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:04:05.668759    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh" event={"ID":"f670e915-f703-417f-aad7-ebec5bedc07b","Type":"ContainerDied","Data":"2df9a35fd2fa0210767b984c049ec85ddd54cf47edc20fcca8a299730d801d68"}
Sep 18 17:04:05.668833 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:04:05.668782    2585 scope.go:117] "RemoveContainer" containerID="8dc11d6926f4ba4a231af59bd1e13df4dedddad8ddcacee02dea8c80040b0be0"
Sep 18 17:04:05.677640 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:04:05.677621    2585 scope.go:117] "RemoveContainer" containerID="8dc11d6926f4ba4a231af59bd1e13df4dedddad8ddcacee02dea8c80040b0be0"
Sep 18 17:04:05.677951 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:04:05.677931    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dc11d6926f4ba4a231af59bd1e13df4dedddad8ddcacee02dea8c80040b0be0\": container with ID starting with 8dc11d6926f4ba4a231af59bd1e13df4dedddad8ddcacee02dea8c80040b0be0 not found: ID does not exist" containerID="8dc11d6926f4ba4a231af59bd1e13df4dedddad8ddcacee02dea8c80040b0be0"
Sep 18 17:04:05.678031 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:04:05.677959    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc11d6926f4ba4a231af59bd1e13df4dedddad8ddcacee02dea8c80040b0be0"} err="failed to get container status \"8dc11d6926f4ba4a231af59bd1e13df4dedddad8ddcacee02dea8c80040b0be0\": rpc error: code = NotFound desc = could not find container \"8dc11d6926f4ba4a231af59bd1e13df4dedddad8ddcacee02dea8c80040b0be0\": container with ID starting with 8dc11d6926f4ba4a231af59bd1e13df4dedddad8ddcacee02dea8c80040b0be0 not found: ID does not exist"
Sep 18 17:04:05.692863 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:04:05.692834    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh"]
Sep 18 17:04:05.696727 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:04:05.696704    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-scheqqxnh"]
Sep 18 17:04:06.393985 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:04:06.393946    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f670e915-f703-417f-aad7-ebec5bedc07b" path="/var/lib/kubelet/pods/f670e915-f703-417f-aad7-ebec5bedc07b/volumes"
Sep 18 17:04:07.488683 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:04:07.488638    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc" podUID="a47b2735-b04f-4626-874b-7bd05fbdfa64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 18 17:04:17.498037 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:04:17.498002    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:04:17.507672 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:04:17.507638    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:05:04.273587 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:04.273552    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"]
Sep 18 17:05:19.296654 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:19.296548    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc" podUID="a47b2735-b04f-4626-874b-7bd05fbdfa64" containerName="main" containerID="cri-o://35b1e7556932b8fdbe7ac1729740cb1a6a9f9b4aafa0714a906bc147dadc34b8" gracePeriod=45
Sep 18 17:05:19.499514 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:19.499471    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc" podUID="a47b2735-b04f-4626-874b-7bd05fbdfa64" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 18 17:05:19.950391 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:19.950368    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:05:19.982369 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:19.982339    2585 generic.go:356] "Generic (PLEG): container finished" podID="a47b2735-b04f-4626-874b-7bd05fbdfa64" containerID="35b1e7556932b8fdbe7ac1729740cb1a6a9f9b4aafa0714a906bc147dadc34b8" exitCode=0
Sep 18 17:05:19.982547 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:19.982406    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"
Sep 18 17:05:19.982547 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:19.982411    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc" event={"ID":"a47b2735-b04f-4626-874b-7bd05fbdfa64","Type":"ContainerDied","Data":"35b1e7556932b8fdbe7ac1729740cb1a6a9f9b4aafa0714a906bc147dadc34b8"}
Sep 18 17:05:19.982672 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:19.982549    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc" event={"ID":"a47b2735-b04f-4626-874b-7bd05fbdfa64","Type":"ContainerDied","Data":"8c35f080439eec42edf3fd8b91993c312736263c8d8a6759e80a04dc88de9164"}
Sep 18 17:05:19.982672 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:19.982571    2585 scope.go:117] "RemoveContainer" containerID="35b1e7556932b8fdbe7ac1729740cb1a6a9f9b4aafa0714a906bc147dadc34b8"
Sep 18 17:05:19.991864 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:19.991841    2585 scope.go:117] "RemoveContainer" containerID="0fef7c16cbb253cf1d886a89f1e1486e1328e990416e4f1c6709d26cdaa704fc"
Sep 18 17:05:20.000082 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.000066    2585 scope.go:117] "RemoveContainer" containerID="35b1e7556932b8fdbe7ac1729740cb1a6a9f9b4aafa0714a906bc147dadc34b8"
Sep 18 17:05:20.000383 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:05:20.000353    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35b1e7556932b8fdbe7ac1729740cb1a6a9f9b4aafa0714a906bc147dadc34b8\": container with ID starting with 35b1e7556932b8fdbe7ac1729740cb1a6a9f9b4aafa0714a906bc147dadc34b8 not found: ID does not exist" containerID="35b1e7556932b8fdbe7ac1729740cb1a6a9f9b4aafa0714a906bc147dadc34b8"
Sep 18 17:05:20.000504 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.000383    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35b1e7556932b8fdbe7ac1729740cb1a6a9f9b4aafa0714a906bc147dadc34b8"} err="failed to get container status \"35b1e7556932b8fdbe7ac1729740cb1a6a9f9b4aafa0714a906bc147dadc34b8\": rpc error: code = NotFound desc = could not find container \"35b1e7556932b8fdbe7ac1729740cb1a6a9f9b4aafa0714a906bc147dadc34b8\": container with ID starting with 35b1e7556932b8fdbe7ac1729740cb1a6a9f9b4aafa0714a906bc147dadc34b8 not found: ID does not exist"
Sep 18 17:05:20.000504 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.000399    2585 scope.go:117] "RemoveContainer" containerID="0fef7c16cbb253cf1d886a89f1e1486e1328e990416e4f1c6709d26cdaa704fc"
Sep 18 17:05:20.000660 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:05:20.000643    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fef7c16cbb253cf1d886a89f1e1486e1328e990416e4f1c6709d26cdaa704fc\": container with ID starting with 0fef7c16cbb253cf1d886a89f1e1486e1328e990416e4f1c6709d26cdaa704fc not found: ID does not exist" containerID="0fef7c16cbb253cf1d886a89f1e1486e1328e990416e4f1c6709d26cdaa704fc"
Sep 18 17:05:20.000712 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.000671    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fef7c16cbb253cf1d886a89f1e1486e1328e990416e4f1c6709d26cdaa704fc"} err="failed to get container status \"0fef7c16cbb253cf1d886a89f1e1486e1328e990416e4f1c6709d26cdaa704fc\": rpc error: code = NotFound desc = could not find container \"0fef7c16cbb253cf1d886a89f1e1486e1328e990416e4f1c6709d26cdaa704fc\": container with ID starting with 0fef7c16cbb253cf1d886a89f1e1486e1328e990416e4f1c6709d26cdaa704fc not found: ID does not exist"
Sep 18 17:05:20.063555 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.063532    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a47b2735-b04f-4626-874b-7bd05fbdfa64-tmp-dir\") pod \"a47b2735-b04f-4626-874b-7bd05fbdfa64\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") "
Sep 18 17:05:20.063642 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.063560    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a47b2735-b04f-4626-874b-7bd05fbdfa64-tls-certs\") pod \"a47b2735-b04f-4626-874b-7bd05fbdfa64\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") "
Sep 18 17:05:20.063642 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.063586    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a47b2735-b04f-4626-874b-7bd05fbdfa64-cabundle-cert\") pod \"a47b2735-b04f-4626-874b-7bd05fbdfa64\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") "
Sep 18 17:05:20.063642 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.063629    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfmds\" (UniqueName: \"kubernetes.io/projected/a47b2735-b04f-4626-874b-7bd05fbdfa64-kube-api-access-hfmds\") pod \"a47b2735-b04f-4626-874b-7bd05fbdfa64\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") "
Sep 18 17:05:20.063791 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.063664    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a47b2735-b04f-4626-874b-7bd05fbdfa64-model-cache\") pod \"a47b2735-b04f-4626-874b-7bd05fbdfa64\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") "
Sep 18 17:05:20.063791 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.063706    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a47b2735-b04f-4626-874b-7bd05fbdfa64-dshm\") pod \"a47b2735-b04f-4626-874b-7bd05fbdfa64\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") "
Sep 18 17:05:20.063791 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.063744    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a47b2735-b04f-4626-874b-7bd05fbdfa64-kserve-provision-location\") pod \"a47b2735-b04f-4626-874b-7bd05fbdfa64\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") "
Sep 18 17:05:20.063991 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.063796    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a47b2735-b04f-4626-874b-7bd05fbdfa64-home\") pod \"a47b2735-b04f-4626-874b-7bd05fbdfa64\" (UID: \"a47b2735-b04f-4626-874b-7bd05fbdfa64\") "
Sep 18 17:05:20.064052 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.063991    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a47b2735-b04f-4626-874b-7bd05fbdfa64-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a47b2735-b04f-4626-874b-7bd05fbdfa64" (UID: "a47b2735-b04f-4626-874b-7bd05fbdfa64"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:05:20.064131 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.064102    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a47b2735-b04f-4626-874b-7bd05fbdfa64-model-cache" (OuterVolumeSpecName: "model-cache") pod "a47b2735-b04f-4626-874b-7bd05fbdfa64" (UID: "a47b2735-b04f-4626-874b-7bd05fbdfa64"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:05:20.064234 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.064202    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a47b2735-b04f-4626-874b-7bd05fbdfa64-cabundle-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:05:20.064349 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.064289    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a47b2735-b04f-4626-874b-7bd05fbdfa64-home" (OuterVolumeSpecName: "home") pod "a47b2735-b04f-4626-874b-7bd05fbdfa64" (UID: "a47b2735-b04f-4626-874b-7bd05fbdfa64"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:05:20.066218 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.066187    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a47b2735-b04f-4626-874b-7bd05fbdfa64-kube-api-access-hfmds" (OuterVolumeSpecName: "kube-api-access-hfmds") pod "a47b2735-b04f-4626-874b-7bd05fbdfa64" (UID: "a47b2735-b04f-4626-874b-7bd05fbdfa64"). InnerVolumeSpecName "kube-api-access-hfmds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:05:20.066640 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.066607    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a47b2735-b04f-4626-874b-7bd05fbdfa64-dshm" (OuterVolumeSpecName: "dshm") pod "a47b2735-b04f-4626-874b-7bd05fbdfa64" (UID: "a47b2735-b04f-4626-874b-7bd05fbdfa64"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:05:20.066723 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.066661    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a47b2735-b04f-4626-874b-7bd05fbdfa64-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a47b2735-b04f-4626-874b-7bd05fbdfa64" (UID: "a47b2735-b04f-4626-874b-7bd05fbdfa64"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:05:20.075270 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.075246    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a47b2735-b04f-4626-874b-7bd05fbdfa64-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a47b2735-b04f-4626-874b-7bd05fbdfa64" (UID: "a47b2735-b04f-4626-874b-7bd05fbdfa64"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:05:20.086581 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.086555    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a47b2735-b04f-4626-874b-7bd05fbdfa64-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a47b2735-b04f-4626-874b-7bd05fbdfa64" (UID: "a47b2735-b04f-4626-874b-7bd05fbdfa64"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:05:20.164705 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.164670    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a47b2735-b04f-4626-874b-7bd05fbdfa64-kserve-provision-location\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:05:20.164705 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.164703    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a47b2735-b04f-4626-874b-7bd05fbdfa64-home\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:05:20.164887 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.164714    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a47b2735-b04f-4626-874b-7bd05fbdfa64-tmp-dir\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:05:20.164887 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.164724    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a47b2735-b04f-4626-874b-7bd05fbdfa64-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:05:20.164887 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.164732    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfmds\" (UniqueName: \"kubernetes.io/projected/a47b2735-b04f-4626-874b-7bd05fbdfa64-kube-api-access-hfmds\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:05:20.164887 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.164739    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a47b2735-b04f-4626-874b-7bd05fbdfa64-model-cache\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:05:20.164887 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.164747    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a47b2735-b04f-4626-874b-7bd05fbdfa64-dshm\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:05:20.307138 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.307107    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"]
Sep 18 17:05:20.312326 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.312304    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-56ccc664c8-m65mc"]
Sep 18 17:05:20.392053 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:05:20.392016    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a47b2735-b04f-4626-874b-7bd05fbdfa64" path="/var/lib/kubelet/pods/a47b2735-b04f-4626-874b-7bd05fbdfa64/volumes"
Sep 18 17:06:00.808383 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:00.808349    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"]
Sep 18 17:06:00.808857 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:00.808641    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f670e915-f703-417f-aad7-ebec5bedc07b" containerName="main"
Sep 18 17:06:00.808857 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:00.808653    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f670e915-f703-417f-aad7-ebec5bedc07b" containerName="main"
Sep 18 17:06:00.808857 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:00.808662    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a47b2735-b04f-4626-874b-7bd05fbdfa64" containerName="storage-initializer"
Sep 18 17:06:00.808857 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:00.808668    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a47b2735-b04f-4626-874b-7bd05fbdfa64" containerName="storage-initializer"
Sep 18 17:06:00.808857 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:00.808693    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a47b2735-b04f-4626-874b-7bd05fbdfa64" containerName="main"
Sep 18 17:06:00.808857 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:00.808699    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a47b2735-b04f-4626-874b-7bd05fbdfa64" containerName="main"
Sep 18 17:06:00.808857 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:00.808762    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f670e915-f703-417f-aad7-ebec5bedc07b" containerName="main"
Sep 18 17:06:00.808857 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:00.808777    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a47b2735-b04f-4626-874b-7bd05fbdfa64" containerName="main"
Sep 18 17:06:00.811711 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:00.811693    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:06:00.814437 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:00.814416    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 18 17:06:00.814662 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:00.814640    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 18 17:06:00.814744 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:00.814672    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 18 17:06:00.814744 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:00.814724    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 18 17:06:00.814873 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:00.814852    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-q6d7q\""
Sep 18 17:06:00.815183 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:00.815166    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 17:06:00.822896 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:00.822875    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"]
Sep 18 17:06:00.976924 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:00.976879    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:06:00.977102 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:00.976948    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:06:00.977102 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:00.977035    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:06:00.977102 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:00.977072    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:06:00.977216 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:00.977098    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:06:00.977216 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:00.977137    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:06:00.977216 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:00.977153    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:06:00.977216 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:00.977196    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5hr9\" (UniqueName: \"kubernetes.io/projected/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-kube-api-access-k5hr9\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:06:01.078233 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:01.078143    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:06:01.078233 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:01.078196    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:06:01.078233 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:01.078228    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:06:01.078487 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:01.078261    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:06:01.078487 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:01.078285    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:06:01.078487 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:01.078448    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5hr9\" (UniqueName: \"kubernetes.io/projected/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-kube-api-access-k5hr9\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:06:01.078644 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:01.078548    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:06:01.078644 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:01.078585    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:06:01.078644 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:01.078591    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:06:01.078793 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:01.078645    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:06:01.078793 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:01.078698    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:06:01.078891 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:01.078869    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:06:01.078994 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:01.078977    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:06:01.080759 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:01.080738    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:06:01.081079 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:01.081062    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:06:01.086508 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:01.086486    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5hr9\" (UniqueName: \"kubernetes.io/projected/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-kube-api-access-k5hr9\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:06:01.122554 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:01.122524    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:06:01.399084 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:01.398998    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"]
Sep 18 17:06:01.402495 ip-10-0-133-147 kubenswrapper[2585]: W0918 17:06:01.402466    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4178f8b9_46c9_4ce2_8e2c_b6b2b3fe8272.slice/crio-9bcf9845a481b0e9165cd294bbda97edafcca65e8ffbaae9ae733a19ce83376e WatchSource:0}: Error finding container 9bcf9845a481b0e9165cd294bbda97edafcca65e8ffbaae9ae733a19ce83376e: Status 404 returned error can't find the container with id 9bcf9845a481b0e9165cd294bbda97edafcca65e8ffbaae9ae733a19ce83376e
Sep 18 17:06:02.159356 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:02.159318    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5" event={"ID":"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272","Type":"ContainerStarted","Data":"c339be19e10a771e8158999e9d60e7c264f3fa3bf08cf537000d3567253e5c0c"}
Sep 18 17:06:02.159780 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:02.159379    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5" event={"ID":"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272","Type":"ContainerStarted","Data":"9bcf9845a481b0e9165cd294bbda97edafcca65e8ffbaae9ae733a19ce83376e"}
Sep 18 17:06:02.526348 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:02.526314    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 17:06:02.530686 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:02.530664    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 17:06:02.532755 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:02.532731    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 17:06:02.537350 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:02.537329    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 17:06:05.173000 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:05.172966    2585 generic.go:356] "Generic (PLEG): container finished" podID="4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272" containerID="c339be19e10a771e8158999e9d60e7c264f3fa3bf08cf537000d3567253e5c0c" exitCode=0
Sep 18 17:06:05.173351 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:05.173044    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5" event={"ID":"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272","Type":"ContainerDied","Data":"c339be19e10a771e8158999e9d60e7c264f3fa3bf08cf537000d3567253e5c0c"}
Sep 18 17:06:06.179047 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:06.179015    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5" event={"ID":"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272","Type":"ContainerStarted","Data":"26f075ce1eba3f72cc414bfb7fc8ad1598b2efa47f741180be2e9e9edec14b30"}
Sep 18 17:06:06.204831 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:06.204750    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5" podStartSLOduration=3.393937154 podStartE2EDuration="6.204732592s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.810795438s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:06:06.203255393 +0000 UTC m=+3005.986257416" watchObservedRunningTime="2026-09-18 17:06:06.204732592 +0000 UTC m=+3005.987734615"
Sep 18 17:06:07.122744 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:07.122649    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:06:11.123454 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:11.123409    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:06:11.125089 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:11.125059    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5" podUID="4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 18 17:06:21.123611 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:21.123555    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5" podUID="4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 18 17:06:31.123220 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:31.123171    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5" podUID="4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 18 17:06:41.124010 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:41.123959    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5" podUID="4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 18 17:06:51.123792 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:06:51.123692    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5" podUID="4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 18 17:07:01.123487 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:01.123446    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5" podUID="4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 18 17:07:11.123610 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:11.123554    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5" podUID="4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 18 17:07:21.132976 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:21.132940    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:07:21.140743 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:21.140717    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:07:22.798737 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:22.798702    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"]
Sep 18 17:07:37.821679 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:37.821637    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5" podUID="4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272" containerName="main" containerID="cri-o://26f075ce1eba3f72cc414bfb7fc8ad1598b2efa47f741180be2e9e9edec14b30" gracePeriod=45
Sep 18 17:07:38.133986 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.133862    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5" podUID="4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 18 17:07:38.471398 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.471376    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:07:38.564148 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.564114    2585 generic.go:356] "Generic (PLEG): container finished" podID="4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272" containerID="26f075ce1eba3f72cc414bfb7fc8ad1598b2efa47f741180be2e9e9edec14b30" exitCode=0
Sep 18 17:07:38.564316 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.564169    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"
Sep 18 17:07:38.564316 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.564185    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5" event={"ID":"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272","Type":"ContainerDied","Data":"26f075ce1eba3f72cc414bfb7fc8ad1598b2efa47f741180be2e9e9edec14b30"}
Sep 18 17:07:38.564316 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.564228    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5" event={"ID":"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272","Type":"ContainerDied","Data":"9bcf9845a481b0e9165cd294bbda97edafcca65e8ffbaae9ae733a19ce83376e"}
Sep 18 17:07:38.564316 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.564250    2585 scope.go:117] "RemoveContainer" containerID="26f075ce1eba3f72cc414bfb7fc8ad1598b2efa47f741180be2e9e9edec14b30"
Sep 18 17:07:38.572746 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.572728    2585 scope.go:117] "RemoveContainer" containerID="c339be19e10a771e8158999e9d60e7c264f3fa3bf08cf537000d3567253e5c0c"
Sep 18 17:07:38.580695 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.580675    2585 scope.go:117] "RemoveContainer" containerID="26f075ce1eba3f72cc414bfb7fc8ad1598b2efa47f741180be2e9e9edec14b30"
Sep 18 17:07:38.581002 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:07:38.580978    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26f075ce1eba3f72cc414bfb7fc8ad1598b2efa47f741180be2e9e9edec14b30\": container with ID starting with 26f075ce1eba3f72cc414bfb7fc8ad1598b2efa47f741180be2e9e9edec14b30 not found: ID does not exist" containerID="26f075ce1eba3f72cc414bfb7fc8ad1598b2efa47f741180be2e9e9edec14b30"
Sep 18 17:07:38.581060 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.581012    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26f075ce1eba3f72cc414bfb7fc8ad1598b2efa47f741180be2e9e9edec14b30"} err="failed to get container status \"26f075ce1eba3f72cc414bfb7fc8ad1598b2efa47f741180be2e9e9edec14b30\": rpc error: code = NotFound desc = could not find container \"26f075ce1eba3f72cc414bfb7fc8ad1598b2efa47f741180be2e9e9edec14b30\": container with ID starting with 26f075ce1eba3f72cc414bfb7fc8ad1598b2efa47f741180be2e9e9edec14b30 not found: ID does not exist"
Sep 18 17:07:38.581060 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.581031    2585 scope.go:117] "RemoveContainer" containerID="c339be19e10a771e8158999e9d60e7c264f3fa3bf08cf537000d3567253e5c0c"
Sep 18 17:07:38.581301 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:07:38.581283    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c339be19e10a771e8158999e9d60e7c264f3fa3bf08cf537000d3567253e5c0c\": container with ID starting with c339be19e10a771e8158999e9d60e7c264f3fa3bf08cf537000d3567253e5c0c not found: ID does not exist" containerID="c339be19e10a771e8158999e9d60e7c264f3fa3bf08cf537000d3567253e5c0c"
Sep 18 17:07:38.581353 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.581309    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c339be19e10a771e8158999e9d60e7c264f3fa3bf08cf537000d3567253e5c0c"} err="failed to get container status \"c339be19e10a771e8158999e9d60e7c264f3fa3bf08cf537000d3567253e5c0c\": rpc error: code = NotFound desc = could not find container \"c339be19e10a771e8158999e9d60e7c264f3fa3bf08cf537000d3567253e5c0c\": container with ID starting with c339be19e10a771e8158999e9d60e7c264f3fa3bf08cf537000d3567253e5c0c not found: ID does not exist"
Sep 18 17:07:38.598569 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.598543    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-model-cache\") pod \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") "
Sep 18 17:07:38.598719 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.598575    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5hr9\" (UniqueName: \"kubernetes.io/projected/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-kube-api-access-k5hr9\") pod \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") "
Sep 18 17:07:38.598719 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.598601    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-home\") pod \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") "
Sep 18 17:07:38.598719 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.598619    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-cabundle-cert\") pod \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") "
Sep 18 17:07:38.598719 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.598644    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-dshm\") pod \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") "
Sep 18 17:07:38.598719 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.598690    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-tls-certs\") pod \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") "
Sep 18 17:07:38.599015 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.598737    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-kserve-provision-location\") pod \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") "
Sep 18 17:07:38.599015 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.598794    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-tmp-dir\") pod \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\" (UID: \"4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272\") "
Sep 18 17:07:38.599134 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.599108    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272" (UID: "4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:07:38.599190 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.599126    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-home" (OuterVolumeSpecName: "home") pod "4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272" (UID: "4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:07:38.599351 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.599324    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272" (UID: "4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:07:38.599422 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.599365    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-model-cache" (OuterVolumeSpecName: "model-cache") pod "4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272" (UID: "4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:07:38.601440 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.601416    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-kube-api-access-k5hr9" (OuterVolumeSpecName: "kube-api-access-k5hr9") pod "4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272" (UID: "4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272"). InnerVolumeSpecName "kube-api-access-k5hr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:07:38.601705 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.601685    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272" (UID: "4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:07:38.601782 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.601693    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-dshm" (OuterVolumeSpecName: "dshm") pod "4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272" (UID: "4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:07:38.622510 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.622470    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272" (UID: "4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:07:38.700310 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.700273    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-tmp-dir\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:07:38.700310 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.700304    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-model-cache\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:07:38.700310 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.700315    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5hr9\" (UniqueName: \"kubernetes.io/projected/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-kube-api-access-k5hr9\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:07:38.700552 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.700326    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-home\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:07:38.700552 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.700335    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-cabundle-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:07:38.700552 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.700343    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-dshm\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:07:38.700552 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.700350    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:07:38.700552 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.700357    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272-kserve-provision-location\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:07:38.890019 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.889981    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"]
Sep 18 17:07:38.893967 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:38.893937    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-5864bc8b5-98zm5"]
Sep 18 17:07:40.392615 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:40.392579    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272" path="/var/lib/kubelet/pods/4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272/volumes"
Sep 18 17:07:40.996389 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:40.996357    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"]
Sep 18 17:07:40.996661 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:40.996648    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272" containerName="storage-initializer"
Sep 18 17:07:40.996706 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:40.996663    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272" containerName="storage-initializer"
Sep 18 17:07:40.996706 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:40.996676    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272" containerName="main"
Sep 18 17:07:40.996706 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:40.996683    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272" containerName="main"
Sep 18 17:07:40.996804 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:40.996730    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="4178f8b9-46c9-4ce2-8e2c-b6b2b3fe8272" containerName="main"
Sep 18 17:07:40.999475 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:40.999456    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:07:41.002286 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.002262    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-x5nt8\""
Sep 18 17:07:41.002643 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.002513    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 18 17:07:41.002643 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.002513    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 18 17:07:41.002643 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.002551    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 17:07:41.002643 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.002553    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Sep 18 17:07:41.002955 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.002650    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 18 17:07:41.011471 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.011447    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"]
Sep 18 17:07:41.121000 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.120959    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/265b26d9-1760-4b95-9dfe-784823c2afa5-cabundle-cert\") pod \"custom-route-timeout-test-kserve-667bcdf585-6jsg8\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:07:41.121000 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.121005    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/265b26d9-1760-4b95-9dfe-784823c2afa5-model-cache\") pod \"custom-route-timeout-test-kserve-667bcdf585-6jsg8\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:07:41.121213 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.121028    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/265b26d9-1760-4b95-9dfe-784823c2afa5-home\") pod \"custom-route-timeout-test-kserve-667bcdf585-6jsg8\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:07:41.121213 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.121044    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/265b26d9-1760-4b95-9dfe-784823c2afa5-dshm\") pod \"custom-route-timeout-test-kserve-667bcdf585-6jsg8\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:07:41.121213 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.121087    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdvkj\" (UniqueName: \"kubernetes.io/projected/265b26d9-1760-4b95-9dfe-784823c2afa5-kube-api-access-gdvkj\") pod \"custom-route-timeout-test-kserve-667bcdf585-6jsg8\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:07:41.121213 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.121169    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/265b26d9-1760-4b95-9dfe-784823c2afa5-tmp-dir\") pod \"custom-route-timeout-test-kserve-667bcdf585-6jsg8\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:07:41.121213 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.121196    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/265b26d9-1760-4b95-9dfe-784823c2afa5-tls-certs\") pod \"custom-route-timeout-test-kserve-667bcdf585-6jsg8\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:07:41.121375 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.121223    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/265b26d9-1760-4b95-9dfe-784823c2afa5-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-667bcdf585-6jsg8\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:07:41.222128 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.222093    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/265b26d9-1760-4b95-9dfe-784823c2afa5-cabundle-cert\") pod \"custom-route-timeout-test-kserve-667bcdf585-6jsg8\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:07:41.222341 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.222151    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/265b26d9-1760-4b95-9dfe-784823c2afa5-model-cache\") pod \"custom-route-timeout-test-kserve-667bcdf585-6jsg8\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:07:41.222341 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.222192    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/265b26d9-1760-4b95-9dfe-784823c2afa5-home\") pod \"custom-route-timeout-test-kserve-667bcdf585-6jsg8\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:07:41.222341 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.222217    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/265b26d9-1760-4b95-9dfe-784823c2afa5-dshm\") pod \"custom-route-timeout-test-kserve-667bcdf585-6jsg8\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:07:41.222523 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.222405    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdvkj\" (UniqueName: \"kubernetes.io/projected/265b26d9-1760-4b95-9dfe-784823c2afa5-kube-api-access-gdvkj\") pod \"custom-route-timeout-test-kserve-667bcdf585-6jsg8\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:07:41.222523 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.222491    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/265b26d9-1760-4b95-9dfe-784823c2afa5-tmp-dir\") pod \"custom-route-timeout-test-kserve-667bcdf585-6jsg8\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:07:41.222632 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.222526    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/265b26d9-1760-4b95-9dfe-784823c2afa5-tls-certs\") pod \"custom-route-timeout-test-kserve-667bcdf585-6jsg8\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:07:41.222632 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.222571    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/265b26d9-1760-4b95-9dfe-784823c2afa5-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-667bcdf585-6jsg8\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:07:41.222751 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.222651    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/265b26d9-1760-4b95-9dfe-784823c2afa5-model-cache\") pod \"custom-route-timeout-test-kserve-667bcdf585-6jsg8\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:07:41.223038 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.223007    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/265b26d9-1760-4b95-9dfe-784823c2afa5-cabundle-cert\") pod \"custom-route-timeout-test-kserve-667bcdf585-6jsg8\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:07:41.223183 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.223137    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/265b26d9-1760-4b95-9dfe-784823c2afa5-tmp-dir\") pod \"custom-route-timeout-test-kserve-667bcdf585-6jsg8\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:07:41.223183 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.223169    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/265b26d9-1760-4b95-9dfe-784823c2afa5-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-667bcdf585-6jsg8\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:07:41.223307 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.223294    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/265b26d9-1760-4b95-9dfe-784823c2afa5-home\") pod \"custom-route-timeout-test-kserve-667bcdf585-6jsg8\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:07:41.225513 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.225489    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/265b26d9-1760-4b95-9dfe-784823c2afa5-tls-certs\") pod \"custom-route-timeout-test-kserve-667bcdf585-6jsg8\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:07:41.225619 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.225558    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/265b26d9-1760-4b95-9dfe-784823c2afa5-dshm\") pod \"custom-route-timeout-test-kserve-667bcdf585-6jsg8\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:07:41.234711 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.234687    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdvkj\" (UniqueName: \"kubernetes.io/projected/265b26d9-1760-4b95-9dfe-784823c2afa5-kube-api-access-gdvkj\") pod \"custom-route-timeout-test-kserve-667bcdf585-6jsg8\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:07:41.310989 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.310864    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:07:41.594282 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:41.594256    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"]
Sep 18 17:07:41.596573 ip-10-0-133-147 kubenswrapper[2585]: W0918 17:07:41.596546    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod265b26d9_1760_4b95_9dfe_784823c2afa5.slice/crio-ee7f36fe32f5a0675ca5804a2e956839a06dbc19318faa86afa7bd195c33bb45 WatchSource:0}: Error finding container ee7f36fe32f5a0675ca5804a2e956839a06dbc19318faa86afa7bd195c33bb45: Status 404 returned error can't find the container with id ee7f36fe32f5a0675ca5804a2e956839a06dbc19318faa86afa7bd195c33bb45
Sep 18 17:07:42.582044 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:42.581995    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8" event={"ID":"265b26d9-1760-4b95-9dfe-784823c2afa5","Type":"ContainerStarted","Data":"b1cb2a3eb1333846690c5db3554f6b8ec4eaf5950f3bd449ddd0b5fcc3d59d8a"}
Sep 18 17:07:42.582231 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:42.582063    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8" event={"ID":"265b26d9-1760-4b95-9dfe-784823c2afa5","Type":"ContainerStarted","Data":"ee7f36fe32f5a0675ca5804a2e956839a06dbc19318faa86afa7bd195c33bb45"}
Sep 18 17:07:44.592681 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:44.592646    2585 generic.go:356] "Generic (PLEG): container finished" podID="265b26d9-1760-4b95-9dfe-784823c2afa5" containerID="b1cb2a3eb1333846690c5db3554f6b8ec4eaf5950f3bd449ddd0b5fcc3d59d8a" exitCode=0
Sep 18 17:07:44.592681 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:44.592684    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8" event={"ID":"265b26d9-1760-4b95-9dfe-784823c2afa5","Type":"ContainerDied","Data":"b1cb2a3eb1333846690c5db3554f6b8ec4eaf5950f3bd449ddd0b5fcc3d59d8a"}
Sep 18 17:07:45.598778 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:45.598741    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8" event={"ID":"265b26d9-1760-4b95-9dfe-784823c2afa5","Type":"ContainerStarted","Data":"f08fcc3542e7694fde887e6ff1cd316a7bc05e50aaeaed3dc1a33655b0c36051"}
Sep 18 17:07:45.628238 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:45.628180    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8" podStartSLOduration=3.435772713 podStartE2EDuration="5.628162952s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.192390239s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:07:45.626026788 +0000 UTC m=+3105.409028813" watchObservedRunningTime="2026-09-18 17:07:45.628162952 +0000 UTC m=+3105.411164976"
Sep 18 17:07:46.311494 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:46.311455    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:07:51.311204 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:51.311161    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:07:51.312292 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:07:51.312266    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8" podUID="265b26d9-1760-4b95-9dfe-784823c2afa5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 18 17:08:01.311789 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:08:01.311743    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8" podUID="265b26d9-1760-4b95-9dfe-784823c2afa5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 18 17:08:11.311958 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:08:11.311879    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8" podUID="265b26d9-1760-4b95-9dfe-784823c2afa5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 18 17:08:21.311981 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:08:21.311935    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8" podUID="265b26d9-1760-4b95-9dfe-784823c2afa5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 18 17:08:31.311693 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:08:31.311645    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8" podUID="265b26d9-1760-4b95-9dfe-784823c2afa5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 18 17:08:41.312159 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:08:41.312111    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8" podUID="265b26d9-1760-4b95-9dfe-784823c2afa5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 18 17:08:51.312147 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:08:51.312103    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8" podUID="265b26d9-1760-4b95-9dfe-784823c2afa5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 18 17:09:01.311807 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:09:01.311753    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8" podUID="265b26d9-1760-4b95-9dfe-784823c2afa5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 18 17:09:11.312010 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:09:11.311959    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8" podUID="265b26d9-1760-4b95-9dfe-784823c2afa5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 18 17:09:21.312038 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:09:21.311988    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8" podUID="265b26d9-1760-4b95-9dfe-784823c2afa5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 18 17:09:31.311779 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:09:31.311725    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8" podUID="265b26d9-1760-4b95-9dfe-784823c2afa5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 18 17:09:41.322141 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:09:41.322107    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:09:41.329824 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:09:41.329803    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:09:46.720675 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:09:46.720642    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"]
Sep 18 17:10:01.743520 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:01.743447    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8" podUID="265b26d9-1760-4b95-9dfe-784823c2afa5" containerName="main" containerID="cri-o://f08fcc3542e7694fde887e6ff1cd316a7bc05e50aaeaed3dc1a33655b0c36051" gracePeriod=45
Sep 18 17:10:02.322725 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:02.322682    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8" podUID="265b26d9-1760-4b95-9dfe-784823c2afa5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 18 17:10:02.494574 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:02.494548    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:10:02.604085 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:02.603989    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/265b26d9-1760-4b95-9dfe-784823c2afa5-model-cache\") pod \"265b26d9-1760-4b95-9dfe-784823c2afa5\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") "
Sep 18 17:10:02.604250 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:02.604083    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdvkj\" (UniqueName: \"kubernetes.io/projected/265b26d9-1760-4b95-9dfe-784823c2afa5-kube-api-access-gdvkj\") pod \"265b26d9-1760-4b95-9dfe-784823c2afa5\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") "
Sep 18 17:10:02.604250 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:02.604110    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/265b26d9-1760-4b95-9dfe-784823c2afa5-tls-certs\") pod \"265b26d9-1760-4b95-9dfe-784823c2afa5\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") "
Sep 18 17:10:02.604250 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:02.604137    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/265b26d9-1760-4b95-9dfe-784823c2afa5-tmp-dir\") pod \"265b26d9-1760-4b95-9dfe-784823c2afa5\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") "
Sep 18 17:10:02.604250 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:02.604172    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/265b26d9-1760-4b95-9dfe-784823c2afa5-home\") pod \"265b26d9-1760-4b95-9dfe-784823c2afa5\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") "
Sep 18 17:10:02.604250 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:02.604194    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/265b26d9-1760-4b95-9dfe-784823c2afa5-kserve-provision-location\") pod \"265b26d9-1760-4b95-9dfe-784823c2afa5\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") "
Sep 18 17:10:02.604250 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:02.604227    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/265b26d9-1760-4b95-9dfe-784823c2afa5-dshm\") pod \"265b26d9-1760-4b95-9dfe-784823c2afa5\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") "
Sep 18 17:10:02.604250 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:02.604251    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/265b26d9-1760-4b95-9dfe-784823c2afa5-cabundle-cert\") pod \"265b26d9-1760-4b95-9dfe-784823c2afa5\" (UID: \"265b26d9-1760-4b95-9dfe-784823c2afa5\") "
Sep 18 17:10:02.604613 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:02.604291    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265b26d9-1760-4b95-9dfe-784823c2afa5-model-cache" (OuterVolumeSpecName: "model-cache") pod "265b26d9-1760-4b95-9dfe-784823c2afa5" (UID: "265b26d9-1760-4b95-9dfe-784823c2afa5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:10:02.604613 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:02.604482    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/265b26d9-1760-4b95-9dfe-784823c2afa5-model-cache\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:10:02.604823 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:02.604801    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/265b26d9-1760-4b95-9dfe-784823c2afa5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "265b26d9-1760-4b95-9dfe-784823c2afa5" (UID: "265b26d9-1760-4b95-9dfe-784823c2afa5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:10:02.604946 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:02.604901    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265b26d9-1760-4b95-9dfe-784823c2afa5-home" (OuterVolumeSpecName: "home") pod "265b26d9-1760-4b95-9dfe-784823c2afa5" (UID: "265b26d9-1760-4b95-9dfe-784823c2afa5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:10:02.607084 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:02.607050    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/265b26d9-1760-4b95-9dfe-784823c2afa5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "265b26d9-1760-4b95-9dfe-784823c2afa5" (UID: "265b26d9-1760-4b95-9dfe-784823c2afa5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:10:02.607283 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:02.607261    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265b26d9-1760-4b95-9dfe-784823c2afa5-dshm" (OuterVolumeSpecName: "dshm") pod "265b26d9-1760-4b95-9dfe-784823c2afa5" (UID: "265b26d9-1760-4b95-9dfe-784823c2afa5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:10:02.607283 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:02.607277    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/265b26d9-1760-4b95-9dfe-784823c2afa5-kube-api-access-gdvkj" (OuterVolumeSpecName: "kube-api-access-gdvkj") pod "265b26d9-1760-4b95-9dfe-784823c2afa5" (UID: "265b26d9-1760-4b95-9dfe-784823c2afa5"). InnerVolumeSpecName "kube-api-access-gdvkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:10:02.617095 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:02.617064    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265b26d9-1760-4b95-9dfe-784823c2afa5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "265b26d9-1760-4b95-9dfe-784823c2afa5" (UID: "265b26d9-1760-4b95-9dfe-784823c2afa5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:10:02.628677 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:02.628640    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265b26d9-1760-4b95-9dfe-784823c2afa5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "265b26d9-1760-4b95-9dfe-784823c2afa5" (UID: "265b26d9-1760-4b95-9dfe-784823c2afa5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:10:02.705231 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:02.705193    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdvkj\" (UniqueName: \"kubernetes.io/projected/265b26d9-1760-4b95-9dfe-784823c2afa5-kube-api-access-gdvkj\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:10:02.705231 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:02.705227    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/265b26d9-1760-4b95-9dfe-784823c2afa5-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:10:02.705231 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:02.705240    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/265b26d9-1760-4b95-9dfe-784823c2afa5-tmp-dir\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:10:02.705477 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:02.705252    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/265b26d9-1760-4b95-9dfe-784823c2afa5-home\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:10:02.705477 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:02.705262    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/265b26d9-1760-4b95-9dfe-784823c2afa5-kserve-provision-location\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:10:02.705477 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:02.705273    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/265b26d9-1760-4b95-9dfe-784823c2afa5-dshm\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:10:02.705477 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:02.705284    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/265b26d9-1760-4b95-9dfe-784823c2afa5-cabundle-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:10:03.173255 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:03.173223    2585 generic.go:356] "Generic (PLEG): container finished" podID="265b26d9-1760-4b95-9dfe-784823c2afa5" containerID="f08fcc3542e7694fde887e6ff1cd316a7bc05e50aaeaed3dc1a33655b0c36051" exitCode=0
Sep 18 17:10:03.173740 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:03.173273    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"
Sep 18 17:10:03.173740 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:03.173307    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8" event={"ID":"265b26d9-1760-4b95-9dfe-784823c2afa5","Type":"ContainerDied","Data":"f08fcc3542e7694fde887e6ff1cd316a7bc05e50aaeaed3dc1a33655b0c36051"}
Sep 18 17:10:03.173740 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:03.173374    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8" event={"ID":"265b26d9-1760-4b95-9dfe-784823c2afa5","Type":"ContainerDied","Data":"ee7f36fe32f5a0675ca5804a2e956839a06dbc19318faa86afa7bd195c33bb45"}
Sep 18 17:10:03.173740 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:03.173410    2585 scope.go:117] "RemoveContainer" containerID="f08fcc3542e7694fde887e6ff1cd316a7bc05e50aaeaed3dc1a33655b0c36051"
Sep 18 17:10:03.182075 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:03.182049    2585 scope.go:117] "RemoveContainer" containerID="b1cb2a3eb1333846690c5db3554f6b8ec4eaf5950f3bd449ddd0b5fcc3d59d8a"
Sep 18 17:10:03.189300 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:03.189284    2585 scope.go:117] "RemoveContainer" containerID="f08fcc3542e7694fde887e6ff1cd316a7bc05e50aaeaed3dc1a33655b0c36051"
Sep 18 17:10:03.189539 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:10:03.189519    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f08fcc3542e7694fde887e6ff1cd316a7bc05e50aaeaed3dc1a33655b0c36051\": container with ID starting with f08fcc3542e7694fde887e6ff1cd316a7bc05e50aaeaed3dc1a33655b0c36051 not found: ID does not exist" containerID="f08fcc3542e7694fde887e6ff1cd316a7bc05e50aaeaed3dc1a33655b0c36051"
Sep 18 17:10:03.189612 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:03.189552    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f08fcc3542e7694fde887e6ff1cd316a7bc05e50aaeaed3dc1a33655b0c36051"} err="failed to get container status \"f08fcc3542e7694fde887e6ff1cd316a7bc05e50aaeaed3dc1a33655b0c36051\": rpc error: code = NotFound desc = could not find container \"f08fcc3542e7694fde887e6ff1cd316a7bc05e50aaeaed3dc1a33655b0c36051\": container with ID starting with f08fcc3542e7694fde887e6ff1cd316a7bc05e50aaeaed3dc1a33655b0c36051 not found: ID does not exist"
Sep 18 17:10:03.189612 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:03.189576    2585 scope.go:117] "RemoveContainer" containerID="b1cb2a3eb1333846690c5db3554f6b8ec4eaf5950f3bd449ddd0b5fcc3d59d8a"
Sep 18 17:10:03.189815 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:10:03.189800    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1cb2a3eb1333846690c5db3554f6b8ec4eaf5950f3bd449ddd0b5fcc3d59d8a\": container with ID starting with b1cb2a3eb1333846690c5db3554f6b8ec4eaf5950f3bd449ddd0b5fcc3d59d8a not found: ID does not exist" containerID="b1cb2a3eb1333846690c5db3554f6b8ec4eaf5950f3bd449ddd0b5fcc3d59d8a"
Sep 18 17:10:03.189854 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:03.189821    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1cb2a3eb1333846690c5db3554f6b8ec4eaf5950f3bd449ddd0b5fcc3d59d8a"} err="failed to get container status \"b1cb2a3eb1333846690c5db3554f6b8ec4eaf5950f3bd449ddd0b5fcc3d59d8a\": rpc error: code = NotFound desc = could not find container \"b1cb2a3eb1333846690c5db3554f6b8ec4eaf5950f3bd449ddd0b5fcc3d59d8a\": container with ID starting with b1cb2a3eb1333846690c5db3554f6b8ec4eaf5950f3bd449ddd0b5fcc3d59d8a not found: ID does not exist"
Sep 18 17:10:03.197297 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:03.197278    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"]
Sep 18 17:10:03.202448 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:03.202424    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-6jsg8"]
Sep 18 17:10:04.392675 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:04.392639    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="265b26d9-1760-4b95-9dfe-784823c2afa5" path="/var/lib/kubelet/pods/265b26d9-1760-4b95-9dfe-784823c2afa5/volumes"
Sep 18 17:10:36.918879 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:36.918841    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"]
Sep 18 17:10:36.919516 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:36.919149    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="265b26d9-1760-4b95-9dfe-784823c2afa5" containerName="storage-initializer"
Sep 18 17:10:36.919516 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:36.919162    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="265b26d9-1760-4b95-9dfe-784823c2afa5" containerName="storage-initializer"
Sep 18 17:10:36.919516 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:36.919172    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="265b26d9-1760-4b95-9dfe-784823c2afa5" containerName="main"
Sep 18 17:10:36.919516 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:36.919178    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="265b26d9-1760-4b95-9dfe-784823c2afa5" containerName="main"
Sep 18 17:10:36.919516 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:36.919236    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="265b26d9-1760-4b95-9dfe-784823c2afa5" containerName="main"
Sep 18 17:10:36.921855 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:36.921838    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:36.925018 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:36.924645    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Sep 18 17:10:36.925103 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:36.925059    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 18 17:10:36.925169 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:36.925096    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 18 17:10:36.925371 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:36.925353    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-rzwwx\""
Sep 18 17:10:36.935478 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:36.935452    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"]
Sep 18 17:10:36.982851 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:36.982816    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92eed10-8069-46ec-a43a-3058c1f79db3-credential-socket\") pod \"router-gateway-1-openshift-default-df55df88d-d7ct5\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:36.982851 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:36.982851    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b92eed10-8069-46ec-a43a-3058c1f79db3-workload-certs\") pod \"router-gateway-1-openshift-default-df55df88d-d7ct5\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:36.983127 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:36.982869    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxdf4\" (UniqueName: \"kubernetes.io/projected/b92eed10-8069-46ec-a43a-3058c1f79db3-kube-api-access-jxdf4\") pod \"router-gateway-1-openshift-default-df55df88d-d7ct5\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:36.983127 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:36.982887    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b92eed10-8069-46ec-a43a-3058c1f79db3-istio-data\") pod \"router-gateway-1-openshift-default-df55df88d-d7ct5\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:36.983127 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:36.983029    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b92eed10-8069-46ec-a43a-3058c1f79db3-istio-token\") pod \"router-gateway-1-openshift-default-df55df88d-d7ct5\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:36.983127 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:36.983095    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b92eed10-8069-46ec-a43a-3058c1f79db3-istio-podinfo\") pod \"router-gateway-1-openshift-default-df55df88d-d7ct5\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:36.983127 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:36.983123    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b92eed10-8069-46ec-a43a-3058c1f79db3-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-df55df88d-d7ct5\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:36.983354 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:36.983172    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b92eed10-8069-46ec-a43a-3058c1f79db3-istio-envoy\") pod \"router-gateway-1-openshift-default-df55df88d-d7ct5\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:36.983354 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:36.983203    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92eed10-8069-46ec-a43a-3058c1f79db3-workload-socket\") pod \"router-gateway-1-openshift-default-df55df88d-d7ct5\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:37.083866 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:37.083817    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b92eed10-8069-46ec-a43a-3058c1f79db3-istio-podinfo\") pod \"router-gateway-1-openshift-default-df55df88d-d7ct5\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:37.083866 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:37.083859    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b92eed10-8069-46ec-a43a-3058c1f79db3-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-df55df88d-d7ct5\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:37.084147 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:37.083954    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b92eed10-8069-46ec-a43a-3058c1f79db3-istio-envoy\") pod \"router-gateway-1-openshift-default-df55df88d-d7ct5\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:37.084147 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:37.083981    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92eed10-8069-46ec-a43a-3058c1f79db3-workload-socket\") pod \"router-gateway-1-openshift-default-df55df88d-d7ct5\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:37.084147 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:37.084006    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92eed10-8069-46ec-a43a-3058c1f79db3-credential-socket\") pod \"router-gateway-1-openshift-default-df55df88d-d7ct5\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:37.084147 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:37.084025    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b92eed10-8069-46ec-a43a-3058c1f79db3-workload-certs\") pod \"router-gateway-1-openshift-default-df55df88d-d7ct5\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:37.084147 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:37.084048    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxdf4\" (UniqueName: \"kubernetes.io/projected/b92eed10-8069-46ec-a43a-3058c1f79db3-kube-api-access-jxdf4\") pod \"router-gateway-1-openshift-default-df55df88d-d7ct5\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:37.084147 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:37.084073    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b92eed10-8069-46ec-a43a-3058c1f79db3-istio-data\") pod \"router-gateway-1-openshift-default-df55df88d-d7ct5\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:37.084147 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:37.084120    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b92eed10-8069-46ec-a43a-3058c1f79db3-istio-token\") pod \"router-gateway-1-openshift-default-df55df88d-d7ct5\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:37.084571 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:37.084545    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92eed10-8069-46ec-a43a-3058c1f79db3-credential-socket\") pod \"router-gateway-1-openshift-default-df55df88d-d7ct5\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:37.084640 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:37.084552    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b92eed10-8069-46ec-a43a-3058c1f79db3-workload-certs\") pod \"router-gateway-1-openshift-default-df55df88d-d7ct5\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:37.084640 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:37.084611    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b92eed10-8069-46ec-a43a-3058c1f79db3-istio-data\") pod \"router-gateway-1-openshift-default-df55df88d-d7ct5\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:37.084717 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:37.084653    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92eed10-8069-46ec-a43a-3058c1f79db3-workload-socket\") pod \"router-gateway-1-openshift-default-df55df88d-d7ct5\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:37.085066 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:37.085048    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b92eed10-8069-46ec-a43a-3058c1f79db3-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-df55df88d-d7ct5\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:37.086603 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:37.086581    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b92eed10-8069-46ec-a43a-3058c1f79db3-istio-envoy\") pod \"router-gateway-1-openshift-default-df55df88d-d7ct5\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:37.086785 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:37.086767    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b92eed10-8069-46ec-a43a-3058c1f79db3-istio-podinfo\") pod \"router-gateway-1-openshift-default-df55df88d-d7ct5\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:37.093056 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:37.093016    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b92eed10-8069-46ec-a43a-3058c1f79db3-istio-token\") pod \"router-gateway-1-openshift-default-df55df88d-d7ct5\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:37.093598 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:37.093568    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxdf4\" (UniqueName: \"kubernetes.io/projected/b92eed10-8069-46ec-a43a-3058c1f79db3-kube-api-access-jxdf4\") pod \"router-gateway-1-openshift-default-df55df88d-d7ct5\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:37.234236 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:37.234202    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:37.515316 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:37.515225    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"]
Sep 18 17:10:37.524568 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:37.524528    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 17:10:37.524708 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:37.524614    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 17:10:37.524708 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:37.524652    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 17:10:38.309925 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:38.309864    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5" event={"ID":"b92eed10-8069-46ec-a43a-3058c1f79db3","Type":"ContainerStarted","Data":"9970212a55986a211b127d7ef60a5ccfd607024ad44f141cfb08c21d03d2fd6a"}
Sep 18 17:10:38.310344 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:38.309950    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5" event={"ID":"b92eed10-8069-46ec-a43a-3058c1f79db3","Type":"ContainerStarted","Data":"1519a7cfcaa21a1956638a03e4f805680294e2e2990275eee5aa05d6c588bcc3"}
Sep 18 17:10:38.339833 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:38.339774    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5" podStartSLOduration=2.339736738 podStartE2EDuration="2.339736738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:10:38.338961758 +0000 UTC m=+3278.121963786" watchObservedRunningTime="2026-09-18 17:10:38.339736738 +0000 UTC m=+3278.122738761"
Sep 18 17:10:39.234769 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:39.234733    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:39.239365 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:39.239338    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:39.314493 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:39.314464    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:39.315641 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:39.315619    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:50.368951 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:50.368893    2585 generic.go:356] "Generic (PLEG): container finished" podID="b92eed10-8069-46ec-a43a-3058c1f79db3" containerID="9970212a55986a211b127d7ef60a5ccfd607024ad44f141cfb08c21d03d2fd6a" exitCode=0
Sep 18 17:10:50.369440 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:50.368965    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5" event={"ID":"b92eed10-8069-46ec-a43a-3058c1f79db3","Type":"ContainerDied","Data":"9970212a55986a211b127d7ef60a5ccfd607024ad44f141cfb08c21d03d2fd6a"}
Sep 18 17:10:50.369440 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:50.369325    2585 scope.go:117] "RemoveContainer" containerID="9970212a55986a211b127d7ef60a5ccfd607024ad44f141cfb08c21d03d2fd6a"
Sep 18 17:10:50.370254 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:50.370228    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 17:10:50.370318 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:50.370299    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 17:10:50.370355 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:50.370329    2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 17:10:51.375167 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:51.375124    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5" event={"ID":"b92eed10-8069-46ec-a43a-3058c1f79db3","Type":"ContainerStarted","Data":"d478b997adc512779e53a882c989e0405c8e7fbeeb61471ac077a8f195c2f550"}
Sep 18 17:10:52.236283 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:52.235433    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:52.236667 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:52.236636    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5" podUID="b92eed10-8069-46ec-a43a-3058c1f79db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 18 17:10:53.235047 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:53.235000    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5" podUID="b92eed10-8069-46ec-a43a-3058c1f79db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 18 17:10:54.235036 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:54.234990    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5" podUID="b92eed10-8069-46ec-a43a-3058c1f79db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 18 17:10:54.315037 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:54.315006    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:55.235656 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:55.235600    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5" podUID="b92eed10-8069-46ec-a43a-3058c1f79db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 18 17:10:56.235042 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:56.234995    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5" podUID="b92eed10-8069-46ec-a43a-3058c1f79db3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 18 17:10:57.238462 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:57.238426    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:10:57.239410 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:10:57.239392    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:11:00.196932 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.196891    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"]
Sep 18 17:11:00.199866 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.199846    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:11:00.202512 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.202479    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 18 17:11:00.202629 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.202513    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 18 17:11:00.202629 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.202496    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 18 17:11:00.202629 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.202513    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 18 17:11:00.202629 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.202480    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 17:11:00.202813 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.202776    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-qc6zq\""
Sep 18 17:11:00.212022 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.211999    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"]
Sep 18 17:11:00.301065 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.301037    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f126f79f-3f34-458b-b5fe-f499ef120527-kserve-provision-location\") pod \"stop-feature-test-kserve-6bfc568595-kzb54\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:11:00.301065 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.301071    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f126f79f-3f34-458b-b5fe-f499ef120527-model-cache\") pod \"stop-feature-test-kserve-6bfc568595-kzb54\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:11:00.301277 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.301120    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f126f79f-3f34-458b-b5fe-f499ef120527-cabundle-cert\") pod \"stop-feature-test-kserve-6bfc568595-kzb54\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:11:00.301277 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.301152    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f126f79f-3f34-458b-b5fe-f499ef120527-home\") pod \"stop-feature-test-kserve-6bfc568595-kzb54\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:11:00.301277 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.301173    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f126f79f-3f34-458b-b5fe-f499ef120527-tls-certs\") pod \"stop-feature-test-kserve-6bfc568595-kzb54\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:11:00.301277 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.301189    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f126f79f-3f34-458b-b5fe-f499ef120527-tmp-dir\") pod \"stop-feature-test-kserve-6bfc568595-kzb54\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:11:00.301277 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.301217    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f126f79f-3f34-458b-b5fe-f499ef120527-dshm\") pod \"stop-feature-test-kserve-6bfc568595-kzb54\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:11:00.301277 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.301232    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngk6n\" (UniqueName: \"kubernetes.io/projected/f126f79f-3f34-458b-b5fe-f499ef120527-kube-api-access-ngk6n\") pod \"stop-feature-test-kserve-6bfc568595-kzb54\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:11:00.402093 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.402046    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f126f79f-3f34-458b-b5fe-f499ef120527-tls-certs\") pod \"stop-feature-test-kserve-6bfc568595-kzb54\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:11:00.402302 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.402115    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f126f79f-3f34-458b-b5fe-f499ef120527-tmp-dir\") pod \"stop-feature-test-kserve-6bfc568595-kzb54\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:11:00.402302 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.402173    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f126f79f-3f34-458b-b5fe-f499ef120527-dshm\") pod \"stop-feature-test-kserve-6bfc568595-kzb54\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:11:00.402302 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.402202    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngk6n\" (UniqueName: \"kubernetes.io/projected/f126f79f-3f34-458b-b5fe-f499ef120527-kube-api-access-ngk6n\") pod \"stop-feature-test-kserve-6bfc568595-kzb54\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:11:00.402302 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.402298    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f126f79f-3f34-458b-b5fe-f499ef120527-kserve-provision-location\") pod \"stop-feature-test-kserve-6bfc568595-kzb54\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:11:00.402512 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.402332    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f126f79f-3f34-458b-b5fe-f499ef120527-model-cache\") pod \"stop-feature-test-kserve-6bfc568595-kzb54\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:11:00.402512 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.402388    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f126f79f-3f34-458b-b5fe-f499ef120527-cabundle-cert\") pod \"stop-feature-test-kserve-6bfc568595-kzb54\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:11:00.402512 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.402445    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f126f79f-3f34-458b-b5fe-f499ef120527-home\") pod \"stop-feature-test-kserve-6bfc568595-kzb54\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:11:00.402906 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.402876    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f126f79f-3f34-458b-b5fe-f499ef120527-home\") pod \"stop-feature-test-kserve-6bfc568595-kzb54\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:11:00.403880 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.403848    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f126f79f-3f34-458b-b5fe-f499ef120527-tmp-dir\") pod \"stop-feature-test-kserve-6bfc568595-kzb54\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:11:00.408565 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.404469    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f126f79f-3f34-458b-b5fe-f499ef120527-kserve-provision-location\") pod \"stop-feature-test-kserve-6bfc568595-kzb54\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:11:00.408565 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.404807    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f126f79f-3f34-458b-b5fe-f499ef120527-model-cache\") pod \"stop-feature-test-kserve-6bfc568595-kzb54\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:11:00.408565 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.405196    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f126f79f-3f34-458b-b5fe-f499ef120527-cabundle-cert\") pod \"stop-feature-test-kserve-6bfc568595-kzb54\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:11:00.408565 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.406798    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f126f79f-3f34-458b-b5fe-f499ef120527-dshm\") pod \"stop-feature-test-kserve-6bfc568595-kzb54\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:11:00.410800 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.410780    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f126f79f-3f34-458b-b5fe-f499ef120527-tls-certs\") pod \"stop-feature-test-kserve-6bfc568595-kzb54\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:11:00.413449 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.413429    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngk6n\" (UniqueName: \"kubernetes.io/projected/f126f79f-3f34-458b-b5fe-f499ef120527-kube-api-access-ngk6n\") pod \"stop-feature-test-kserve-6bfc568595-kzb54\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:11:00.511147 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.511111    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:11:00.794722 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:00.794699    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"]
Sep 18 17:11:00.797357 ip-10-0-133-147 kubenswrapper[2585]: W0918 17:11:00.797329    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf126f79f_3f34_458b_b5fe_f499ef120527.slice/crio-972ba39705fdc15f205e7cb06cec3667f37f00bcfc12990d0a5399b21c2abf37 WatchSource:0}: Error finding container 972ba39705fdc15f205e7cb06cec3667f37f00bcfc12990d0a5399b21c2abf37: Status 404 returned error can't find the container with id 972ba39705fdc15f205e7cb06cec3667f37f00bcfc12990d0a5399b21c2abf37
Sep 18 17:11:01.418267 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:01.418225    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54" event={"ID":"f126f79f-3f34-458b-b5fe-f499ef120527","Type":"ContainerStarted","Data":"6a2abaa4cf92adebe292aa359cea341a59e54ac8a999a54db62201f64b6726e6"}
Sep 18 17:11:01.418732 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:01.418298    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54" event={"ID":"f126f79f-3f34-458b-b5fe-f499ef120527","Type":"ContainerStarted","Data":"972ba39705fdc15f205e7cb06cec3667f37f00bcfc12990d0a5399b21c2abf37"}
Sep 18 17:11:02.560481 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:02.560454    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 17:11:02.565939 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:02.565900    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 17:11:02.569420 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:02.569395    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 17:11:02.574694 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:02.574676    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 17:11:03.429367 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:03.429332    2585 generic.go:356] "Generic (PLEG): container finished" podID="f126f79f-3f34-458b-b5fe-f499ef120527" containerID="6a2abaa4cf92adebe292aa359cea341a59e54ac8a999a54db62201f64b6726e6" exitCode=0
Sep 18 17:11:03.429519 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:03.429380    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54" event={"ID":"f126f79f-3f34-458b-b5fe-f499ef120527","Type":"ContainerDied","Data":"6a2abaa4cf92adebe292aa359cea341a59e54ac8a999a54db62201f64b6726e6"}
Sep 18 17:11:04.436040 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:04.436007    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54" event={"ID":"f126f79f-3f34-458b-b5fe-f499ef120527","Type":"ContainerStarted","Data":"92ec061f44593b302793637b511bd5ea866080c1ee44f71e73a498306bfa1f81"}
Sep 18 17:11:04.459491 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:04.459433    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54" podStartSLOduration=2.340064387 podStartE2EDuration="4.459417399s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.119353012s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:11:04.456870022 +0000 UTC m=+3304.239872050" watchObservedRunningTime="2026-09-18 17:11:04.459417399 +0000 UTC m=+3304.242419403"
Sep 18 17:11:04.511974 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:04.511934    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:11:10.511677 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:10.511640    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:11:10.513389 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:10.513354    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54" podUID="f126f79f-3f34-458b-b5fe-f499ef120527" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 18 17:11:11.062092 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:11.062060    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm"]
Sep 18 17:11:11.064812 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:11.064790    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm"
Sep 18 17:11:11.069659 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:11.069632    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 18 17:11:11.069820 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:11.069633    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-2t64g\""
Sep 18 17:11:11.069820 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:11.069671    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-6jqtg\""
Sep 18 17:11:11.074904 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:11.074880    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm"]
Sep 18 17:11:11.095123 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:11.095099    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rddc\" (UniqueName: \"kubernetes.io/projected/837ed00d-0881-43a9-aa4f-921dd763dd58-kube-api-access-9rddc\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm\" (UID: \"837ed00d-0881-43a9-aa4f-921dd763dd58\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm"
Sep 18 17:11:11.095264 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:11.095241    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/837ed00d-0881-43a9-aa4f-921dd763dd58-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm\" (UID: \"837ed00d-0881-43a9-aa4f-921dd763dd58\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm"
Sep 18 17:11:11.195816 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:11.195771    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/837ed00d-0881-43a9-aa4f-921dd763dd58-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm\" (UID: \"837ed00d-0881-43a9-aa4f-921dd763dd58\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm"
Sep 18 17:11:11.196004 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:11.195849    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rddc\" (UniqueName: \"kubernetes.io/projected/837ed00d-0881-43a9-aa4f-921dd763dd58-kube-api-access-9rddc\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm\" (UID: \"837ed00d-0881-43a9-aa4f-921dd763dd58\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm"
Sep 18 17:11:11.198692 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:11.198662    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/837ed00d-0881-43a9-aa4f-921dd763dd58-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm\" (UID: \"837ed00d-0881-43a9-aa4f-921dd763dd58\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm"
Sep 18 17:11:11.205717 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:11.205684    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rddc\" (UniqueName: \"kubernetes.io/projected/837ed00d-0881-43a9-aa4f-921dd763dd58-kube-api-access-9rddc\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm\" (UID: \"837ed00d-0881-43a9-aa4f-921dd763dd58\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm"
Sep 18 17:11:11.376530 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:11.376444    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm"
Sep 18 17:11:11.651158 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:11.651031    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm"]
Sep 18 17:11:11.653755 ip-10-0-133-147 kubenswrapper[2585]: W0918 17:11:11.653725    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837ed00d_0881_43a9_aa4f_921dd763dd58.slice/crio-f55e656ed640db929606d3a294dc48b2834ab479fc24242032323c34c87ee671 WatchSource:0}: Error finding container f55e656ed640db929606d3a294dc48b2834ab479fc24242032323c34c87ee671: Status 404 returned error can't find the container with id f55e656ed640db929606d3a294dc48b2834ab479fc24242032323c34c87ee671
Sep 18 17:11:12.470959 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:12.470905    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm" event={"ID":"837ed00d-0881-43a9-aa4f-921dd763dd58","Type":"ContainerStarted","Data":"14108c3ed9c4d0b3136e7ace73f5b6e1858cf65ebe772d840b862e495b608af3"}
Sep 18 17:11:12.471133 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:12.470976    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm" event={"ID":"837ed00d-0881-43a9-aa4f-921dd763dd58","Type":"ContainerStarted","Data":"f55e656ed640db929606d3a294dc48b2834ab479fc24242032323c34c87ee671"}
Sep 18 17:11:12.471133 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:12.471021    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm"
Sep 18 17:11:12.488323 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:12.488272    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm" podStartSLOduration=1.488258005 podStartE2EDuration="1.488258005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:11:12.487627227 +0000 UTC m=+3312.270629253" watchObservedRunningTime="2026-09-18 17:11:12.488258005 +0000 UTC m=+3312.271260062"
Sep 18 17:11:20.512031 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:20.511989    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54" podUID="f126f79f-3f34-458b-b5fe-f499ef120527" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 18 17:11:30.511952 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:30.511884    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54" podUID="f126f79f-3f34-458b-b5fe-f499ef120527" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 18 17:11:40.511765 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:40.511715    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54" podUID="f126f79f-3f34-458b-b5fe-f499ef120527" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 18 17:11:43.476561 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:43.476529    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm"
Sep 18 17:11:50.511669 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:11:50.511626    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54" podUID="f126f79f-3f34-458b-b5fe-f499ef120527" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 18 17:12:00.512896 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:12:00.512847    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54" podUID="f126f79f-3f34-458b-b5fe-f499ef120527" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 18 17:12:10.512333 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:12:10.512281    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54" podUID="f126f79f-3f34-458b-b5fe-f499ef120527" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 18 17:12:20.512187 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:12:20.512140    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54" podUID="f126f79f-3f34-458b-b5fe-f499ef120527" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 18 17:12:30.511972 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:12:30.511928    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54" podUID="f126f79f-3f34-458b-b5fe-f499ef120527" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 18 17:12:40.512360 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:12:40.512311    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54" podUID="f126f79f-3f34-458b-b5fe-f499ef120527" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 18 17:12:50.511624 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:12:50.511513    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54" podUID="f126f79f-3f34-458b-b5fe-f499ef120527" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 18 17:13:00.522777 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:00.522742    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:13:00.530561 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:00.530522    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:13:01.983161 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:01.983116    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"]
Sep 18 17:13:17.005852 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:17.005805    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54" podUID="f126f79f-3f34-458b-b5fe-f499ef120527" containerName="main" containerID="cri-o://92ec061f44593b302793637b511bd5ea866080c1ee44f71e73a498306bfa1f81" gracePeriod=45
Sep 18 17:13:17.523515 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:17.523475    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54" podUID="f126f79f-3f34-458b-b5fe-f499ef120527" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 18 17:13:17.748829 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:17.748805    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:13:17.851828 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:17.851752    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f126f79f-3f34-458b-b5fe-f499ef120527-model-cache\") pod \"f126f79f-3f34-458b-b5fe-f499ef120527\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") "
Sep 18 17:13:17.851828 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:17.851787    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f126f79f-3f34-458b-b5fe-f499ef120527-tmp-dir\") pod \"f126f79f-3f34-458b-b5fe-f499ef120527\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") "
Sep 18 17:13:17.851828 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:17.851820    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f126f79f-3f34-458b-b5fe-f499ef120527-tls-certs\") pod \"f126f79f-3f34-458b-b5fe-f499ef120527\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") "
Sep 18 17:13:17.852113 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:17.851863    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f126f79f-3f34-458b-b5fe-f499ef120527-kserve-provision-location\") pod \"f126f79f-3f34-458b-b5fe-f499ef120527\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") "
Sep 18 17:13:17.852113 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:17.851888    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f126f79f-3f34-458b-b5fe-f499ef120527-home\") pod \"f126f79f-3f34-458b-b5fe-f499ef120527\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") "
Sep 18 17:13:17.852113 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:17.851944    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f126f79f-3f34-458b-b5fe-f499ef120527-cabundle-cert\") pod \"f126f79f-3f34-458b-b5fe-f499ef120527\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") "
Sep 18 17:13:17.852113 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:17.851971    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngk6n\" (UniqueName: \"kubernetes.io/projected/f126f79f-3f34-458b-b5fe-f499ef120527-kube-api-access-ngk6n\") pod \"f126f79f-3f34-458b-b5fe-f499ef120527\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") "
Sep 18 17:13:17.852113 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:17.852014    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f126f79f-3f34-458b-b5fe-f499ef120527-dshm\") pod \"f126f79f-3f34-458b-b5fe-f499ef120527\" (UID: \"f126f79f-3f34-458b-b5fe-f499ef120527\") "
Sep 18 17:13:17.852113 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:17.852089    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f126f79f-3f34-458b-b5fe-f499ef120527-model-cache" (OuterVolumeSpecName: "model-cache") pod "f126f79f-3f34-458b-b5fe-f499ef120527" (UID: "f126f79f-3f34-458b-b5fe-f499ef120527"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:13:17.852474 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:17.852403    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f126f79f-3f34-458b-b5fe-f499ef120527-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f126f79f-3f34-458b-b5fe-f499ef120527" (UID: "f126f79f-3f34-458b-b5fe-f499ef120527"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:13:17.852474 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:17.852448    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f126f79f-3f34-458b-b5fe-f499ef120527-model-cache\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:13:17.852965 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:17.852906    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f126f79f-3f34-458b-b5fe-f499ef120527-home" (OuterVolumeSpecName: "home") pod "f126f79f-3f34-458b-b5fe-f499ef120527" (UID: "f126f79f-3f34-458b-b5fe-f499ef120527"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:13:17.854488 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:17.854459    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f126f79f-3f34-458b-b5fe-f499ef120527-dshm" (OuterVolumeSpecName: "dshm") pod "f126f79f-3f34-458b-b5fe-f499ef120527" (UID: "f126f79f-3f34-458b-b5fe-f499ef120527"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:13:17.854595 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:17.854519    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f126f79f-3f34-458b-b5fe-f499ef120527-kube-api-access-ngk6n" (OuterVolumeSpecName: "kube-api-access-ngk6n") pod "f126f79f-3f34-458b-b5fe-f499ef120527" (UID: "f126f79f-3f34-458b-b5fe-f499ef120527"). InnerVolumeSpecName "kube-api-access-ngk6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:13:17.854595 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:17.854533    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f126f79f-3f34-458b-b5fe-f499ef120527-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f126f79f-3f34-458b-b5fe-f499ef120527" (UID: "f126f79f-3f34-458b-b5fe-f499ef120527"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:13:17.869809 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:17.869785    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f126f79f-3f34-458b-b5fe-f499ef120527-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f126f79f-3f34-458b-b5fe-f499ef120527" (UID: "f126f79f-3f34-458b-b5fe-f499ef120527"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:13:17.882271 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:17.882245    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f126f79f-3f34-458b-b5fe-f499ef120527-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f126f79f-3f34-458b-b5fe-f499ef120527" (UID: "f126f79f-3f34-458b-b5fe-f499ef120527"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:13:17.953057 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:17.953034    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f126f79f-3f34-458b-b5fe-f499ef120527-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:13:17.953057 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:17.953059    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f126f79f-3f34-458b-b5fe-f499ef120527-kserve-provision-location\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:13:17.953199 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:17.953068    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f126f79f-3f34-458b-b5fe-f499ef120527-home\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:13:17.953199 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:17.953077    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f126f79f-3f34-458b-b5fe-f499ef120527-cabundle-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:13:17.953199 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:17.953087    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngk6n\" (UniqueName: \"kubernetes.io/projected/f126f79f-3f34-458b-b5fe-f499ef120527-kube-api-access-ngk6n\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:13:17.953199 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:17.953094    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f126f79f-3f34-458b-b5fe-f499ef120527-dshm\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:13:17.953199 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:17.953102    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f126f79f-3f34-458b-b5fe-f499ef120527-tmp-dir\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:13:18.013166 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:18.013136    2585 generic.go:356] "Generic (PLEG): container finished" podID="f126f79f-3f34-458b-b5fe-f499ef120527" containerID="92ec061f44593b302793637b511bd5ea866080c1ee44f71e73a498306bfa1f81" exitCode=0
Sep 18 17:13:18.013483 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:18.013189    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"
Sep 18 17:13:18.013483 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:18.013223    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54" event={"ID":"f126f79f-3f34-458b-b5fe-f499ef120527","Type":"ContainerDied","Data":"92ec061f44593b302793637b511bd5ea866080c1ee44f71e73a498306bfa1f81"}
Sep 18 17:13:18.013483 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:18.013284    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54" event={"ID":"f126f79f-3f34-458b-b5fe-f499ef120527","Type":"ContainerDied","Data":"972ba39705fdc15f205e7cb06cec3667f37f00bcfc12990d0a5399b21c2abf37"}
Sep 18 17:13:18.013483 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:18.013306    2585 scope.go:117] "RemoveContainer" containerID="92ec061f44593b302793637b511bd5ea866080c1ee44f71e73a498306bfa1f81"
Sep 18 17:13:18.022725 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:18.022710    2585 scope.go:117] "RemoveContainer" containerID="6a2abaa4cf92adebe292aa359cea341a59e54ac8a999a54db62201f64b6726e6"
Sep 18 17:13:18.030253 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:18.030238    2585 scope.go:117] "RemoveContainer" containerID="92ec061f44593b302793637b511bd5ea866080c1ee44f71e73a498306bfa1f81"
Sep 18 17:13:18.030501 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:13:18.030481    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92ec061f44593b302793637b511bd5ea866080c1ee44f71e73a498306bfa1f81\": container with ID starting with 92ec061f44593b302793637b511bd5ea866080c1ee44f71e73a498306bfa1f81 not found: ID does not exist" containerID="92ec061f44593b302793637b511bd5ea866080c1ee44f71e73a498306bfa1f81"
Sep 18 17:13:18.030548 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:18.030510    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92ec061f44593b302793637b511bd5ea866080c1ee44f71e73a498306bfa1f81"} err="failed to get container status \"92ec061f44593b302793637b511bd5ea866080c1ee44f71e73a498306bfa1f81\": rpc error: code = NotFound desc = could not find container \"92ec061f44593b302793637b511bd5ea866080c1ee44f71e73a498306bfa1f81\": container with ID starting with 92ec061f44593b302793637b511bd5ea866080c1ee44f71e73a498306bfa1f81 not found: ID does not exist"
Sep 18 17:13:18.030548 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:18.030526    2585 scope.go:117] "RemoveContainer" containerID="6a2abaa4cf92adebe292aa359cea341a59e54ac8a999a54db62201f64b6726e6"
Sep 18 17:13:18.030752 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:13:18.030733    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a2abaa4cf92adebe292aa359cea341a59e54ac8a999a54db62201f64b6726e6\": container with ID starting with 6a2abaa4cf92adebe292aa359cea341a59e54ac8a999a54db62201f64b6726e6 not found: ID does not exist" containerID="6a2abaa4cf92adebe292aa359cea341a59e54ac8a999a54db62201f64b6726e6"
Sep 18 17:13:18.030790 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:18.030760    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a2abaa4cf92adebe292aa359cea341a59e54ac8a999a54db62201f64b6726e6"} err="failed to get container status \"6a2abaa4cf92adebe292aa359cea341a59e54ac8a999a54db62201f64b6726e6\": rpc error: code = NotFound desc = could not find container \"6a2abaa4cf92adebe292aa359cea341a59e54ac8a999a54db62201f64b6726e6\": container with ID starting with 6a2abaa4cf92adebe292aa359cea341a59e54ac8a999a54db62201f64b6726e6 not found: ID does not exist"
Sep 18 17:13:18.036945 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:18.036901    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"]
Sep 18 17:13:18.040724 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:18.040702    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-kzb54"]
Sep 18 17:13:18.392302 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:18.392268    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f126f79f-3f34-458b-b5fe-f499ef120527" path="/var/lib/kubelet/pods/f126f79f-3f34-458b-b5fe-f499ef120527/volumes"
Sep 18 17:13:26.148267 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:26.148234    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm"]
Sep 18 17:13:33.476789 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:33.476752    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm" podUID="837ed00d-0881-43a9-aa4f-921dd763dd58" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:13:41.168056 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:41.168013    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm" podUID="837ed00d-0881-43a9-aa4f-921dd763dd58" containerName="main" containerID="cri-o://14108c3ed9c4d0b3136e7ace73f5b6e1858cf65ebe772d840b862e495b608af3" gracePeriod=45
Sep 18 17:13:43.476543 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:43.476509    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm" podUID="837ed00d-0881-43a9-aa4f-921dd763dd58" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:13:53.476790 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:53.476749    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm" podUID="837ed00d-0881-43a9-aa4f-921dd763dd58" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:13:53.477356 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:13:53.476870    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm"
Sep 18 17:14:03.476813 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:03.476769    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm" podUID="837ed00d-0881-43a9-aa4f-921dd763dd58" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:14:11.406464 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:11.406438    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm"
Sep 18 17:14:11.473629 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:11.473560    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/837ed00d-0881-43a9-aa4f-921dd763dd58-tls-certs\") pod \"837ed00d-0881-43a9-aa4f-921dd763dd58\" (UID: \"837ed00d-0881-43a9-aa4f-921dd763dd58\") "
Sep 18 17:14:11.473793 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:11.473633    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rddc\" (UniqueName: \"kubernetes.io/projected/837ed00d-0881-43a9-aa4f-921dd763dd58-kube-api-access-9rddc\") pod \"837ed00d-0881-43a9-aa4f-921dd763dd58\" (UID: \"837ed00d-0881-43a9-aa4f-921dd763dd58\") "
Sep 18 17:14:11.476018 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:11.475983    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/837ed00d-0881-43a9-aa4f-921dd763dd58-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "837ed00d-0881-43a9-aa4f-921dd763dd58" (UID: "837ed00d-0881-43a9-aa4f-921dd763dd58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:14:11.476135 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:11.476111    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/837ed00d-0881-43a9-aa4f-921dd763dd58-kube-api-access-9rddc" (OuterVolumeSpecName: "kube-api-access-9rddc") pod "837ed00d-0881-43a9-aa4f-921dd763dd58" (UID: "837ed00d-0881-43a9-aa4f-921dd763dd58"). InnerVolumeSpecName "kube-api-access-9rddc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:14:11.574268 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:11.574235    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rddc\" (UniqueName: \"kubernetes.io/projected/837ed00d-0881-43a9-aa4f-921dd763dd58-kube-api-access-9rddc\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:14:11.574268 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:11.574264    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/837ed00d-0881-43a9-aa4f-921dd763dd58-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:14:12.236119 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:12.236077    2585 generic.go:356] "Generic (PLEG): container finished" podID="837ed00d-0881-43a9-aa4f-921dd763dd58" containerID="14108c3ed9c4d0b3136e7ace73f5b6e1858cf65ebe772d840b862e495b608af3" exitCode=0
Sep 18 17:14:12.236320 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:12.236127    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm"
Sep 18 17:14:12.236320 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:12.236162    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm" event={"ID":"837ed00d-0881-43a9-aa4f-921dd763dd58","Type":"ContainerDied","Data":"14108c3ed9c4d0b3136e7ace73f5b6e1858cf65ebe772d840b862e495b608af3"}
Sep 18 17:14:12.236320 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:12.236213    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm" event={"ID":"837ed00d-0881-43a9-aa4f-921dd763dd58","Type":"ContainerDied","Data":"f55e656ed640db929606d3a294dc48b2834ab479fc24242032323c34c87ee671"}
Sep 18 17:14:12.236320 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:12.236236    2585 scope.go:117] "RemoveContainer" containerID="14108c3ed9c4d0b3136e7ace73f5b6e1858cf65ebe772d840b862e495b608af3"
Sep 18 17:14:12.245490 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:12.245472    2585 scope.go:117] "RemoveContainer" containerID="14108c3ed9c4d0b3136e7ace73f5b6e1858cf65ebe772d840b862e495b608af3"
Sep 18 17:14:12.245795 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:14:12.245776    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14108c3ed9c4d0b3136e7ace73f5b6e1858cf65ebe772d840b862e495b608af3\": container with ID starting with 14108c3ed9c4d0b3136e7ace73f5b6e1858cf65ebe772d840b862e495b608af3 not found: ID does not exist" containerID="14108c3ed9c4d0b3136e7ace73f5b6e1858cf65ebe772d840b862e495b608af3"
Sep 18 17:14:12.245847 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:12.245802    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14108c3ed9c4d0b3136e7ace73f5b6e1858cf65ebe772d840b862e495b608af3"} err="failed to get container status \"14108c3ed9c4d0b3136e7ace73f5b6e1858cf65ebe772d840b862e495b608af3\": rpc error: code = NotFound desc = could not find container \"14108c3ed9c4d0b3136e7ace73f5b6e1858cf65ebe772d840b862e495b608af3\": container with ID starting with 14108c3ed9c4d0b3136e7ace73f5b6e1858cf65ebe772d840b862e495b608af3 not found: ID does not exist"
Sep 18 17:14:12.257197 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:12.257169    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm"]
Sep 18 17:14:12.261462 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:12.261441    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-kphwm"]
Sep 18 17:14:12.391707 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:12.391671    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="837ed00d-0881-43a9-aa4f-921dd763dd58" path="/var/lib/kubelet/pods/837ed00d-0881-43a9-aa4f-921dd763dd58/volumes"
Sep 18 17:14:21.177164 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:21.177127    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"]
Sep 18 17:14:21.177570 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:21.177344    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5" podUID="b92eed10-8069-46ec-a43a-3058c1f79db3" containerName="istio-proxy" containerID="cri-o://d478b997adc512779e53a882c989e0405c8e7fbeeb61471ac077a8f195c2f550" gracePeriod=30
Sep 18 17:14:23.909200 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:23.909157    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"]
Sep 18 17:14:23.909674 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:23.909424    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f126f79f-3f34-458b-b5fe-f499ef120527" containerName="storage-initializer"
Sep 18 17:14:23.909674 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:23.909435    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f126f79f-3f34-458b-b5fe-f499ef120527" containerName="storage-initializer"
Sep 18 17:14:23.909674 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:23.909456    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="837ed00d-0881-43a9-aa4f-921dd763dd58" containerName="main"
Sep 18 17:14:23.909674 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:23.909461    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="837ed00d-0881-43a9-aa4f-921dd763dd58" containerName="main"
Sep 18 17:14:23.909674 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:23.909468    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f126f79f-3f34-458b-b5fe-f499ef120527" containerName="main"
Sep 18 17:14:23.909674 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:23.909473    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f126f79f-3f34-458b-b5fe-f499ef120527" containerName="main"
Sep 18 17:14:23.909674 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:23.909517    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f126f79f-3f34-458b-b5fe-f499ef120527" containerName="main"
Sep 18 17:14:23.909674 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:23.909526    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="837ed00d-0881-43a9-aa4f-921dd763dd58" containerName="main"
Sep 18 17:14:23.913502 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:23.913482    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:14:23.917061 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:23.917036    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-r5vk4\""
Sep 18 17:14:23.917251 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:23.917058    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-j4h4t\""
Sep 18 17:14:23.917360 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:23.917181    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 18 17:14:23.917438 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:23.917181    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 18 17:14:23.917520 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:23.917190    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 17:14:23.917889 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:23.917821    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 18 17:14:23.917889 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:23.917837    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 18 17:14:23.930548 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:23.930524    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"]
Sep 18 17:14:23.975696 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:23.975672    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/354562a6-430a-4270-ab8a-58a13454ecb2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:14:23.975826 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:23.975704    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/354562a6-430a-4270-ab8a-58a13454ecb2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:14:23.975826 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:23.975728    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsxz8\" (UniqueName: \"kubernetes.io/projected/354562a6-430a-4270-ab8a-58a13454ecb2-kube-api-access-qsxz8\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:14:23.975826 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:23.975754    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/354562a6-430a-4270-ab8a-58a13454ecb2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:14:23.975826 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:23.975784    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/354562a6-430a-4270-ab8a-58a13454ecb2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:14:23.975826 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:23.975805    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/354562a6-430a-4270-ab8a-58a13454ecb2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:14:23.975826 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:23.975821    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/354562a6-430a-4270-ab8a-58a13454ecb2-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:14:23.976076 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:23.975859    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/354562a6-430a-4270-ab8a-58a13454ecb2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:14:24.077289 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:24.077256    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/354562a6-430a-4270-ab8a-58a13454ecb2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:14:24.077289 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:24.077292    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/354562a6-430a-4270-ab8a-58a13454ecb2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:14:24.077540 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:24.077316    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsxz8\" (UniqueName: \"kubernetes.io/projected/354562a6-430a-4270-ab8a-58a13454ecb2-kube-api-access-qsxz8\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:14:24.077540 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:24.077356    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/354562a6-430a-4270-ab8a-58a13454ecb2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:14:24.077540 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:24.077385    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/354562a6-430a-4270-ab8a-58a13454ecb2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:14:24.077540 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:24.077413    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/354562a6-430a-4270-ab8a-58a13454ecb2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:14:24.077540 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:24.077440    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/354562a6-430a-4270-ab8a-58a13454ecb2-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:14:24.077540 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:24.077490    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/354562a6-430a-4270-ab8a-58a13454ecb2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:14:24.077933 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:24.077889    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/354562a6-430a-4270-ab8a-58a13454ecb2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:14:24.078205 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:24.077947    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/354562a6-430a-4270-ab8a-58a13454ecb2-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:14:24.078205 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:24.078041    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/354562a6-430a-4270-ab8a-58a13454ecb2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:14:24.078205 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:24.078232    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/354562a6-430a-4270-ab8a-58a13454ecb2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:14:24.078554 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:24.078290    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/354562a6-430a-4270-ab8a-58a13454ecb2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:14:24.080108 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:24.080091    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/354562a6-430a-4270-ab8a-58a13454ecb2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:14:24.080430 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:24.080389    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/354562a6-430a-4270-ab8a-58a13454ecb2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:14:24.089580 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:24.089558    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsxz8\" (UniqueName: \"kubernetes.io/projected/354562a6-430a-4270-ab8a-58a13454ecb2-kube-api-access-qsxz8\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:14:24.223774 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:24.223702    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:14:24.505153 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:24.505130    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"]
Sep 18 17:14:24.507731 ip-10-0-133-147 kubenswrapper[2585]: W0918 17:14:24.507709    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod354562a6_430a_4270_ab8a_58a13454ecb2.slice/crio-eadd8f41b50c06fb962b090267441d58a2676d6ecfa8b99de996e9e43d6982f2 WatchSource:0}: Error finding container eadd8f41b50c06fb962b090267441d58a2676d6ecfa8b99de996e9e43d6982f2: Status 404 returned error can't find the container with id eadd8f41b50c06fb962b090267441d58a2676d6ecfa8b99de996e9e43d6982f2
Sep 18 17:14:24.509684 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:24.509665    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 17:14:25.295956 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:25.295896    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml" event={"ID":"354562a6-430a-4270-ab8a-58a13454ecb2","Type":"ContainerStarted","Data":"eadd8f41b50c06fb962b090267441d58a2676d6ecfa8b99de996e9e43d6982f2"}
Sep 18 17:14:26.302809 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.302782    2585 generic.go:356] "Generic (PLEG): container finished" podID="b92eed10-8069-46ec-a43a-3058c1f79db3" containerID="d478b997adc512779e53a882c989e0405c8e7fbeeb61471ac077a8f195c2f550" exitCode=0
Sep 18 17:14:26.303156 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.302862    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5" event={"ID":"b92eed10-8069-46ec-a43a-3058c1f79db3","Type":"ContainerDied","Data":"d478b997adc512779e53a882c989e0405c8e7fbeeb61471ac077a8f195c2f550"}
Sep 18 17:14:26.303156 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.302934    2585 scope.go:117] "RemoveContainer" containerID="9970212a55986a211b127d7ef60a5ccfd607024ad44f141cfb08c21d03d2fd6a"
Sep 18 17:14:26.439201 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.439141    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:14:26.506157 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.506130    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92eed10-8069-46ec-a43a-3058c1f79db3-workload-socket\") pod \"b92eed10-8069-46ec-a43a-3058c1f79db3\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") "
Sep 18 17:14:26.506299 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.506179    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b92eed10-8069-46ec-a43a-3058c1f79db3-istio-envoy\") pod \"b92eed10-8069-46ec-a43a-3058c1f79db3\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") "
Sep 18 17:14:26.506299 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.506203    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b92eed10-8069-46ec-a43a-3058c1f79db3-workload-certs\") pod \"b92eed10-8069-46ec-a43a-3058c1f79db3\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") "
Sep 18 17:14:26.506299 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.506240    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92eed10-8069-46ec-a43a-3058c1f79db3-credential-socket\") pod \"b92eed10-8069-46ec-a43a-3058c1f79db3\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") "
Sep 18 17:14:26.506299 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.506261    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxdf4\" (UniqueName: \"kubernetes.io/projected/b92eed10-8069-46ec-a43a-3058c1f79db3-kube-api-access-jxdf4\") pod \"b92eed10-8069-46ec-a43a-3058c1f79db3\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") "
Sep 18 17:14:26.506299 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.506279    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b92eed10-8069-46ec-a43a-3058c1f79db3-istio-podinfo\") pod \"b92eed10-8069-46ec-a43a-3058c1f79db3\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") "
Sep 18 17:14:26.506563 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.506315    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b92eed10-8069-46ec-a43a-3058c1f79db3-istio-token\") pod \"b92eed10-8069-46ec-a43a-3058c1f79db3\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") "
Sep 18 17:14:26.506563 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.506351    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b92eed10-8069-46ec-a43a-3058c1f79db3-istiod-ca-cert\") pod \"b92eed10-8069-46ec-a43a-3058c1f79db3\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") "
Sep 18 17:14:26.506563 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.506391    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b92eed10-8069-46ec-a43a-3058c1f79db3-istio-data\") pod \"b92eed10-8069-46ec-a43a-3058c1f79db3\" (UID: \"b92eed10-8069-46ec-a43a-3058c1f79db3\") "
Sep 18 17:14:26.506563 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.506392    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92eed10-8069-46ec-a43a-3058c1f79db3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b92eed10-8069-46ec-a43a-3058c1f79db3" (UID: "b92eed10-8069-46ec-a43a-3058c1f79db3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:14:26.506563 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.506533    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92eed10-8069-46ec-a43a-3058c1f79db3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b92eed10-8069-46ec-a43a-3058c1f79db3" (UID: "b92eed10-8069-46ec-a43a-3058c1f79db3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:14:26.506834 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.506688    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92eed10-8069-46ec-a43a-3058c1f79db3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b92eed10-8069-46ec-a43a-3058c1f79db3" (UID: "b92eed10-8069-46ec-a43a-3058c1f79db3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:14:26.506834 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.506824    2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b92eed10-8069-46ec-a43a-3058c1f79db3-workload-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:14:26.507222 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.506844    2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92eed10-8069-46ec-a43a-3058c1f79db3-credential-socket\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:14:26.507222 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.506860    2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92eed10-8069-46ec-a43a-3058c1f79db3-workload-socket\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:14:26.507222 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.507064    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b92eed10-8069-46ec-a43a-3058c1f79db3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b92eed10-8069-46ec-a43a-3058c1f79db3" (UID: "b92eed10-8069-46ec-a43a-3058c1f79db3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:14:26.507388 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.507339    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92eed10-8069-46ec-a43a-3058c1f79db3-istio-data" (OuterVolumeSpecName: "istio-data") pod "b92eed10-8069-46ec-a43a-3058c1f79db3" (UID: "b92eed10-8069-46ec-a43a-3058c1f79db3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:14:26.509047 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.509012    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b92eed10-8069-46ec-a43a-3058c1f79db3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b92eed10-8069-46ec-a43a-3058c1f79db3" (UID: "b92eed10-8069-46ec-a43a-3058c1f79db3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:14:26.509312 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.509284    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b92eed10-8069-46ec-a43a-3058c1f79db3-istio-token" (OuterVolumeSpecName: "istio-token") pod "b92eed10-8069-46ec-a43a-3058c1f79db3" (UID: "b92eed10-8069-46ec-a43a-3058c1f79db3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:14:26.509477 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.509458    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b92eed10-8069-46ec-a43a-3058c1f79db3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b92eed10-8069-46ec-a43a-3058c1f79db3" (UID: "b92eed10-8069-46ec-a43a-3058c1f79db3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 17:14:26.509477 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.509467    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b92eed10-8069-46ec-a43a-3058c1f79db3-kube-api-access-jxdf4" (OuterVolumeSpecName: "kube-api-access-jxdf4") pod "b92eed10-8069-46ec-a43a-3058c1f79db3" (UID: "b92eed10-8069-46ec-a43a-3058c1f79db3"). InnerVolumeSpecName "kube-api-access-jxdf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:14:26.607656 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.607628    2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b92eed10-8069-46ec-a43a-3058c1f79db3-istio-podinfo\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:14:26.607656 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.607655    2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b92eed10-8069-46ec-a43a-3058c1f79db3-istio-token\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:14:26.607656 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.607664    2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b92eed10-8069-46ec-a43a-3058c1f79db3-istiod-ca-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:14:26.607862 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.607672    2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b92eed10-8069-46ec-a43a-3058c1f79db3-istio-data\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:14:26.607862 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.607679    2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b92eed10-8069-46ec-a43a-3058c1f79db3-istio-envoy\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:14:26.607862 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:26.607687    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxdf4\" (UniqueName: \"kubernetes.io/projected/b92eed10-8069-46ec-a43a-3058c1f79db3-kube-api-access-jxdf4\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:14:27.309136 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:27.309102    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"
Sep 18 17:14:27.309589 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:27.309141    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5" event={"ID":"b92eed10-8069-46ec-a43a-3058c1f79db3","Type":"ContainerDied","Data":"1519a7cfcaa21a1956638a03e4f805680294e2e2990275eee5aa05d6c588bcc3"}
Sep 18 17:14:27.309589 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:27.309191    2585 scope.go:117] "RemoveContainer" containerID="d478b997adc512779e53a882c989e0405c8e7fbeeb61471ac077a8f195c2f550"
Sep 18 17:14:27.310956 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:27.310929    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml" event={"ID":"354562a6-430a-4270-ab8a-58a13454ecb2","Type":"ContainerStarted","Data":"ca3f37fd383e783a838b3449c9f01e0cb580d5504585e99b94c47184da1b2bbc"}
Sep 18 17:14:27.311094 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:27.311048    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:14:27.347657 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:27.347634    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"]
Sep 18 17:14:27.354193 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:27.354172    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-d7ct5"]
Sep 18 17:14:28.318803 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:28.318766    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml" event={"ID":"354562a6-430a-4270-ab8a-58a13454ecb2","Type":"ContainerStarted","Data":"2e8b3487adfa93ff3e1267ed2d3b94191be4960a5c14cf6ba46fbde1668d573b"}
Sep 18 17:14:28.392520 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:28.392487    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b92eed10-8069-46ec-a43a-3058c1f79db3" path="/var/lib/kubelet/pods/b92eed10-8069-46ec-a43a-3058c1f79db3/volumes"
Sep 18 17:14:30.329931 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:30.329884    2585 generic.go:356] "Generic (PLEG): container finished" podID="354562a6-430a-4270-ab8a-58a13454ecb2" containerID="2e8b3487adfa93ff3e1267ed2d3b94191be4960a5c14cf6ba46fbde1668d573b" exitCode=0
Sep 18 17:14:30.330394 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:30.329956    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml" event={"ID":"354562a6-430a-4270-ab8a-58a13454ecb2","Type":"ContainerDied","Data":"2e8b3487adfa93ff3e1267ed2d3b94191be4960a5c14cf6ba46fbde1668d573b"}
Sep 18 17:14:31.336969 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:31.336939    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml" event={"ID":"354562a6-430a-4270-ab8a-58a13454ecb2","Type":"ContainerStarted","Data":"a939c518650d25929ef6ab295f94ef70b9d74ecb284c141e74fe5083302d4f82"}
Sep 18 17:14:31.365386 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:31.365326    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml" podStartSLOduration=4.433569053 podStartE2EDuration="8.365306702s" totalImagesPullingTime="1.766868287s" totalInitContainerRuntime="2.164869362s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:14:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:14:31.363811215 +0000 UTC m=+3511.146813239" watchObservedRunningTime="2026-09-18 17:14:31.365306702 +0000 UTC m=+3511.148308739"
Sep 18 17:14:34.224032 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:34.223973    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:14:34.224593 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:34.224072    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:14:34.225500 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:34.225470    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml" podUID="354562a6-430a-4270-ab8a-58a13454ecb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8001/health\": dial tcp 10.133.0.60:8001: connect: connection refused"
Sep 18 17:14:44.224704 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:44.224659    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml" podUID="354562a6-430a-4270-ab8a-58a13454ecb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8001/health\": dial tcp 10.133.0.60:8001: connect: connection refused"
Sep 18 17:14:44.361289 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:44.361259    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:14:54.224364 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:14:54.224316    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml" podUID="354562a6-430a-4270-ab8a-58a13454ecb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8001/health\": dial tcp 10.133.0.60:8001: connect: connection refused"
Sep 18 17:15:04.224236 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:15:04.224176    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml" podUID="354562a6-430a-4270-ab8a-58a13454ecb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8001/health\": dial tcp 10.133.0.60:8001: connect: connection refused"
Sep 18 17:15:14.224606 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:15:14.224561    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml" podUID="354562a6-430a-4270-ab8a-58a13454ecb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8001/health\": dial tcp 10.133.0.60:8001: connect: connection refused"
Sep 18 17:15:24.224516 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:15:24.224464    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml" podUID="354562a6-430a-4270-ab8a-58a13454ecb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8001/health\": dial tcp 10.133.0.60:8001: connect: connection refused"
Sep 18 17:15:34.224209 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:15:34.224160    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml" podUID="354562a6-430a-4270-ab8a-58a13454ecb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8001/health\": dial tcp 10.133.0.60:8001: connect: connection refused"
Sep 18 17:15:44.225216 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:15:44.225097    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml" podUID="354562a6-430a-4270-ab8a-58a13454ecb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8001/health\": dial tcp 10.133.0.60:8001: connect: connection refused"
Sep 18 17:15:54.224450 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:15:54.224395    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml" podUID="354562a6-430a-4270-ab8a-58a13454ecb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8001/health\": dial tcp 10.133.0.60:8001: connect: connection refused"
Sep 18 17:16:02.591529 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:02.591503    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 17:16:02.596034 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:02.596011    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 17:16:02.601938 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:02.601903    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 17:16:02.605978 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:02.605959    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 17:16:04.225050 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:04.224989    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml" podUID="354562a6-430a-4270-ab8a-58a13454ecb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8001/health\": dial tcp 10.133.0.60:8001: connect: connection refused"
Sep 18 17:16:14.224309 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:14.224258    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml" podUID="354562a6-430a-4270-ab8a-58a13454ecb2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8001/health\": dial tcp 10.133.0.60:8001: connect: connection refused"
Sep 18 17:16:24.239177 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:24.239144    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:16:24.257839 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:24.257810    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:16:34.672382 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.672344    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"]
Sep 18 17:16:34.672772 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.672755    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b92eed10-8069-46ec-a43a-3058c1f79db3" containerName="istio-proxy"
Sep 18 17:16:34.672818 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.672776    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b92eed10-8069-46ec-a43a-3058c1f79db3" containerName="istio-proxy"
Sep 18 17:16:34.672818 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.672804    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b92eed10-8069-46ec-a43a-3058c1f79db3" containerName="istio-proxy"
Sep 18 17:16:34.672818 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.672812    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b92eed10-8069-46ec-a43a-3058c1f79db3" containerName="istio-proxy"
Sep 18 17:16:34.672932 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.672882    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b92eed10-8069-46ec-a43a-3058c1f79db3" containerName="istio-proxy"
Sep 18 17:16:34.673042 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.673031    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b92eed10-8069-46ec-a43a-3058c1f79db3" containerName="istio-proxy"
Sep 18 17:16:34.675813 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.675792    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:16:34.678438 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.678411    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-v48mg\""
Sep 18 17:16:34.679943 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.679904    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 18 17:16:34.680071 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.680007    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 18 17:16:34.680071 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.680056    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 18 17:16:34.680192 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.680111    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 18 17:16:34.680192 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.680139    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 17:16:34.688149 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.688127    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"]
Sep 18 17:16:34.869534 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.869505    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/440327bd-970b-433f-a6c5-49527b47b86d-model-cache\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:16:34.869534 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.869538    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/440327bd-970b-433f-a6c5-49527b47b86d-dshm\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:16:34.869813 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.869556    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/440327bd-970b-433f-a6c5-49527b47b86d-home\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:16:34.869813 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.869586    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/440327bd-970b-433f-a6c5-49527b47b86d-tls-certs\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:16:34.869813 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.869674    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/440327bd-970b-433f-a6c5-49527b47b86d-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:16:34.869813 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.869742    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/440327bd-970b-433f-a6c5-49527b47b86d-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:16:34.869813 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.869792    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q4kj\" (UniqueName: \"kubernetes.io/projected/440327bd-970b-433f-a6c5-49527b47b86d-kube-api-access-7q4kj\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:16:34.870042 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.869820    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/440327bd-970b-433f-a6c5-49527b47b86d-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:16:34.971217 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.971133    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/440327bd-970b-433f-a6c5-49527b47b86d-tls-certs\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:16:34.971217 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.971174    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/440327bd-970b-433f-a6c5-49527b47b86d-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:16:34.971446 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.971292    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/440327bd-970b-433f-a6c5-49527b47b86d-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:16:34.971446 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.971337    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q4kj\" (UniqueName: \"kubernetes.io/projected/440327bd-970b-433f-a6c5-49527b47b86d-kube-api-access-7q4kj\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:16:34.971446 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.971362    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/440327bd-970b-433f-a6c5-49527b47b86d-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:16:34.971612 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.971536    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/440327bd-970b-433f-a6c5-49527b47b86d-model-cache\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:16:34.971612 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.971577    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/440327bd-970b-433f-a6c5-49527b47b86d-dshm\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:16:34.971730 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.971676    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/440327bd-970b-433f-a6c5-49527b47b86d-home\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:16:34.971789 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.971737    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/440327bd-970b-433f-a6c5-49527b47b86d-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:16:34.971841 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.971818    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/440327bd-970b-433f-a6c5-49527b47b86d-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:16:34.971897 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.971857    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/440327bd-970b-433f-a6c5-49527b47b86d-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:16:34.972094 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.971986    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/440327bd-970b-433f-a6c5-49527b47b86d-model-cache\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:16:34.972094 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.972000    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/440327bd-970b-433f-a6c5-49527b47b86d-home\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:16:34.973810 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.973788    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/440327bd-970b-433f-a6c5-49527b47b86d-dshm\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:16:34.974097 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.974079    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/440327bd-970b-433f-a6c5-49527b47b86d-tls-certs\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:16:34.984461 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.984437    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q4kj\" (UniqueName: \"kubernetes.io/projected/440327bd-970b-433f-a6c5-49527b47b86d-kube-api-access-7q4kj\") pod \"lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:16:34.986248 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:34.986230    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:16:35.267987 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:35.267955    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"]
Sep 18 17:16:35.269168 ip-10-0-133-147 kubenswrapper[2585]: W0918 17:16:35.269139    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440327bd_970b_433f_a6c5_49527b47b86d.slice/crio-ac556a8402f8cafb03259dacc8132925e10af20ead8e7af68267b741c1005ad9 WatchSource:0}: Error finding container ac556a8402f8cafb03259dacc8132925e10af20ead8e7af68267b741c1005ad9: Status 404 returned error can't find the container with id ac556a8402f8cafb03259dacc8132925e10af20ead8e7af68267b741c1005ad9
Sep 18 17:16:35.865068 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:35.865025    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw" event={"ID":"440327bd-970b-433f-a6c5-49527b47b86d","Type":"ContainerStarted","Data":"4397c21c77704742eb5688ccbbcdbea2d79275dc244a8efb6d04c6e8eeef30af"}
Sep 18 17:16:35.865586 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:35.865119    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw" event={"ID":"440327bd-970b-433f-a6c5-49527b47b86d","Type":"ContainerStarted","Data":"ac556a8402f8cafb03259dacc8132925e10af20ead8e7af68267b741c1005ad9"}
Sep 18 17:16:38.880430 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:38.880391    2585 generic.go:356] "Generic (PLEG): container finished" podID="440327bd-970b-433f-a6c5-49527b47b86d" containerID="4397c21c77704742eb5688ccbbcdbea2d79275dc244a8efb6d04c6e8eeef30af" exitCode=0
Sep 18 17:16:38.880843 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:38.880467    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw" event={"ID":"440327bd-970b-433f-a6c5-49527b47b86d","Type":"ContainerDied","Data":"4397c21c77704742eb5688ccbbcdbea2d79275dc244a8efb6d04c6e8eeef30af"}
Sep 18 17:16:39.889021 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:39.888981    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw" event={"ID":"440327bd-970b-433f-a6c5-49527b47b86d","Type":"ContainerStarted","Data":"3df139ab389fb3e868c1a0cff5d108d7bea7d1bfe53127cf0c39e857485d91da"}
Sep 18 17:16:39.945045 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:39.944980    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw" podStartSLOduration=3.098449587 podStartE2EDuration="5.944957219s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.846507632s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:16:39.943030729 +0000 UTC m=+3639.726032764" watchObservedRunningTime="2026-09-18 17:16:39.944957219 +0000 UTC m=+3639.727959243"
Sep 18 17:16:39.986699 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:39.986665    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:16:44.986964 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:44.986925    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:16:44.988643 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:44.988613    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw" podUID="440327bd-970b-433f-a6c5-49527b47b86d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Sep 18 17:16:54.222443 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:54.222404    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"]
Sep 18 17:16:54.986708 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:16:54.986666    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw" podUID="440327bd-970b-433f-a6c5-49527b47b86d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Sep 18 17:17:04.986988 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:04.986935    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw" podUID="440327bd-970b-433f-a6c5-49527b47b86d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Sep 18 17:17:09.249511 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:09.249452    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml" podUID="354562a6-430a-4270-ab8a-58a13454ecb2" containerName="main" containerID="cri-o://a939c518650d25929ef6ab295f94ef70b9d74ecb284c141e74fe5083302d4f82" gracePeriod=45
Sep 18 17:17:09.861115 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:09.861076    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml" podUID="354562a6-430a-4270-ab8a-58a13454ecb2" containerName="llm-d-routing-sidecar" containerID="cri-o://ca3f37fd383e783a838b3449c9f01e0cb580d5504585e99b94c47184da1b2bbc" gracePeriod=45
Sep 18 17:17:10.025628 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:10.025539    2585 generic.go:356] "Generic (PLEG): container finished" podID="354562a6-430a-4270-ab8a-58a13454ecb2" containerID="a939c518650d25929ef6ab295f94ef70b9d74ecb284c141e74fe5083302d4f82" exitCode=0
Sep 18 17:17:10.025628 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:10.025567    2585 generic.go:356] "Generic (PLEG): container finished" podID="354562a6-430a-4270-ab8a-58a13454ecb2" containerID="ca3f37fd383e783a838b3449c9f01e0cb580d5504585e99b94c47184da1b2bbc" exitCode=0
Sep 18 17:17:10.025628 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:10.025608    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml" event={"ID":"354562a6-430a-4270-ab8a-58a13454ecb2","Type":"ContainerDied","Data":"a939c518650d25929ef6ab295f94ef70b9d74ecb284c141e74fe5083302d4f82"}
Sep 18 17:17:10.025829 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:10.025675    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml" event={"ID":"354562a6-430a-4270-ab8a-58a13454ecb2","Type":"ContainerDied","Data":"ca3f37fd383e783a838b3449c9f01e0cb580d5504585e99b94c47184da1b2bbc"}
Sep 18 17:17:10.118261 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:10.118237    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:17:10.282775 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:10.282695    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/354562a6-430a-4270-ab8a-58a13454ecb2-tmp-dir\") pod \"354562a6-430a-4270-ab8a-58a13454ecb2\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") "
Sep 18 17:17:10.282775 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:10.282740    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsxz8\" (UniqueName: \"kubernetes.io/projected/354562a6-430a-4270-ab8a-58a13454ecb2-kube-api-access-qsxz8\") pod \"354562a6-430a-4270-ab8a-58a13454ecb2\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") "
Sep 18 17:17:10.282775 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:10.282769    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/354562a6-430a-4270-ab8a-58a13454ecb2-tls-certs\") pod \"354562a6-430a-4270-ab8a-58a13454ecb2\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") "
Sep 18 17:17:10.283344 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:10.282802    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/354562a6-430a-4270-ab8a-58a13454ecb2-kserve-provision-location\") pod \"354562a6-430a-4270-ab8a-58a13454ecb2\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") "
Sep 18 17:17:10.283344 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:10.282842    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/354562a6-430a-4270-ab8a-58a13454ecb2-dshm\") pod \"354562a6-430a-4270-ab8a-58a13454ecb2\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") "
Sep 18 17:17:10.283344 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:10.282868    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/354562a6-430a-4270-ab8a-58a13454ecb2-home\") pod \"354562a6-430a-4270-ab8a-58a13454ecb2\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") "
Sep 18 17:17:10.283344 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:10.282945    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/354562a6-430a-4270-ab8a-58a13454ecb2-model-cache\") pod \"354562a6-430a-4270-ab8a-58a13454ecb2\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") "
Sep 18 17:17:10.283344 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:10.282979    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/354562a6-430a-4270-ab8a-58a13454ecb2-cabundle-cert\") pod \"354562a6-430a-4270-ab8a-58a13454ecb2\" (UID: \"354562a6-430a-4270-ab8a-58a13454ecb2\") "
Sep 18 17:17:10.284049 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:10.283607    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/354562a6-430a-4270-ab8a-58a13454ecb2-model-cache" (OuterVolumeSpecName: "model-cache") pod "354562a6-430a-4270-ab8a-58a13454ecb2" (UID: "354562a6-430a-4270-ab8a-58a13454ecb2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:17:10.284049 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:10.283631    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/354562a6-430a-4270-ab8a-58a13454ecb2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "354562a6-430a-4270-ab8a-58a13454ecb2" (UID: "354562a6-430a-4270-ab8a-58a13454ecb2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:17:10.284049 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:10.284015    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/354562a6-430a-4270-ab8a-58a13454ecb2-home" (OuterVolumeSpecName: "home") pod "354562a6-430a-4270-ab8a-58a13454ecb2" (UID: "354562a6-430a-4270-ab8a-58a13454ecb2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:17:10.286068 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:10.286023    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/354562a6-430a-4270-ab8a-58a13454ecb2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "354562a6-430a-4270-ab8a-58a13454ecb2" (UID: "354562a6-430a-4270-ab8a-58a13454ecb2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:17:10.286068 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:10.286023    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/354562a6-430a-4270-ab8a-58a13454ecb2-dshm" (OuterVolumeSpecName: "dshm") pod "354562a6-430a-4270-ab8a-58a13454ecb2" (UID: "354562a6-430a-4270-ab8a-58a13454ecb2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:17:10.286241 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:10.286124    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/354562a6-430a-4270-ab8a-58a13454ecb2-kube-api-access-qsxz8" (OuterVolumeSpecName: "kube-api-access-qsxz8") pod "354562a6-430a-4270-ab8a-58a13454ecb2" (UID: "354562a6-430a-4270-ab8a-58a13454ecb2"). InnerVolumeSpecName "kube-api-access-qsxz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:17:10.297649 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:10.297624    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/354562a6-430a-4270-ab8a-58a13454ecb2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "354562a6-430a-4270-ab8a-58a13454ecb2" (UID: "354562a6-430a-4270-ab8a-58a13454ecb2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:17:10.305089 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:10.305068    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/354562a6-430a-4270-ab8a-58a13454ecb2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "354562a6-430a-4270-ab8a-58a13454ecb2" (UID: "354562a6-430a-4270-ab8a-58a13454ecb2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:17:10.384240 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:10.384213    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/354562a6-430a-4270-ab8a-58a13454ecb2-dshm\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:17:10.384240 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:10.384235    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/354562a6-430a-4270-ab8a-58a13454ecb2-home\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:17:10.384377 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:10.384243    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/354562a6-430a-4270-ab8a-58a13454ecb2-model-cache\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:17:10.384377 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:10.384253    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/354562a6-430a-4270-ab8a-58a13454ecb2-cabundle-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:17:10.384377 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:10.384261    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/354562a6-430a-4270-ab8a-58a13454ecb2-tmp-dir\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:17:10.384377 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:10.384269    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsxz8\" (UniqueName: \"kubernetes.io/projected/354562a6-430a-4270-ab8a-58a13454ecb2-kube-api-access-qsxz8\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:17:10.384377 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:10.384277    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/354562a6-430a-4270-ab8a-58a13454ecb2-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:17:10.384377 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:10.384285    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/354562a6-430a-4270-ab8a-58a13454ecb2-kserve-provision-location\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:17:11.032078 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:11.032036    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml" event={"ID":"354562a6-430a-4270-ab8a-58a13454ecb2","Type":"ContainerDied","Data":"eadd8f41b50c06fb962b090267441d58a2676d6ecfa8b99de996e9e43d6982f2"}
Sep 18 17:17:11.032247 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:11.032127    2585 scope.go:117] "RemoveContainer" containerID="a939c518650d25929ef6ab295f94ef70b9d74ecb284c141e74fe5083302d4f82"
Sep 18 17:17:11.032247 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:11.032051    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"
Sep 18 17:17:11.040750 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:11.040729    2585 scope.go:117] "RemoveContainer" containerID="2e8b3487adfa93ff3e1267ed2d3b94191be4960a5c14cf6ba46fbde1668d573b"
Sep 18 17:17:11.048382 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:11.048365    2585 scope.go:117] "RemoveContainer" containerID="ca3f37fd383e783a838b3449c9f01e0cb580d5504585e99b94c47184da1b2bbc"
Sep 18 17:17:11.055102 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:11.055076    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"]
Sep 18 17:17:11.057779 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:11.057756    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5d7565d6b7tjtml"]
Sep 18 17:17:12.394367 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:12.394270    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="354562a6-430a-4270-ab8a-58a13454ecb2" path="/var/lib/kubelet/pods/354562a6-430a-4270-ab8a-58a13454ecb2/volumes"
Sep 18 17:17:14.987406 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:14.987363    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw" podUID="440327bd-970b-433f-a6c5-49527b47b86d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Sep 18 17:17:24.987161 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:24.987107    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw" podUID="440327bd-970b-433f-a6c5-49527b47b86d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Sep 18 17:17:34.987372 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:34.987319    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw" podUID="440327bd-970b-433f-a6c5-49527b47b86d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Sep 18 17:17:44.987694 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:44.987650    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw" podUID="440327bd-970b-433f-a6c5-49527b47b86d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Sep 18 17:17:54.996999 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:54.996963    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:17:55.004637 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:17:55.004605    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:18:05.349413 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.349384    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"]
Sep 18 17:18:05.349813 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.349660    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="354562a6-430a-4270-ab8a-58a13454ecb2" containerName="storage-initializer"
Sep 18 17:18:05.349813 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.349671    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="354562a6-430a-4270-ab8a-58a13454ecb2" containerName="storage-initializer"
Sep 18 17:18:05.349813 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.349682    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="354562a6-430a-4270-ab8a-58a13454ecb2" containerName="main"
Sep 18 17:18:05.349813 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.349689    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="354562a6-430a-4270-ab8a-58a13454ecb2" containerName="main"
Sep 18 17:18:05.349813 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.349709    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="354562a6-430a-4270-ab8a-58a13454ecb2" containerName="llm-d-routing-sidecar"
Sep 18 17:18:05.349813 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.349719    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="354562a6-430a-4270-ab8a-58a13454ecb2" containerName="llm-d-routing-sidecar"
Sep 18 17:18:05.349813 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.349796    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="354562a6-430a-4270-ab8a-58a13454ecb2" containerName="llm-d-routing-sidecar"
Sep 18 17:18:05.349813 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.349810    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="354562a6-430a-4270-ab8a-58a13454ecb2" containerName="main"
Sep 18 17:18:05.352304 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.352286    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:18:05.355081 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.355058    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 17:18:05.356405 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.356173    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-jlp6s\""
Sep 18 17:18:05.356405 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.356184    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 18 17:18:05.356405 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.356210    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-44kfd\""
Sep 18 17:18:05.356405 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.356287    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 18 17:18:05.356405 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.356210    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 18 17:18:05.356405 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.356218    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 18 17:18:05.371994 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.371972    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"]
Sep 18 17:18:05.436856 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.436829    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/747aec53-b00c-4aba-b278-b49104058d80-home\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-rwppj\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:18:05.437034 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.436887    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/747aec53-b00c-4aba-b278-b49104058d80-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-rwppj\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:18:05.437034 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.436940    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/747aec53-b00c-4aba-b278-b49104058d80-dshm\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-rwppj\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:18:05.437034 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.436996    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn8fx\" (UniqueName: \"kubernetes.io/projected/747aec53-b00c-4aba-b278-b49104058d80-kube-api-access-jn8fx\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-rwppj\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:18:05.437153 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.437038    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/747aec53-b00c-4aba-b278-b49104058d80-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-rwppj\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:18:05.437153 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.437094    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/747aec53-b00c-4aba-b278-b49104058d80-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-rwppj\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:18:05.437153 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.437121    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/747aec53-b00c-4aba-b278-b49104058d80-model-cache\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-rwppj\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:18:05.437153 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.437145    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/747aec53-b00c-4aba-b278-b49104058d80-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-rwppj\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:18:05.537694 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.537660    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/747aec53-b00c-4aba-b278-b49104058d80-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-rwppj\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:18:05.537880 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.537705    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/747aec53-b00c-4aba-b278-b49104058d80-dshm\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-rwppj\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:18:05.537880 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.537725    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn8fx\" (UniqueName: \"kubernetes.io/projected/747aec53-b00c-4aba-b278-b49104058d80-kube-api-access-jn8fx\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-rwppj\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:18:05.537880 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.537803    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/747aec53-b00c-4aba-b278-b49104058d80-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-rwppj\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:18:05.538159 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.537993    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/747aec53-b00c-4aba-b278-b49104058d80-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-rwppj\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:18:05.538159 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.538038    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/747aec53-b00c-4aba-b278-b49104058d80-model-cache\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-rwppj\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:18:05.538159 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.538065    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/747aec53-b00c-4aba-b278-b49104058d80-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-rwppj\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:18:05.538320 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.538163    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/747aec53-b00c-4aba-b278-b49104058d80-home\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-rwppj\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:18:05.538474 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.538448    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/747aec53-b00c-4aba-b278-b49104058d80-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-rwppj\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:18:05.538661 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.538494    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/747aec53-b00c-4aba-b278-b49104058d80-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-rwppj\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:18:05.538661 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.538532    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/747aec53-b00c-4aba-b278-b49104058d80-model-cache\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-rwppj\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:18:05.538808 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.538551    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/747aec53-b00c-4aba-b278-b49104058d80-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-rwppj\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:18:05.538808 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.538722    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/747aec53-b00c-4aba-b278-b49104058d80-home\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-rwppj\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:18:05.540891 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.540867    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/747aec53-b00c-4aba-b278-b49104058d80-dshm\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-rwppj\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:18:05.541104 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.541088    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/747aec53-b00c-4aba-b278-b49104058d80-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-rwppj\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:18:05.547775 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.547755    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn8fx\" (UniqueName: \"kubernetes.io/projected/747aec53-b00c-4aba-b278-b49104058d80-kube-api-access-jn8fx\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-rwppj\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:18:05.663630 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.663547    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:18:05.941816 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:05.941686    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"]
Sep 18 17:18:05.944828 ip-10-0-133-147 kubenswrapper[2585]: W0918 17:18:05.944785    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod747aec53_b00c_4aba_b278_b49104058d80.slice/crio-b0925ef52952cc2cff7a5ca1a6421f2de58a4d867ac3c77080ac868ceb3f5353 WatchSource:0}: Error finding container b0925ef52952cc2cff7a5ca1a6421f2de58a4d867ac3c77080ac868ceb3f5353: Status 404 returned error can't find the container with id b0925ef52952cc2cff7a5ca1a6421f2de58a4d867ac3c77080ac868ceb3f5353
Sep 18 17:18:06.260817 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:06.260780    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj" event={"ID":"747aec53-b00c-4aba-b278-b49104058d80","Type":"ContainerStarted","Data":"c71008eae2ca43aa4d136f145776809a37ffb1e8b72c524f90f9a71883609ed4"}
Sep 18 17:18:06.261052 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:06.260839    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj" event={"ID":"747aec53-b00c-4aba-b278-b49104058d80","Type":"ContainerStarted","Data":"b0925ef52952cc2cff7a5ca1a6421f2de58a4d867ac3c77080ac868ceb3f5353"}
Sep 18 17:18:06.261052 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:06.260942    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:18:07.266555 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:07.266513    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj" event={"ID":"747aec53-b00c-4aba-b278-b49104058d80","Type":"ContainerStarted","Data":"10428e9782c45c1e00dce3972d6ff268a360126655a9c6af9f01ef365ad1935e"}
Sep 18 17:18:09.277240 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:09.277199    2585 generic.go:356] "Generic (PLEG): container finished" podID="747aec53-b00c-4aba-b278-b49104058d80" containerID="10428e9782c45c1e00dce3972d6ff268a360126655a9c6af9f01ef365ad1935e" exitCode=0
Sep 18 17:18:09.277625 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:09.277249    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj" event={"ID":"747aec53-b00c-4aba-b278-b49104058d80","Type":"ContainerDied","Data":"10428e9782c45c1e00dce3972d6ff268a360126655a9c6af9f01ef365ad1935e"}
Sep 18 17:18:10.284661 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:10.284629    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj" event={"ID":"747aec53-b00c-4aba-b278-b49104058d80","Type":"ContainerStarted","Data":"f78b7a023f616e9ad0225779120d94101e5bc2bb06d59756500ed67927200781"}
Sep 18 17:18:10.309743 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:10.309677    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj" podStartSLOduration=3.200988579 podStartE2EDuration="5.309655597s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.108667018s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:18:10.306584249 +0000 UTC m=+3730.089586272" watchObservedRunningTime="2026-09-18 17:18:10.309655597 +0000 UTC m=+3730.092657622"
Sep 18 17:18:15.664276 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:15.664226    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:18:15.664839 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:15.664317    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:18:15.665701 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:15.665663    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj" podUID="747aec53-b00c-4aba-b278-b49104058d80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 18 17:18:15.684102 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:15.684080    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:18:25.664156 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:25.664094    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj" podUID="747aec53-b00c-4aba-b278-b49104058d80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 18 17:18:35.664609 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:35.664562    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj" podUID="747aec53-b00c-4aba-b278-b49104058d80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 18 17:18:45.665072 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:45.664960    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj" podUID="747aec53-b00c-4aba-b278-b49104058d80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 18 17:18:46.381677 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:46.381641    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw_440327bd-970b-433f-a6c5-49527b47b86d/storage-initializer/0.log"
Sep 18 17:18:46.393397 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:46.393364    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw_440327bd-970b-433f-a6c5-49527b47b86d/main/0.log"
Sep 18 17:18:47.420526 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:47.420494    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"]
Sep 18 17:18:55.664347 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:18:55.664297    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj" podUID="747aec53-b00c-4aba-b278-b49104058d80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 18 17:19:02.443350 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:02.443285    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw" podUID="440327bd-970b-433f-a6c5-49527b47b86d" containerName="main" containerID="cri-o://3df139ab389fb3e868c1a0cff5d108d7bea7d1bfe53127cf0c39e857485d91da" gracePeriod=45
Sep 18 17:19:02.997630 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:02.997585    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw" podUID="440327bd-970b-433f-a6c5-49527b47b86d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Sep 18 17:19:03.099216 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.099193    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:19:03.255079 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.255045    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/440327bd-970b-433f-a6c5-49527b47b86d-dshm\") pod \"440327bd-970b-433f-a6c5-49527b47b86d\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") "
Sep 18 17:19:03.255268 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.255091    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/440327bd-970b-433f-a6c5-49527b47b86d-model-cache\") pod \"440327bd-970b-433f-a6c5-49527b47b86d\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") "
Sep 18 17:19:03.255268 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.255109    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/440327bd-970b-433f-a6c5-49527b47b86d-tls-certs\") pod \"440327bd-970b-433f-a6c5-49527b47b86d\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") "
Sep 18 17:19:03.255268 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.255141    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/440327bd-970b-433f-a6c5-49527b47b86d-tmp-dir\") pod \"440327bd-970b-433f-a6c5-49527b47b86d\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") "
Sep 18 17:19:03.255268 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.255165    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/440327bd-970b-433f-a6c5-49527b47b86d-cabundle-cert\") pod \"440327bd-970b-433f-a6c5-49527b47b86d\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") "
Sep 18 17:19:03.255268 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.255190    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7q4kj\" (UniqueName: \"kubernetes.io/projected/440327bd-970b-433f-a6c5-49527b47b86d-kube-api-access-7q4kj\") pod \"440327bd-970b-433f-a6c5-49527b47b86d\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") "
Sep 18 17:19:03.255552 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.255389    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440327bd-970b-433f-a6c5-49527b47b86d-model-cache" (OuterVolumeSpecName: "model-cache") pod "440327bd-970b-433f-a6c5-49527b47b86d" (UID: "440327bd-970b-433f-a6c5-49527b47b86d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:19:03.255552 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.255474    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/440327bd-970b-433f-a6c5-49527b47b86d-kserve-provision-location\") pod \"440327bd-970b-433f-a6c5-49527b47b86d\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") "
Sep 18 17:19:03.255552 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.255540    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/440327bd-970b-433f-a6c5-49527b47b86d-home\") pod \"440327bd-970b-433f-a6c5-49527b47b86d\" (UID: \"440327bd-970b-433f-a6c5-49527b47b86d\") "
Sep 18 17:19:03.255722 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.255579    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440327bd-970b-433f-a6c5-49527b47b86d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "440327bd-970b-433f-a6c5-49527b47b86d" (UID: "440327bd-970b-433f-a6c5-49527b47b86d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:19:03.255840 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.255791    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/440327bd-970b-433f-a6c5-49527b47b86d-cabundle-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:19:03.255840 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.255815    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/440327bd-970b-433f-a6c5-49527b47b86d-model-cache\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:19:03.256072 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.255969    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440327bd-970b-433f-a6c5-49527b47b86d-home" (OuterVolumeSpecName: "home") pod "440327bd-970b-433f-a6c5-49527b47b86d" (UID: "440327bd-970b-433f-a6c5-49527b47b86d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:19:03.258289 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.258254    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440327bd-970b-433f-a6c5-49527b47b86d-dshm" (OuterVolumeSpecName: "dshm") pod "440327bd-970b-433f-a6c5-49527b47b86d" (UID: "440327bd-970b-433f-a6c5-49527b47b86d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:19:03.258490 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.258467    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/440327bd-970b-433f-a6c5-49527b47b86d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "440327bd-970b-433f-a6c5-49527b47b86d" (UID: "440327bd-970b-433f-a6c5-49527b47b86d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:19:03.258599 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.258487    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440327bd-970b-433f-a6c5-49527b47b86d-kube-api-access-7q4kj" (OuterVolumeSpecName: "kube-api-access-7q4kj") pod "440327bd-970b-433f-a6c5-49527b47b86d" (UID: "440327bd-970b-433f-a6c5-49527b47b86d"). InnerVolumeSpecName "kube-api-access-7q4kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:19:03.266342 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.266313    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440327bd-970b-433f-a6c5-49527b47b86d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "440327bd-970b-433f-a6c5-49527b47b86d" (UID: "440327bd-970b-433f-a6c5-49527b47b86d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:19:03.277651 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.277619    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440327bd-970b-433f-a6c5-49527b47b86d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "440327bd-970b-433f-a6c5-49527b47b86d" (UID: "440327bd-970b-433f-a6c5-49527b47b86d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:19:03.357252 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.357208    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/440327bd-970b-433f-a6c5-49527b47b86d-dshm\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:19:03.357252 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.357251    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/440327bd-970b-433f-a6c5-49527b47b86d-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:19:03.357431 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.357261    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/440327bd-970b-433f-a6c5-49527b47b86d-tmp-dir\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:19:03.357431 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.357270    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q4kj\" (UniqueName: \"kubernetes.io/projected/440327bd-970b-433f-a6c5-49527b47b86d-kube-api-access-7q4kj\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:19:03.357431 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.357279    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/440327bd-970b-433f-a6c5-49527b47b86d-kserve-provision-location\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:19:03.357431 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.357287    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/440327bd-970b-433f-a6c5-49527b47b86d-home\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:19:03.533147 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.533047    2585 generic.go:356] "Generic (PLEG): container finished" podID="440327bd-970b-433f-a6c5-49527b47b86d" containerID="3df139ab389fb3e868c1a0cff5d108d7bea7d1bfe53127cf0c39e857485d91da" exitCode=0
Sep 18 17:19:03.533147 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.533090    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw" event={"ID":"440327bd-970b-433f-a6c5-49527b47b86d","Type":"ContainerDied","Data":"3df139ab389fb3e868c1a0cff5d108d7bea7d1bfe53127cf0c39e857485d91da"}
Sep 18 17:19:03.533147 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.533098    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"
Sep 18 17:19:03.533147 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.533137    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw" event={"ID":"440327bd-970b-433f-a6c5-49527b47b86d","Type":"ContainerDied","Data":"ac556a8402f8cafb03259dacc8132925e10af20ead8e7af68267b741c1005ad9"}
Sep 18 17:19:03.533859 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.533166    2585 scope.go:117] "RemoveContainer" containerID="3df139ab389fb3e868c1a0cff5d108d7bea7d1bfe53127cf0c39e857485d91da"
Sep 18 17:19:03.542066 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.542049    2585 scope.go:117] "RemoveContainer" containerID="4397c21c77704742eb5688ccbbcdbea2d79275dc244a8efb6d04c6e8eeef30af"
Sep 18 17:19:03.551835 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.551816    2585 scope.go:117] "RemoveContainer" containerID="3df139ab389fb3e868c1a0cff5d108d7bea7d1bfe53127cf0c39e857485d91da"
Sep 18 17:19:03.552196 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:19:03.552176    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3df139ab389fb3e868c1a0cff5d108d7bea7d1bfe53127cf0c39e857485d91da\": container with ID starting with 3df139ab389fb3e868c1a0cff5d108d7bea7d1bfe53127cf0c39e857485d91da not found: ID does not exist" containerID="3df139ab389fb3e868c1a0cff5d108d7bea7d1bfe53127cf0c39e857485d91da"
Sep 18 17:19:03.552258 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.552206    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3df139ab389fb3e868c1a0cff5d108d7bea7d1bfe53127cf0c39e857485d91da"} err="failed to get container status \"3df139ab389fb3e868c1a0cff5d108d7bea7d1bfe53127cf0c39e857485d91da\": rpc error: code = NotFound desc = could not find container \"3df139ab389fb3e868c1a0cff5d108d7bea7d1bfe53127cf0c39e857485d91da\": container with ID starting with 3df139ab389fb3e868c1a0cff5d108d7bea7d1bfe53127cf0c39e857485d91da not found: ID does not exist"
Sep 18 17:19:03.552258 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.552223    2585 scope.go:117] "RemoveContainer" containerID="4397c21c77704742eb5688ccbbcdbea2d79275dc244a8efb6d04c6e8eeef30af"
Sep 18 17:19:03.552496 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:19:03.552477    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4397c21c77704742eb5688ccbbcdbea2d79275dc244a8efb6d04c6e8eeef30af\": container with ID starting with 4397c21c77704742eb5688ccbbcdbea2d79275dc244a8efb6d04c6e8eeef30af not found: ID does not exist" containerID="4397c21c77704742eb5688ccbbcdbea2d79275dc244a8efb6d04c6e8eeef30af"
Sep 18 17:19:03.552553 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.552505    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4397c21c77704742eb5688ccbbcdbea2d79275dc244a8efb6d04c6e8eeef30af"} err="failed to get container status \"4397c21c77704742eb5688ccbbcdbea2d79275dc244a8efb6d04c6e8eeef30af\": rpc error: code = NotFound desc = could not find container \"4397c21c77704742eb5688ccbbcdbea2d79275dc244a8efb6d04c6e8eeef30af\": container with ID starting with 4397c21c77704742eb5688ccbbcdbea2d79275dc244a8efb6d04c6e8eeef30af not found: ID does not exist"
Sep 18 17:19:03.557713 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.557691    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"]
Sep 18 17:19:03.561889 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:03.561859    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-59c46bcd6b-89dzw"]
Sep 18 17:19:04.392316 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:04.392285    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="440327bd-970b-433f-a6c5-49527b47b86d" path="/var/lib/kubelet/pods/440327bd-970b-433f-a6c5-49527b47b86d/volumes"
Sep 18 17:19:05.664149 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:05.664104    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj" podUID="747aec53-b00c-4aba-b278-b49104058d80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 18 17:19:15.664209 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:15.664164    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj" podUID="747aec53-b00c-4aba-b278-b49104058d80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 18 17:19:25.665007 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:25.664955    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj" podUID="747aec53-b00c-4aba-b278-b49104058d80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 18 17:19:34.194152 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.194113    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"]
Sep 18 17:19:34.194559 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.194542    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="440327bd-970b-433f-a6c5-49527b47b86d" containerName="main"
Sep 18 17:19:34.194604 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.194563    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="440327bd-970b-433f-a6c5-49527b47b86d" containerName="main"
Sep 18 17:19:34.194604 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.194583    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="440327bd-970b-433f-a6c5-49527b47b86d" containerName="storage-initializer"
Sep 18 17:19:34.194604 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.194593    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="440327bd-970b-433f-a6c5-49527b47b86d" containerName="storage-initializer"
Sep 18 17:19:34.194698 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.194675    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="440327bd-970b-433f-a6c5-49527b47b86d" containerName="main"
Sep 18 17:19:34.196758 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.196743    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:34.199357 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.199318    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 17:19:34.199500 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.199417    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 18 17:19:34.199618 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.199597    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 18 17:19:34.199699 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.199596    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-t7p8k\""
Sep 18 17:19:34.199766 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.199743    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 18 17:19:34.200684 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.200667    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"lora-pvc-single-adapter-test-kserve-self-signed-certs\""
Sep 18 17:19:34.208602 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.208581    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"]
Sep 18 17:19:34.317349 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.317315    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2277f96e-d009-4897-a1a5-286644feb0a0-dshm\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:34.317570 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.317363    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2277f96e-d009-4897-a1a5-286644feb0a0-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:34.317570 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.317387    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2277f96e-d009-4897-a1a5-286644feb0a0-home\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:34.317570 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.317402    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2277f96e-d009-4897-a1a5-286644feb0a0-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:34.317570 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.317469    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2277f96e-d009-4897-a1a5-286644feb0a0-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:34.317570 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.317554    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2277f96e-d009-4897-a1a5-286644feb0a0-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:34.317940 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.317585    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2277f96e-d009-4897-a1a5-286644feb0a0-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:34.317940 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.317664    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54npp\" (UniqueName: \"kubernetes.io/projected/2277f96e-d009-4897-a1a5-286644feb0a0-kube-api-access-54npp\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:34.418274 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.418229    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2277f96e-d009-4897-a1a5-286644feb0a0-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:34.418274 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.418270    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2277f96e-d009-4897-a1a5-286644feb0a0-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:34.418556 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.418330    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54npp\" (UniqueName: \"kubernetes.io/projected/2277f96e-d009-4897-a1a5-286644feb0a0-kube-api-access-54npp\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:34.418556 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.418376    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2277f96e-d009-4897-a1a5-286644feb0a0-dshm\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:34.418556 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.418415    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2277f96e-d009-4897-a1a5-286644feb0a0-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:34.418773 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.418743    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2277f96e-d009-4897-a1a5-286644feb0a0-home\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:34.418840 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.418791    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2277f96e-d009-4897-a1a5-286644feb0a0-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:34.418934 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.418841    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2277f96e-d009-4897-a1a5-286644feb0a0-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:34.419011 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.418952    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2277f96e-d009-4897-a1a5-286644feb0a0-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:34.419011 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.418982    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2277f96e-d009-4897-a1a5-286644feb0a0-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:34.419139 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.419025    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2277f96e-d009-4897-a1a5-286644feb0a0-home\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:34.419378 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.419353    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2277f96e-d009-4897-a1a5-286644feb0a0-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:34.419670 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.419646    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2277f96e-d009-4897-a1a5-286644feb0a0-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:34.421032 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.421004    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2277f96e-d009-4897-a1a5-286644feb0a0-dshm\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:34.421309 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.421293    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2277f96e-d009-4897-a1a5-286644feb0a0-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:34.426620 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:34.426596    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54npp\" (UniqueName: \"kubernetes.io/projected/2277f96e-d009-4897-a1a5-286644feb0a0-kube-api-access-54npp\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:35.664934 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:35.664870    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj" podUID="747aec53-b00c-4aba-b278-b49104058d80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 18 17:19:39.058885 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:39.058837    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-8b8c8904-89ff-4e63-bbe0-4abeeecc0063\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0221ad248af1f2266\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:39.065409 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:39.065374    2585 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-8b8c8904-89ff-4e63-bbe0-4abeeecc0063\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0221ad248af1f2266\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:39.159682 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:39.159643    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-8b8c8904-89ff-4e63-bbe0-4abeeecc0063\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0221ad248af1f2266\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:39.159942 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:39.159769    2585 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-8b8c8904-89ff-4e63-bbe0-4abeeecc0063\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0221ad248af1f2266\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:39.163252 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:39.163220    2585 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-8b8c8904-89ff-4e63-bbe0-4abeeecc0063\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0221ad248af1f2266\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") DevicePath \"csi-7dc35f80f9db91f369f8e4dde4abacf691cde4331c6c0b75999f0d7f84b6fb9c\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:39.262032 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:39.262004    2585 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-8b8c8904-89ff-4e63-bbe0-4abeeecc0063\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0221ad248af1f2266\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/8e8e8e00a2139f55fc05574dd20b4fe91023a1019de5796549727172032e0e2b/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:39.281977 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:39.281946    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-8b8c8904-89ff-4e63-bbe0-4abeeecc0063\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0221ad248af1f2266\") pod \"lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:39.307238 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:39.307202    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:39.583786 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:39.583653    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"]
Sep 18 17:19:39.587585 ip-10-0-133-147 kubenswrapper[2585]: W0918 17:19:39.587558    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2277f96e_d009_4897_a1a5_286644feb0a0.slice/crio-a2642c285a9b4d3d49f3fe7baa03f7bf52e9a5a8409f1677493f10ac328e1553 WatchSource:0}: Error finding container a2642c285a9b4d3d49f3fe7baa03f7bf52e9a5a8409f1677493f10ac328e1553: Status 404 returned error can't find the container with id a2642c285a9b4d3d49f3fe7baa03f7bf52e9a5a8409f1677493f10ac328e1553
Sep 18 17:19:39.691261 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:39.691224    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks" event={"ID":"2277f96e-d009-4897-a1a5-286644feb0a0","Type":"ContainerStarted","Data":"88276b90b5bf69f974af0328bb97c07d61a68a3d31131b268790773fcf2f39b9"}
Sep 18 17:19:39.691411 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:39.691285    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks" event={"ID":"2277f96e-d009-4897-a1a5-286644feb0a0","Type":"ContainerStarted","Data":"a2642c285a9b4d3d49f3fe7baa03f7bf52e9a5a8409f1677493f10ac328e1553"}
Sep 18 17:19:42.708215 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:42.708183    2585 generic.go:356] "Generic (PLEG): container finished" podID="2277f96e-d009-4897-a1a5-286644feb0a0" containerID="88276b90b5bf69f974af0328bb97c07d61a68a3d31131b268790773fcf2f39b9" exitCode=0
Sep 18 17:19:42.708627 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:42.708279    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks" event={"ID":"2277f96e-d009-4897-a1a5-286644feb0a0","Type":"ContainerDied","Data":"88276b90b5bf69f974af0328bb97c07d61a68a3d31131b268790773fcf2f39b9"}
Sep 18 17:19:43.715351 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:43.715258    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks" event={"ID":"2277f96e-d009-4897-a1a5-286644feb0a0","Type":"ContainerStarted","Data":"b07f695167455189af8cde65a866ea0994513519e383ceb09bedf407af1a11d7"}
Sep 18 17:19:43.740277 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:43.740227    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks" podStartSLOduration=7.509775967 podStartE2EDuration="9.740207579s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.230431612s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:19:43.737820561 +0000 UTC m=+3823.520822586" watchObservedRunningTime="2026-09-18 17:19:43.740207579 +0000 UTC m=+3823.523209613"
Sep 18 17:19:44.308368 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:44.308330    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:45.664461 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:45.664399    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj" podUID="747aec53-b00c-4aba-b278-b49104058d80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 18 17:19:49.308151 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:49.308114    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:19:49.309531 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:49.309498    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks" podUID="2277f96e-d009-4897-a1a5-286644feb0a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 18 17:19:55.664155 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:55.664101    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj" podUID="747aec53-b00c-4aba-b278-b49104058d80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8001/health\": dial tcp 10.133.0.62:8001: connect: connection refused"
Sep 18 17:19:59.308483 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:19:59.308436    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks" podUID="2277f96e-d009-4897-a1a5-286644feb0a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 18 17:20:05.674439 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:05.674404    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:20:05.686990 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:05.686964    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:20:09.307808 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:09.307753    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks" podUID="2277f96e-d009-4897-a1a5-286644feb0a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 18 17:20:19.307736 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:19.307640    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks" podUID="2277f96e-d009-4897-a1a5-286644feb0a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 18 17:20:25.130270 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:25.130236    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"]
Sep 18 17:20:29.307904 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:29.307863    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks" podUID="2277f96e-d009-4897-a1a5-286644feb0a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 18 17:20:39.307796 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:39.307752    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks" podUID="2277f96e-d009-4897-a1a5-286644feb0a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 18 17:20:40.153818 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:40.153754    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj" podUID="747aec53-b00c-4aba-b278-b49104058d80" containerName="main" containerID="cri-o://f78b7a023f616e9ad0225779120d94101e5bc2bb06d59756500ed67927200781" gracePeriod=45
Sep 18 17:20:40.770950 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:40.770894    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj" podUID="747aec53-b00c-4aba-b278-b49104058d80" containerName="llm-d-routing-sidecar" containerID="cri-o://c71008eae2ca43aa4d136f145776809a37ffb1e8b72c524f90f9a71883609ed4" gracePeriod=45
Sep 18 17:20:40.974577 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:40.974542    2585 generic.go:356] "Generic (PLEG): container finished" podID="747aec53-b00c-4aba-b278-b49104058d80" containerID="f78b7a023f616e9ad0225779120d94101e5bc2bb06d59756500ed67927200781" exitCode=0
Sep 18 17:20:40.974577 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:40.974573    2585 generic.go:356] "Generic (PLEG): container finished" podID="747aec53-b00c-4aba-b278-b49104058d80" containerID="c71008eae2ca43aa4d136f145776809a37ffb1e8b72c524f90f9a71883609ed4" exitCode=0
Sep 18 17:20:40.974834 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:40.974594    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj" event={"ID":"747aec53-b00c-4aba-b278-b49104058d80","Type":"ContainerDied","Data":"f78b7a023f616e9ad0225779120d94101e5bc2bb06d59756500ed67927200781"}
Sep 18 17:20:40.974834 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:40.974660    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj" event={"ID":"747aec53-b00c-4aba-b278-b49104058d80","Type":"ContainerDied","Data":"c71008eae2ca43aa4d136f145776809a37ffb1e8b72c524f90f9a71883609ed4"}
Sep 18 17:20:41.017923 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:41.017887    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:20:41.097051 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:41.097019    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/747aec53-b00c-4aba-b278-b49104058d80-tls-certs\") pod \"747aec53-b00c-4aba-b278-b49104058d80\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") "
Sep 18 17:20:41.097222 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:41.097060    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/747aec53-b00c-4aba-b278-b49104058d80-kserve-provision-location\") pod \"747aec53-b00c-4aba-b278-b49104058d80\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") "
Sep 18 17:20:41.097222 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:41.097079    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/747aec53-b00c-4aba-b278-b49104058d80-home\") pod \"747aec53-b00c-4aba-b278-b49104058d80\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") "
Sep 18 17:20:41.097222 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:41.097097    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/747aec53-b00c-4aba-b278-b49104058d80-cabundle-cert\") pod \"747aec53-b00c-4aba-b278-b49104058d80\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") "
Sep 18 17:20:41.097222 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:41.097122    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jn8fx\" (UniqueName: \"kubernetes.io/projected/747aec53-b00c-4aba-b278-b49104058d80-kube-api-access-jn8fx\") pod \"747aec53-b00c-4aba-b278-b49104058d80\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") "
Sep 18 17:20:41.097222 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:41.097149    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/747aec53-b00c-4aba-b278-b49104058d80-tmp-dir\") pod \"747aec53-b00c-4aba-b278-b49104058d80\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") "
Sep 18 17:20:41.097222 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:41.097187    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/747aec53-b00c-4aba-b278-b49104058d80-model-cache\") pod \"747aec53-b00c-4aba-b278-b49104058d80\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") "
Sep 18 17:20:41.097222 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:41.097214    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/747aec53-b00c-4aba-b278-b49104058d80-dshm\") pod \"747aec53-b00c-4aba-b278-b49104058d80\" (UID: \"747aec53-b00c-4aba-b278-b49104058d80\") "
Sep 18 17:20:41.097848 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:41.097794    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/747aec53-b00c-4aba-b278-b49104058d80-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "747aec53-b00c-4aba-b278-b49104058d80" (UID: "747aec53-b00c-4aba-b278-b49104058d80"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:20:41.098020 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:41.097840    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747aec53-b00c-4aba-b278-b49104058d80-model-cache" (OuterVolumeSpecName: "model-cache") pod "747aec53-b00c-4aba-b278-b49104058d80" (UID: "747aec53-b00c-4aba-b278-b49104058d80"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:20:41.098348 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:41.098314    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747aec53-b00c-4aba-b278-b49104058d80-home" (OuterVolumeSpecName: "home") pod "747aec53-b00c-4aba-b278-b49104058d80" (UID: "747aec53-b00c-4aba-b278-b49104058d80"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:20:41.100218 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:41.100189    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/747aec53-b00c-4aba-b278-b49104058d80-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "747aec53-b00c-4aba-b278-b49104058d80" (UID: "747aec53-b00c-4aba-b278-b49104058d80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:20:41.100322 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:41.100301    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747aec53-b00c-4aba-b278-b49104058d80-kube-api-access-jn8fx" (OuterVolumeSpecName: "kube-api-access-jn8fx") pod "747aec53-b00c-4aba-b278-b49104058d80" (UID: "747aec53-b00c-4aba-b278-b49104058d80"). InnerVolumeSpecName "kube-api-access-jn8fx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:20:41.100385 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:41.100362    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747aec53-b00c-4aba-b278-b49104058d80-dshm" (OuterVolumeSpecName: "dshm") pod "747aec53-b00c-4aba-b278-b49104058d80" (UID: "747aec53-b00c-4aba-b278-b49104058d80"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:20:41.109513 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:41.109487    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747aec53-b00c-4aba-b278-b49104058d80-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "747aec53-b00c-4aba-b278-b49104058d80" (UID: "747aec53-b00c-4aba-b278-b49104058d80"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:20:41.122466 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:41.122441    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747aec53-b00c-4aba-b278-b49104058d80-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "747aec53-b00c-4aba-b278-b49104058d80" (UID: "747aec53-b00c-4aba-b278-b49104058d80"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:20:41.198566 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:41.198485    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/747aec53-b00c-4aba-b278-b49104058d80-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:20:41.198566 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:41.198519    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/747aec53-b00c-4aba-b278-b49104058d80-kserve-provision-location\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:20:41.198566 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:41.198531    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/747aec53-b00c-4aba-b278-b49104058d80-home\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:20:41.198566 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:41.198540    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/747aec53-b00c-4aba-b278-b49104058d80-cabundle-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:20:41.198566 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:41.198549    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn8fx\" (UniqueName: \"kubernetes.io/projected/747aec53-b00c-4aba-b278-b49104058d80-kube-api-access-jn8fx\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:20:41.198566 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:41.198557    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/747aec53-b00c-4aba-b278-b49104058d80-tmp-dir\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:20:41.198566 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:41.198565    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/747aec53-b00c-4aba-b278-b49104058d80-model-cache\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:20:41.198566 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:41.198572    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/747aec53-b00c-4aba-b278-b49104058d80-dshm\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:20:41.980304 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:41.980266    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj" event={"ID":"747aec53-b00c-4aba-b278-b49104058d80","Type":"ContainerDied","Data":"b0925ef52952cc2cff7a5ca1a6421f2de58a4d867ac3c77080ac868ceb3f5353"}
Sep 18 17:20:41.980708 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:41.980312    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"
Sep 18 17:20:41.980708 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:41.980336    2585 scope.go:117] "RemoveContainer" containerID="f78b7a023f616e9ad0225779120d94101e5bc2bb06d59756500ed67927200781"
Sep 18 17:20:41.989446 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:41.989429    2585 scope.go:117] "RemoveContainer" containerID="10428e9782c45c1e00dce3972d6ff268a360126655a9c6af9f01ef365ad1935e"
Sep 18 17:20:41.998896 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:41.998873    2585 scope.go:117] "RemoveContainer" containerID="c71008eae2ca43aa4d136f145776809a37ffb1e8b72c524f90f9a71883609ed4"
Sep 18 17:20:42.004392 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:42.004366    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"]
Sep 18 17:20:42.010830 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:42.010807    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-rwppj"]
Sep 18 17:20:42.392425 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:42.392346    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="747aec53-b00c-4aba-b278-b49104058d80" path="/var/lib/kubelet/pods/747aec53-b00c-4aba-b278-b49104058d80/volumes"
Sep 18 17:20:49.308149 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:49.308103    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks" podUID="2277f96e-d009-4897-a1a5-286644feb0a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 18 17:20:59.318506 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:59.318471    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:20:59.326498 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:20:59.326474    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:21:02.625901 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:02.625860    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 17:21:02.629542 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:02.629519    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 17:21:02.633209 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:02.633188    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 17:21:02.637335 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:02.637318    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 17:21:23.850420 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:23.850388    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks_2277f96e-d009-4897-a1a5-286644feb0a0/storage-initializer/0.log"
Sep 18 17:21:23.859267 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:23.859245    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks_2277f96e-d009-4897-a1a5-286644feb0a0/main/0.log"
Sep 18 17:21:30.733053 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.733017    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"]
Sep 18 17:21:30.733473 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.733403    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="747aec53-b00c-4aba-b278-b49104058d80" containerName="main"
Sep 18 17:21:30.733473 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.733420    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="747aec53-b00c-4aba-b278-b49104058d80" containerName="main"
Sep 18 17:21:30.733473 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.733439    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="747aec53-b00c-4aba-b278-b49104058d80" containerName="storage-initializer"
Sep 18 17:21:30.733473 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.733448    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="747aec53-b00c-4aba-b278-b49104058d80" containerName="storage-initializer"
Sep 18 17:21:30.733661 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.733476    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="747aec53-b00c-4aba-b278-b49104058d80" containerName="llm-d-routing-sidecar"
Sep 18 17:21:30.733661 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.733485    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="747aec53-b00c-4aba-b278-b49104058d80" containerName="llm-d-routing-sidecar"
Sep 18 17:21:30.733661 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.733563    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="747aec53-b00c-4aba-b278-b49104058d80" containerName="main"
Sep 18 17:21:30.733661 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.733573    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="747aec53-b00c-4aba-b278-b49104058d80" containerName="llm-d-routing-sidecar"
Sep 18 17:21:30.739714 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.739687    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:21:30.743358 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.743331    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-v5kh6\""
Sep 18 17:21:30.743522 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.743502    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 18 17:21:30.743959 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.743940    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 18 17:21:30.744065 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.744024    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 17:21:30.744313 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.744286    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 18 17:21:30.744754 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.744736    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 18 17:21:30.744855 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.744765    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-kvtm9\""
Sep 18 17:21:30.755558 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.755533    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"]
Sep 18 17:21:30.823515 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.823481    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a7189d4-e59c-492b-b892-9d5909759bbf-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-6757b868d4-l8wmf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:21:30.823684 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.823584    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a7189d4-e59c-492b-b892-9d5909759bbf-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6757b868d4-l8wmf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:21:30.823684 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.823611    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a7189d4-e59c-492b-b892-9d5909759bbf-tls-certs\") pod \"router-with-refs-pd-test-kserve-6757b868d4-l8wmf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:21:30.823684 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.823631    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9r25\" (UniqueName: \"kubernetes.io/projected/4a7189d4-e59c-492b-b892-9d5909759bbf-kube-api-access-t9r25\") pod \"router-with-refs-pd-test-kserve-6757b868d4-l8wmf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:21:30.823830 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.823692    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a7189d4-e59c-492b-b892-9d5909759bbf-home\") pod \"router-with-refs-pd-test-kserve-6757b868d4-l8wmf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:21:30.823830 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.823728    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a7189d4-e59c-492b-b892-9d5909759bbf-dshm\") pod \"router-with-refs-pd-test-kserve-6757b868d4-l8wmf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:21:30.823830 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.823749    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a7189d4-e59c-492b-b892-9d5909759bbf-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6757b868d4-l8wmf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:21:30.823830 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.823778    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a7189d4-e59c-492b-b892-9d5909759bbf-model-cache\") pod \"router-with-refs-pd-test-kserve-6757b868d4-l8wmf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:21:30.925097 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.925066    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a7189d4-e59c-492b-b892-9d5909759bbf-dshm\") pod \"router-with-refs-pd-test-kserve-6757b868d4-l8wmf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:21:30.925282 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.925102    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a7189d4-e59c-492b-b892-9d5909759bbf-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6757b868d4-l8wmf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:21:30.925282 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.925123    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a7189d4-e59c-492b-b892-9d5909759bbf-model-cache\") pod \"router-with-refs-pd-test-kserve-6757b868d4-l8wmf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:21:30.925282 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.925150    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a7189d4-e59c-492b-b892-9d5909759bbf-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-6757b868d4-l8wmf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:21:30.925282 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.925185    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a7189d4-e59c-492b-b892-9d5909759bbf-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6757b868d4-l8wmf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:21:30.925282 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.925200    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a7189d4-e59c-492b-b892-9d5909759bbf-tls-certs\") pod \"router-with-refs-pd-test-kserve-6757b868d4-l8wmf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:21:30.925282 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.925215    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9r25\" (UniqueName: \"kubernetes.io/projected/4a7189d4-e59c-492b-b892-9d5909759bbf-kube-api-access-t9r25\") pod \"router-with-refs-pd-test-kserve-6757b868d4-l8wmf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:21:30.925282 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.925244    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a7189d4-e59c-492b-b892-9d5909759bbf-home\") pod \"router-with-refs-pd-test-kserve-6757b868d4-l8wmf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:21:30.925633 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.925609    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a7189d4-e59c-492b-b892-9d5909759bbf-model-cache\") pod \"router-with-refs-pd-test-kserve-6757b868d4-l8wmf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:21:30.925695 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.925668    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a7189d4-e59c-492b-b892-9d5909759bbf-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6757b868d4-l8wmf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:21:30.925786 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.925764    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a7189d4-e59c-492b-b892-9d5909759bbf-home\") pod \"router-with-refs-pd-test-kserve-6757b868d4-l8wmf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:21:30.925845 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.925802    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a7189d4-e59c-492b-b892-9d5909759bbf-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6757b868d4-l8wmf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:21:30.926025 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.926002    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a7189d4-e59c-492b-b892-9d5909759bbf-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-6757b868d4-l8wmf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:21:30.927578 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.927549    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a7189d4-e59c-492b-b892-9d5909759bbf-dshm\") pod \"router-with-refs-pd-test-kserve-6757b868d4-l8wmf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:21:30.927844 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.927825    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a7189d4-e59c-492b-b892-9d5909759bbf-tls-certs\") pod \"router-with-refs-pd-test-kserve-6757b868d4-l8wmf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:21:30.937313 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:30.937294    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9r25\" (UniqueName: \"kubernetes.io/projected/4a7189d4-e59c-492b-b892-9d5909759bbf-kube-api-access-t9r25\") pod \"router-with-refs-pd-test-kserve-6757b868d4-l8wmf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:21:31.050293 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:31.050210    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:21:31.330100 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:31.330013    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"]
Sep 18 17:21:31.334006 ip-10-0-133-147 kubenswrapper[2585]: W0918 17:21:31.333964    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a7189d4_e59c_492b_b892_9d5909759bbf.slice/crio-2a2bfe849a450c45f56b3f16952d6c0d2de70a2ae4879e625d2b863c789e5e36 WatchSource:0}: Error finding container 2a2bfe849a450c45f56b3f16952d6c0d2de70a2ae4879e625d2b863c789e5e36: Status 404 returned error can't find the container with id 2a2bfe849a450c45f56b3f16952d6c0d2de70a2ae4879e625d2b863c789e5e36
Sep 18 17:21:32.192999 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:32.192958    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf" event={"ID":"4a7189d4-e59c-492b-b892-9d5909759bbf","Type":"ContainerStarted","Data":"50511ab85709e56f6de93408d06e374273f3b781d98dab35b226ffa44b7932aa"}
Sep 18 17:21:32.193416 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:32.193055    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:21:32.193416 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:32.193073    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf" event={"ID":"4a7189d4-e59c-492b-b892-9d5909759bbf","Type":"ContainerStarted","Data":"2a2bfe849a450c45f56b3f16952d6c0d2de70a2ae4879e625d2b863c789e5e36"}
Sep 18 17:21:33.198707 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:33.198671    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf" event={"ID":"4a7189d4-e59c-492b-b892-9d5909759bbf","Type":"ContainerStarted","Data":"9800942718d345424ccaa88fbe145cc00bc5ec9b3d47bd87a3bc3a7cfe755aed"}
Sep 18 17:21:35.209080 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:35.209041    2585 generic.go:356] "Generic (PLEG): container finished" podID="4a7189d4-e59c-492b-b892-9d5909759bbf" containerID="9800942718d345424ccaa88fbe145cc00bc5ec9b3d47bd87a3bc3a7cfe755aed" exitCode=0
Sep 18 17:21:35.209486 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:35.209110    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf" event={"ID":"4a7189d4-e59c-492b-b892-9d5909759bbf","Type":"ContainerDied","Data":"9800942718d345424ccaa88fbe145cc00bc5ec9b3d47bd87a3bc3a7cfe755aed"}
Sep 18 17:21:36.216030 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:36.215985    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf" event={"ID":"4a7189d4-e59c-492b-b892-9d5909759bbf","Type":"ContainerStarted","Data":"d0ef09419d0aac89f0d62525eebd33cf4c061919f0320cb899dcc21a559e690a"}
Sep 18 17:21:36.240201 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:36.240149    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf" podStartSLOduration=4.09450565 podStartE2EDuration="6.24013388s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.14562823s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:21:36.237023045 +0000 UTC m=+3936.020025071" watchObservedRunningTime="2026-09-18 17:21:36.24013388 +0000 UTC m=+3936.023135904"
Sep 18 17:21:39.511829 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:39.511785    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"]
Sep 18 17:21:41.050713 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:41.050671    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:21:41.051234 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:41.050755    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:21:41.052111 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:41.052069    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf" podUID="4a7189d4-e59c-492b-b892-9d5909759bbf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8001/health\": dial tcp 10.133.0.64:8001: connect: connection refused"
Sep 18 17:21:41.064592 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:41.064569    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:21:51.051529 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:51.051413    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf" podUID="4a7189d4-e59c-492b-b892-9d5909759bbf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8001/health\": dial tcp 10.133.0.64:8001: connect: connection refused"
Sep 18 17:21:54.535591 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:54.535550    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks" podUID="2277f96e-d009-4897-a1a5-286644feb0a0" containerName="main" containerID="cri-o://b07f695167455189af8cde65a866ea0994513519e383ceb09bedf407af1a11d7" gracePeriod=45
Sep 18 17:21:55.198709 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.198679    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:21:55.296703 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.296617    2585 generic.go:356] "Generic (PLEG): container finished" podID="2277f96e-d009-4897-a1a5-286644feb0a0" containerID="b07f695167455189af8cde65a866ea0994513519e383ceb09bedf407af1a11d7" exitCode=0
Sep 18 17:21:55.296703 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.296664    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"
Sep 18 17:21:55.296883 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.296697    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks" event={"ID":"2277f96e-d009-4897-a1a5-286644feb0a0","Type":"ContainerDied","Data":"b07f695167455189af8cde65a866ea0994513519e383ceb09bedf407af1a11d7"}
Sep 18 17:21:55.296883 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.296775    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks" event={"ID":"2277f96e-d009-4897-a1a5-286644feb0a0","Type":"ContainerDied","Data":"a2642c285a9b4d3d49f3fe7baa03f7bf52e9a5a8409f1677493f10ac328e1553"}
Sep 18 17:21:55.296883 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.296798    2585 scope.go:117] "RemoveContainer" containerID="b07f695167455189af8cde65a866ea0994513519e383ceb09bedf407af1a11d7"
Sep 18 17:21:55.305130 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.305115    2585 scope.go:117] "RemoveContainer" containerID="88276b90b5bf69f974af0328bb97c07d61a68a3d31131b268790773fcf2f39b9"
Sep 18 17:21:55.313852 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.313822    2585 scope.go:117] "RemoveContainer" containerID="b07f695167455189af8cde65a866ea0994513519e383ceb09bedf407af1a11d7"
Sep 18 17:21:55.314175 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:21:55.314155    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b07f695167455189af8cde65a866ea0994513519e383ceb09bedf407af1a11d7\": container with ID starting with b07f695167455189af8cde65a866ea0994513519e383ceb09bedf407af1a11d7 not found: ID does not exist" containerID="b07f695167455189af8cde65a866ea0994513519e383ceb09bedf407af1a11d7"
Sep 18 17:21:55.314268 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.314186    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b07f695167455189af8cde65a866ea0994513519e383ceb09bedf407af1a11d7"} err="failed to get container status \"b07f695167455189af8cde65a866ea0994513519e383ceb09bedf407af1a11d7\": rpc error: code = NotFound desc = could not find container \"b07f695167455189af8cde65a866ea0994513519e383ceb09bedf407af1a11d7\": container with ID starting with b07f695167455189af8cde65a866ea0994513519e383ceb09bedf407af1a11d7 not found: ID does not exist"
Sep 18 17:21:55.314268 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.314211    2585 scope.go:117] "RemoveContainer" containerID="88276b90b5bf69f974af0328bb97c07d61a68a3d31131b268790773fcf2f39b9"
Sep 18 17:21:55.314542 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:21:55.314524    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88276b90b5bf69f974af0328bb97c07d61a68a3d31131b268790773fcf2f39b9\": container with ID starting with 88276b90b5bf69f974af0328bb97c07d61a68a3d31131b268790773fcf2f39b9 not found: ID does not exist" containerID="88276b90b5bf69f974af0328bb97c07d61a68a3d31131b268790773fcf2f39b9"
Sep 18 17:21:55.314585 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.314549    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88276b90b5bf69f974af0328bb97c07d61a68a3d31131b268790773fcf2f39b9"} err="failed to get container status \"88276b90b5bf69f974af0328bb97c07d61a68a3d31131b268790773fcf2f39b9\": rpc error: code = NotFound desc = could not find container \"88276b90b5bf69f974af0328bb97c07d61a68a3d31131b268790773fcf2f39b9\": container with ID starting with 88276b90b5bf69f974af0328bb97c07d61a68a3d31131b268790773fcf2f39b9 not found: ID does not exist"
Sep 18 17:21:55.337451 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.337425    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2277f96e-d009-4897-a1a5-286644feb0a0-dshm\") pod \"2277f96e-d009-4897-a1a5-286644feb0a0\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") "
Sep 18 17:21:55.337549 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.337454    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2277f96e-d009-4897-a1a5-286644feb0a0-cabundle-cert\") pod \"2277f96e-d009-4897-a1a5-286644feb0a0\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") "
Sep 18 17:21:55.337549 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.337480    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54npp\" (UniqueName: \"kubernetes.io/projected/2277f96e-d009-4897-a1a5-286644feb0a0-kube-api-access-54npp\") pod \"2277f96e-d009-4897-a1a5-286644feb0a0\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") "
Sep 18 17:21:55.337549 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.337504    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2277f96e-d009-4897-a1a5-286644feb0a0-home\") pod \"2277f96e-d009-4897-a1a5-286644feb0a0\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") "
Sep 18 17:21:55.337549 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.337526    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2277f96e-d009-4897-a1a5-286644feb0a0-model-cache\") pod \"2277f96e-d009-4897-a1a5-286644feb0a0\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") "
Sep 18 17:21:55.337762 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.337584    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2277f96e-d009-4897-a1a5-286644feb0a0-kserve-provision-location\") pod \"2277f96e-d009-4897-a1a5-286644feb0a0\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") "
Sep 18 17:21:55.337762 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.337607    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2277f96e-d009-4897-a1a5-286644feb0a0-tls-certs\") pod \"2277f96e-d009-4897-a1a5-286644feb0a0\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") "
Sep 18 17:21:55.337762 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.337634    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2277f96e-d009-4897-a1a5-286644feb0a0-tmp-dir\") pod \"2277f96e-d009-4897-a1a5-286644feb0a0\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") "
Sep 18 17:21:55.337762 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.337747    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"lora-pvc-lora-adapter-1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0221ad248af1f2266\") pod \"2277f96e-d009-4897-a1a5-286644feb0a0\" (UID: \"2277f96e-d009-4897-a1a5-286644feb0a0\") "
Sep 18 17:21:55.338010 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.337839    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2277f96e-d009-4897-a1a5-286644feb0a0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2277f96e-d009-4897-a1a5-286644feb0a0" (UID: "2277f96e-d009-4897-a1a5-286644feb0a0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:21:55.338010 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.337899    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2277f96e-d009-4897-a1a5-286644feb0a0-home" (OuterVolumeSpecName: "home") pod "2277f96e-d009-4897-a1a5-286644feb0a0" (UID: "2277f96e-d009-4897-a1a5-286644feb0a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:21:55.338161 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.338136    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2277f96e-d009-4897-a1a5-286644feb0a0-cabundle-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:21:55.338395 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.338170    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2277f96e-d009-4897-a1a5-286644feb0a0-home\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:21:55.338689 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.338664    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2277f96e-d009-4897-a1a5-286644feb0a0-model-cache" (OuterVolumeSpecName: "model-cache") pod "2277f96e-d009-4897-a1a5-286644feb0a0" (UID: "2277f96e-d009-4897-a1a5-286644feb0a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:21:55.340298 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.340226    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2277f96e-d009-4897-a1a5-286644feb0a0-dshm" (OuterVolumeSpecName: "dshm") pod "2277f96e-d009-4897-a1a5-286644feb0a0" (UID: "2277f96e-d009-4897-a1a5-286644feb0a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:21:55.340550 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.340506    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2277f96e-d009-4897-a1a5-286644feb0a0-kube-api-access-54npp" (OuterVolumeSpecName: "kube-api-access-54npp") pod "2277f96e-d009-4897-a1a5-286644feb0a0" (UID: "2277f96e-d009-4897-a1a5-286644feb0a0"). InnerVolumeSpecName "kube-api-access-54npp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:21:55.340792 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.340763    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2277f96e-d009-4897-a1a5-286644feb0a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2277f96e-d009-4897-a1a5-286644feb0a0" (UID: "2277f96e-d009-4897-a1a5-286644feb0a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:21:55.344743 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.344718    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0221ad248af1f2266" (OuterVolumeSpecName: "lora-pvc-lora-adapter-1") pod "2277f96e-d009-4897-a1a5-286644feb0a0" (UID: "2277f96e-d009-4897-a1a5-286644feb0a0"). InnerVolumeSpecName "pvc-8b8c8904-89ff-4e63-bbe0-4abeeecc0063". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 18 17:21:55.352838 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.352805    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2277f96e-d009-4897-a1a5-286644feb0a0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2277f96e-d009-4897-a1a5-286644feb0a0" (UID: "2277f96e-d009-4897-a1a5-286644feb0a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:21:55.361639 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.361604    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2277f96e-d009-4897-a1a5-286644feb0a0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2277f96e-d009-4897-a1a5-286644feb0a0" (UID: "2277f96e-d009-4897-a1a5-286644feb0a0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:21:55.439295 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.439261    2585 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-8b8c8904-89ff-4e63-bbe0-4abeeecc0063\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0221ad248af1f2266\") on node \"ip-10-0-133-147.ec2.internal\" "
Sep 18 17:21:55.439295 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.439289    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2277f96e-d009-4897-a1a5-286644feb0a0-dshm\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:21:55.439295 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.439301    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54npp\" (UniqueName: \"kubernetes.io/projected/2277f96e-d009-4897-a1a5-286644feb0a0-kube-api-access-54npp\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:21:55.439499 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.439311    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2277f96e-d009-4897-a1a5-286644feb0a0-model-cache\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:21:55.439499 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.439319    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2277f96e-d009-4897-a1a5-286644feb0a0-kserve-provision-location\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:21:55.439499 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.439328    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2277f96e-d009-4897-a1a5-286644feb0a0-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:21:55.439499 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.439335    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2277f96e-d009-4897-a1a5-286644feb0a0-tmp-dir\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:21:55.450322 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.450299    2585 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-8b8c8904-89ff-4e63-bbe0-4abeeecc0063" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0221ad248af1f2266") on node "ip-10-0-133-147.ec2.internal"
Sep 18 17:21:55.539735 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.539708    2585 reconciler_common.go:299] "Volume detached for volume \"pvc-8b8c8904-89ff-4e63-bbe0-4abeeecc0063\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0221ad248af1f2266\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"csi-7dc35f80f9db91f369f8e4dde4abacf691cde4331c6c0b75999f0d7f84b6fb9c\""
Sep 18 17:21:55.622032 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.622001    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"]
Sep 18 17:21:55.625189 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:55.625164    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-fff6ff677-4mdks"]
Sep 18 17:21:56.392396 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:21:56.392316    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2277f96e-d009-4897-a1a5-286644feb0a0" path="/var/lib/kubelet/pods/2277f96e-d009-4897-a1a5-286644feb0a0/volumes"
Sep 18 17:22:01.050788 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:01.050739    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf" podUID="4a7189d4-e59c-492b-b892-9d5909759bbf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8001/health\": dial tcp 10.133.0.64:8001: connect: connection refused"
Sep 18 17:22:03.240780 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:03.240743    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-mqllb"]
Sep 18 17:22:03.241205 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:03.241148    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2277f96e-d009-4897-a1a5-286644feb0a0" containerName="main"
Sep 18 17:22:03.241205 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:03.241165    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2277f96e-d009-4897-a1a5-286644feb0a0" containerName="main"
Sep 18 17:22:03.241205 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:03.241174    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2277f96e-d009-4897-a1a5-286644feb0a0" containerName="storage-initializer"
Sep 18 17:22:03.241205 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:03.241180    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2277f96e-d009-4897-a1a5-286644feb0a0" containerName="storage-initializer"
Sep 18 17:22:03.241348 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:03.241228    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2277f96e-d009-4897-a1a5-286644feb0a0" containerName="main"
Sep 18 17:22:03.245334 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:03.245315    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-mqllb"
Sep 18 17:22:03.248103 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:03.248077    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 18 17:22:03.248232 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:03.248124    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 18 17:22:03.249401 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:03.249214    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"seaweedfs-s3-creds\""
Sep 18 17:22:03.249401 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:03.249247    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-m4skn\""
Sep 18 17:22:03.251578 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:03.251555    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-mqllb"]
Sep 18 17:22:03.410639 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:03.410604    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj9p6\" (UniqueName: \"kubernetes.io/projected/3548dc85-90e9-4af1-be42-aae4e2373710-kube-api-access-hj9p6\") pod \"e2e-pvc-lora-download-mqllb\" (UID: \"3548dc85-90e9-4af1-be42-aae4e2373710\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-mqllb"
Sep 18 17:22:03.511691 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:03.511601    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj9p6\" (UniqueName: \"kubernetes.io/projected/3548dc85-90e9-4af1-be42-aae4e2373710-kube-api-access-hj9p6\") pod \"e2e-pvc-lora-download-mqllb\" (UID: \"3548dc85-90e9-4af1-be42-aae4e2373710\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-mqllb"
Sep 18 17:22:03.521234 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:03.521205    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj9p6\" (UniqueName: \"kubernetes.io/projected/3548dc85-90e9-4af1-be42-aae4e2373710-kube-api-access-hj9p6\") pod \"e2e-pvc-lora-download-mqllb\" (UID: \"3548dc85-90e9-4af1-be42-aae4e2373710\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-mqllb"
Sep 18 17:22:11.051429 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:11.051380    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf" podUID="4a7189d4-e59c-492b-b892-9d5909759bbf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8001/health\": dial tcp 10.133.0.64:8001: connect: connection refused"
Sep 18 17:22:11.274416 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:11.274373    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-698b13c9-8bf6-4a7e-af55-cf2a5ff668f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-079c14f72df9eec65\") pod \"e2e-pvc-lora-download-mqllb\" (UID: \"3548dc85-90e9-4af1-be42-aae4e2373710\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-mqllb"
Sep 18 17:22:11.279968 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:11.279938    2585 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-698b13c9-8bf6-4a7e-af55-cf2a5ff668f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-079c14f72df9eec65\") pod \"e2e-pvc-lora-download-mqllb\" (UID: \"3548dc85-90e9-4af1-be42-aae4e2373710\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-mqllb"
Sep 18 17:22:11.375245 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:11.375152    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-698b13c9-8bf6-4a7e-af55-cf2a5ff668f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-079c14f72df9eec65\") pod \"e2e-pvc-lora-download-mqllb\" (UID: \"3548dc85-90e9-4af1-be42-aae4e2373710\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-mqllb"
Sep 18 17:22:11.375399 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:11.375297    2585 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-698b13c9-8bf6-4a7e-af55-cf2a5ff668f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-079c14f72df9eec65\") pod \"e2e-pvc-lora-download-mqllb\" (UID: \"3548dc85-90e9-4af1-be42-aae4e2373710\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-mqllb"
Sep 18 17:22:11.378420 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:11.378401    2585 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-698b13c9-8bf6-4a7e-af55-cf2a5ff668f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-079c14f72df9eec65\") pod \"e2e-pvc-lora-download-mqllb\" (UID: \"3548dc85-90e9-4af1-be42-aae4e2373710\") DevicePath \"csi-b06beff56aedabdcecd272013a7f3d3b395414b9a7256efa727015218310ddc4\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-mqllb"
Sep 18 17:22:11.599609 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:11.599574    2585 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-698b13c9-8bf6-4a7e-af55-cf2a5ff668f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-079c14f72df9eec65\") pod \"e2e-pvc-lora-download-mqllb\" (UID: \"3548dc85-90e9-4af1-be42-aae4e2373710\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/9d615f203ca9bc2eb4423728807638f6b015c6defc17b8fe44837f745c8a01b6/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-mqllb"
Sep 18 17:22:11.613219 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:11.613192    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-698b13c9-8bf6-4a7e-af55-cf2a5ff668f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-079c14f72df9eec65\") pod \"e2e-pvc-lora-download-mqllb\" (UID: \"3548dc85-90e9-4af1-be42-aae4e2373710\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-mqllb"
Sep 18 17:22:11.655782 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:11.655693    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-mqllb"
Sep 18 17:22:11.936866 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:11.936838    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-mqllb"]
Sep 18 17:22:11.940210 ip-10-0-133-147 kubenswrapper[2585]: W0918 17:22:11.940181    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3548dc85_90e9_4af1_be42_aae4e2373710.slice/crio-a4d396497dd1fa12fbd6ff71feb1ada89c8eb27e42319b07c4687e067cfc0fb9 WatchSource:0}: Error finding container a4d396497dd1fa12fbd6ff71feb1ada89c8eb27e42319b07c4687e067cfc0fb9: Status 404 returned error can't find the container with id a4d396497dd1fa12fbd6ff71feb1ada89c8eb27e42319b07c4687e067cfc0fb9
Sep 18 17:22:12.392275 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:12.392233    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-mqllb" event={"ID":"3548dc85-90e9-4af1-be42-aae4e2373710","Type":"ContainerStarted","Data":"6065028c11367be37e816d5c310d9c50cd3b66c2d6f92edc19d037ddc89af7ca"}
Sep 18 17:22:12.392778 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:12.392287    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-mqllb" event={"ID":"3548dc85-90e9-4af1-be42-aae4e2373710","Type":"ContainerStarted","Data":"a4d396497dd1fa12fbd6ff71feb1ada89c8eb27e42319b07c4687e067cfc0fb9"}
Sep 18 17:22:14.397420 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:14.397378    2585 generic.go:356] "Generic (PLEG): container finished" podID="3548dc85-90e9-4af1-be42-aae4e2373710" containerID="6065028c11367be37e816d5c310d9c50cd3b66c2d6f92edc19d037ddc89af7ca" exitCode=0
Sep 18 17:22:14.397420 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:14.397419    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-mqllb" event={"ID":"3548dc85-90e9-4af1-be42-aae4e2373710","Type":"ContainerDied","Data":"6065028c11367be37e816d5c310d9c50cd3b66c2d6f92edc19d037ddc89af7ca"}
Sep 18 17:22:15.548240 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:15.548209    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-mqllb"
Sep 18 17:22:15.711785 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:15.711675    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hj9p6\" (UniqueName: \"kubernetes.io/projected/3548dc85-90e9-4af1-be42-aae4e2373710-kube-api-access-hj9p6\") pod \"3548dc85-90e9-4af1-be42-aae4e2373710\" (UID: \"3548dc85-90e9-4af1-be42-aae4e2373710\") "
Sep 18 17:22:15.711995 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:15.711823    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-079c14f72df9eec65\") pod \"3548dc85-90e9-4af1-be42-aae4e2373710\" (UID: \"3548dc85-90e9-4af1-be42-aae4e2373710\") "
Sep 18 17:22:15.714404 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:15.714362    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3548dc85-90e9-4af1-be42-aae4e2373710-kube-api-access-hj9p6" (OuterVolumeSpecName: "kube-api-access-hj9p6") pod "3548dc85-90e9-4af1-be42-aae4e2373710" (UID: "3548dc85-90e9-4af1-be42-aae4e2373710"). InnerVolumeSpecName "kube-api-access-hj9p6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:22:15.715261 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:15.715237    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-079c14f72df9eec65" (OuterVolumeSpecName: "model-storage") pod "3548dc85-90e9-4af1-be42-aae4e2373710" (UID: "3548dc85-90e9-4af1-be42-aae4e2373710"). InnerVolumeSpecName "pvc-698b13c9-8bf6-4a7e-af55-cf2a5ff668f0". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 18 17:22:15.813132 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:15.813092    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj9p6\" (UniqueName: \"kubernetes.io/projected/3548dc85-90e9-4af1-be42-aae4e2373710-kube-api-access-hj9p6\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:22:15.813304 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:15.813158    2585 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-698b13c9-8bf6-4a7e-af55-cf2a5ff668f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-079c14f72df9eec65\") on node \"ip-10-0-133-147.ec2.internal\" "
Sep 18 17:22:16.405899 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:16.405857    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-mqllb" event={"ID":"3548dc85-90e9-4af1-be42-aae4e2373710","Type":"ContainerDied","Data":"a4d396497dd1fa12fbd6ff71feb1ada89c8eb27e42319b07c4687e067cfc0fb9"}
Sep 18 17:22:16.406045 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:16.405924    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4d396497dd1fa12fbd6ff71feb1ada89c8eb27e42319b07c4687e067cfc0fb9"
Sep 18 17:22:16.406045 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:16.405873    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-mqllb"
Sep 18 17:22:16.796105 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:16.796083    2585 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-698b13c9-8bf6-4a7e-af55-cf2a5ff668f0" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-079c14f72df9eec65") on node "ip-10-0-133-147.ec2.internal"
Sep 18 17:22:16.822346 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:16.822316    2585 reconciler_common.go:299] "Volume detached for volume \"pvc-698b13c9-8bf6-4a7e-af55-cf2a5ff668f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-079c14f72df9eec65\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"csi-b06beff56aedabdcecd272013a7f3d3b395414b9a7256efa727015218310ddc4\""
Sep 18 17:22:21.051313 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:21.051265    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf" podUID="4a7189d4-e59c-492b-b892-9d5909759bbf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8001/health\": dial tcp 10.133.0.64:8001: connect: connection refused"
Sep 18 17:22:31.050743 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:31.050697    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf" podUID="4a7189d4-e59c-492b-b892-9d5909759bbf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8001/health\": dial tcp 10.133.0.64:8001: connect: connection refused"
Sep 18 17:22:41.051182 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:41.051118    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf" podUID="4a7189d4-e59c-492b-b892-9d5909759bbf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8001/health\": dial tcp 10.133.0.64:8001: connect: connection refused"
Sep 18 17:22:51.051338 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:22:51.051283    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf" podUID="4a7189d4-e59c-492b-b892-9d5909759bbf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8001/health\": dial tcp 10.133.0.64:8001: connect: connection refused"
Sep 18 17:23:01.051001 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:23:01.050938    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf" podUID="4a7189d4-e59c-492b-b892-9d5909759bbf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8001/health\": dial tcp 10.133.0.64:8001: connect: connection refused"
Sep 18 17:23:11.051290 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:23:11.051238    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf" podUID="4a7189d4-e59c-492b-b892-9d5909759bbf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8001/health\": dial tcp 10.133.0.64:8001: connect: connection refused"
Sep 18 17:23:21.051447 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:23:21.051332    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf" podUID="4a7189d4-e59c-492b-b892-9d5909759bbf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8001/health\": dial tcp 10.133.0.64:8001: connect: connection refused"
Sep 18 17:23:31.060955 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:23:31.060900    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:23:31.077085 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:23:31.077057    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:23:50.452483 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:23:50.452444    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"]
Sep 18 17:24:05.484602 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:05.484531    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf" podUID="4a7189d4-e59c-492b-b892-9d5909759bbf" containerName="main" containerID="cri-o://d0ef09419d0aac89f0d62525eebd33cf4c061919f0320cb899dcc21a559e690a" gracePeriod=45
Sep 18 17:24:06.095631 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.095598    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf" podUID="4a7189d4-e59c-492b-b892-9d5909759bbf" containerName="llm-d-routing-sidecar" containerID="cri-o://50511ab85709e56f6de93408d06e374273f3b781d98dab35b226ffa44b7932aa" gracePeriod=45
Sep 18 17:24:06.337555 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.337525    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:24:06.438826 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.438785    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a7189d4-e59c-492b-b892-9d5909759bbf-model-cache\") pod \"4a7189d4-e59c-492b-b892-9d5909759bbf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") "
Sep 18 17:24:06.439000 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.438838    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a7189d4-e59c-492b-b892-9d5909759bbf-home\") pod \"4a7189d4-e59c-492b-b892-9d5909759bbf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") "
Sep 18 17:24:06.439000 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.438892    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a7189d4-e59c-492b-b892-9d5909759bbf-tmp-dir\") pod \"4a7189d4-e59c-492b-b892-9d5909759bbf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") "
Sep 18 17:24:06.439000 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.438929    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a7189d4-e59c-492b-b892-9d5909759bbf-tls-certs\") pod \"4a7189d4-e59c-492b-b892-9d5909759bbf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") "
Sep 18 17:24:06.439000 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.438955    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a7189d4-e59c-492b-b892-9d5909759bbf-cabundle-cert\") pod \"4a7189d4-e59c-492b-b892-9d5909759bbf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") "
Sep 18 17:24:06.439000 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.438982    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a7189d4-e59c-492b-b892-9d5909759bbf-kserve-provision-location\") pod \"4a7189d4-e59c-492b-b892-9d5909759bbf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") "
Sep 18 17:24:06.439274 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.439014    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a7189d4-e59c-492b-b892-9d5909759bbf-dshm\") pod \"4a7189d4-e59c-492b-b892-9d5909759bbf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") "
Sep 18 17:24:06.439274 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.439042    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9r25\" (UniqueName: \"kubernetes.io/projected/4a7189d4-e59c-492b-b892-9d5909759bbf-kube-api-access-t9r25\") pod \"4a7189d4-e59c-492b-b892-9d5909759bbf\" (UID: \"4a7189d4-e59c-492b-b892-9d5909759bbf\") "
Sep 18 17:24:06.439274 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.439106    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a7189d4-e59c-492b-b892-9d5909759bbf-model-cache" (OuterVolumeSpecName: "model-cache") pod "4a7189d4-e59c-492b-b892-9d5909759bbf" (UID: "4a7189d4-e59c-492b-b892-9d5909759bbf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:24:06.439431 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.439276    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a7189d4-e59c-492b-b892-9d5909759bbf-model-cache\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:24:06.439491 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.439426    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a7189d4-e59c-492b-b892-9d5909759bbf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4a7189d4-e59c-492b-b892-9d5909759bbf" (UID: "4a7189d4-e59c-492b-b892-9d5909759bbf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:24:06.439598 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.439576    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a7189d4-e59c-492b-b892-9d5909759bbf-home" (OuterVolumeSpecName: "home") pod "4a7189d4-e59c-492b-b892-9d5909759bbf" (UID: "4a7189d4-e59c-492b-b892-9d5909759bbf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:24:06.442069 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.442037    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a7189d4-e59c-492b-b892-9d5909759bbf-dshm" (OuterVolumeSpecName: "dshm") pod "4a7189d4-e59c-492b-b892-9d5909759bbf" (UID: "4a7189d4-e59c-492b-b892-9d5909759bbf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:24:06.442191 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.442118    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a7189d4-e59c-492b-b892-9d5909759bbf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a7189d4-e59c-492b-b892-9d5909759bbf" (UID: "4a7189d4-e59c-492b-b892-9d5909759bbf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:24:06.442191 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.442160    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a7189d4-e59c-492b-b892-9d5909759bbf-kube-api-access-t9r25" (OuterVolumeSpecName: "kube-api-access-t9r25") pod "4a7189d4-e59c-492b-b892-9d5909759bbf" (UID: "4a7189d4-e59c-492b-b892-9d5909759bbf"). InnerVolumeSpecName "kube-api-access-t9r25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:24:06.452692 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.452670    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a7189d4-e59c-492b-b892-9d5909759bbf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4a7189d4-e59c-492b-b892-9d5909759bbf" (UID: "4a7189d4-e59c-492b-b892-9d5909759bbf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:24:06.460882 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.460860    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a7189d4-e59c-492b-b892-9d5909759bbf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4a7189d4-e59c-492b-b892-9d5909759bbf" (UID: "4a7189d4-e59c-492b-b892-9d5909759bbf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:24:06.540059 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.540002    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a7189d4-e59c-492b-b892-9d5909759bbf-tmp-dir\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:24:06.540059 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.540024    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a7189d4-e59c-492b-b892-9d5909759bbf-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:24:06.540059 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.540034    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a7189d4-e59c-492b-b892-9d5909759bbf-cabundle-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:24:06.540059 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.540042    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a7189d4-e59c-492b-b892-9d5909759bbf-kserve-provision-location\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:24:06.540059 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.540049    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a7189d4-e59c-492b-b892-9d5909759bbf-dshm\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:24:06.540059 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.540057    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9r25\" (UniqueName: \"kubernetes.io/projected/4a7189d4-e59c-492b-b892-9d5909759bbf-kube-api-access-t9r25\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:24:06.540059 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.540065    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a7189d4-e59c-492b-b892-9d5909759bbf-home\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:24:06.859315 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.859236    2585 generic.go:356] "Generic (PLEG): container finished" podID="4a7189d4-e59c-492b-b892-9d5909759bbf" containerID="d0ef09419d0aac89f0d62525eebd33cf4c061919f0320cb899dcc21a559e690a" exitCode=0
Sep 18 17:24:06.859315 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.859259    2585 generic.go:356] "Generic (PLEG): container finished" podID="4a7189d4-e59c-492b-b892-9d5909759bbf" containerID="50511ab85709e56f6de93408d06e374273f3b781d98dab35b226ffa44b7932aa" exitCode=0
Sep 18 17:24:06.859315 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.859287    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"
Sep 18 17:24:06.859537 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.859316    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf" event={"ID":"4a7189d4-e59c-492b-b892-9d5909759bbf","Type":"ContainerDied","Data":"d0ef09419d0aac89f0d62525eebd33cf4c061919f0320cb899dcc21a559e690a"}
Sep 18 17:24:06.859537 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.859370    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf" event={"ID":"4a7189d4-e59c-492b-b892-9d5909759bbf","Type":"ContainerDied","Data":"50511ab85709e56f6de93408d06e374273f3b781d98dab35b226ffa44b7932aa"}
Sep 18 17:24:06.859537 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.859387    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf" event={"ID":"4a7189d4-e59c-492b-b892-9d5909759bbf","Type":"ContainerDied","Data":"2a2bfe849a450c45f56b3f16952d6c0d2de70a2ae4879e625d2b863c789e5e36"}
Sep 18 17:24:06.859537 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.859406    2585 scope.go:117] "RemoveContainer" containerID="d0ef09419d0aac89f0d62525eebd33cf4c061919f0320cb899dcc21a559e690a"
Sep 18 17:24:06.868193 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.868175    2585 scope.go:117] "RemoveContainer" containerID="9800942718d345424ccaa88fbe145cc00bc5ec9b3d47bd87a3bc3a7cfe755aed"
Sep 18 17:24:06.875721 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.875705    2585 scope.go:117] "RemoveContainer" containerID="50511ab85709e56f6de93408d06e374273f3b781d98dab35b226ffa44b7932aa"
Sep 18 17:24:06.883219 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.883195    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"]
Sep 18 17:24:06.883486 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.883466    2585 scope.go:117] "RemoveContainer" containerID="d0ef09419d0aac89f0d62525eebd33cf4c061919f0320cb899dcc21a559e690a"
Sep 18 17:24:06.883724 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:24:06.883705    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0ef09419d0aac89f0d62525eebd33cf4c061919f0320cb899dcc21a559e690a\": container with ID starting with d0ef09419d0aac89f0d62525eebd33cf4c061919f0320cb899dcc21a559e690a not found: ID does not exist" containerID="d0ef09419d0aac89f0d62525eebd33cf4c061919f0320cb899dcc21a559e690a"
Sep 18 17:24:06.883781 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.883732    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0ef09419d0aac89f0d62525eebd33cf4c061919f0320cb899dcc21a559e690a"} err="failed to get container status \"d0ef09419d0aac89f0d62525eebd33cf4c061919f0320cb899dcc21a559e690a\": rpc error: code = NotFound desc = could not find container \"d0ef09419d0aac89f0d62525eebd33cf4c061919f0320cb899dcc21a559e690a\": container with ID starting with d0ef09419d0aac89f0d62525eebd33cf4c061919f0320cb899dcc21a559e690a not found: ID does not exist"
Sep 18 17:24:06.883781 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.883749    2585 scope.go:117] "RemoveContainer" containerID="9800942718d345424ccaa88fbe145cc00bc5ec9b3d47bd87a3bc3a7cfe755aed"
Sep 18 17:24:06.883996 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:24:06.883977    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9800942718d345424ccaa88fbe145cc00bc5ec9b3d47bd87a3bc3a7cfe755aed\": container with ID starting with 9800942718d345424ccaa88fbe145cc00bc5ec9b3d47bd87a3bc3a7cfe755aed not found: ID does not exist" containerID="9800942718d345424ccaa88fbe145cc00bc5ec9b3d47bd87a3bc3a7cfe755aed"
Sep 18 17:24:06.884042 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.884001    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9800942718d345424ccaa88fbe145cc00bc5ec9b3d47bd87a3bc3a7cfe755aed"} err="failed to get container status \"9800942718d345424ccaa88fbe145cc00bc5ec9b3d47bd87a3bc3a7cfe755aed\": rpc error: code = NotFound desc = could not find container \"9800942718d345424ccaa88fbe145cc00bc5ec9b3d47bd87a3bc3a7cfe755aed\": container with ID starting with 9800942718d345424ccaa88fbe145cc00bc5ec9b3d47bd87a3bc3a7cfe755aed not found: ID does not exist"
Sep 18 17:24:06.884042 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.884015    2585 scope.go:117] "RemoveContainer" containerID="50511ab85709e56f6de93408d06e374273f3b781d98dab35b226ffa44b7932aa"
Sep 18 17:24:06.884270 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:24:06.884254    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50511ab85709e56f6de93408d06e374273f3b781d98dab35b226ffa44b7932aa\": container with ID starting with 50511ab85709e56f6de93408d06e374273f3b781d98dab35b226ffa44b7932aa not found: ID does not exist" containerID="50511ab85709e56f6de93408d06e374273f3b781d98dab35b226ffa44b7932aa"
Sep 18 17:24:06.884360 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.884276    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50511ab85709e56f6de93408d06e374273f3b781d98dab35b226ffa44b7932aa"} err="failed to get container status \"50511ab85709e56f6de93408d06e374273f3b781d98dab35b226ffa44b7932aa\": rpc error: code = NotFound desc = could not find container \"50511ab85709e56f6de93408d06e374273f3b781d98dab35b226ffa44b7932aa\": container with ID starting with 50511ab85709e56f6de93408d06e374273f3b781d98dab35b226ffa44b7932aa not found: ID does not exist"
Sep 18 17:24:06.884360 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.884295    2585 scope.go:117] "RemoveContainer" containerID="d0ef09419d0aac89f0d62525eebd33cf4c061919f0320cb899dcc21a559e690a"
Sep 18 17:24:06.884498 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.884481    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0ef09419d0aac89f0d62525eebd33cf4c061919f0320cb899dcc21a559e690a"} err="failed to get container status \"d0ef09419d0aac89f0d62525eebd33cf4c061919f0320cb899dcc21a559e690a\": rpc error: code = NotFound desc = could not find container \"d0ef09419d0aac89f0d62525eebd33cf4c061919f0320cb899dcc21a559e690a\": container with ID starting with d0ef09419d0aac89f0d62525eebd33cf4c061919f0320cb899dcc21a559e690a not found: ID does not exist"
Sep 18 17:24:06.884540 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.884499    2585 scope.go:117] "RemoveContainer" containerID="9800942718d345424ccaa88fbe145cc00bc5ec9b3d47bd87a3bc3a7cfe755aed"
Sep 18 17:24:06.884745 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.884727    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9800942718d345424ccaa88fbe145cc00bc5ec9b3d47bd87a3bc3a7cfe755aed"} err="failed to get container status \"9800942718d345424ccaa88fbe145cc00bc5ec9b3d47bd87a3bc3a7cfe755aed\": rpc error: code = NotFound desc = could not find container \"9800942718d345424ccaa88fbe145cc00bc5ec9b3d47bd87a3bc3a7cfe755aed\": container with ID starting with 9800942718d345424ccaa88fbe145cc00bc5ec9b3d47bd87a3bc3a7cfe755aed not found: ID does not exist"
Sep 18 17:24:06.884797 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.884746    2585 scope.go:117] "RemoveContainer" containerID="50511ab85709e56f6de93408d06e374273f3b781d98dab35b226ffa44b7932aa"
Sep 18 17:24:06.885044 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.885024    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50511ab85709e56f6de93408d06e374273f3b781d98dab35b226ffa44b7932aa"} err="failed to get container status \"50511ab85709e56f6de93408d06e374273f3b781d98dab35b226ffa44b7932aa\": rpc error: code = NotFound desc = could not find container \"50511ab85709e56f6de93408d06e374273f3b781d98dab35b226ffa44b7932aa\": container with ID starting with 50511ab85709e56f6de93408d06e374273f3b781d98dab35b226ffa44b7932aa not found: ID does not exist"
Sep 18 17:24:06.887662 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:06.887640    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-l8wmf"]
Sep 18 17:24:08.392191 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:08.392158    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a7189d4-e59c-492b-b892-9d5909759bbf" path="/var/lib/kubelet/pods/4a7189d4-e59c-492b-b892-9d5909759bbf/volumes"
Sep 18 17:24:45.585532 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.585455    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"]
Sep 18 17:24:45.586023 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.585748    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a7189d4-e59c-492b-b892-9d5909759bbf" containerName="storage-initializer"
Sep 18 17:24:45.586023 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.585759    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a7189d4-e59c-492b-b892-9d5909759bbf" containerName="storage-initializer"
Sep 18 17:24:45.586023 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.585773    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a7189d4-e59c-492b-b892-9d5909759bbf" containerName="llm-d-routing-sidecar"
Sep 18 17:24:45.586023 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.585778    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a7189d4-e59c-492b-b892-9d5909759bbf" containerName="llm-d-routing-sidecar"
Sep 18 17:24:45.586023 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.585786    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a7189d4-e59c-492b-b892-9d5909759bbf" containerName="main"
Sep 18 17:24:45.586023 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.585791    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a7189d4-e59c-492b-b892-9d5909759bbf" containerName="main"
Sep 18 17:24:45.586023 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.585802    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3548dc85-90e9-4af1-be42-aae4e2373710" containerName="storage-initializer"
Sep 18 17:24:45.586023 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.585808    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3548dc85-90e9-4af1-be42-aae4e2373710" containerName="storage-initializer"
Sep 18 17:24:45.586023 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.585858    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a7189d4-e59c-492b-b892-9d5909759bbf" containerName="llm-d-routing-sidecar"
Sep 18 17:24:45.586023 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.585864    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a7189d4-e59c-492b-b892-9d5909759bbf" containerName="main"
Sep 18 17:24:45.586023 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.585879    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3548dc85-90e9-4af1-be42-aae4e2373710" containerName="storage-initializer"
Sep 18 17:24:45.587667 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.587649    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:24:45.590395 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.590364    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 18 17:24:45.590395 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.590381    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 18 17:24:45.590395 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.590390    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 18 17:24:45.590886 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.590867    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 18 17:24:45.591741 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.591720    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-9cd65\""
Sep 18 17:24:45.591824 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.591742    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 17:24:45.604264 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.602759    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"]
Sep 18 17:24:45.644166 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.644134    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f4cadb2-3e72-436d-baf2-d91561428ea2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:24:45.644166 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.644167    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f4cadb2-3e72-436d-baf2-d91561428ea2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:24:45.644369 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.644189    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f4cadb2-3e72-436d-baf2-d91561428ea2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:24:45.644369 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.644239    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f4cadb2-3e72-436d-baf2-d91561428ea2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:24:45.644369 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.644268    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7zsj\" (UniqueName: \"kubernetes.io/projected/3f4cadb2-3e72-436d-baf2-d91561428ea2-kube-api-access-n7zsj\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:24:45.644532 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.644374    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f4cadb2-3e72-436d-baf2-d91561428ea2-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:24:45.644532 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.644413    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f4cadb2-3e72-436d-baf2-d91561428ea2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:24:45.644532 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.644440    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f4cadb2-3e72-436d-baf2-d91561428ea2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:24:45.744856 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.744821    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7zsj\" (UniqueName: \"kubernetes.io/projected/3f4cadb2-3e72-436d-baf2-d91561428ea2-kube-api-access-n7zsj\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:24:45.745056 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.744865    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f4cadb2-3e72-436d-baf2-d91561428ea2-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:24:45.745056 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.744948    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f4cadb2-3e72-436d-baf2-d91561428ea2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:24:45.745056 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.744964    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f4cadb2-3e72-436d-baf2-d91561428ea2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:24:45.745056 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.745038    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f4cadb2-3e72-436d-baf2-d91561428ea2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:24:45.745294 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.745062    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f4cadb2-3e72-436d-baf2-d91561428ea2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:24:45.745744 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.745452    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f4cadb2-3e72-436d-baf2-d91561428ea2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:24:45.746040 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.745653    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f4cadb2-3e72-436d-baf2-d91561428ea2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:24:45.746280 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.745726    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f4cadb2-3e72-436d-baf2-d91561428ea2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:24:45.746509 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.746461    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f4cadb2-3e72-436d-baf2-d91561428ea2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:24:45.746627 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.746533    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f4cadb2-3e72-436d-baf2-d91561428ea2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:24:45.746627 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.746269    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f4cadb2-3e72-436d-baf2-d91561428ea2-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:24:45.746828 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.746796    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f4cadb2-3e72-436d-baf2-d91561428ea2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:24:45.747730 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.747706    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f4cadb2-3e72-436d-baf2-d91561428ea2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:24:45.748347 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.748326    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f4cadb2-3e72-436d-baf2-d91561428ea2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:24:45.753521 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.753499    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7zsj\" (UniqueName: \"kubernetes.io/projected/3f4cadb2-3e72-436d-baf2-d91561428ea2-kube-api-access-n7zsj\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:24:45.897732 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:45.897656    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:24:46.179161 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:46.179129    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"]
Sep 18 17:24:46.180155 ip-10-0-133-147 kubenswrapper[2585]: W0918 17:24:46.180123    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f4cadb2_3e72_436d_baf2_d91561428ea2.slice/crio-c668de49d7cf65ef16ab8124402c2e192a263c5d8b5afdef6146ee9ca00b1168 WatchSource:0}: Error finding container c668de49d7cf65ef16ab8124402c2e192a263c5d8b5afdef6146ee9ca00b1168: Status 404 returned error can't find the container with id c668de49d7cf65ef16ab8124402c2e192a263c5d8b5afdef6146ee9ca00b1168
Sep 18 17:24:47.021220 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:47.021186    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r" event={"ID":"3f4cadb2-3e72-436d-baf2-d91561428ea2","Type":"ContainerStarted","Data":"99bf0e948283278723324ea166c4e9348312d9ea54a347a4f2eb0643e6e60c16"}
Sep 18 17:24:47.021713 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:47.021235    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r" event={"ID":"3f4cadb2-3e72-436d-baf2-d91561428ea2","Type":"ContainerStarted","Data":"c668de49d7cf65ef16ab8124402c2e192a263c5d8b5afdef6146ee9ca00b1168"}
Sep 18 17:24:49.031117 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:49.031084    2585 generic.go:356] "Generic (PLEG): container finished" podID="3f4cadb2-3e72-436d-baf2-d91561428ea2" containerID="99bf0e948283278723324ea166c4e9348312d9ea54a347a4f2eb0643e6e60c16" exitCode=0
Sep 18 17:24:49.031579 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:49.031156    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r" event={"ID":"3f4cadb2-3e72-436d-baf2-d91561428ea2","Type":"ContainerDied","Data":"99bf0e948283278723324ea166c4e9348312d9ea54a347a4f2eb0643e6e60c16"}
Sep 18 17:24:50.037279 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:50.037241    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r" event={"ID":"3f4cadb2-3e72-436d-baf2-d91561428ea2","Type":"ContainerStarted","Data":"6810ed6c3ecda447a5e0819cf6dd0cf395f6c95f58c458b0328a97be6236e832"}
Sep 18 17:24:50.061094 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:50.061037    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r" podStartSLOduration=2.911853823 podStartE2EDuration="5.061022607s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.149168784s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:24:50.058470642 +0000 UTC m=+4129.841472685" watchObservedRunningTime="2026-09-18 17:24:50.061022607 +0000 UTC m=+4129.844024630"
Sep 18 17:24:50.898138 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:50.898103    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:24:55.898105 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:55.898067    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:24:55.899672 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:24:55.899639    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r" podUID="3f4cadb2-3e72-436d-baf2-d91561428ea2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 18 17:25:05.898266 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:25:05.898213    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r" podUID="3f4cadb2-3e72-436d-baf2-d91561428ea2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 18 17:25:15.898814 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:25:15.898757    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r" podUID="3f4cadb2-3e72-436d-baf2-d91561428ea2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 18 17:25:25.898675 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:25:25.898626    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r" podUID="3f4cadb2-3e72-436d-baf2-d91561428ea2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 18 17:25:28.587231 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:25:28.587189    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-mqllb"]
Sep 18 17:25:28.592431 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:25:28.592403    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-mqllb"]
Sep 18 17:25:30.392345 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:25:30.392311    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3548dc85-90e9-4af1-be42-aae4e2373710" path="/var/lib/kubelet/pods/3548dc85-90e9-4af1-be42-aae4e2373710/volumes"
Sep 18 17:25:35.898553 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:25:35.898509    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r" podUID="3f4cadb2-3e72-436d-baf2-d91561428ea2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 18 17:25:45.898270 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:25:45.898224    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r" podUID="3f4cadb2-3e72-436d-baf2-d91561428ea2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 18 17:25:55.898562 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:25:55.898504    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r" podUID="3f4cadb2-3e72-436d-baf2-d91561428ea2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 18 17:26:02.652009 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:26:02.651982    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 17:26:02.656281 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:26:02.656255    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 17:26:02.661061 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:26:02.661040    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 17:26:02.664889 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:26:02.664870    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 17:26:05.898386 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:26:05.898339    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r" podUID="3f4cadb2-3e72-436d-baf2-d91561428ea2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 18 17:26:15.898884 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:26:15.898789    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r" podUID="3f4cadb2-3e72-436d-baf2-d91561428ea2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 18 17:26:25.898625 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:26:25.898581    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r" podUID="3f4cadb2-3e72-436d-baf2-d91561428ea2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 18 17:26:35.898474 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:26:35.898425    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r" podUID="3f4cadb2-3e72-436d-baf2-d91561428ea2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 18 17:26:45.908628 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:26:45.908586    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:26:45.916968 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:26:45.916941    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:26:50.091953 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:26:50.091905    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"]
Sep 18 17:27:05.114831 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:05.114766    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r" podUID="3f4cadb2-3e72-436d-baf2-d91561428ea2" containerName="main" containerID="cri-o://6810ed6c3ecda447a5e0819cf6dd0cf395f6c95f58c458b0328a97be6236e832" gracePeriod=45
Sep 18 17:27:05.851208 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:05.851133    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:27:05.924572 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:05.924477    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f4cadb2-3e72-436d-baf2-d91561428ea2-dshm\") pod \"3f4cadb2-3e72-436d-baf2-d91561428ea2\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") "
Sep 18 17:27:05.924572 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:05.924524    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f4cadb2-3e72-436d-baf2-d91561428ea2-model-cache\") pod \"3f4cadb2-3e72-436d-baf2-d91561428ea2\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") "
Sep 18 17:27:05.924572 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:05.924544    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f4cadb2-3e72-436d-baf2-d91561428ea2-tls-certs\") pod \"3f4cadb2-3e72-436d-baf2-d91561428ea2\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") "
Sep 18 17:27:05.924572 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:05.924571    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f4cadb2-3e72-436d-baf2-d91561428ea2-tmp-dir\") pod \"3f4cadb2-3e72-436d-baf2-d91561428ea2\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") "
Sep 18 17:27:05.924947 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:05.924595    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f4cadb2-3e72-436d-baf2-d91561428ea2-home\") pod \"3f4cadb2-3e72-436d-baf2-d91561428ea2\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") "
Sep 18 17:27:05.924947 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:05.924641    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7zsj\" (UniqueName: \"kubernetes.io/projected/3f4cadb2-3e72-436d-baf2-d91561428ea2-kube-api-access-n7zsj\") pod \"3f4cadb2-3e72-436d-baf2-d91561428ea2\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") "
Sep 18 17:27:05.924947 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:05.924672    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f4cadb2-3e72-436d-baf2-d91561428ea2-cabundle-cert\") pod \"3f4cadb2-3e72-436d-baf2-d91561428ea2\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") "
Sep 18 17:27:05.924947 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:05.924706    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f4cadb2-3e72-436d-baf2-d91561428ea2-kserve-provision-location\") pod \"3f4cadb2-3e72-436d-baf2-d91561428ea2\" (UID: \"3f4cadb2-3e72-436d-baf2-d91561428ea2\") "
Sep 18 17:27:05.924947 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:05.924838    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4cadb2-3e72-436d-baf2-d91561428ea2-model-cache" (OuterVolumeSpecName: "model-cache") pod "3f4cadb2-3e72-436d-baf2-d91561428ea2" (UID: "3f4cadb2-3e72-436d-baf2-d91561428ea2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:27:05.925224 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:05.924995    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f4cadb2-3e72-436d-baf2-d91561428ea2-model-cache\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:27:05.925345 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:05.925317    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f4cadb2-3e72-436d-baf2-d91561428ea2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3f4cadb2-3e72-436d-baf2-d91561428ea2" (UID: "3f4cadb2-3e72-436d-baf2-d91561428ea2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:27:05.925439 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:05.925385    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4cadb2-3e72-436d-baf2-d91561428ea2-home" (OuterVolumeSpecName: "home") pod "3f4cadb2-3e72-436d-baf2-d91561428ea2" (UID: "3f4cadb2-3e72-436d-baf2-d91561428ea2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:27:05.927356 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:05.927325    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4cadb2-3e72-436d-baf2-d91561428ea2-dshm" (OuterVolumeSpecName: "dshm") pod "3f4cadb2-3e72-436d-baf2-d91561428ea2" (UID: "3f4cadb2-3e72-436d-baf2-d91561428ea2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:27:05.927588 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:05.927452    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f4cadb2-3e72-436d-baf2-d91561428ea2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f4cadb2-3e72-436d-baf2-d91561428ea2" (UID: "3f4cadb2-3e72-436d-baf2-d91561428ea2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:27:05.927588 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:05.927463    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4cadb2-3e72-436d-baf2-d91561428ea2-kube-api-access-n7zsj" (OuterVolumeSpecName: "kube-api-access-n7zsj") pod "3f4cadb2-3e72-436d-baf2-d91561428ea2" (UID: "3f4cadb2-3e72-436d-baf2-d91561428ea2"). InnerVolumeSpecName "kube-api-access-n7zsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:27:05.943434 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:05.943401    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4cadb2-3e72-436d-baf2-d91561428ea2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3f4cadb2-3e72-436d-baf2-d91561428ea2" (UID: "3f4cadb2-3e72-436d-baf2-d91561428ea2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:27:05.950645 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:05.950614    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4cadb2-3e72-436d-baf2-d91561428ea2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3f4cadb2-3e72-436d-baf2-d91561428ea2" (UID: "3f4cadb2-3e72-436d-baf2-d91561428ea2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:27:06.025907 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:06.025871    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f4cadb2-3e72-436d-baf2-d91561428ea2-tmp-dir\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:27:06.025907 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:06.025905    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f4cadb2-3e72-436d-baf2-d91561428ea2-home\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:27:06.026106 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:06.025942    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7zsj\" (UniqueName: \"kubernetes.io/projected/3f4cadb2-3e72-436d-baf2-d91561428ea2-kube-api-access-n7zsj\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:27:06.026106 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:06.025951    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f4cadb2-3e72-436d-baf2-d91561428ea2-cabundle-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:27:06.026106 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:06.025961    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f4cadb2-3e72-436d-baf2-d91561428ea2-kserve-provision-location\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:27:06.026106 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:06.025969    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f4cadb2-3e72-436d-baf2-d91561428ea2-dshm\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:27:06.026106 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:06.025977    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f4cadb2-3e72-436d-baf2-d91561428ea2-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:27:06.600799 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:06.600762    2585 generic.go:356] "Generic (PLEG): container finished" podID="3f4cadb2-3e72-436d-baf2-d91561428ea2" containerID="6810ed6c3ecda447a5e0819cf6dd0cf395f6c95f58c458b0328a97be6236e832" exitCode=0
Sep 18 17:27:06.601324 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:06.600812    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"
Sep 18 17:27:06.601324 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:06.600846    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r" event={"ID":"3f4cadb2-3e72-436d-baf2-d91561428ea2","Type":"ContainerDied","Data":"6810ed6c3ecda447a5e0819cf6dd0cf395f6c95f58c458b0328a97be6236e832"}
Sep 18 17:27:06.601324 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:06.600906    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r" event={"ID":"3f4cadb2-3e72-436d-baf2-d91561428ea2","Type":"ContainerDied","Data":"c668de49d7cf65ef16ab8124402c2e192a263c5d8b5afdef6146ee9ca00b1168"}
Sep 18 17:27:06.601324 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:06.600950    2585 scope.go:117] "RemoveContainer" containerID="6810ed6c3ecda447a5e0819cf6dd0cf395f6c95f58c458b0328a97be6236e832"
Sep 18 17:27:06.609259 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:06.609242    2585 scope.go:117] "RemoveContainer" containerID="99bf0e948283278723324ea166c4e9348312d9ea54a347a4f2eb0643e6e60c16"
Sep 18 17:27:06.616740 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:06.616720    2585 scope.go:117] "RemoveContainer" containerID="6810ed6c3ecda447a5e0819cf6dd0cf395f6c95f58c458b0328a97be6236e832"
Sep 18 17:27:06.617037 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:27:06.617008    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6810ed6c3ecda447a5e0819cf6dd0cf395f6c95f58c458b0328a97be6236e832\": container with ID starting with 6810ed6c3ecda447a5e0819cf6dd0cf395f6c95f58c458b0328a97be6236e832 not found: ID does not exist" containerID="6810ed6c3ecda447a5e0819cf6dd0cf395f6c95f58c458b0328a97be6236e832"
Sep 18 17:27:06.617155 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:06.617047    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6810ed6c3ecda447a5e0819cf6dd0cf395f6c95f58c458b0328a97be6236e832"} err="failed to get container status \"6810ed6c3ecda447a5e0819cf6dd0cf395f6c95f58c458b0328a97be6236e832\": rpc error: code = NotFound desc = could not find container \"6810ed6c3ecda447a5e0819cf6dd0cf395f6c95f58c458b0328a97be6236e832\": container with ID starting with 6810ed6c3ecda447a5e0819cf6dd0cf395f6c95f58c458b0328a97be6236e832 not found: ID does not exist"
Sep 18 17:27:06.617155 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:06.617069    2585 scope.go:117] "RemoveContainer" containerID="99bf0e948283278723324ea166c4e9348312d9ea54a347a4f2eb0643e6e60c16"
Sep 18 17:27:06.617320 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:27:06.617303    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99bf0e948283278723324ea166c4e9348312d9ea54a347a4f2eb0643e6e60c16\": container with ID starting with 99bf0e948283278723324ea166c4e9348312d9ea54a347a4f2eb0643e6e60c16 not found: ID does not exist" containerID="99bf0e948283278723324ea166c4e9348312d9ea54a347a4f2eb0643e6e60c16"
Sep 18 17:27:06.617365 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:06.617326    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99bf0e948283278723324ea166c4e9348312d9ea54a347a4f2eb0643e6e60c16"} err="failed to get container status \"99bf0e948283278723324ea166c4e9348312d9ea54a347a4f2eb0643e6e60c16\": rpc error: code = NotFound desc = could not find container \"99bf0e948283278723324ea166c4e9348312d9ea54a347a4f2eb0643e6e60c16\": container with ID starting with 99bf0e948283278723324ea166c4e9348312d9ea54a347a4f2eb0643e6e60c16 not found: ID does not exist"
Sep 18 17:27:06.620234 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:06.620207    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"]
Sep 18 17:27:06.630165 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:06.630142    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b9447574r8r"]
Sep 18 17:27:08.391882 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:08.391837    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f4cadb2-3e72-436d-baf2-d91561428ea2" path="/var/lib/kubelet/pods/3f4cadb2-3e72-436d-baf2-d91561428ea2/volumes"
Sep 18 17:27:17.903699 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:17.903664    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 18 17:27:17.904253 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:17.904087    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f4cadb2-3e72-436d-baf2-d91561428ea2" containerName="storage-initializer"
Sep 18 17:27:17.904253 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:17.904107    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f4cadb2-3e72-436d-baf2-d91561428ea2" containerName="storage-initializer"
Sep 18 17:27:17.904253 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:17.904124    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f4cadb2-3e72-436d-baf2-d91561428ea2" containerName="main"
Sep 18 17:27:17.904253 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:17.904132    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f4cadb2-3e72-436d-baf2-d91561428ea2" containerName="main"
Sep 18 17:27:17.904253 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:17.904202    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f4cadb2-3e72-436d-baf2-d91561428ea2" containerName="main"
Sep 18 17:27:17.906781 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:17.906760    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:27:17.909733 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:17.909710    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 18 17:27:17.909857 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:17.909838    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-g8sr7\""
Sep 18 17:27:17.909992 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:17.909927    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 18 17:27:17.911177 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:17.911161    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 18 17:27:17.911177 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:17.911183    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-679vs\""
Sep 18 17:27:17.911177 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:17.911209    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 17:27:17.911515 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:17.911211    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 18 17:27:17.924185 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:17.924157    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 18 17:27:18.016297 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.016245    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b190fc1e-65e9-4558-9f19-a4c01f3960c5-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:27:18.016297 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.016288    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b190fc1e-65e9-4558-9f19-a4c01f3960c5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:27:18.016578 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.016312    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b190fc1e-65e9-4558-9f19-a4c01f3960c5-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:27:18.016578 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.016337    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b190fc1e-65e9-4558-9f19-a4c01f3960c5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:27:18.016578 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.016415    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dml9w\" (UniqueName: \"kubernetes.io/projected/b190fc1e-65e9-4558-9f19-a4c01f3960c5-kube-api-access-dml9w\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:27:18.016578 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.016519    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b190fc1e-65e9-4558-9f19-a4c01f3960c5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:27:18.016578 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.016561    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b190fc1e-65e9-4558-9f19-a4c01f3960c5-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:27:18.016775 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.016626    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b190fc1e-65e9-4558-9f19-a4c01f3960c5-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:27:18.117205 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.117172    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b190fc1e-65e9-4558-9f19-a4c01f3960c5-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:27:18.117380 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.117218    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b190fc1e-65e9-4558-9f19-a4c01f3960c5-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:27:18.117380 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.117245    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b190fc1e-65e9-4558-9f19-a4c01f3960c5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:27:18.117380 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.117268    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b190fc1e-65e9-4558-9f19-a4c01f3960c5-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:27:18.117380 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.117289    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b190fc1e-65e9-4558-9f19-a4c01f3960c5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:27:18.117380 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.117346    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dml9w\" (UniqueName: \"kubernetes.io/projected/b190fc1e-65e9-4558-9f19-a4c01f3960c5-kube-api-access-dml9w\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:27:18.117662 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.117400    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b190fc1e-65e9-4558-9f19-a4c01f3960c5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:27:18.117662 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.117437    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b190fc1e-65e9-4558-9f19-a4c01f3960c5-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:27:18.117662 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.117643    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b190fc1e-65e9-4558-9f19-a4c01f3960c5-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:27:18.117823 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.117675    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b190fc1e-65e9-4558-9f19-a4c01f3960c5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:27:18.117823 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.117768    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b190fc1e-65e9-4558-9f19-a4c01f3960c5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:27:18.117970 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.117831    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b190fc1e-65e9-4558-9f19-a4c01f3960c5-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:27:18.118127 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.118107    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b190fc1e-65e9-4558-9f19-a4c01f3960c5-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:27:18.119790 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.119768    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b190fc1e-65e9-4558-9f19-a4c01f3960c5-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:27:18.119991 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.119973    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b190fc1e-65e9-4558-9f19-a4c01f3960c5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:27:18.129239 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.129218    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dml9w\" (UniqueName: \"kubernetes.io/projected/b190fc1e-65e9-4558-9f19-a4c01f3960c5-kube-api-access-dml9w\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:27:18.163699 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.163637    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k"]
Sep 18 17:27:18.167873 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.167851    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k"
Sep 18 17:27:18.170499 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.170477    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-kcg8w\""
Sep 18 17:27:18.177485 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.177462    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k"]
Sep 18 17:27:18.216428 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.216406    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:27:18.218842 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.218818    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcs8p\" (UniqueName: \"kubernetes.io/projected/7bab1aaa-99ce-4ef5-af19-c3043647bb11-kube-api-access-hcs8p\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k\" (UID: \"7bab1aaa-99ce-4ef5-af19-c3043647bb11\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k"
Sep 18 17:27:18.218948 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.218881    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bab1aaa-99ce-4ef5-af19-c3043647bb11-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k\" (UID: \"7bab1aaa-99ce-4ef5-af19-c3043647bb11\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k"
Sep 18 17:27:18.320088 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.320058    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bab1aaa-99ce-4ef5-af19-c3043647bb11-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k\" (UID: \"7bab1aaa-99ce-4ef5-af19-c3043647bb11\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k"
Sep 18 17:27:18.320275 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.320142    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcs8p\" (UniqueName: \"kubernetes.io/projected/7bab1aaa-99ce-4ef5-af19-c3043647bb11-kube-api-access-hcs8p\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k\" (UID: \"7bab1aaa-99ce-4ef5-af19-c3043647bb11\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k"
Sep 18 17:27:18.322685 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.322664    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bab1aaa-99ce-4ef5-af19-c3043647bb11-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k\" (UID: \"7bab1aaa-99ce-4ef5-af19-c3043647bb11\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k"
Sep 18 17:27:18.328831 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.328812    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcs8p\" (UniqueName: \"kubernetes.io/projected/7bab1aaa-99ce-4ef5-af19-c3043647bb11-kube-api-access-hcs8p\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k\" (UID: \"7bab1aaa-99ce-4ef5-af19-c3043647bb11\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k"
Sep 18 17:27:18.477972 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.477940    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k"
Sep 18 17:27:18.492624 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.492594    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 18 17:27:18.495473 ip-10-0-133-147 kubenswrapper[2585]: W0918 17:27:18.495441    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb190fc1e_65e9_4558_9f19_a4c01f3960c5.slice/crio-03a66fb52fa6373f0060592fc3555a1e1869bb374171a81b1132113b06ea336c WatchSource:0}: Error finding container 03a66fb52fa6373f0060592fc3555a1e1869bb374171a81b1132113b06ea336c: Status 404 returned error can't find the container with id 03a66fb52fa6373f0060592fc3555a1e1869bb374171a81b1132113b06ea336c
Sep 18 17:27:18.649120 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.649082    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"b190fc1e-65e9-4558-9f19-a4c01f3960c5","Type":"ContainerStarted","Data":"1421be74eea08a372dd8a65986c663ab9b1768e6ad384b500e5a78d0cd038eab"}
Sep 18 17:27:18.649299 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.649144    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"b190fc1e-65e9-4558-9f19-a4c01f3960c5","Type":"ContainerStarted","Data":"03a66fb52fa6373f0060592fc3555a1e1869bb374171a81b1132113b06ea336c"}
Sep 18 17:27:18.752787 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:18.752757    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k"]
Sep 18 17:27:18.753906 ip-10-0-133-147 kubenswrapper[2585]: W0918 17:27:18.753878    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bab1aaa_99ce_4ef5_af19_c3043647bb11.slice/crio-7b135effbdc749b5be46f1d3a79e636ad7cd8316470477c7e54b5ee9d83c87ce WatchSource:0}: Error finding container 7b135effbdc749b5be46f1d3a79e636ad7cd8316470477c7e54b5ee9d83c87ce: Status 404 returned error can't find the container with id 7b135effbdc749b5be46f1d3a79e636ad7cd8316470477c7e54b5ee9d83c87ce
Sep 18 17:27:19.655062 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:19.655033    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k" event={"ID":"7bab1aaa-99ce-4ef5-af19-c3043647bb11","Type":"ContainerStarted","Data":"d320e93d5b31060037f753b14527185a6492d0bfb3e4544699a9138531d55f61"}
Sep 18 17:27:19.655495 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:19.655085    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k" event={"ID":"7bab1aaa-99ce-4ef5-af19-c3043647bb11","Type":"ContainerStarted","Data":"7b135effbdc749b5be46f1d3a79e636ad7cd8316470477c7e54b5ee9d83c87ce"}
Sep 18 17:27:19.655495 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:19.655338    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k"
Sep 18 17:27:21.664380 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:21.664341    2585 generic.go:356] "Generic (PLEG): container finished" podID="b190fc1e-65e9-4558-9f19-a4c01f3960c5" containerID="1421be74eea08a372dd8a65986c663ab9b1768e6ad384b500e5a78d0cd038eab" exitCode=0
Sep 18 17:27:21.664780 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:21.664403    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"b190fc1e-65e9-4558-9f19-a4c01f3960c5","Type":"ContainerDied","Data":"1421be74eea08a372dd8a65986c663ab9b1768e6ad384b500e5a78d0cd038eab"}
Sep 18 17:27:21.687788 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:21.687743    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k" podStartSLOduration=3.687729051 podStartE2EDuration="3.687729051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:27:19.689699766 +0000 UTC m=+4279.472701792" watchObservedRunningTime="2026-09-18 17:27:21.687729051 +0000 UTC m=+4281.470731076"
Sep 18 17:27:22.670494 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:22.670457    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"b190fc1e-65e9-4558-9f19-a4c01f3960c5","Type":"ContainerStarted","Data":"60437a438c156606dfc525079118886bd2f3830c7b43fb9a19c0401535cebe81"}
Sep 18 17:27:22.694902 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:22.694837    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=3.5120871510000002 podStartE2EDuration="5.694814682s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.182727531s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:27:22.691592359 +0000 UTC m=+4282.474594384" watchObservedRunningTime="2026-09-18 17:27:22.694814682 +0000 UTC m=+4282.477816706"
Sep 18 17:27:23.217298 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:23.217261    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:27:28.216636 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:28.216595    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:27:28.218237 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:28.218207    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="b190fc1e-65e9-4558-9f19-a4c01f3960c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 18 17:27:38.217197 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:38.217154    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="b190fc1e-65e9-4558-9f19-a4c01f3960c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 18 17:27:48.217354 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:48.217310    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="b190fc1e-65e9-4558-9f19-a4c01f3960c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 18 17:27:50.661776 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:50.661737    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k"
Sep 18 17:27:58.217053 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:27:58.217012    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="b190fc1e-65e9-4558-9f19-a4c01f3960c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 18 17:28:08.217841 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:28:08.217789    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="b190fc1e-65e9-4558-9f19-a4c01f3960c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 18 17:28:18.217374 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:28:18.217321    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="b190fc1e-65e9-4558-9f19-a4c01f3960c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 18 17:28:28.217056 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:28:28.217011    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="b190fc1e-65e9-4558-9f19-a4c01f3960c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 18 17:28:38.217190 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:28:38.217146    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="b190fc1e-65e9-4558-9f19-a4c01f3960c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 18 17:28:48.217701 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:28:48.217651    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="b190fc1e-65e9-4558-9f19-a4c01f3960c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 18 17:28:58.226574 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:28:58.226535    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:28:58.234616 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:28:58.234589    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:29:02.892284 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:02.892249    2585 scope.go:117] "RemoveContainer" containerID="6065028c11367be37e816d5c310d9c50cd3b66c2d6f92edc19d037ddc89af7ca"
Sep 18 17:29:07.775643 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:07.775609    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k"]
Sep 18 17:29:07.782293 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:07.782266    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 18 17:29:10.661554 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:10.661515    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k" podUID="7bab1aaa-99ce-4ef5-af19-c3043647bb11" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:29:20.661555 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:20.661455    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k" podUID="7bab1aaa-99ce-4ef5-af19-c3043647bb11" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:29:22.795867 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:22.795801    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k" podUID="7bab1aaa-99ce-4ef5-af19-c3043647bb11" containerName="main" containerID="cri-o://d320e93d5b31060037f753b14527185a6492d0bfb3e4544699a9138531d55f61" gracePeriod=45
Sep 18 17:29:22.804160 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:22.804110    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="b190fc1e-65e9-4558-9f19-a4c01f3960c5" containerName="main" containerID="cri-o://60437a438c156606dfc525079118886bd2f3830c7b43fb9a19c0401535cebe81" gracePeriod=45
Sep 18 17:29:23.228016 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:23.227959    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="b190fc1e-65e9-4558-9f19-a4c01f3960c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 18 17:29:23.546274 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:23.546251    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:29:23.683367 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:23.683332    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b190fc1e-65e9-4558-9f19-a4c01f3960c5-kserve-provision-location\") pod \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") "
Sep 18 17:29:23.683584 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:23.683401    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b190fc1e-65e9-4558-9f19-a4c01f3960c5-tls-certs\") pod \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") "
Sep 18 17:29:23.683584 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:23.683432    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b190fc1e-65e9-4558-9f19-a4c01f3960c5-cabundle-cert\") pod \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") "
Sep 18 17:29:23.683584 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:23.683450    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b190fc1e-65e9-4558-9f19-a4c01f3960c5-home\") pod \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") "
Sep 18 17:29:23.683584 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:23.683479    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b190fc1e-65e9-4558-9f19-a4c01f3960c5-dshm\") pod \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") "
Sep 18 17:29:23.683584 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:23.683531    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b190fc1e-65e9-4558-9f19-a4c01f3960c5-model-cache\") pod \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") "
Sep 18 17:29:23.683584 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:23.683561    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dml9w\" (UniqueName: \"kubernetes.io/projected/b190fc1e-65e9-4558-9f19-a4c01f3960c5-kube-api-access-dml9w\") pod \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") "
Sep 18 17:29:23.683953 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:23.683615    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b190fc1e-65e9-4558-9f19-a4c01f3960c5-tmp-dir\") pod \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\" (UID: \"b190fc1e-65e9-4558-9f19-a4c01f3960c5\") "
Sep 18 17:29:23.684021 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:23.683959    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b190fc1e-65e9-4558-9f19-a4c01f3960c5-model-cache" (OuterVolumeSpecName: "model-cache") pod "b190fc1e-65e9-4558-9f19-a4c01f3960c5" (UID: "b190fc1e-65e9-4558-9f19-a4c01f3960c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:29:23.684248 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:23.684220    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b190fc1e-65e9-4558-9f19-a4c01f3960c5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b190fc1e-65e9-4558-9f19-a4c01f3960c5" (UID: "b190fc1e-65e9-4558-9f19-a4c01f3960c5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:29:23.684423 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:23.684320    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b190fc1e-65e9-4558-9f19-a4c01f3960c5-home" (OuterVolumeSpecName: "home") pod "b190fc1e-65e9-4558-9f19-a4c01f3960c5" (UID: "b190fc1e-65e9-4558-9f19-a4c01f3960c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:29:23.686371 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:23.686347    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b190fc1e-65e9-4558-9f19-a4c01f3960c5-dshm" (OuterVolumeSpecName: "dshm") pod "b190fc1e-65e9-4558-9f19-a4c01f3960c5" (UID: "b190fc1e-65e9-4558-9f19-a4c01f3960c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:29:23.686937 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:23.686888    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b190fc1e-65e9-4558-9f19-a4c01f3960c5-kube-api-access-dml9w" (OuterVolumeSpecName: "kube-api-access-dml9w") pod "b190fc1e-65e9-4558-9f19-a4c01f3960c5" (UID: "b190fc1e-65e9-4558-9f19-a4c01f3960c5"). InnerVolumeSpecName "kube-api-access-dml9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:29:23.687022 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:23.686998    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b190fc1e-65e9-4558-9f19-a4c01f3960c5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b190fc1e-65e9-4558-9f19-a4c01f3960c5" (UID: "b190fc1e-65e9-4558-9f19-a4c01f3960c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:29:23.702436 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:23.702404    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b190fc1e-65e9-4558-9f19-a4c01f3960c5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b190fc1e-65e9-4558-9f19-a4c01f3960c5" (UID: "b190fc1e-65e9-4558-9f19-a4c01f3960c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:29:23.709622 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:23.709597    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b190fc1e-65e9-4558-9f19-a4c01f3960c5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b190fc1e-65e9-4558-9f19-a4c01f3960c5" (UID: "b190fc1e-65e9-4558-9f19-a4c01f3960c5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:29:23.784247 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:23.784178    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b190fc1e-65e9-4558-9f19-a4c01f3960c5-model-cache\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:29:23.784247 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:23.784203    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dml9w\" (UniqueName: \"kubernetes.io/projected/b190fc1e-65e9-4558-9f19-a4c01f3960c5-kube-api-access-dml9w\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:29:23.784247 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:23.784237    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b190fc1e-65e9-4558-9f19-a4c01f3960c5-tmp-dir\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:29:23.784247 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:23.784248    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b190fc1e-65e9-4558-9f19-a4c01f3960c5-kserve-provision-location\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:29:23.784468 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:23.784258    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b190fc1e-65e9-4558-9f19-a4c01f3960c5-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:29:23.784468 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:23.784267    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b190fc1e-65e9-4558-9f19-a4c01f3960c5-cabundle-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:29:23.784468 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:23.784276    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b190fc1e-65e9-4558-9f19-a4c01f3960c5-home\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:29:23.784468 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:23.784283    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b190fc1e-65e9-4558-9f19-a4c01f3960c5-dshm\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:29:24.208845 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:24.208765    2585 generic.go:356] "Generic (PLEG): container finished" podID="b190fc1e-65e9-4558-9f19-a4c01f3960c5" containerID="60437a438c156606dfc525079118886bd2f3830c7b43fb9a19c0401535cebe81" exitCode=0
Sep 18 17:29:24.208845 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:24.208825    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 18 17:29:24.209295 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:24.208854    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"b190fc1e-65e9-4558-9f19-a4c01f3960c5","Type":"ContainerDied","Data":"60437a438c156606dfc525079118886bd2f3830c7b43fb9a19c0401535cebe81"}
Sep 18 17:29:24.209295 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:24.208936    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"b190fc1e-65e9-4558-9f19-a4c01f3960c5","Type":"ContainerDied","Data":"03a66fb52fa6373f0060592fc3555a1e1869bb374171a81b1132113b06ea336c"}
Sep 18 17:29:24.209295 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:24.208960    2585 scope.go:117] "RemoveContainer" containerID="60437a438c156606dfc525079118886bd2f3830c7b43fb9a19c0401535cebe81"
Sep 18 17:29:24.217947 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:24.217927    2585 scope.go:117] "RemoveContainer" containerID="1421be74eea08a372dd8a65986c663ab9b1768e6ad384b500e5a78d0cd038eab"
Sep 18 17:29:24.225564 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:24.225548    2585 scope.go:117] "RemoveContainer" containerID="60437a438c156606dfc525079118886bd2f3830c7b43fb9a19c0401535cebe81"
Sep 18 17:29:24.225842 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:29:24.225823    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60437a438c156606dfc525079118886bd2f3830c7b43fb9a19c0401535cebe81\": container with ID starting with 60437a438c156606dfc525079118886bd2f3830c7b43fb9a19c0401535cebe81 not found: ID does not exist" containerID="60437a438c156606dfc525079118886bd2f3830c7b43fb9a19c0401535cebe81"
Sep 18 17:29:24.225899 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:24.225850    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60437a438c156606dfc525079118886bd2f3830c7b43fb9a19c0401535cebe81"} err="failed to get container status \"60437a438c156606dfc525079118886bd2f3830c7b43fb9a19c0401535cebe81\": rpc error: code = NotFound desc = could not find container \"60437a438c156606dfc525079118886bd2f3830c7b43fb9a19c0401535cebe81\": container with ID starting with 60437a438c156606dfc525079118886bd2f3830c7b43fb9a19c0401535cebe81 not found: ID does not exist"
Sep 18 17:29:24.225899 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:24.225890    2585 scope.go:117] "RemoveContainer" containerID="1421be74eea08a372dd8a65986c663ab9b1768e6ad384b500e5a78d0cd038eab"
Sep 18 17:29:24.226177 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:29:24.226159    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1421be74eea08a372dd8a65986c663ab9b1768e6ad384b500e5a78d0cd038eab\": container with ID starting with 1421be74eea08a372dd8a65986c663ab9b1768e6ad384b500e5a78d0cd038eab not found: ID does not exist" containerID="1421be74eea08a372dd8a65986c663ab9b1768e6ad384b500e5a78d0cd038eab"
Sep 18 17:29:24.226240 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:24.226196    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1421be74eea08a372dd8a65986c663ab9b1768e6ad384b500e5a78d0cd038eab"} err="failed to get container status \"1421be74eea08a372dd8a65986c663ab9b1768e6ad384b500e5a78d0cd038eab\": rpc error: code = NotFound desc = could not find container \"1421be74eea08a372dd8a65986c663ab9b1768e6ad384b500e5a78d0cd038eab\": container with ID starting with 1421be74eea08a372dd8a65986c663ab9b1768e6ad384b500e5a78d0cd038eab not found: ID does not exist"
Sep 18 17:29:24.233451 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:24.233430    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 18 17:29:24.241907 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:24.241888    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 18 17:29:24.392032 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:24.392004    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b190fc1e-65e9-4558-9f19-a4c01f3960c5" path="/var/lib/kubelet/pods/b190fc1e-65e9-4558-9f19-a4c01f3960c5/volumes"
Sep 18 17:29:30.661039 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:30.660997    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k" podUID="7bab1aaa-99ce-4ef5-af19-c3043647bb11" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:29:30.661528 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:30.661101    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k"
Sep 18 17:29:40.660994 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:40.660956    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k" podUID="7bab1aaa-99ce-4ef5-af19-c3043647bb11" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:29:50.661040 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:50.660989    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k" podUID="7bab1aaa-99ce-4ef5-af19-c3043647bb11" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:29:53.033891 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:53.033859    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k"
Sep 18 17:29:53.111624 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:53.111549    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcs8p\" (UniqueName: \"kubernetes.io/projected/7bab1aaa-99ce-4ef5-af19-c3043647bb11-kube-api-access-hcs8p\") pod \"7bab1aaa-99ce-4ef5-af19-c3043647bb11\" (UID: \"7bab1aaa-99ce-4ef5-af19-c3043647bb11\") "
Sep 18 17:29:53.111624 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:53.111616    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bab1aaa-99ce-4ef5-af19-c3043647bb11-tls-certs\") pod \"7bab1aaa-99ce-4ef5-af19-c3043647bb11\" (UID: \"7bab1aaa-99ce-4ef5-af19-c3043647bb11\") "
Sep 18 17:29:53.113838 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:53.113810    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bab1aaa-99ce-4ef5-af19-c3043647bb11-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7bab1aaa-99ce-4ef5-af19-c3043647bb11" (UID: "7bab1aaa-99ce-4ef5-af19-c3043647bb11"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:29:53.113944 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:53.113839    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bab1aaa-99ce-4ef5-af19-c3043647bb11-kube-api-access-hcs8p" (OuterVolumeSpecName: "kube-api-access-hcs8p") pod "7bab1aaa-99ce-4ef5-af19-c3043647bb11" (UID: "7bab1aaa-99ce-4ef5-af19-c3043647bb11"). InnerVolumeSpecName "kube-api-access-hcs8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:29:53.212901 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:53.212861    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcs8p\" (UniqueName: \"kubernetes.io/projected/7bab1aaa-99ce-4ef5-af19-c3043647bb11-kube-api-access-hcs8p\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:29:53.212901 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:53.212893    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bab1aaa-99ce-4ef5-af19-c3043647bb11-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:29:53.328078 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:53.328045    2585 generic.go:356] "Generic (PLEG): container finished" podID="7bab1aaa-99ce-4ef5-af19-c3043647bb11" containerID="d320e93d5b31060037f753b14527185a6492d0bfb3e4544699a9138531d55f61" exitCode=0
Sep 18 17:29:53.328217 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:53.328097    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k"
Sep 18 17:29:53.328217 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:53.328104    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k" event={"ID":"7bab1aaa-99ce-4ef5-af19-c3043647bb11","Type":"ContainerDied","Data":"d320e93d5b31060037f753b14527185a6492d0bfb3e4544699a9138531d55f61"}
Sep 18 17:29:53.328217 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:53.328150    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k" event={"ID":"7bab1aaa-99ce-4ef5-af19-c3043647bb11","Type":"ContainerDied","Data":"7b135effbdc749b5be46f1d3a79e636ad7cd8316470477c7e54b5ee9d83c87ce"}
Sep 18 17:29:53.328217 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:53.328172    2585 scope.go:117] "RemoveContainer" containerID="d320e93d5b31060037f753b14527185a6492d0bfb3e4544699a9138531d55f61"
Sep 18 17:29:53.337158 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:53.337142    2585 scope.go:117] "RemoveContainer" containerID="d320e93d5b31060037f753b14527185a6492d0bfb3e4544699a9138531d55f61"
Sep 18 17:29:53.337409 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:29:53.337391    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d320e93d5b31060037f753b14527185a6492d0bfb3e4544699a9138531d55f61\": container with ID starting with d320e93d5b31060037f753b14527185a6492d0bfb3e4544699a9138531d55f61 not found: ID does not exist" containerID="d320e93d5b31060037f753b14527185a6492d0bfb3e4544699a9138531d55f61"
Sep 18 17:29:53.337473 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:53.337421    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d320e93d5b31060037f753b14527185a6492d0bfb3e4544699a9138531d55f61"} err="failed to get container status \"d320e93d5b31060037f753b14527185a6492d0bfb3e4544699a9138531d55f61\": rpc error: code = NotFound desc = could not find container \"d320e93d5b31060037f753b14527185a6492d0bfb3e4544699a9138531d55f61\": container with ID starting with d320e93d5b31060037f753b14527185a6492d0bfb3e4544699a9138531d55f61 not found: ID does not exist"
Sep 18 17:29:53.349809 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:53.349785    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k"]
Sep 18 17:29:53.356929 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:53.356895    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem2x6k"]
Sep 18 17:29:54.391850 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:29:54.391813    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bab1aaa-99ce-4ef5-af19-c3043647bb11" path="/var/lib/kubelet/pods/7bab1aaa-99ce-4ef5-af19-c3043647bb11/volumes"
Sep 18 17:30:09.778019 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.777985    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"]
Sep 18 17:30:09.778596 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.778256    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b190fc1e-65e9-4558-9f19-a4c01f3960c5" containerName="storage-initializer"
Sep 18 17:30:09.778596 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.778268    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b190fc1e-65e9-4558-9f19-a4c01f3960c5" containerName="storage-initializer"
Sep 18 17:30:09.778596 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.778277    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b190fc1e-65e9-4558-9f19-a4c01f3960c5" containerName="main"
Sep 18 17:30:09.778596 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.778282    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b190fc1e-65e9-4558-9f19-a4c01f3960c5" containerName="main"
Sep 18 17:30:09.778596 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.778294    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bab1aaa-99ce-4ef5-af19-c3043647bb11" containerName="main"
Sep 18 17:30:09.778596 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.778299    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bab1aaa-99ce-4ef5-af19-c3043647bb11" containerName="main"
Sep 18 17:30:09.778596 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.778368    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bab1aaa-99ce-4ef5-af19-c3043647bb11" containerName="main"
Sep 18 17:30:09.778596 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.778379    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b190fc1e-65e9-4558-9f19-a4c01f3960c5" containerName="main"
Sep 18 17:30:09.782734 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.782711    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"
Sep 18 17:30:09.785709 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.785682    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 18 17:30:09.786790 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.786726    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 18 17:30:09.786790 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.786744    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 18 17:30:09.787030 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.786789    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-86wfl\""
Sep 18 17:30:09.788792 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.788770    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"]
Sep 18 17:30:09.838159 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.838132    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-s28xb\" (UID: \"475ed6c9-e19e-40e1-be90-dfe4c5ed9345\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"
Sep 18 17:30:09.838283 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.838187    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-s28xb\" (UID: \"475ed6c9-e19e-40e1-be90-dfe4c5ed9345\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"
Sep 18 17:30:09.838283 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.838214    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-s28xb\" (UID: \"475ed6c9-e19e-40e1-be90-dfe4c5ed9345\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"
Sep 18 17:30:09.838283 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.838241    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-s28xb\" (UID: \"475ed6c9-e19e-40e1-be90-dfe4c5ed9345\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"
Sep 18 17:30:09.838283 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.838259    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbcs9\" (UniqueName: \"kubernetes.io/projected/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-kube-api-access-cbcs9\") pod \"scheduler-inline-config-test-kserve-8655bc757-s28xb\" (UID: \"475ed6c9-e19e-40e1-be90-dfe4c5ed9345\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"
Sep 18 17:30:09.838283 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.838274    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-s28xb\" (UID: \"475ed6c9-e19e-40e1-be90-dfe4c5ed9345\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"
Sep 18 17:30:09.939274 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.939236    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-s28xb\" (UID: \"475ed6c9-e19e-40e1-be90-dfe4c5ed9345\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"
Sep 18 17:30:09.939456 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.939281    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-s28xb\" (UID: \"475ed6c9-e19e-40e1-be90-dfe4c5ed9345\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"
Sep 18 17:30:09.939456 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.939305    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-s28xb\" (UID: \"475ed6c9-e19e-40e1-be90-dfe4c5ed9345\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"
Sep 18 17:30:09.939456 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.939324    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbcs9\" (UniqueName: \"kubernetes.io/projected/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-kube-api-access-cbcs9\") pod \"scheduler-inline-config-test-kserve-8655bc757-s28xb\" (UID: \"475ed6c9-e19e-40e1-be90-dfe4c5ed9345\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"
Sep 18 17:30:09.939456 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.939342    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-s28xb\" (UID: \"475ed6c9-e19e-40e1-be90-dfe4c5ed9345\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"
Sep 18 17:30:09.939456 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.939370    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-s28xb\" (UID: \"475ed6c9-e19e-40e1-be90-dfe4c5ed9345\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"
Sep 18 17:30:09.939737 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.939696    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-s28xb\" (UID: \"475ed6c9-e19e-40e1-be90-dfe4c5ed9345\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"
Sep 18 17:30:09.939784 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.939750    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-s28xb\" (UID: \"475ed6c9-e19e-40e1-be90-dfe4c5ed9345\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"
Sep 18 17:30:09.939820 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.939786    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-s28xb\" (UID: \"475ed6c9-e19e-40e1-be90-dfe4c5ed9345\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"
Sep 18 17:30:09.941870 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.941841    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-s28xb\" (UID: \"475ed6c9-e19e-40e1-be90-dfe4c5ed9345\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"
Sep 18 17:30:09.942030 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.942014    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-s28xb\" (UID: \"475ed6c9-e19e-40e1-be90-dfe4c5ed9345\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"
Sep 18 17:30:09.948832 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:09.948813    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbcs9\" (UniqueName: \"kubernetes.io/projected/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-kube-api-access-cbcs9\") pod \"scheduler-inline-config-test-kserve-8655bc757-s28xb\" (UID: \"475ed6c9-e19e-40e1-be90-dfe4c5ed9345\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"
Sep 18 17:30:10.093661 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:10.093592    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"
Sep 18 17:30:10.363260 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:10.363199    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"]
Sep 18 17:30:10.366028 ip-10-0-133-147 kubenswrapper[2585]: W0918 17:30:10.366003    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod475ed6c9_e19e_40e1_be90_dfe4c5ed9345.slice/crio-d06935c69f1d594e6acd20734d9ade897af29bc6eef273534214616c9530ca43 WatchSource:0}: Error finding container d06935c69f1d594e6acd20734d9ade897af29bc6eef273534214616c9530ca43: Status 404 returned error can't find the container with id d06935c69f1d594e6acd20734d9ade897af29bc6eef273534214616c9530ca43
Sep 18 17:30:10.393956 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:10.393930    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb" event={"ID":"475ed6c9-e19e-40e1-be90-dfe4c5ed9345","Type":"ContainerStarted","Data":"d06935c69f1d594e6acd20734d9ade897af29bc6eef273534214616c9530ca43"}
Sep 18 17:30:11.398992 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:11.398947    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb" event={"ID":"475ed6c9-e19e-40e1-be90-dfe4c5ed9345","Type":"ContainerStarted","Data":"58becf757be2ce27ab8999d13af02d8516d0490442661e399467c6c0f706bb0d"}
Sep 18 17:30:11.414569 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:11.414519    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb" podStartSLOduration=2.414502069 podStartE2EDuration="2.414502069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:30:11.414276397 +0000 UTC m=+4451.197278425" watchObservedRunningTime="2026-09-18 17:30:11.414502069 +0000 UTC m=+4451.197504094"
Sep 18 17:30:12.094383 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:12.094337    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"
Sep 18 17:30:20.094665 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:20.094611    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"
Sep 18 17:30:20.107276 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:20.107242    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"
Sep 18 17:30:20.443287 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:20.443199    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"
Sep 18 17:30:43.349510 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:43.349313    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"]
Sep 18 17:30:43.359809 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:30:43.359775    2585 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 17:30:43.359809 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 17:30:43.359809 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 17:30:43.359809 ip-10-0-133-147 kubenswrapper[2585]:  > containerID="58becf757be2ce27ab8999d13af02d8516d0490442661e399467c6c0f706bb0d" cmd=["/bin/sleep","15"]
Sep 18 17:30:43.360115 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:43.359826    2585 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb" message="" err=<
Sep 18 17:30:43.360115 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 17:30:43.360115 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 17:30:43.360115 ip-10-0-133-147 kubenswrapper[2585]:  >
Sep 18 17:30:43.360115 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:30:43.359838    2585 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 17:30:43.360115 ip-10-0-133-147 kubenswrapper[2585]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 17:30:43.360115 ip-10-0-133-147 kubenswrapper[2585]:         , stdout: , stderr: , exit code -1
Sep 18 17:30:43.360115 ip-10-0-133-147 kubenswrapper[2585]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb" podUID="475ed6c9-e19e-40e1-be90-dfe4c5ed9345" containerName="main" containerID="cri-o://58becf757be2ce27ab8999d13af02d8516d0490442661e399467c6c0f706bb0d"
Sep 18 17:30:43.360115 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:43.359886    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb" podUID="475ed6c9-e19e-40e1-be90-dfe4c5ed9345" containerName="main" containerID="cri-o://58becf757be2ce27ab8999d13af02d8516d0490442661e399467c6c0f706bb0d" gracePeriod=60
Sep 18 17:30:43.433542 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:43.433498    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb" podUID="475ed6c9-e19e-40e1-be90-dfe4c5ed9345" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 18 17:30:43.697240 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:43.697217    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"
Sep 18 17:30:43.821044 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:43.821008    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-dshm\") pod \"475ed6c9-e19e-40e1-be90-dfe4c5ed9345\" (UID: \"475ed6c9-e19e-40e1-be90-dfe4c5ed9345\") "
Sep 18 17:30:43.821196 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:43.821077    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-model-cache\") pod \"475ed6c9-e19e-40e1-be90-dfe4c5ed9345\" (UID: \"475ed6c9-e19e-40e1-be90-dfe4c5ed9345\") "
Sep 18 17:30:43.821196 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:43.821100    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-home\") pod \"475ed6c9-e19e-40e1-be90-dfe4c5ed9345\" (UID: \"475ed6c9-e19e-40e1-be90-dfe4c5ed9345\") "
Sep 18 17:30:43.821196 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:43.821131    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-tls-certs\") pod \"475ed6c9-e19e-40e1-be90-dfe4c5ed9345\" (UID: \"475ed6c9-e19e-40e1-be90-dfe4c5ed9345\") "
Sep 18 17:30:43.821196 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:43.821164    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-tmp-dir\") pod \"475ed6c9-e19e-40e1-be90-dfe4c5ed9345\" (UID: \"475ed6c9-e19e-40e1-be90-dfe4c5ed9345\") "
Sep 18 17:30:43.821196 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:43.821191    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbcs9\" (UniqueName: \"kubernetes.io/projected/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-kube-api-access-cbcs9\") pod \"475ed6c9-e19e-40e1-be90-dfe4c5ed9345\" (UID: \"475ed6c9-e19e-40e1-be90-dfe4c5ed9345\") "
Sep 18 17:30:43.821517 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:43.821383    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-model-cache" (OuterVolumeSpecName: "model-cache") pod "475ed6c9-e19e-40e1-be90-dfe4c5ed9345" (UID: "475ed6c9-e19e-40e1-be90-dfe4c5ed9345"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:30:43.821517 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:43.821427    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-home" (OuterVolumeSpecName: "home") pod "475ed6c9-e19e-40e1-be90-dfe4c5ed9345" (UID: "475ed6c9-e19e-40e1-be90-dfe4c5ed9345"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:30:43.821517 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:43.821441    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "475ed6c9-e19e-40e1-be90-dfe4c5ed9345" (UID: "475ed6c9-e19e-40e1-be90-dfe4c5ed9345"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:30:43.823643 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:43.823616    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "475ed6c9-e19e-40e1-be90-dfe4c5ed9345" (UID: "475ed6c9-e19e-40e1-be90-dfe4c5ed9345"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:30:43.823760 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:43.823648    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-dshm" (OuterVolumeSpecName: "dshm") pod "475ed6c9-e19e-40e1-be90-dfe4c5ed9345" (UID: "475ed6c9-e19e-40e1-be90-dfe4c5ed9345"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:30:43.823760 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:43.823661    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-kube-api-access-cbcs9" (OuterVolumeSpecName: "kube-api-access-cbcs9") pod "475ed6c9-e19e-40e1-be90-dfe4c5ed9345" (UID: "475ed6c9-e19e-40e1-be90-dfe4c5ed9345"). InnerVolumeSpecName "kube-api-access-cbcs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:30:43.922607 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:43.922524    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-model-cache\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:30:43.922607 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:43.922551    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-home\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:30:43.922607 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:43.922560    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:30:43.922607 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:43.922569    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-tmp-dir\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:30:43.922607 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:43.922577    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbcs9\" (UniqueName: \"kubernetes.io/projected/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-kube-api-access-cbcs9\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:30:43.922607 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:43.922587    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/475ed6c9-e19e-40e1-be90-dfe4c5ed9345-dshm\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:30:44.524262 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:44.524220    2585 generic.go:356] "Generic (PLEG): container finished" podID="475ed6c9-e19e-40e1-be90-dfe4c5ed9345" containerID="58becf757be2ce27ab8999d13af02d8516d0490442661e399467c6c0f706bb0d" exitCode=0
Sep 18 17:30:44.524663 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:44.524282    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"
Sep 18 17:30:44.524663 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:44.524301    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb" event={"ID":"475ed6c9-e19e-40e1-be90-dfe4c5ed9345","Type":"ContainerDied","Data":"58becf757be2ce27ab8999d13af02d8516d0490442661e399467c6c0f706bb0d"}
Sep 18 17:30:44.524663 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:44.524363    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb" event={"ID":"475ed6c9-e19e-40e1-be90-dfe4c5ed9345","Type":"ContainerDied","Data":"d06935c69f1d594e6acd20734d9ade897af29bc6eef273534214616c9530ca43"}
Sep 18 17:30:44.524663 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:44.524388    2585 scope.go:117] "RemoveContainer" containerID="58becf757be2ce27ab8999d13af02d8516d0490442661e399467c6c0f706bb0d"
Sep 18 17:30:44.533168 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:44.533150    2585 scope.go:117] "RemoveContainer" containerID="58becf757be2ce27ab8999d13af02d8516d0490442661e399467c6c0f706bb0d"
Sep 18 17:30:44.533462 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:30:44.533441    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58becf757be2ce27ab8999d13af02d8516d0490442661e399467c6c0f706bb0d\": container with ID starting with 58becf757be2ce27ab8999d13af02d8516d0490442661e399467c6c0f706bb0d not found: ID does not exist" containerID="58becf757be2ce27ab8999d13af02d8516d0490442661e399467c6c0f706bb0d"
Sep 18 17:30:44.533524 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:44.533475    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58becf757be2ce27ab8999d13af02d8516d0490442661e399467c6c0f706bb0d"} err="failed to get container status \"58becf757be2ce27ab8999d13af02d8516d0490442661e399467c6c0f706bb0d\": rpc error: code = NotFound desc = could not find container \"58becf757be2ce27ab8999d13af02d8516d0490442661e399467c6c0f706bb0d\": container with ID starting with 58becf757be2ce27ab8999d13af02d8516d0490442661e399467c6c0f706bb0d not found: ID does not exist"
Sep 18 17:30:44.540011 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:44.539989    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"]
Sep 18 17:30:44.544431 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:44.544410    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-s28xb"]
Sep 18 17:30:46.392598 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:30:46.392560    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="475ed6c9-e19e-40e1-be90-dfe4c5ed9345" path="/var/lib/kubelet/pods/475ed6c9-e19e-40e1-be90-dfe4c5ed9345/volumes"
Sep 18 17:31:02.676891 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:02.676778    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 17:31:02.692410 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:02.680932    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 17:31:02.692410 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:02.686560    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 17:31:02.692410 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:02.689808    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 17:31:05.706349 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.706313    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"]
Sep 18 17:31:05.706786 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.706608    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="475ed6c9-e19e-40e1-be90-dfe4c5ed9345" containerName="main"
Sep 18 17:31:05.706786 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.706620    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="475ed6c9-e19e-40e1-be90-dfe4c5ed9345" containerName="main"
Sep 18 17:31:05.706786 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.706681    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="475ed6c9-e19e-40e1-be90-dfe4c5ed9345" containerName="main"
Sep 18 17:31:05.710846 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.710826    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"
Sep 18 17:31:05.713428 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.713400    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 18 17:31:05.713558 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.713404    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 18 17:31:05.713558 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.713439    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 18 17:31:05.713668 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.713595    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 17:31:05.713737 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.713682    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 18 17:31:05.713737 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.713710    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-sfd7z\""
Sep 18 17:31:05.721229 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.721209    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"]
Sep 18 17:31:05.785469 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.785436    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a62f043-246b-490d-81ac-4c6de96f91e6-tmp-dir\") pod \"prestop-hook-test-kserve-d957cb699-qwwwx\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"
Sep 18 17:31:05.785602 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.785479    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a62f043-246b-490d-81ac-4c6de96f91e6-cabundle-cert\") pod \"prestop-hook-test-kserve-d957cb699-qwwwx\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"
Sep 18 17:31:05.785602 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.785523    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a62f043-246b-490d-81ac-4c6de96f91e6-home\") pod \"prestop-hook-test-kserve-d957cb699-qwwwx\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"
Sep 18 17:31:05.785602 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.785568    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a62f043-246b-490d-81ac-4c6de96f91e6-dshm\") pod \"prestop-hook-test-kserve-d957cb699-qwwwx\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"
Sep 18 17:31:05.785602 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.785590    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a62f043-246b-490d-81ac-4c6de96f91e6-tls-certs\") pod \"prestop-hook-test-kserve-d957cb699-qwwwx\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"
Sep 18 17:31:05.785752 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.785630    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a62f043-246b-490d-81ac-4c6de96f91e6-model-cache\") pod \"prestop-hook-test-kserve-d957cb699-qwwwx\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"
Sep 18 17:31:05.785752 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.785669    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a62f043-246b-490d-81ac-4c6de96f91e6-kserve-provision-location\") pod \"prestop-hook-test-kserve-d957cb699-qwwwx\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"
Sep 18 17:31:05.785752 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.785691    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csnrw\" (UniqueName: \"kubernetes.io/projected/1a62f043-246b-490d-81ac-4c6de96f91e6-kube-api-access-csnrw\") pod \"prestop-hook-test-kserve-d957cb699-qwwwx\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"
Sep 18 17:31:05.886087 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.886041    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a62f043-246b-490d-81ac-4c6de96f91e6-tmp-dir\") pod \"prestop-hook-test-kserve-d957cb699-qwwwx\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"
Sep 18 17:31:05.886087 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.886093    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a62f043-246b-490d-81ac-4c6de96f91e6-cabundle-cert\") pod \"prestop-hook-test-kserve-d957cb699-qwwwx\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"
Sep 18 17:31:05.886295 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.886135    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a62f043-246b-490d-81ac-4c6de96f91e6-home\") pod \"prestop-hook-test-kserve-d957cb699-qwwwx\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"
Sep 18 17:31:05.886295 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.886166    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a62f043-246b-490d-81ac-4c6de96f91e6-dshm\") pod \"prestop-hook-test-kserve-d957cb699-qwwwx\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"
Sep 18 17:31:05.886368 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.886306    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a62f043-246b-490d-81ac-4c6de96f91e6-tls-certs\") pod \"prestop-hook-test-kserve-d957cb699-qwwwx\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"
Sep 18 17:31:05.886414 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.886390    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a62f043-246b-490d-81ac-4c6de96f91e6-model-cache\") pod \"prestop-hook-test-kserve-d957cb699-qwwwx\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"
Sep 18 17:31:05.886458 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.886430    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a62f043-246b-490d-81ac-4c6de96f91e6-kserve-provision-location\") pod \"prestop-hook-test-kserve-d957cb699-qwwwx\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"
Sep 18 17:31:05.886516 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.886465    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csnrw\" (UniqueName: \"kubernetes.io/projected/1a62f043-246b-490d-81ac-4c6de96f91e6-kube-api-access-csnrw\") pod \"prestop-hook-test-kserve-d957cb699-qwwwx\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"
Sep 18 17:31:05.886574 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.886521    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a62f043-246b-490d-81ac-4c6de96f91e6-tmp-dir\") pod \"prestop-hook-test-kserve-d957cb699-qwwwx\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"
Sep 18 17:31:05.886630 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.886580    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a62f043-246b-490d-81ac-4c6de96f91e6-home\") pod \"prestop-hook-test-kserve-d957cb699-qwwwx\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"
Sep 18 17:31:05.886859 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.886818    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a62f043-246b-490d-81ac-4c6de96f91e6-kserve-provision-location\") pod \"prestop-hook-test-kserve-d957cb699-qwwwx\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"
Sep 18 17:31:05.887025 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.886933    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a62f043-246b-490d-81ac-4c6de96f91e6-model-cache\") pod \"prestop-hook-test-kserve-d957cb699-qwwwx\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"
Sep 18 17:31:05.887025 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.886949    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a62f043-246b-490d-81ac-4c6de96f91e6-cabundle-cert\") pod \"prestop-hook-test-kserve-d957cb699-qwwwx\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"
Sep 18 17:31:05.888599 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.888579    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a62f043-246b-490d-81ac-4c6de96f91e6-dshm\") pod \"prestop-hook-test-kserve-d957cb699-qwwwx\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"
Sep 18 17:31:05.888971 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.888955    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a62f043-246b-490d-81ac-4c6de96f91e6-tls-certs\") pod \"prestop-hook-test-kserve-d957cb699-qwwwx\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"
Sep 18 17:31:05.894945 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:05.894896    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csnrw\" (UniqueName: \"kubernetes.io/projected/1a62f043-246b-490d-81ac-4c6de96f91e6-kube-api-access-csnrw\") pod \"prestop-hook-test-kserve-d957cb699-qwwwx\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"
Sep 18 17:31:06.021631 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:06.021596    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"
Sep 18 17:31:06.298434 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:06.298340    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"]
Sep 18 17:31:06.301102 ip-10-0-133-147 kubenswrapper[2585]: W0918 17:31:06.301073    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a62f043_246b_490d_81ac_4c6de96f91e6.slice/crio-6d9cce7336012be789f815fcbcb87fe66cd9f73ed396e6332fdedad36cf6d96b WatchSource:0}: Error finding container 6d9cce7336012be789f815fcbcb87fe66cd9f73ed396e6332fdedad36cf6d96b: Status 404 returned error can't find the container with id 6d9cce7336012be789f815fcbcb87fe66cd9f73ed396e6332fdedad36cf6d96b
Sep 18 17:31:06.612218 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:06.612113    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx" event={"ID":"1a62f043-246b-490d-81ac-4c6de96f91e6","Type":"ContainerStarted","Data":"bc79229e1f8d9168bf49070ccfd212f66ef848f277c8fd30a5be10345b9d2aa5"}
Sep 18 17:31:06.612218 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:06.612186    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx" event={"ID":"1a62f043-246b-490d-81ac-4c6de96f91e6","Type":"ContainerStarted","Data":"6d9cce7336012be789f815fcbcb87fe66cd9f73ed396e6332fdedad36cf6d96b"}
Sep 18 17:31:08.621131 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:08.621100    2585 generic.go:356] "Generic (PLEG): container finished" podID="1a62f043-246b-490d-81ac-4c6de96f91e6" containerID="bc79229e1f8d9168bf49070ccfd212f66ef848f277c8fd30a5be10345b9d2aa5" exitCode=0
Sep 18 17:31:08.621576 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:08.621174    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx" event={"ID":"1a62f043-246b-490d-81ac-4c6de96f91e6","Type":"ContainerDied","Data":"bc79229e1f8d9168bf49070ccfd212f66ef848f277c8fd30a5be10345b9d2aa5"}
Sep 18 17:31:09.627976 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:09.627940    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx" event={"ID":"1a62f043-246b-490d-81ac-4c6de96f91e6","Type":"ContainerStarted","Data":"8c4aadcdc5f43a9023850074e31ed2036305d99e5e9c02b09173c7f91755e521"}
Sep 18 17:31:09.652623 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:09.652554    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx" podStartSLOduration=2.574927871 podStartE2EDuration="4.652532343s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.077604472s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:31:09.650535389 +0000 UTC m=+4509.433537412" watchObservedRunningTime="2026-09-18 17:31:09.652532343 +0000 UTC m=+4509.435534370"
Sep 18 17:31:10.021958 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:10.021907    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"
Sep 18 17:31:16.021946 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:16.021888    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"
Sep 18 17:31:16.023450 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:16.023416    2585 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx" podUID="1a62f043-246b-490d-81ac-4c6de96f91e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Sep 18 17:31:24.950243 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:24.950207    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"]
Sep 18 17:31:39.973105 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:31:39.973056    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx" podUID="1a62f043-246b-490d-81ac-4c6de96f91e6" containerName="main" containerID="cri-o://8c4aadcdc5f43a9023850074e31ed2036305d99e5e9c02b09173c7f91755e521" gracePeriod=45
Sep 18 17:32:25.616734 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.616707    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-d957cb699-qwwwx_1a62f043-246b-490d-81ac-4c6de96f91e6/main/0.log"
Sep 18 17:32:25.619419 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.619401    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"
Sep 18 17:32:25.731155 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.731119    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a62f043-246b-490d-81ac-4c6de96f91e6-tls-certs\") pod \"1a62f043-246b-490d-81ac-4c6de96f91e6\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") "
Sep 18 17:32:25.731155 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.731155    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a62f043-246b-490d-81ac-4c6de96f91e6-model-cache\") pod \"1a62f043-246b-490d-81ac-4c6de96f91e6\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") "
Sep 18 17:32:25.731396 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.731179    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a62f043-246b-490d-81ac-4c6de96f91e6-tmp-dir\") pod \"1a62f043-246b-490d-81ac-4c6de96f91e6\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") "
Sep 18 17:32:25.731396 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.731229    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a62f043-246b-490d-81ac-4c6de96f91e6-cabundle-cert\") pod \"1a62f043-246b-490d-81ac-4c6de96f91e6\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") "
Sep 18 17:32:25.731396 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.731246    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a62f043-246b-490d-81ac-4c6de96f91e6-dshm\") pod \"1a62f043-246b-490d-81ac-4c6de96f91e6\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") "
Sep 18 17:32:25.731396 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.731270    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csnrw\" (UniqueName: \"kubernetes.io/projected/1a62f043-246b-490d-81ac-4c6de96f91e6-kube-api-access-csnrw\") pod \"1a62f043-246b-490d-81ac-4c6de96f91e6\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") "
Sep 18 17:32:25.731396 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.731298    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a62f043-246b-490d-81ac-4c6de96f91e6-kserve-provision-location\") pod \"1a62f043-246b-490d-81ac-4c6de96f91e6\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") "
Sep 18 17:32:25.731396 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.731339    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a62f043-246b-490d-81ac-4c6de96f91e6-home\") pod \"1a62f043-246b-490d-81ac-4c6de96f91e6\" (UID: \"1a62f043-246b-490d-81ac-4c6de96f91e6\") "
Sep 18 17:32:25.731692 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.731440    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a62f043-246b-490d-81ac-4c6de96f91e6-model-cache" (OuterVolumeSpecName: "model-cache") pod "1a62f043-246b-490d-81ac-4c6de96f91e6" (UID: "1a62f043-246b-490d-81ac-4c6de96f91e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:32:25.731692 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.731553    2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a62f043-246b-490d-81ac-4c6de96f91e6-model-cache\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:32:25.731692 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.731598    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a62f043-246b-490d-81ac-4c6de96f91e6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1a62f043-246b-490d-81ac-4c6de96f91e6" (UID: "1a62f043-246b-490d-81ac-4c6de96f91e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:32:25.731879 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.731744    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a62f043-246b-490d-81ac-4c6de96f91e6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1a62f043-246b-490d-81ac-4c6de96f91e6" (UID: "1a62f043-246b-490d-81ac-4c6de96f91e6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:32:25.731879 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.731783    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a62f043-246b-490d-81ac-4c6de96f91e6-home" (OuterVolumeSpecName: "home") pod "1a62f043-246b-490d-81ac-4c6de96f91e6" (UID: "1a62f043-246b-490d-81ac-4c6de96f91e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:32:25.733730 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.733697    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a62f043-246b-490d-81ac-4c6de96f91e6-kube-api-access-csnrw" (OuterVolumeSpecName: "kube-api-access-csnrw") pod "1a62f043-246b-490d-81ac-4c6de96f91e6" (UID: "1a62f043-246b-490d-81ac-4c6de96f91e6"). InnerVolumeSpecName "kube-api-access-csnrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:32:25.733730 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.733710    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a62f043-246b-490d-81ac-4c6de96f91e6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a62f043-246b-490d-81ac-4c6de96f91e6" (UID: "1a62f043-246b-490d-81ac-4c6de96f91e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:32:25.733886 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.733784    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a62f043-246b-490d-81ac-4c6de96f91e6-dshm" (OuterVolumeSpecName: "dshm") pod "1a62f043-246b-490d-81ac-4c6de96f91e6" (UID: "1a62f043-246b-490d-81ac-4c6de96f91e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:32:25.754590 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.754532    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a62f043-246b-490d-81ac-4c6de96f91e6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1a62f043-246b-490d-81ac-4c6de96f91e6" (UID: "1a62f043-246b-490d-81ac-4c6de96f91e6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:32:25.832135 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.832105    2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a62f043-246b-490d-81ac-4c6de96f91e6-tmp-dir\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:32:25.832135 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.832132    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a62f043-246b-490d-81ac-4c6de96f91e6-cabundle-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:32:25.832292 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.832144    2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a62f043-246b-490d-81ac-4c6de96f91e6-dshm\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:32:25.832292 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.832157    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csnrw\" (UniqueName: \"kubernetes.io/projected/1a62f043-246b-490d-81ac-4c6de96f91e6-kube-api-access-csnrw\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:32:25.832292 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.832171    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a62f043-246b-490d-81ac-4c6de96f91e6-kserve-provision-location\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:32:25.832292 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.832182    2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a62f043-246b-490d-81ac-4c6de96f91e6-home\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:32:25.832292 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.832193    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a62f043-246b-490d-81ac-4c6de96f91e6-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:32:25.949934 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.949887    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-d957cb699-qwwwx_1a62f043-246b-490d-81ac-4c6de96f91e6/main/0.log"
Sep 18 17:32:25.950272 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.950250    2585 generic.go:356] "Generic (PLEG): container finished" podID="1a62f043-246b-490d-81ac-4c6de96f91e6" containerID="8c4aadcdc5f43a9023850074e31ed2036305d99e5e9c02b09173c7f91755e521" exitCode=137
Sep 18 17:32:25.950326 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.950308    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"
Sep 18 17:32:25.950404 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.950312    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx" event={"ID":"1a62f043-246b-490d-81ac-4c6de96f91e6","Type":"ContainerDied","Data":"8c4aadcdc5f43a9023850074e31ed2036305d99e5e9c02b09173c7f91755e521"}
Sep 18 17:32:25.950441 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.950433    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx" event={"ID":"1a62f043-246b-490d-81ac-4c6de96f91e6","Type":"ContainerDied","Data":"6d9cce7336012be789f815fcbcb87fe66cd9f73ed396e6332fdedad36cf6d96b"}
Sep 18 17:32:25.950475 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.950454    2585 scope.go:117] "RemoveContainer" containerID="8c4aadcdc5f43a9023850074e31ed2036305d99e5e9c02b09173c7f91755e521"
Sep 18 17:32:25.964027 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.964006    2585 scope.go:117] "RemoveContainer" containerID="bc79229e1f8d9168bf49070ccfd212f66ef848f277c8fd30a5be10345b9d2aa5"
Sep 18 17:32:25.971846 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.971829    2585 scope.go:117] "RemoveContainer" containerID="8c4aadcdc5f43a9023850074e31ed2036305d99e5e9c02b09173c7f91755e521"
Sep 18 17:32:25.972158 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:32:25.972135    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c4aadcdc5f43a9023850074e31ed2036305d99e5e9c02b09173c7f91755e521\": container with ID starting with 8c4aadcdc5f43a9023850074e31ed2036305d99e5e9c02b09173c7f91755e521 not found: ID does not exist" containerID="8c4aadcdc5f43a9023850074e31ed2036305d99e5e9c02b09173c7f91755e521"
Sep 18 17:32:25.972256 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.972163    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c4aadcdc5f43a9023850074e31ed2036305d99e5e9c02b09173c7f91755e521"} err="failed to get container status \"8c4aadcdc5f43a9023850074e31ed2036305d99e5e9c02b09173c7f91755e521\": rpc error: code = NotFound desc = could not find container \"8c4aadcdc5f43a9023850074e31ed2036305d99e5e9c02b09173c7f91755e521\": container with ID starting with 8c4aadcdc5f43a9023850074e31ed2036305d99e5e9c02b09173c7f91755e521 not found: ID does not exist"
Sep 18 17:32:25.972256 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.972178    2585 scope.go:117] "RemoveContainer" containerID="bc79229e1f8d9168bf49070ccfd212f66ef848f277c8fd30a5be10345b9d2aa5"
Sep 18 17:32:25.972452 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:32:25.972432    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc79229e1f8d9168bf49070ccfd212f66ef848f277c8fd30a5be10345b9d2aa5\": container with ID starting with bc79229e1f8d9168bf49070ccfd212f66ef848f277c8fd30a5be10345b9d2aa5 not found: ID does not exist" containerID="bc79229e1f8d9168bf49070ccfd212f66ef848f277c8fd30a5be10345b9d2aa5"
Sep 18 17:32:25.972503 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.972461    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc79229e1f8d9168bf49070ccfd212f66ef848f277c8fd30a5be10345b9d2aa5"} err="failed to get container status \"bc79229e1f8d9168bf49070ccfd212f66ef848f277c8fd30a5be10345b9d2aa5\": rpc error: code = NotFound desc = could not find container \"bc79229e1f8d9168bf49070ccfd212f66ef848f277c8fd30a5be10345b9d2aa5\": container with ID starting with bc79229e1f8d9168bf49070ccfd212f66ef848f277c8fd30a5be10345b9d2aa5 not found: ID does not exist"
Sep 18 17:32:25.976871 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.976849    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"]
Sep 18 17:32:25.980684 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:25.980661    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-qwwwx"]
Sep 18 17:32:26.391951 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:26.391897    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a62f043-246b-490d-81ac-4c6de96f91e6" path="/var/lib/kubelet/pods/1a62f043-246b-490d-81ac-4c6de96f91e6/volumes"
Sep 18 17:32:29.780248 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:29.780219    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs"]
Sep 18 17:32:29.780640 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:29.780499    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a62f043-246b-490d-81ac-4c6de96f91e6" containerName="main"
Sep 18 17:32:29.780640 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:29.780511    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a62f043-246b-490d-81ac-4c6de96f91e6" containerName="main"
Sep 18 17:32:29.780640 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:29.780526    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a62f043-246b-490d-81ac-4c6de96f91e6" containerName="storage-initializer"
Sep 18 17:32:29.780640 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:29.780532    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a62f043-246b-490d-81ac-4c6de96f91e6" containerName="storage-initializer"
Sep 18 17:32:29.780640 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:29.780581    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a62f043-246b-490d-81ac-4c6de96f91e6" containerName="main"
Sep 18 17:32:29.784755 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:29.784728    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs"
Sep 18 17:32:29.787517 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:29.787491    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-lvz65\""
Sep 18 17:32:29.787662 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:29.787554    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 18 17:32:29.788905 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:29.788888    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 18 17:32:29.789009 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:29.788934    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 18 17:32:29.789054 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:29.789018    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-657px\""
Sep 18 17:32:29.791666 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:29.791647    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs"]
Sep 18 17:32:29.862028 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:29.862003    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1744dda-7ebc-475c-a440-b4c958fb64af-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs\" (UID: \"a1744dda-7ebc-475c-a440-b4c958fb64af\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs"
Sep 18 17:32:29.862185 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:29.862054    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zrc5\" (UniqueName: \"kubernetes.io/projected/a1744dda-7ebc-475c-a440-b4c958fb64af-kube-api-access-8zrc5\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs\" (UID: \"a1744dda-7ebc-475c-a440-b4c958fb64af\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs"
Sep 18 17:32:29.963118 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:29.963078    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zrc5\" (UniqueName: \"kubernetes.io/projected/a1744dda-7ebc-475c-a440-b4c958fb64af-kube-api-access-8zrc5\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs\" (UID: \"a1744dda-7ebc-475c-a440-b4c958fb64af\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs"
Sep 18 17:32:29.963290 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:29.963171    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1744dda-7ebc-475c-a440-b4c958fb64af-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs\" (UID: \"a1744dda-7ebc-475c-a440-b4c958fb64af\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs"
Sep 18 17:32:29.966065 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:29.966035    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1744dda-7ebc-475c-a440-b4c958fb64af-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs\" (UID: \"a1744dda-7ebc-475c-a440-b4c958fb64af\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs"
Sep 18 17:32:29.972286 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:29.972263    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zrc5\" (UniqueName: \"kubernetes.io/projected/a1744dda-7ebc-475c-a440-b4c958fb64af-kube-api-access-8zrc5\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs\" (UID: \"a1744dda-7ebc-475c-a440-b4c958fb64af\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs"
Sep 18 17:32:30.096200 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:30.096130    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs"
Sep 18 17:32:30.370523 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:30.370497    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs"]
Sep 18 17:32:30.372974 ip-10-0-133-147 kubenswrapper[2585]: W0918 17:32:30.372942    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1744dda_7ebc_475c_a440_b4c958fb64af.slice/crio-e4c0a1127be1adf281c0df13511cbca23218840cdae3b3180d17e23d42844f5b WatchSource:0}: Error finding container e4c0a1127be1adf281c0df13511cbca23218840cdae3b3180d17e23d42844f5b: Status 404 returned error can't find the container with id e4c0a1127be1adf281c0df13511cbca23218840cdae3b3180d17e23d42844f5b
Sep 18 17:32:30.972735 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:30.972695    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs" event={"ID":"a1744dda-7ebc-475c-a440-b4c958fb64af","Type":"ContainerStarted","Data":"9d3e6dde4ca5d150d24072df49246b98f404dad3680875a644bf1e7bab180835"}
Sep 18 17:32:30.973127 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:30.972752    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs" event={"ID":"a1744dda-7ebc-475c-a440-b4c958fb64af","Type":"ContainerStarted","Data":"e4c0a1127be1adf281c0df13511cbca23218840cdae3b3180d17e23d42844f5b"}
Sep 18 17:32:30.973127 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:30.972811    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs"
Sep 18 17:32:30.990319 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:32:30.990273    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs" podStartSLOduration=1.990241632 podStartE2EDuration="1.990241632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:32:30.988606217 +0000 UTC m=+4590.771608240" watchObservedRunningTime="2026-09-18 17:32:30.990241632 +0000 UTC m=+4590.773243690"
Sep 18 17:33:01.978884 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:33:01.978853    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs"
Sep 18 17:33:28.327869 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:33:28.327822    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs"]
Sep 18 17:33:31.979155 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:33:31.979116    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs" podUID="a1744dda-7ebc-475c-a440-b4c958fb64af" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:33:41.978800 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:33:41.978709    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs" podUID="a1744dda-7ebc-475c-a440-b4c958fb64af" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:33:43.353407 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:33:43.353358    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs" podUID="a1744dda-7ebc-475c-a440-b4c958fb64af" containerName="main" containerID="cri-o://9d3e6dde4ca5d150d24072df49246b98f404dad3680875a644bf1e7bab180835" gracePeriod=45
Sep 18 17:33:51.979287 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:33:51.979245    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs" podUID="a1744dda-7ebc-475c-a440-b4c958fb64af" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:33:51.979737 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:33:51.979375    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs"
Sep 18 17:34:01.978338 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:01.978288    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs" podUID="a1744dda-7ebc-475c-a440-b4c958fb64af" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:34:11.979017 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:11.978967    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs" podUID="a1744dda-7ebc-475c-a440-b4c958fb64af" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 18 17:34:13.598173 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:13.598147    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs"
Sep 18 17:34:13.687034 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:13.686958    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1744dda-7ebc-475c-a440-b4c958fb64af-tls-certs\") pod \"a1744dda-7ebc-475c-a440-b4c958fb64af\" (UID: \"a1744dda-7ebc-475c-a440-b4c958fb64af\") "
Sep 18 17:34:13.687034 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:13.687003    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zrc5\" (UniqueName: \"kubernetes.io/projected/a1744dda-7ebc-475c-a440-b4c958fb64af-kube-api-access-8zrc5\") pod \"a1744dda-7ebc-475c-a440-b4c958fb64af\" (UID: \"a1744dda-7ebc-475c-a440-b4c958fb64af\") "
Sep 18 17:34:13.689405 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:13.689377    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1744dda-7ebc-475c-a440-b4c958fb64af-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a1744dda-7ebc-475c-a440-b4c958fb64af" (UID: "a1744dda-7ebc-475c-a440-b4c958fb64af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:34:13.689521 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:13.689469    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1744dda-7ebc-475c-a440-b4c958fb64af-kube-api-access-8zrc5" (OuterVolumeSpecName: "kube-api-access-8zrc5") pod "a1744dda-7ebc-475c-a440-b4c958fb64af" (UID: "a1744dda-7ebc-475c-a440-b4c958fb64af"). InnerVolumeSpecName "kube-api-access-8zrc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:34:13.788534 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:13.788507    2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1744dda-7ebc-475c-a440-b4c958fb64af-tls-certs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:34:13.788534 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:13.788533    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zrc5\" (UniqueName: \"kubernetes.io/projected/a1744dda-7ebc-475c-a440-b4c958fb64af-kube-api-access-8zrc5\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\""
Sep 18 17:34:14.375877 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:14.375841    2585 generic.go:356] "Generic (PLEG): container finished" podID="a1744dda-7ebc-475c-a440-b4c958fb64af" containerID="9d3e6dde4ca5d150d24072df49246b98f404dad3680875a644bf1e7bab180835" exitCode=0
Sep 18 17:34:14.376093 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:14.375880    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs"
Sep 18 17:34:14.376093 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:14.375955    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs" event={"ID":"a1744dda-7ebc-475c-a440-b4c958fb64af","Type":"ContainerDied","Data":"9d3e6dde4ca5d150d24072df49246b98f404dad3680875a644bf1e7bab180835"}
Sep 18 17:34:14.376093 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:14.376016    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs" event={"ID":"a1744dda-7ebc-475c-a440-b4c958fb64af","Type":"ContainerDied","Data":"e4c0a1127be1adf281c0df13511cbca23218840cdae3b3180d17e23d42844f5b"}
Sep 18 17:34:14.376093 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:14.376044    2585 scope.go:117] "RemoveContainer" containerID="9d3e6dde4ca5d150d24072df49246b98f404dad3680875a644bf1e7bab180835"
Sep 18 17:34:14.385442 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:14.385418    2585 scope.go:117] "RemoveContainer" containerID="9d3e6dde4ca5d150d24072df49246b98f404dad3680875a644bf1e7bab180835"
Sep 18 17:34:14.385704 ip-10-0-133-147 kubenswrapper[2585]: E0918 17:34:14.385683    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d3e6dde4ca5d150d24072df49246b98f404dad3680875a644bf1e7bab180835\": container with ID starting with 9d3e6dde4ca5d150d24072df49246b98f404dad3680875a644bf1e7bab180835 not found: ID does not exist" containerID="9d3e6dde4ca5d150d24072df49246b98f404dad3680875a644bf1e7bab180835"
Sep 18 17:34:14.385772 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:14.385714    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d3e6dde4ca5d150d24072df49246b98f404dad3680875a644bf1e7bab180835"} err="failed to get container status \"9d3e6dde4ca5d150d24072df49246b98f404dad3680875a644bf1e7bab180835\": rpc error: code = NotFound desc = could not find container \"9d3e6dde4ca5d150d24072df49246b98f404dad3680875a644bf1e7bab180835\": container with ID starting with 9d3e6dde4ca5d150d24072df49246b98f404dad3680875a644bf1e7bab180835 not found: ID does not exist"
Sep 18 17:34:14.398895 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:14.398861    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs"]
Sep 18 17:34:14.402806 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:14.402784    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-qpqzs"]
Sep 18 17:34:16.391406 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:16.391369    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1744dda-7ebc-475c-a440-b4c958fb64af" path="/var/lib/kubelet/pods/a1744dda-7ebc-475c-a440-b4c958fb64af/volumes"
Sep 18 17:34:39.791801 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:39.791771    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d_3a522b31-c301-4372-a729-93e898489653/istio-proxy/0.log"
Sep 18 17:34:39.794751 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:39.794731    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d_3a522b31-c301-4372-a729-93e898489653/istio-proxy/1.log"
Sep 18 17:34:40.627353 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:40.627318    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d_3a522b31-c301-4372-a729-93e898489653/istio-proxy/0.log"
Sep 18 17:34:40.629777 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:40.629757    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d_3a522b31-c301-4372-a729-93e898489653/istio-proxy/1.log"
Sep 18 17:34:41.512603 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:41.512576    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-r28wr_dee1080d-684c-452a-a15f-5f39b67634e9/kuadrant-console-plugin/0.log"
Sep 18 17:34:47.526849 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:47.526809    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4jhnn/must-gather-xr98x"]
Sep 18 17:34:47.527446 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:47.527221    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1744dda-7ebc-475c-a440-b4c958fb64af" containerName="main"
Sep 18 17:34:47.527446 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:47.527239    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1744dda-7ebc-475c-a440-b4c958fb64af" containerName="main"
Sep 18 17:34:47.527446 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:47.527329    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1744dda-7ebc-475c-a440-b4c958fb64af" containerName="main"
Sep 18 17:34:47.530296 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:47.530275    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4jhnn/must-gather-xr98x"
Sep 18 17:34:47.533189 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:47.533169    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4jhnn\"/\"default-dockercfg-mqxfb\""
Sep 18 17:34:47.533544 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:47.533520    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4jhnn\"/\"kube-root-ca.crt\""
Sep 18 17:34:47.533651 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:47.533585    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4jhnn\"/\"openshift-service-ca.crt\""
Sep 18 17:34:47.538237 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:47.538213    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4jhnn/must-gather-xr98x"]
Sep 18 17:34:47.644965 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:47.644906    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7c695c39-b8b3-47c2-a558-1d33948cc3d7-must-gather-output\") pod \"must-gather-xr98x\" (UID: \"7c695c39-b8b3-47c2-a558-1d33948cc3d7\") " pod="openshift-must-gather-4jhnn/must-gather-xr98x"
Sep 18 17:34:47.645118 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:47.644993    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sndzw\" (UniqueName: \"kubernetes.io/projected/7c695c39-b8b3-47c2-a558-1d33948cc3d7-kube-api-access-sndzw\") pod \"must-gather-xr98x\" (UID: \"7c695c39-b8b3-47c2-a558-1d33948cc3d7\") " pod="openshift-must-gather-4jhnn/must-gather-xr98x"
Sep 18 17:34:47.746108 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:47.746076    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7c695c39-b8b3-47c2-a558-1d33948cc3d7-must-gather-output\") pod \"must-gather-xr98x\" (UID: \"7c695c39-b8b3-47c2-a558-1d33948cc3d7\") " pod="openshift-must-gather-4jhnn/must-gather-xr98x"
Sep 18 17:34:47.746239 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:47.746125    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sndzw\" (UniqueName: \"kubernetes.io/projected/7c695c39-b8b3-47c2-a558-1d33948cc3d7-kube-api-access-sndzw\") pod \"must-gather-xr98x\" (UID: \"7c695c39-b8b3-47c2-a558-1d33948cc3d7\") " pod="openshift-must-gather-4jhnn/must-gather-xr98x"
Sep 18 17:34:47.746408 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:47.746391    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7c695c39-b8b3-47c2-a558-1d33948cc3d7-must-gather-output\") pod \"must-gather-xr98x\" (UID: \"7c695c39-b8b3-47c2-a558-1d33948cc3d7\") " pod="openshift-must-gather-4jhnn/must-gather-xr98x"
Sep 18 17:34:47.755233 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:47.755208    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sndzw\" (UniqueName: \"kubernetes.io/projected/7c695c39-b8b3-47c2-a558-1d33948cc3d7-kube-api-access-sndzw\") pod \"must-gather-xr98x\" (UID: \"7c695c39-b8b3-47c2-a558-1d33948cc3d7\") " pod="openshift-must-gather-4jhnn/must-gather-xr98x"
Sep 18 17:34:47.840014 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:47.839939    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4jhnn/must-gather-xr98x"
Sep 18 17:34:48.110757 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:48.110666    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4jhnn/must-gather-xr98x"]
Sep 18 17:34:48.113013 ip-10-0-133-147 kubenswrapper[2585]: W0918 17:34:48.112979    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c695c39_b8b3_47c2_a558_1d33948cc3d7.slice/crio-05c2d12f61a02925de09ee65ed543ca6b9639a00e3b7077f950e86fa83866f9e WatchSource:0}: Error finding container 05c2d12f61a02925de09ee65ed543ca6b9639a00e3b7077f950e86fa83866f9e: Status 404 returned error can't find the container with id 05c2d12f61a02925de09ee65ed543ca6b9639a00e3b7077f950e86fa83866f9e
Sep 18 17:34:48.115058 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:48.115039    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 17:34:48.501604 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:48.501569    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4jhnn/must-gather-xr98x" event={"ID":"7c695c39-b8b3-47c2-a558-1d33948cc3d7","Type":"ContainerStarted","Data":"05c2d12f61a02925de09ee65ed543ca6b9639a00e3b7077f950e86fa83866f9e"}
Sep 18 17:34:49.509389 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:49.509344    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4jhnn/must-gather-xr98x" event={"ID":"7c695c39-b8b3-47c2-a558-1d33948cc3d7","Type":"ContainerStarted","Data":"fde6f665c522b83cd4365523d842e521cefd3406957d20dde296af49b173cd4f"}
Sep 18 17:34:49.509975 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:49.509401    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4jhnn/must-gather-xr98x" event={"ID":"7c695c39-b8b3-47c2-a558-1d33948cc3d7","Type":"ContainerStarted","Data":"41669a6c709d5df24db95843d38cda8bcf7e0c0d76f5bc7b4015fb626afb3d99"}
Sep 18 17:34:49.525301 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:49.525246    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4jhnn/must-gather-xr98x" podStartSLOduration=1.430463079 podStartE2EDuration="2.525229519s" totalImagesPullingTime="1.09476644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:34:49.524249155 +0000 UTC m=+4729.307251180" watchObservedRunningTime="2026-09-18 17:34:49.525229519 +0000 UTC m=+4729.308231543"
Sep 18 17:34:50.790031 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:50.789997    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-w8drz_d3ab8ecb-15ee-4992-96d1-eb456a8a0d1d/global-pull-secret-syncer/0.log"
Sep 18 17:34:50.879009 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:50.878979    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rxjxh_e20b935d-4164-44ca-8b01-8bd08e10b3ac/konnectivity-agent/0.log"
Sep 18 17:34:50.982488 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:50.982451    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-147.ec2.internal_72458550952f102887362406515f2735/haproxy/0.log"
Sep 18 17:34:55.859194 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:55.859149    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-r28wr_dee1080d-684c-452a-a15f-5f39b67634e9/kuadrant-console-plugin/0.log"
Sep 18 17:34:57.619149 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:57.619118    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5gwsw_a2a7a976-02e8-47ac-8d0e-b081f8ceb225/node-exporter/0.log"
Sep 18 17:34:57.640038 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:57.640006    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5gwsw_a2a7a976-02e8-47ac-8d0e-b081f8ceb225/kube-rbac-proxy/0.log"
Sep 18 17:34:57.662174 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:57.662135    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5gwsw_a2a7a976-02e8-47ac-8d0e-b081f8ceb225/init-textfile/0.log"
Sep 18 17:34:57.955311 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:57.955228    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-strvg_96d83cf1-2b3e-46e0-9deb-a666cc9fa27d/kube-rbac-proxy-main/0.log"
Sep 18 17:34:57.979987 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:57.979960    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-strvg_96d83cf1-2b3e-46e0-9deb-a666cc9fa27d/kube-rbac-proxy-self/0.log"
Sep 18 17:34:58.004713 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:34:58.004687    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-strvg_96d83cf1-2b3e-46e0-9deb-a666cc9fa27d/openshift-state-metrics/0.log"
Sep 18 17:35:00.323017 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:00.322982    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/2.log"
Sep 18 17:35:00.327708 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:00.327677    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-cfzmp_f0cae29c-d3b9-4c3f-ac8d-d151e05916af/console-operator/3.log"
Sep 18 17:35:00.867405 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:00.867355    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4jhnn/perf-node-gather-daemonset-5p98d"]
Sep 18 17:35:00.871369 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:00.871341    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-5p98d"
Sep 18 17:35:00.881309 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:00.881280    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4jhnn/perf-node-gather-daemonset-5p98d"]
Sep 18 17:35:00.972967 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:00.972905    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/37771198-cbce-4e0a-b84b-b140d829e28d-podres\") pod \"perf-node-gather-daemonset-5p98d\" (UID: \"37771198-cbce-4e0a-b84b-b140d829e28d\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-5p98d"
Sep 18 17:35:00.973300 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:00.973272    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37771198-cbce-4e0a-b84b-b140d829e28d-lib-modules\") pod \"perf-node-gather-daemonset-5p98d\" (UID: \"37771198-cbce-4e0a-b84b-b140d829e28d\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-5p98d"
Sep 18 17:35:00.973459 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:00.973445    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-762gk\" (UniqueName: \"kubernetes.io/projected/37771198-cbce-4e0a-b84b-b140d829e28d-kube-api-access-762gk\") pod \"perf-node-gather-daemonset-5p98d\" (UID: \"37771198-cbce-4e0a-b84b-b140d829e28d\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-5p98d"
Sep 18 17:35:00.973616 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:00.973603    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37771198-cbce-4e0a-b84b-b140d829e28d-sys\") pod \"perf-node-gather-daemonset-5p98d\" (UID: \"37771198-cbce-4e0a-b84b-b140d829e28d\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-5p98d"
Sep 18 17:35:00.973757 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:00.973744    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/37771198-cbce-4e0a-b84b-b140d829e28d-proc\") pod \"perf-node-gather-daemonset-5p98d\" (UID: \"37771198-cbce-4e0a-b84b-b140d829e28d\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-5p98d"
Sep 18 17:35:01.074495 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:01.074458    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/37771198-cbce-4e0a-b84b-b140d829e28d-proc\") pod \"perf-node-gather-daemonset-5p98d\" (UID: \"37771198-cbce-4e0a-b84b-b140d829e28d\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-5p98d"
Sep 18 17:35:01.074703 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:01.074523    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/37771198-cbce-4e0a-b84b-b140d829e28d-podres\") pod \"perf-node-gather-daemonset-5p98d\" (UID: \"37771198-cbce-4e0a-b84b-b140d829e28d\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-5p98d"
Sep 18 17:35:01.074703 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:01.074573    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37771198-cbce-4e0a-b84b-b140d829e28d-lib-modules\") pod \"perf-node-gather-daemonset-5p98d\" (UID: \"37771198-cbce-4e0a-b84b-b140d829e28d\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-5p98d"
Sep 18 17:35:01.074703 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:01.074597    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/37771198-cbce-4e0a-b84b-b140d829e28d-proc\") pod \"perf-node-gather-daemonset-5p98d\" (UID: \"37771198-cbce-4e0a-b84b-b140d829e28d\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-5p98d"
Sep 18 17:35:01.074703 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:01.074610    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-762gk\" (UniqueName: \"kubernetes.io/projected/37771198-cbce-4e0a-b84b-b140d829e28d-kube-api-access-762gk\") pod \"perf-node-gather-daemonset-5p98d\" (UID: \"37771198-cbce-4e0a-b84b-b140d829e28d\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-5p98d"
Sep 18 17:35:01.074703 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:01.074666    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37771198-cbce-4e0a-b84b-b140d829e28d-sys\") pod \"perf-node-gather-daemonset-5p98d\" (UID: \"37771198-cbce-4e0a-b84b-b140d829e28d\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-5p98d"
Sep 18 17:35:01.074972 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:01.074738    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/37771198-cbce-4e0a-b84b-b140d829e28d-podres\") pod \"perf-node-gather-daemonset-5p98d\" (UID: \"37771198-cbce-4e0a-b84b-b140d829e28d\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-5p98d"
Sep 18 17:35:01.074972 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:01.074760    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/37771198-cbce-4e0a-b84b-b140d829e28d-sys\") pod \"perf-node-gather-daemonset-5p98d\" (UID: \"37771198-cbce-4e0a-b84b-b140d829e28d\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-5p98d"
Sep 18 17:35:01.074972 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:01.074738    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/37771198-cbce-4e0a-b84b-b140d829e28d-lib-modules\") pod \"perf-node-gather-daemonset-5p98d\" (UID: \"37771198-cbce-4e0a-b84b-b140d829e28d\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-5p98d"
Sep 18 17:35:01.084292 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:01.084253    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-762gk\" (UniqueName: \"kubernetes.io/projected/37771198-cbce-4e0a-b84b-b140d829e28d-kube-api-access-762gk\") pod \"perf-node-gather-daemonset-5p98d\" (UID: \"37771198-cbce-4e0a-b84b-b140d829e28d\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-5p98d"
Sep 18 17:35:01.185455 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:01.185114    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-5p98d"
Sep 18 17:35:01.482181 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:01.482143    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4jhnn/perf-node-gather-daemonset-5p98d"]
Sep 18 17:35:01.489185 ip-10-0-133-147 kubenswrapper[2585]: W0918 17:35:01.489147    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod37771198_cbce_4e0a_b84b_b140d829e28d.slice/crio-546c662cbf2091ee6c106c19240d3166202950b4d486c74901a85b617ead3903 WatchSource:0}: Error finding container 546c662cbf2091ee6c106c19240d3166202950b4d486c74901a85b617ead3903: Status 404 returned error can't find the container with id 546c662cbf2091ee6c106c19240d3166202950b4d486c74901a85b617ead3903
Sep 18 17:35:01.580361 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:01.580327    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-5p98d" event={"ID":"37771198-cbce-4e0a-b84b-b140d829e28d","Type":"ContainerStarted","Data":"546c662cbf2091ee6c106c19240d3166202950b4d486c74901a85b617ead3903"}
Sep 18 17:35:02.394684 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:02.394659    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-glsl2_b1157877-3544-413a-8893-f143ad8e234d/dns/0.log"
Sep 18 17:35:02.414867 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:02.414837    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-glsl2_b1157877-3544-413a-8893-f143ad8e234d/kube-rbac-proxy/0.log"
Sep 18 17:35:02.586207 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:02.586167    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-5p98d" event={"ID":"37771198-cbce-4e0a-b84b-b140d829e28d","Type":"ContainerStarted","Data":"5a5a8639d014e517877a89b9e149bff73a645f1f4581b3707e3cdab2ab2639e3"}
Sep 18 17:35:02.586609 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:02.586262    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-5p98d"
Sep 18 17:35:02.592121 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:02.592098    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q9svc_895c0864-ed92-4326-b78f-3a85240437c2/dns-node-resolver/0.log"
Sep 18 17:35:02.601218 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:02.601163    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-5p98d" podStartSLOduration=2.6011471520000002 podStartE2EDuration="2.601147152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:35:02.600905596 +0000 UTC m=+4742.383907621" watchObservedRunningTime="2026-09-18 17:35:02.601147152 +0000 UTC m=+4742.384149178"
Sep 18 17:35:03.111110 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:03.111079    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5876889fd5-2dfl4_4adc28b1-0f25-4090-9014-7ca2c46f57cf/registry/0.log"
Sep 18 17:35:03.158827 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:03.158797    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lh2tk_d8e4139d-8ca0-4a9c-92b1-7a6998c5988b/node-ca/0.log"
Sep 18 17:35:04.251770 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:04.251733    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d_3a522b31-c301-4372-a729-93e898489653/istio-proxy/0.log"
Sep 18 17:35:04.255294 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:04.255270    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d_3a522b31-c301-4372-a729-93e898489653/istio-proxy/1.log"
Sep 18 17:35:04.840078 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:04.840045    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fxkzj_4e73cb49-af8f-408b-aaeb-db96fabe4998/serve-healthcheck-canary/0.log"
Sep 18 17:35:05.276806 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:05.276774    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-rsb8k_6f505047-5eda-41d3-8df8-b8c2a31d0001/insights-operator/0.log"
Sep 18 17:35:05.278830 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:05.278808    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-rsb8k_6f505047-5eda-41d3-8df8-b8c2a31d0001/insights-operator/1.log"
Sep 18 17:35:05.438674 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:05.438650    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gvzrz_eb3be0d0-05e3-4f51-8bd0-1cff06299f5a/kube-rbac-proxy/0.log"
Sep 18 17:35:05.460185 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:05.460155    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gvzrz_eb3be0d0-05e3-4f51-8bd0-1cff06299f5a/exporter/0.log"
Sep 18 17:35:05.482764 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:05.482736    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gvzrz_eb3be0d0-05e3-4f51-8bd0-1cff06299f5a/extractor/0.log"
Sep 18 17:35:08.438723 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:08.438695    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6fd4df8954-lhqzz_c098bf08-79fd-499a-88b0-deec4754cc75/manager/0.log"
Sep 18 17:35:08.601980 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:08.601949    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-5p98d"
Sep 18 17:35:09.468570 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:09.468536    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-6888f9f64c-t7dlt_f11daaa2-53bc-4a9d-b447-87a3e8a90d14/manager/0.log"
Sep 18 17:35:10.052570 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:10.052539    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-6dcvf_ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7/s3-init/0.log"
Sep 18 17:35:10.073489 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:10.073462    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-6dcvf_ef5df3ca-2b86-4b0e-abd2-40cc92dd62c7/download-hf-model/0.log"
Sep 18 17:35:10.459944 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:10.459883    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d_3a522b31-c301-4372-a729-93e898489653/istio-proxy/0.log"
Sep 18 17:35:10.464007 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:10.463978    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d_3a522b31-c301-4372-a729-93e898489653/istio-proxy/1.log"
Sep 18 17:35:11.780079 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:11.779996    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d_3a522b31-c301-4372-a729-93e898489653/istio-proxy/0.log"
Sep 18 17:35:11.783689 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:11.783665    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-mfh2d_3a522b31-c301-4372-a729-93e898489653/istio-proxy/1.log"
Sep 18 17:35:16.575430 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:16.575403    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-92ffq_70b1301b-d0c6-4fc3-aeac-661f45a32d79/kube-multus-additional-cni-plugins/0.log"
Sep 18 17:35:16.597577 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:16.597555    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-92ffq_70b1301b-d0c6-4fc3-aeac-661f45a32d79/egress-router-binary-copy/0.log"
Sep 18 17:35:16.625096 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:16.625072    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-92ffq_70b1301b-d0c6-4fc3-aeac-661f45a32d79/cni-plugins/0.log"
Sep 18 17:35:16.645856 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:16.645831    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-92ffq_70b1301b-d0c6-4fc3-aeac-661f45a32d79/bond-cni-plugin/0.log"
Sep 18 17:35:16.666033 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:16.666010    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-92ffq_70b1301b-d0c6-4fc3-aeac-661f45a32d79/routeoverride-cni/0.log"
Sep 18 17:35:16.686695 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:16.686669    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-92ffq_70b1301b-d0c6-4fc3-aeac-661f45a32d79/whereabouts-cni-bincopy/0.log"
Sep 18 17:35:16.707166 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:16.707142    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-92ffq_70b1301b-d0c6-4fc3-aeac-661f45a32d79/whereabouts-cni/0.log"
Sep 18 17:35:16.973379 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:16.973307    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-tzzf6_cf86bb19-6c76-4d93-b36c-7a4078078ff3/kube-multus/0.log"
Sep 18 17:35:17.050903 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:17.050871    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-c6t6w_53188ed0-31cd-4e9a-b72d-f21a8a738db6/network-metrics-daemon/0.log"
Sep 18 17:35:17.076158 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:17.076125    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-c6t6w_53188ed0-31cd-4e9a-b72d-f21a8a738db6/kube-rbac-proxy/0.log"
Sep 18 17:35:18.428353 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:18.428319    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-controller/0.log"
Sep 18 17:35:18.445602 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:18.445572    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/1.log"
Sep 18 17:35:18.445602 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:18.445579    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/ovn-acl-logging/0.log"
Sep 18 17:35:18.467787 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:18.467765    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/kube-rbac-proxy-node/0.log"
Sep 18 17:35:18.488429 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:18.488402    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/kube-rbac-proxy-ovn-metrics/0.log"
Sep 18 17:35:18.504741 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:18.504719    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/northd/0.log"
Sep 18 17:35:18.524806 ip-10-0-133-147 kubenswrapper[2585]: I0918 17:35:18.524787    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vx9bt_1c5e2210-6de0-4e88-ad43-8c99387acdb4/nbdb/0.log"