Sep 17 18:29:01.959509 ip-10-0-139-75 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 18:29:02.385583 ip-10-0-139-75 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:29:02.385583 ip-10-0-139-75 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 18:29:02.385583 ip-10-0-139-75 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:29:02.385583 ip-10-0-139-75 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 18:29:02.385583 ip-10-0-139-75 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:29:02.386653 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.386576    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 18:29:02.390716 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390700    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:02.390716 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390715    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:02.390779 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390719    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:02.390779 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390723    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:02.390779 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390728    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:02.390779 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390732    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:02.390779 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390736    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:02.390779 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390740    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:02.390779 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390744    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:02.390779 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390747    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:02.390779 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390750    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:02.390779 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390753    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:02.390779 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390757    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:02.390779 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390761    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:02.390779 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390764    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:02.390779 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390767    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:02.390779 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390770    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:02.390779 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390773    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:02.390779 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390776    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:02.390779 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390779    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:02.390779 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390782    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:02.391257 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390785    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:02.391257 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390789    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:02.391257 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390792    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:02.391257 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390795    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:02.391257 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390797    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:02.391257 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390800    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:02.391257 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390803    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:02.391257 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390806    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:02.391257 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390809    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:02.391257 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390811    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:02.391257 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390814    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:02.391257 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390817    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:02.391257 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390819    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:02.391257 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390822    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:02.391257 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390824    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:02.391257 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390827    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:02.391257 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390829    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:02.391257 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390832    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:02.391257 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390834    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:02.391257 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390837    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:02.391730 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390839    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:02.391730 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390841    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:02.391730 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390844    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:02.391730 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390846    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:02.391730 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390849    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:02.391730 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390852    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:02.391730 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390855    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:02.391730 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390857    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:02.391730 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390860    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:02.391730 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390862    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:02.391730 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390865    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:02.391730 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390867    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:02.391730 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390869    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:02.391730 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390873    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:02.391730 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390875    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:02.391730 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390878    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:02.391730 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390880    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:02.391730 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390883    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:02.391730 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390885    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:02.391730 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390888    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:02.392255 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390890    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:02.392255 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390893    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:02.392255 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390896    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:02.392255 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390898    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:02.392255 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390901    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:02.392255 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390904    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:02.392255 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390907    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:02.392255 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390909    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:02.392255 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390912    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:02.392255 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390915    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:02.392255 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390917    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:02.392255 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390920    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:02.392255 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390922    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:02.392255 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390924    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:02.392255 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390928    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:02.392255 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390930    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:02.392255 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390933    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:02.392255 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390936    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:02.392255 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390939    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:02.392255 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390941    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:02.392739 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390943    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:02.392739 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390946    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:02.392739 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390949    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:02.392739 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390951    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:02.392739 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390954    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:02.392739 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390957    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:02.392739 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390959    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:02.392739 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390962    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:02.392739 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390964    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:02.392739 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390967    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:02.392739 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390969    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:02.392739 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390972    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:02.392739 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390974    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:02.392739 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390977    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:02.392739 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390979    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:02.392739 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390982    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:02.392739 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390985    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:02.392739 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.390987    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:02.393581 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393567    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:02.393581 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393580    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:02.393637 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393583    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:02.393637 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393587    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:02.393637 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393590    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:02.393637 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393593    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:02.393637 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393595    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:02.393637 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393598    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:02.393637 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393601    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:02.393637 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393605    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:02.393637 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393607    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:02.393637 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393610    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:02.393637 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393613    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:02.393637 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393615    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:02.393637 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393618    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:02.393637 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393620    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:02.393637 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393623    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:02.393637 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393625    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:02.393637 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393628    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:02.393637 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393630    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:02.393637 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393633    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:02.393637 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393635    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:02.394136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393638    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:02.394136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393640    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:02.394136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393643    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:02.394136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393646    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:02.394136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393649    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:02.394136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393651    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:02.394136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393654    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:02.394136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393657    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:02.394136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393659    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:02.394136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393664    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:02.394136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393667    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:02.394136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393670    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:02.394136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393673    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:02.394136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393675    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:02.394136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393678    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:02.394136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393681    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:02.394136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393684    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:02.394136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393687    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:02.394136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393690    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:02.394629 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393693    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:02.394629 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393696    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:02.394629 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393699    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:02.394629 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393702    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:02.394629 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393704    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:02.394629 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393707    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:02.394629 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393709    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:02.394629 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393712    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:02.394629 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393714    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:02.394629 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393717    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:02.394629 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393720    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:02.394629 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393722    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:02.394629 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393724    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:02.394629 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393727    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:02.394629 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393731    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:02.394629 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393734    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:02.394629 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393738    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:02.394629 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393741    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:02.394629 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393744    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:02.394629 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393747    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:02.395117 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393750    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:02.395117 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393752    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:02.395117 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393755    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:02.395117 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393758    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:02.395117 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393761    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:02.395117 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393763    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:02.395117 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393766    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:02.395117 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393768    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:02.395117 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393771    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:02.395117 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393774    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:02.395117 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393776    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:02.395117 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393779    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:02.395117 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393782    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:02.395117 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393784    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:02.395117 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393787    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:02.395117 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393790    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:02.395117 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393793    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:02.395117 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393796    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:02.395117 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393798    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:02.395117 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393801    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:02.395633 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393804    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:02.395633 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393807    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:02.395633 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393809    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:02.395633 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393812    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:02.395633 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393814    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:02.395633 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393817    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:02.395633 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393820    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:02.395633 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393822    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:02.395633 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393825    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:02.395633 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393827    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:02.395633 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393830    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:02.395633 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393833    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:02.395633 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393836    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:02.395633 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393840    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:02.395633 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393843    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:02.395633 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393845    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:02.395633 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393848    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:02.395633 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.393850    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:02.395633 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394219    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:02.395633 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394224    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:02.396136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394227    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:02.396136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394230    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:02.396136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394235    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:02.396136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394238    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:02.396136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394241    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:02.396136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394244    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:02.396136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394248    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:02.396136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394250    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:02.396136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394254    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:02.396136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394256    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:02.396136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394259    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:02.396136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394262    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:02.396136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394266    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:02.396136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394269    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:02.396136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394271    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:02.396136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394274    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:02.396136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394276    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:02.396136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394279    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:02.396136 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394282    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:02.396624 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394285    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:02.396624 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394288    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:02.396624 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394291    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:02.396624 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394293    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:02.396624 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394296    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:02.396624 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394299    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:02.396624 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394302    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:02.396624 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394304    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:02.396624 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394307    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:02.396624 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394310    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:02.396624 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394312    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:02.396624 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394315    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:02.396624 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394318    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:02.396624 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394320    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:02.396624 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394323    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:02.396624 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394325    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:02.396624 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394328    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:02.396624 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394330    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:02.396624 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394333    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:02.396624 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394335    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:02.396624 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394338    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:02.397138 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394341    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:02.397138 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394344    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:02.397138 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394346    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:02.397138 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394349    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:02.397138 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394352    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:02.397138 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394354    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:02.397138 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394357    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:02.397138 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394360    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:02.397138 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394363    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:02.397138 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394366    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:02.397138 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394369    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:02.397138 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394372    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:02.397138 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394374    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:02.397138 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394377    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:02.397138 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394379    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:02.397138 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394382    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:02.397138 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394384    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:02.397138 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394387    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:02.397138 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394390    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:02.397614 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394392    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:02.397614 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394395    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:02.397614 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394399    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:02.397614 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394402    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:02.397614 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394406    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:02.397614 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394409    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:02.397614 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394411    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:02.397614 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394414    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:02.397614 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394416    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:02.397614 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394419    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:02.397614 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394421    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:02.397614 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394424    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:02.397614 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394426    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:02.397614 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394429    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:02.397614 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394432    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:02.397614 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394434    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:02.397614 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394437    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:02.397614 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394439    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:02.397614 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394442    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:02.398086 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394445    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:02.398086 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394448    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:02.398086 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394450    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:02.398086 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394453    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:02.398086 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394455    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:02.398086 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394458    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:02.398086 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394461    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:02.398086 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394464    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:02.398086 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394466    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:02.398086 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394469    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:02.398086 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394471    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:02.398086 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394475    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:02.398086 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394479    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:02.398086 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394482    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:02.398086 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394485    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:02.398086 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394488    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:02.398086 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394491    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:02.398086 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394493    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:02.398086 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.394496    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:02.398086 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394576    2565 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 18:29:02.398633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394607    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 18:29:02.398633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394614    2565 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 18:29:02.398633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394618    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 18:29:02.398633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394622    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 18:29:02.398633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394625    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 18:29:02.398633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394629    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 18:29:02.398633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394633    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 18:29:02.398633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394637    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 18:29:02.398633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394641    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 18:29:02.398633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394644    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 18:29:02.398633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394648    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 18:29:02.398633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394651    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 18:29:02.398633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394654    2565 flags.go:64] FLAG: --cgroup-root=""
Sep 17 18:29:02.398633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394657    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 18:29:02.398633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394660    2565 flags.go:64] FLAG: --client-ca-file=""
Sep 17 18:29:02.398633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394663    2565 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 18:29:02.398633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394666    2565 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 18:29:02.398633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394670    2565 flags.go:64] FLAG: --cluster-domain=""
Sep 17 18:29:02.398633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394673    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 18:29:02.398633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394676    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 18:29:02.398633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394679    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 18:29:02.398633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394682    2565 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 18:29:02.398633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394685    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 18:29:02.398633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394688    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 18:29:02.399336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394691    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 18:29:02.399336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394695    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 18:29:02.399336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394698    2565 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 18:29:02.399336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394701    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 18:29:02.399336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394704    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 18:29:02.399336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394707    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 18:29:02.399336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394710    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 18:29:02.399336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394713    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 18:29:02.399336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394717    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 18:29:02.399336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394719    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 18:29:02.399336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394722    2565 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 18:29:02.399336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394725    2565 flags.go:64] FLAG: --enable-server="true"
Sep 17 18:29:02.399336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394728    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 18:29:02.399336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394733    2565 flags.go:64] FLAG: --event-burst="100"
Sep 17 18:29:02.399336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394736    2565 flags.go:64] FLAG: --event-qps="50"
Sep 17 18:29:02.399336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394739    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 18:29:02.399336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394743    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 18:29:02.399336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394746    2565 flags.go:64] FLAG: --eviction-hard=""
Sep 17 18:29:02.399336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394749    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 18:29:02.399336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394753    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 18:29:02.399336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394756    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 18:29:02.399336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394758    2565 flags.go:64] FLAG: --eviction-soft=""
Sep 17 18:29:02.399336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394761    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 18:29:02.399336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394764    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 18:29:02.399336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394767    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 18:29:02.399964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394772    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 18:29:02.399964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394775    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 18:29:02.399964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394778    2565 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 18:29:02.399964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394781    2565 flags.go:64] FLAG: --feature-gates=""
Sep 17 18:29:02.399964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394785    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 18:29:02.399964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394788    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 18:29:02.399964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394791    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 18:29:02.399964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394794    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 18:29:02.399964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394796    2565 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 18:29:02.399964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394800    2565 flags.go:64] FLAG: --help="false"
Sep 17 18:29:02.399964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394803    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-139-75.ec2.internal"
Sep 17 18:29:02.399964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394806    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 18:29:02.399964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394809    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 18:29:02.399964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394812    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 18:29:02.399964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394815    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 18:29:02.399964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394818    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 18:29:02.399964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394821    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 18:29:02.399964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394826    2565 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 18:29:02.399964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394828    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 18:29:02.399964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394831    2565 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 18:29:02.399964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394834    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 18:29:02.399964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394837    2565 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 18:29:02.399964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394840    2565 flags.go:64] FLAG: --kube-reserved=""
Sep 17 18:29:02.399964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394843    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 18:29:02.400560 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394846    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 18:29:02.400560 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394849    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 18:29:02.400560 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394852    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 18:29:02.400560 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394855    2565 flags.go:64] FLAG: --lock-file=""
Sep 17 18:29:02.400560 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394858    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 18:29:02.400560 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394861    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 18:29:02.400560 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394864    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 18:29:02.400560 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394869    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 18:29:02.400560 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394873    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 18:29:02.400560 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394876    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 18:29:02.400560 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394879    2565 flags.go:64] FLAG: --logging-format="text"
Sep 17 18:29:02.400560 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394882    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 18:29:02.400560 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394885    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 18:29:02.400560 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394888    2565 flags.go:64] FLAG: --manifest-url=""
Sep 17 18:29:02.400560 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394891    2565 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 18:29:02.400560 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394895    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 18:29:02.400560 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394898    2565 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 18:29:02.400560 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394902    2565 flags.go:64] FLAG: --max-pods="110"
Sep 17 18:29:02.400560 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394905    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 18:29:02.400560 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394908    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 18:29:02.400560 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394911    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 18:29:02.400560 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394914    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 18:29:02.400560 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394917    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 18:29:02.400560 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394919    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 18:29:02.400560 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394923    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 18:29:02.401175 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394930    2565 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 18:29:02.401175 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394933    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 18:29:02.401175 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394936    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 18:29:02.401175 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394939    2565 flags.go:64] FLAG: --pod-cidr=""
Sep 17 18:29:02.401175 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394942    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 18:29:02.401175 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394947    2565 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 18:29:02.401175 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394950    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 18:29:02.401175 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394953    2565 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 18:29:02.401175 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394957    2565 flags.go:64] FLAG: --port="10250"
Sep 17 18:29:02.401175 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394960    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 18:29:02.401175 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394963    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cf7aabe9313f4a92"
Sep 17 18:29:02.401175 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394966    2565 flags.go:64] FLAG: --qos-reserved=""
Sep 17 18:29:02.401175 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394969    2565 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 18:29:02.401175 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394972    2565 flags.go:64] FLAG: --register-node="true"
Sep 17 18:29:02.401175 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394975    2565 flags.go:64] FLAG: --register-with-taints=""
Sep 17 18:29:02.401175 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394980    2565 flags.go:64] FLAG: --registry-burst="10"
Sep 17 18:29:02.401175 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394983    2565 flags.go:64] FLAG: --registry-qps="5"
Sep 17 18:29:02.401175 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394985    2565 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 18:29:02.401175 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394988    2565 flags.go:64] FLAG: --reserved-memory=""
Sep 17 18:29:02.401175 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394992    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 18:29:02.401175 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394995    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 18:29:02.401175 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.394998    2565 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 18:29:02.401175 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.395001    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 18:29:02.401175 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.395003    2565 flags.go:64] FLAG: --runonce="false"
Sep 17 18:29:02.401175 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.395006    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 18:29:02.401824 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.395009    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 18:29:02.401824 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.395012    2565 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 18:29:02.401824 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.395015    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 18:29:02.401824 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.395018    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 18:29:02.401824 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.395021    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 18:29:02.401824 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.395024    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 18:29:02.401824 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.395028    2565 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 18:29:02.401824 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.395031    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 18:29:02.401824 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.395035    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 18:29:02.401824 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.395038    2565 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 18:29:02.401824 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.395041    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 18:29:02.401824 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.395044    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 18:29:02.401824 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.395047    2565 flags.go:64] FLAG: --system-cgroups=""
Sep 17 18:29:02.401824 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.395050    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 18:29:02.401824 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.395055    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 18:29:02.401824 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.395057    2565 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 18:29:02.401824 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.395061    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 18:29:02.401824 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.395065    2565 flags.go:64] FLAG: --tls-min-version=""
Sep 17 18:29:02.401824 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.395067    2565 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 18:29:02.401824 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.395070    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 18:29:02.401824 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.395073    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 18:29:02.401824 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.395076    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 18:29:02.401824 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.395080    2565 flags.go:64] FLAG: --v="2"
Sep 17 18:29:02.401824 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.395084    2565 flags.go:64] FLAG: --version="false"
Sep 17 18:29:02.401824 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.395088    2565 flags.go:64] FLAG: --vmodule=""
Sep 17 18:29:02.402493 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.395091    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 18:29:02.402493 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.395094    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 18:29:02.402493 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395210    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:02.402493 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395214    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:02.402493 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395217    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:02.402493 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395220    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:02.402493 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395222    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:02.402493 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395225    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:02.402493 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395228    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:02.402493 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395230    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:02.402493 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395233    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:02.402493 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395236    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:02.402493 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395238    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:02.402493 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395241    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:02.402493 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395243    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:02.402493 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395249    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:02.402493 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395252    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:02.402493 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395255    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:02.402493 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395258    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:02.402493 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395260    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:02.403011 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395263    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:02.403011 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395266    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:02.403011 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395268    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:02.403011 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395271    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:02.403011 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395274    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:02.403011 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395277    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:02.403011 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395279    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:02.403011 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395282    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:02.403011 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395285    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:02.403011 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395289    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:02.403011 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395291    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:02.403011 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395294    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:02.403011 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395297    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:02.403011 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395299    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:02.403011 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395302    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:02.403011 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395305    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:02.403011 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395307    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:02.403011 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395310    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:02.403011 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395313    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:02.403011 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395315    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:02.403574 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395318    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:02.403574 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395321    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:02.403574 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395324    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:02.403574 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395327    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:02.403574 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395329    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:02.403574 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395332    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:02.403574 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395334    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:02.403574 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395339    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:02.403574 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395341    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:02.403574 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395344    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:02.403574 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395347    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:02.403574 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395350    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:02.403574 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395353    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:02.403574 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395355    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:02.403574 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395358    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:02.403574 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395361    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:02.403574 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395363    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:02.403574 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395366    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:02.403574 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395369    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:02.404066 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395372    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:02.404066 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395374    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:02.404066 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395378    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:02.404066 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395381    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:02.404066 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395383    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:02.404066 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395386    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:02.404066 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395388    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:02.404066 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395391    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:02.404066 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395394    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:02.404066 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395396    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:02.404066 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395399    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:02.404066 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395401    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:02.404066 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395404    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:02.404066 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395407    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:02.404066 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395411    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:02.404066 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395415    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:02.404066 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395418    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:02.404066 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395421    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:02.404066 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395423    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:02.404551 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395426    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:02.404551 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395431    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:02.404551 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395434    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:02.404551 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395438    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:02.404551 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395441    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:02.404551 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395444    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:02.404551 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395447    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:02.404551 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395450    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:02.404551 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395453    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:02.404551 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395456    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:02.404551 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395459    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:02.404551 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395462    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:02.404551 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395465    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:02.404551 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395468    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:02.404551 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395471    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:02.404551 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395474    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:02.404551 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395477    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:02.404551 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395481    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:02.404551 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395484    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:02.405020 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395487    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:02.405020 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395490    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:02.405020 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395493    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:02.405020 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.395496    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:02.405020 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.396284    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:29:02.405020 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.404568    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 18:29:02.405020 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.404582    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 18:29:02.405572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.405561    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 18:29:02.405608 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.405575    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 18:29:02.405642 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405628    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:02.405642 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405633    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:02.405642 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405637    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:02.405642 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405640    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:02.405642 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405644    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:02.405770 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405647    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:02.405770 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405650    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:02.405770 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405653    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:02.405770 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405655    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:02.405770 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405660    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:02.405770 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405664    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:02.405770 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405667    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:02.405770 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405670    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:02.405770 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405673    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:02.405770 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405676    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:02.405770 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405678    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:02.405770 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405681    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:02.405770 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405684    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:02.405770 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405686    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:02.405770 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405689    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:02.405770 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405691    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:02.405770 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405694    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:02.405770 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405697    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:02.405770 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405699    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:02.405770 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405702    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:02.406306 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405705    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:02.406306 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405707    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:02.406306 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405710    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:02.406306 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405712    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:02.406306 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405716    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:02.406306 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405720    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:02.406306 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405724    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:02.406306 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405727    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:02.406306 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405729    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:02.406306 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405732    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:02.406306 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405735    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:02.406306 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405738    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:02.406306 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405740    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:02.406306 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405743    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:02.406306 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405746    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:02.406306 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405749    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:02.406306 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405751    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:02.406306 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405754    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:02.406306 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405756    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:02.406776 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405759    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:02.406776 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405762    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:02.406776 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405764    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:02.406776 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405767    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:02.406776 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405769    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:02.406776 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405772    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:02.406776 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405775    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:02.406776 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405778    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:02.406776 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405780    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:02.406776 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405783    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:02.406776 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405786    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:02.406776 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405788    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:02.406776 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405791    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:02.406776 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405793    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:02.406776 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405796    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:02.406776 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405799    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:02.406776 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405801    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:02.406776 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405804    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:02.406776 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405807    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:02.406776 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405810    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:02.407381 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405812    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:02.407381 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405815    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:02.407381 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405817    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:02.407381 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405820    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:02.407381 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405823    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:02.407381 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405825    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:02.407381 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405829    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:02.407381 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405832    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:02.407381 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405834    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:02.407381 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405837    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:02.407381 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405840    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:02.407381 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405842    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:02.407381 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405845    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:02.407381 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405847    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:02.407381 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405849    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:02.407381 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405853    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:02.407381 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405855    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:02.407381 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405858    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:02.407381 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405860    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:02.407841 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405864    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:02.407841 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405867    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:02.407841 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405871    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:02.407841 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405873    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:02.407841 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405876    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:02.407841 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405879    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:02.407841 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405882    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:02.407841 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405885    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:02.407841 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405887    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:02.407841 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405890    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:02.407841 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405893    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:02.407841 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405895    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:02.407841 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405898    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:02.407841 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405900    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:02.407841 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405903    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:02.407841 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.405906    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:02.408238 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.405911    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:29:02.408238 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406016    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:02.408238 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406021    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:02.408238 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406025    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:02.408238 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406028    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:02.408238 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406031    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:02.408238 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406034    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:02.408238 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406036    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:02.408238 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406039    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:02.408238 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406042    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:02.408238 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406044    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:02.408238 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406047    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:02.408238 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406050    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:02.408238 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406052    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:02.408582 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406055    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:02.408582 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406057    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:02.408582 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406060    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:02.408582 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406063    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:02.408582 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406065    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:02.408582 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406068    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:02.408582 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406071    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:02.408582 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406075    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:02.408582 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406078    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:02.408582 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406081    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:02.408582 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406084    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:02.408582 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406087    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:02.408582 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406090    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:02.408582 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406093    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:02.408582 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406095    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:02.408582 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406098    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:02.408582 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406101    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:02.408582 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406104    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:02.408582 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406107    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:02.409042 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406109    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:02.409042 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406112    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:02.409042 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406115    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:02.409042 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406117    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:02.409042 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406120    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:02.409042 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406122    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:02.409042 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406125    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:02.409042 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406127    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:02.409042 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406130    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:02.409042 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406132    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:02.409042 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406135    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:02.409042 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406138    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:02.409042 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406142    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:02.409042 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406145    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:02.409042 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406148    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:02.409042 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406151    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:02.409042 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406153    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:02.409042 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406156    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:02.409042 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406159    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:02.409042 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406161    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:02.409538 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406164    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:02.409538 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406166    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:02.409538 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406169    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:02.409538 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406171    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:02.409538 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406174    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:02.409538 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406177    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:02.409538 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406181    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:02.409538 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406184    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:02.409538 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406187    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:02.409538 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406190    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:02.409538 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406193    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:02.409538 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406208    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:02.409538 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406213    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:02.409538 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406218    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:02.409538 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406222    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:02.409538 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406226    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:02.409538 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406229    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:02.409538 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406232    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:02.409538 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406234    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:02.409538 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406237    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:02.410027 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406240    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:02.410027 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406242    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:02.410027 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406245    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:02.410027 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406248    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:02.410027 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406250    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:02.410027 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406253    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:02.410027 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406256    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:02.410027 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406258    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:02.410027 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406261    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:02.410027 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406263    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:02.410027 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406266    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:02.410027 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406269    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:02.410027 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406272    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:02.410027 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406275    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:02.410027 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406277    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:02.410027 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406280    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:02.410027 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406283    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:02.410027 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406285    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:02.410027 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406288    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:02.410027 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406290    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:02.410518 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406293    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:02.410518 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406296    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:02.410518 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406299    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:02.410518 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406302    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:02.410518 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406304    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:02.410518 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406307    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:02.410518 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:02.406310    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:02.410518 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.406315    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:29:02.410518 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.406412    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 18:29:02.410518 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.408667    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 18:29:02.410518 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.409732    2565 server.go:1019] "Starting client certificate rotation"
Sep 17 18:29:02.410518 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.409828    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 18:29:02.410518 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.409895    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 18:29:02.415182 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.415163    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 18:29:02.416342 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.416327    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 18:29:02.426230 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.426213    2565 log.go:25] "Validated CRI v1 runtime API"
Sep 17 18:29:02.427975 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.427961    2565 log.go:25] "Validated CRI v1 image API"
Sep 17 18:29:02.430121 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.430105    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 18:29:02.433444 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.433426    2565 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 832a0dbc-65a1-4b45-970d-878e5f38f9ef:/dev/nvme0n1p3 9437988e-1765-49bf-9959-6b5369a6e89f:/dev/nvme0n1p4]
Sep 17 18:29:02.433510 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.433442    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 18:29:02.438743 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.438637    2565 manager.go:216] Machine: {Timestamp:2026-09-17 18:29:02.438138442 +0000 UTC m=+0.370339952 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100668 MemoryCapacity:33164468224 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26e46e2f70c53485d4b1f53db64652 SystemUUID:ec26e46e-2f70-c534-85d4-b1f53db64652 BootID:b0f6152d-d3dd-4e75-a179-f04c0ba46a57 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582234112 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c2:e9:bc:05 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c2:e9:bc:05 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:2c:fa:d3:d7:8e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164468224 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 18:29:02.438743 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.438737    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 18:29:02.438851 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.438805    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 18:29:02.441372 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.441342    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 18:29:02.441496 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.441373    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-75.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 18:29:02.442251 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.442241    2565 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 18:29:02.442287 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.442254    2565 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 18:29:02.442287 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.442262    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 18:29:02.443314 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.443298    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 18:29:02.443370 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.443304    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 18:29:02.443464 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.443453    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 18:29:02.448246 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.448233    2565 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 18:29:02.448366 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.448356    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 18:29:02.448413 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.448387    2565 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 18:29:02.448413 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.448396    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 18:29:02.448413 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.448406    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 18:29:02.448522 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.448416    2565 kubelet.go:398] "Adding apiserver pod source"
Sep 17 18:29:02.448522 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.448426    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 18:29:02.448750 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.448738    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 18:29:02.448792 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.448759    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 18:29:02.450995 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.450973    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 18:29:02.454295 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.454268    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 18:29:02.454526 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.454511    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 18:29:02.456012 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.456000    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 18:29:02.456059 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.456016    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 18:29:02.456059 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.456022    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 18:29:02.456059 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.456026    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 18:29:02.456059 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.456035    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 18:29:02.456059 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.456041    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 18:29:02.456059 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.456060    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 18:29:02.456243 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.456066    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 18:29:02.456243 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.456072    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 18:29:02.456243 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.456078    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 18:29:02.456243 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.456085    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 18:29:02.456243 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.456090    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 18:29:02.456925 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.456915    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 18:29:02.456960 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.456927    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 18:29:02.460010 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.459989    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-75.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 18:29:02.460062 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:02.460033    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-75.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 18:29:02.460062 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:02.460035    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 18:29:02.460720 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.460706    2565 server.go:1268] "Started kubelet"
Sep 17 18:29:02.460813 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.460790    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 18:29:02.460861 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.460785    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 18:29:02.460861 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.460847    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 18:29:02.461555 ip-10-0-139-75 systemd[1]: Started Kubernetes Kubelet.
Sep 17 18:29:02.462424 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.462405    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 18:29:02.464246 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.464226    2565 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 18:29:02.465145 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:02.464936    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 18:29:02.468929 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:02.467788    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-75.ec2.internal.18d62f2610a228e8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-75.ec2.internal,UID:ip-10-0-139-75.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-75.ec2.internal,},FirstTimestamp:2026-09-17 18:29:02.460676328 +0000 UTC m=+0.392877839,LastTimestamp:2026-09-17 18:29:02.460676328 +0000 UTC m=+0.392877839,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-75.ec2.internal,}"
Sep 17 18:29:02.470802 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.470783    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 18:29:02.470802 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.470793    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 18:29:02.470930 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:02.470893    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-75.ec2.internal\" not found"
Sep 17 18:29:02.471541 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.471524    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 18:29:02.471541 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.471525    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 18:29:02.471649 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.471552    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 18:29:02.472522 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.472507    2565 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 18:29:02.472522 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.472518    2565 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 18:29:02.473729 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.473708    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 18:29:02.473729 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.473727    2565 factory.go:55] Registering systemd factory
Sep 17 18:29:02.473871 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.473736    2565 factory.go:223] Registration of the systemd container factory successfully
Sep 17 18:29:02.474342 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.474326    2565 factory.go:153] Registering CRI-O factory
Sep 17 18:29:02.474460 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.474442    2565 factory.go:223] Registration of the crio container factory successfully
Sep 17 18:29:02.474536 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.474465    2565 factory.go:103] Registering Raw factory
Sep 17 18:29:02.474536 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.474481    2565 manager.go:1196] Started watching for new ooms in manager
Sep 17 18:29:02.474793 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:02.474759    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 18:29:02.474881 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:02.474829    2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-75.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 18:29:02.474946 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.474904    2565 manager.go:349] Starting recovery of all containers
Sep 17 18:29:02.479757 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.479728    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 18:29:02.486775 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.486756    2565 manager.go:354] Recovery completed
Sep 17 18:29:02.492301 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.492289    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:02.494448 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.494418    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-75.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:02.494525 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.494459    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-75.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:02.494525 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.494473    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-75.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:02.495040 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.495026    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 18:29:02.495040 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.495038    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 18:29:02.495134 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.495053    2565 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 18:29:02.495134 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.495085    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:02.495825 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.495807    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-75.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:02.495887 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.495845    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-75.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:02.495887 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.495863    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-75.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:02.496537 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:02.496470    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-75.ec2.internal.18d62f2612a560c7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-75.ec2.internal,UID:ip-10-0-139-75.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-75.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-75.ec2.internal,},FirstTimestamp:2026-09-17 18:29:02.494441671 +0000 UTC m=+0.426643185,LastTimestamp:2026-09-17 18:29:02.494441671 +0000 UTC m=+0.426643185,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-75.ec2.internal,}"
Sep 17 18:29:02.496987 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.496970    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tt2zh"
Sep 17 18:29:02.497291 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.497279    2565 policy_none.go:49] "None policy: Start"
Sep 17 18:29:02.497345 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.497294    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 18:29:02.497345 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.497304    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 18:29:02.499260 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.499249    2565 policy_none.go:47] "Start"
Sep 17 18:29:02.503393 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.503275    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tt2zh"
Sep 17 18:29:02.505491 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:02.505428    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-75.ec2.internal.18d62f2612a5bfac  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-75.ec2.internal,UID:ip-10-0-139-75.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-75.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-75.ec2.internal,},FirstTimestamp:2026-09-17 18:29:02.494465964 +0000 UTC m=+0.426667476,LastTimestamp:2026-09-17 18:29:02.494465964 +0000 UTC m=+0.426667476,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-75.ec2.internal,}"
Sep 17 18:29:02.546418 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.535749    2565 manager.go:337] "Starting Device Plugin manager"
Sep 17 18:29:02.546418 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:02.535783    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 18:29:02.546418 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.535792    2565 server.go:87] "Starting device plugin registration server"
Sep 17 18:29:02.546418 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.535982    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 18:29:02.546418 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.535994    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 18:29:02.546418 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.536048    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 18:29:02.546418 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.536109    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 18:29:02.546418 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.536117    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 18:29:02.546418 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:02.536616    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 18:29:02.546418 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:02.536664    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-75.ec2.internal\" not found"
Sep 17 18:29:02.600438 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.600422    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 18:29:02.600522 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.600467    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 18:29:02.600522 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.600487    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 18:29:02.600617 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:02.600548    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 18:29:02.602871 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.602854    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:02.636874 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.636813    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:02.637594 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.637576    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-75.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:02.637689 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.637610    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-75.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:02.637689 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.637629    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-75.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:02.637689 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.637657    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-75.ec2.internal"
Sep 17 18:29:02.646886 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.646872    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-75.ec2.internal"
Sep 17 18:29:02.646986 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:02.646894    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-75.ec2.internal\": node \"ip-10-0-139-75.ec2.internal\" not found"
Sep 17 18:29:02.672660 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:02.672644    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-75.ec2.internal\" not found"
Sep 17 18:29:02.700882 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.700860    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-75.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-75.ec2.internal"]
Sep 17 18:29:02.700953 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.700913    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:02.702095 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.702071    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-75.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:02.702095 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.702093    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-75.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:02.702192 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.702102    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-75.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:02.704478 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.704466    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:02.704613 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.704589    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-75.ec2.internal"
Sep 17 18:29:02.704652 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.704630    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:02.705641 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.705622    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-75.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:02.705723 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.705648    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-75.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:02.705723 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.705664    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-75.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:02.705723 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.705686    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-75.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:02.705723 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.705706    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-75.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:02.705723 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.705718    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-75.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:02.708030 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.708017    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-75.ec2.internal"
Sep 17 18:29:02.708073 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.708041    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:02.708859 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.708843    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-75.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:02.708938 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.708868    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-75.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:02.708938 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.708877    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-75.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:02.737698 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:02.737678    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-75.ec2.internal\" not found" node="ip-10-0-139-75.ec2.internal"
Sep 17 18:29:02.741812 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:02.741796    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-75.ec2.internal\" not found" node="ip-10-0-139-75.ec2.internal"
Sep 17 18:29:02.773359 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:02.773344    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-75.ec2.internal\" not found"
Sep 17 18:29:02.774478 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.774456    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/67e17decd2a851abd4f31ec3df7b2fb3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-75.ec2.internal\" (UID: \"67e17decd2a851abd4f31ec3df7b2fb3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-75.ec2.internal"
Sep 17 18:29:02.774560 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.774480    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67e17decd2a851abd4f31ec3df7b2fb3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-75.ec2.internal\" (UID: \"67e17decd2a851abd4f31ec3df7b2fb3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-75.ec2.internal"
Sep 17 18:29:02.774560 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.774498    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ef83900f88d178fe684eab642563132-config\") pod \"kube-apiserver-proxy-ip-10-0-139-75.ec2.internal\" (UID: \"7ef83900f88d178fe684eab642563132\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-75.ec2.internal"
Sep 17 18:29:02.873991 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:02.873960    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-75.ec2.internal\" not found"
Sep 17 18:29:02.875074 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.875059    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/67e17decd2a851abd4f31ec3df7b2fb3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-75.ec2.internal\" (UID: \"67e17decd2a851abd4f31ec3df7b2fb3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-75.ec2.internal"
Sep 17 18:29:02.875141 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.875085    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67e17decd2a851abd4f31ec3df7b2fb3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-75.ec2.internal\" (UID: \"67e17decd2a851abd4f31ec3df7b2fb3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-75.ec2.internal"
Sep 17 18:29:02.875141 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.875101    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ef83900f88d178fe684eab642563132-config\") pod \"kube-apiserver-proxy-ip-10-0-139-75.ec2.internal\" (UID: \"7ef83900f88d178fe684eab642563132\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-75.ec2.internal"
Sep 17 18:29:02.875261 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.875155    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ef83900f88d178fe684eab642563132-config\") pod \"kube-apiserver-proxy-ip-10-0-139-75.ec2.internal\" (UID: \"7ef83900f88d178fe684eab642563132\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-75.ec2.internal"
Sep 17 18:29:02.875261 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.875171    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/67e17decd2a851abd4f31ec3df7b2fb3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-75.ec2.internal\" (UID: \"67e17decd2a851abd4f31ec3df7b2fb3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-75.ec2.internal"
Sep 17 18:29:02.875261 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:02.875195    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67e17decd2a851abd4f31ec3df7b2fb3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-75.ec2.internal\" (UID: \"67e17decd2a851abd4f31ec3df7b2fb3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-75.ec2.internal"
Sep 17 18:29:02.974621 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:02.974595    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-75.ec2.internal\" not found"
Sep 17 18:29:03.040874 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:03.040850    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-75.ec2.internal"
Sep 17 18:29:03.044759 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:03.044740    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-75.ec2.internal"
Sep 17 18:29:03.075081 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:03.075054    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-75.ec2.internal\" not found"
Sep 17 18:29:03.175580 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:03.175546    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-75.ec2.internal\" not found"
Sep 17 18:29:03.276005 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:03.275948    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-75.ec2.internal\" not found"
Sep 17 18:29:03.377147 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:03.377118    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-75.ec2.internal\" not found"
Sep 17 18:29:03.409615 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:03.409578    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 18:29:03.410617 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:03.409725    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 18:29:03.477243 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:03.477221    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-75.ec2.internal\" not found"
Sep 17 18:29:03.509876 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:03.509837    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 18:24:03 +0000 UTC" deadline="2028-06-18 13:56:35.4395188 +0000 UTC"
Sep 17 18:29:03.509876 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:03.509870    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15355h27m31.929651942s"
Sep 17 18:29:03.577814 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:03.577790    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-75.ec2.internal\" not found"
Sep 17 18:29:03.636514 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:03.636495    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:03.672033 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:03.672010    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-75.ec2.internal"
Sep 17 18:29:03.686231 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:03.686210    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 18:29:03.687627 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:03.687613    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-75.ec2.internal"
Sep 17 18:29:03.691824 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:03.691796    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e17decd2a851abd4f31ec3df7b2fb3.slice/crio-68b221a0cf1d12215d45a766f148b8825d2a5ed1e75e5c8b5d439e8253527f28 WatchSource:0}: Error finding container 68b221a0cf1d12215d45a766f148b8825d2a5ed1e75e5c8b5d439e8253527f28: Status 404 returned error can't find the container with id 68b221a0cf1d12215d45a766f148b8825d2a5ed1e75e5c8b5d439e8253527f28
Sep 17 18:29:03.693643 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:03.693627    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef83900f88d178fe684eab642563132.slice/crio-17f4be9545b46ccb5efc927cb22007c7bd9cf4db58412ecfe3dd0dd234bed55b WatchSource:0}: Error finding container 17f4be9545b46ccb5efc927cb22007c7bd9cf4db58412ecfe3dd0dd234bed55b: Status 404 returned error can't find the container with id 17f4be9545b46ccb5efc927cb22007c7bd9cf4db58412ecfe3dd0dd234bed55b
Sep 17 18:29:03.694704 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:03.694689    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 18:29:03.698820 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:03.698802    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:29:03.719711 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:03.719693    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:03.789091 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:03.789068    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:04.450633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.450338    2565 apiserver.go:52] "Watching apiserver"
Sep 17 18:29:04.461632 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.461419    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 18:29:04.463109 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.463081    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-4zwdx","kube-system/konnectivity-agent-zttbn","kube-system/kube-apiserver-proxy-ip-10-0-139-75.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtbwp","openshift-dns/node-resolver-mkdtq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-75.ec2.internal","openshift-multus/multus-mfxhw","openshift-network-diagnostics/network-check-target-ssztx","openshift-cluster-node-tuning-operator/tuned-pwktj","openshift-image-registry/node-ca-glcw2","openshift-multus/multus-additional-cni-plugins-stwlh","openshift-multus/network-metrics-daemon-w67kg","openshift-network-operator/iptables-alerter-vfv5l","openshift-ovn-kubernetes/ovnkube-node-grgmp"]
Sep 17 18:29:04.465743 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.465721    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.468379 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.468320    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vfv5l"
Sep 17 18:29:04.468473 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.468452    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 18:29:04.468911 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.468889    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 18:29:04.468997 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.468952    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 18:29:04.469312 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.469291    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-td6p7\""
Sep 17 18:29:04.469597 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.469577    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 18:29:04.470554 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.470533    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:29:04.470930 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.470910    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 18:29:04.471142 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.471123    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 18:29:04.471527 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.471496    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hrq5n\""
Sep 17 18:29:04.471746 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.471732    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 18:29:04.471866 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.471850    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.472925 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.472905    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtbwp"
Sep 17 18:29:04.474022 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.473999    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 18:29:04.475148 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.475130    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 18:29:04.475518 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.475502    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 18:29:04.475813 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.475721    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cz4k5\""
Sep 17 18:29:04.476336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.476317    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 18:29:04.476409 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.476354    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 18:29:04.476552 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.476526    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 18:29:04.476654 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.476564    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 18:29:04.476654 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.476630    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 18:29:04.476900 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.476883    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tz82f\""
Sep 17 18:29:04.476949 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.476884    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 18:29:04.477980 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.477926    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mkdtq"
Sep 17 18:29:04.480255 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.480217    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 18:29:04.480610 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.480580    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 18:29:04.480734 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.480671    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-whgvr\""
Sep 17 18:29:04.480993 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.480958    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:29:04.481074 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:04.481040    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssztx" podUID="5d3336f6-e38a-4183-8d58-3c99f1284f33"
Sep 17 18:29:04.482333 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.482302    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-host-slash\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.482427 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.482342    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-run-ovn\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.482427 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.482370    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twrhl\" (UniqueName: \"kubernetes.io/projected/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-kube-api-access-twrhl\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.482427 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.482394    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7fb70a02-f3d7-496f-b1de-445b7165dd01-device-dir\") pod \"aws-ebs-csi-driver-node-xtbwp\" (UID: \"7fb70a02-f3d7-496f-b1de-445b7165dd01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtbwp"
Sep 17 18:29:04.482427 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.482419    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-etc-kubernetes\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.482644 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.482443    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-multus-conf-dir\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.482644 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.482467    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-host-kubelet\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.482644 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.482492    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-etc-openvswitch\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.482644 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.482518    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7fb70a02-f3d7-496f-b1de-445b7165dd01-sys-fs\") pod \"aws-ebs-csi-driver-node-xtbwp\" (UID: \"7fb70a02-f3d7-496f-b1de-445b7165dd01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtbwp"
Sep 17 18:29:04.482644 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.482566    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-host-var-lib-cni-bin\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.482644 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.482592    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-host-run-netns\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.482644 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.482618    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-run-systemd\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.482644 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.482643    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-log-socket\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.483040 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.482667    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-ovnkube-script-lib\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.483040 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.482692    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-cnibin\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.483040 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.482716    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-host-run-multus-certs\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.483040 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.482740    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fdjs\" (UniqueName: \"kubernetes.io/projected/0eeeae49-526f-41f9-8438-3eebbee372f6-kube-api-access-4fdjs\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.483040 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.482789    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc0044f2-c713-4de4-b82e-a211c1f398c3-host-slash\") pod \"iptables-alerter-vfv5l\" (UID: \"bc0044f2-c713-4de4-b82e-a211c1f398c3\") " pod="openshift-network-operator/iptables-alerter-vfv5l"
Sep 17 18:29:04.483040 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.482817    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjjbj\" (UniqueName: \"kubernetes.io/projected/bc0044f2-c713-4de4-b82e-a211c1f398c3-kube-api-access-zjjbj\") pod \"iptables-alerter-vfv5l\" (UID: \"bc0044f2-c713-4de4-b82e-a211c1f398c3\") " pod="openshift-network-operator/iptables-alerter-vfv5l"
Sep 17 18:29:04.483040 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.482843    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-var-lib-openvswitch\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.483040 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.482871    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.483040 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.482902    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-ovn-node-metrics-cert\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.483040 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.482926    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7fb70a02-f3d7-496f-b1de-445b7165dd01-registration-dir\") pod \"aws-ebs-csi-driver-node-xtbwp\" (UID: \"7fb70a02-f3d7-496f-b1de-445b7165dd01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtbwp"
Sep 17 18:29:04.483040 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.482953    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0eeeae49-526f-41f9-8438-3eebbee372f6-cni-binary-copy\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.483040 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.482977    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-host-var-lib-cni-multus\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.483040 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.483007    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-run-openvswitch\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.483040 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.483032    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-host-cni-bin\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.483732 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.483067    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-ovnkube-config\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.483732 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.483093    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7fb70a02-f3d7-496f-b1de-445b7165dd01-etc-selinux\") pod \"aws-ebs-csi-driver-node-xtbwp\" (UID: \"7fb70a02-f3d7-496f-b1de-445b7165dd01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtbwp"
Sep 17 18:29:04.483732 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.483124    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0eeeae49-526f-41f9-8438-3eebbee372f6-multus-daemon-config\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.483732 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.483148    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-node-log\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.483732 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.483173    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-env-overrides\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.483732 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.483196    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7fb70a02-f3d7-496f-b1de-445b7165dd01-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xtbwp\" (UID: \"7fb70a02-f3d7-496f-b1de-445b7165dd01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtbwp"
Sep 17 18:29:04.483732 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.483221    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7fb70a02-f3d7-496f-b1de-445b7165dd01-socket-dir\") pod \"aws-ebs-csi-driver-node-xtbwp\" (UID: \"7fb70a02-f3d7-496f-b1de-445b7165dd01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtbwp"
Sep 17 18:29:04.483732 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.483269    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-multus-cni-dir\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.483732 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.483317    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-multus-socket-dir-parent\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.483732 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.483344    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-host-run-k8s-cni-cncf-io\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.483732 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.483368    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-hostroot\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.483732 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.483398    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bc0044f2-c713-4de4-b82e-a211c1f398c3-iptables-alerter-script\") pod \"iptables-alerter-vfv5l\" (UID: \"bc0044f2-c713-4de4-b82e-a211c1f398c3\") " pod="openshift-network-operator/iptables-alerter-vfv5l"
Sep 17 18:29:04.483732 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.483493    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-host-run-ovn-kubernetes\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.483732 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.483528    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-host-cni-netd\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.483732 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.483557    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tztx2\" (UniqueName: \"kubernetes.io/projected/7fb70a02-f3d7-496f-b1de-445b7165dd01-kube-api-access-tztx2\") pod \"aws-ebs-csi-driver-node-xtbwp\" (UID: \"7fb70a02-f3d7-496f-b1de-445b7165dd01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtbwp"
Sep 17 18:29:04.483732 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.483583    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-system-cni-dir\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.484542 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.483606    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-host-run-netns\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.484542 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.483631    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-host-var-lib-kubelet\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.484542 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.483658    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-systemd-units\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.484542 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.484008    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.486088 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.486064    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 18:29:04.486335 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.486316    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-htfv9\""
Sep 17 18:29:04.486568 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.486548    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:29:04.486648 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.486583    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 18:29:04.486772 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.486732    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:04.486993 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:04.486911    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zwdx" podUID="43264d48-807c-43de-8b8f-52db0225d1c9"
Sep 17 18:29:04.489477 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.489456    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zttbn"
Sep 17 18:29:04.491504 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.491484    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 18:29:04.492067 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.492047    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 18:29:04.492290 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.492266    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hntf5\""
Sep 17 18:29:04.492374 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.492362    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-glcw2"
Sep 17 18:29:04.495615 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.495505    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-stwlh"
Sep 17 18:29:04.495732 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.495713    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f6qmj\""
Sep 17 18:29:04.495996 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.495960    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 18:29:04.496104 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.496086    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 18:29:04.496760 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.496739    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 18:29:04.497893 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.497869    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 18:29:04.498822 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.498799    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nvpw8\""
Sep 17 18:29:04.499166 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.499137    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 18:29:04.500919 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.500902    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:29:04.501032 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:04.500961    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w67kg" podUID="db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4"
Sep 17 18:29:04.515546 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.515524    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mwjrr"
Sep 17 18:29:04.527808 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.527775    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mwjrr"
Sep 17 18:29:04.573242 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.573226    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 18:29:04.584670 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584226    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7fb70a02-f3d7-496f-b1de-445b7165dd01-etc-selinux\") pod \"aws-ebs-csi-driver-node-xtbwp\" (UID: \"7fb70a02-f3d7-496f-b1de-445b7165dd01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtbwp"
Sep 17 18:29:04.584670 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584276    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2szjv\" (UniqueName: \"kubernetes.io/projected/822d9b1e-1a38-49ac-900a-6ad946e6078f-kube-api-access-2szjv\") pod \"node-ca-glcw2\" (UID: \"822d9b1e-1a38-49ac-900a-6ad946e6078f\") " pod="openshift-image-registry/node-ca-glcw2"
Sep 17 18:29:04.584670 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584289    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7fb70a02-f3d7-496f-b1de-445b7165dd01-etc-selinux\") pod \"aws-ebs-csi-driver-node-xtbwp\" (UID: \"7fb70a02-f3d7-496f-b1de-445b7165dd01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtbwp"
Sep 17 18:29:04.584670 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584304    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-node-log\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.584670 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584327    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7fb70a02-f3d7-496f-b1de-445b7165dd01-socket-dir\") pod \"aws-ebs-csi-driver-node-xtbwp\" (UID: \"7fb70a02-f3d7-496f-b1de-445b7165dd01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtbwp"
Sep 17 18:29:04.584670 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584352    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-hostroot\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.584670 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584359    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-node-log\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.584670 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584377    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0058f4db-dbaa-4b8b-942b-06ecbd1301ef-hosts-file\") pod \"node-resolver-mkdtq\" (UID: \"0058f4db-dbaa-4b8b-942b-06ecbd1301ef\") " pod="openshift-dns/node-resolver-mkdtq"
Sep 17 18:29:04.584670 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584401    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-etc-sysctl-d\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.584670 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584421    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/653fa5e9-f667-45f8-a387-1b8a7affdb36-tmp\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.584670 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584441    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/43264d48-807c-43de-8b8f-52db0225d1c9-dbus\") pod \"global-pull-secret-syncer-4zwdx\" (UID: \"43264d48-807c-43de-8b8f-52db0225d1c9\") " pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:04.584670 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584464    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bc0044f2-c713-4de4-b82e-a211c1f398c3-iptables-alerter-script\") pod \"iptables-alerter-vfv5l\" (UID: \"bc0044f2-c713-4de4-b82e-a211c1f398c3\") " pod="openshift-network-operator/iptables-alerter-vfv5l"
Sep 17 18:29:04.584670 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584473    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-hostroot\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.584670 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584486    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-host-run-ovn-kubernetes\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.584670 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584507    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-host-cni-netd\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.584670 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584530    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tztx2\" (UniqueName: \"kubernetes.io/projected/7fb70a02-f3d7-496f-b1de-445b7165dd01-kube-api-access-tztx2\") pod \"aws-ebs-csi-driver-node-xtbwp\" (UID: \"7fb70a02-f3d7-496f-b1de-445b7165dd01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtbwp"
Sep 17 18:29:04.584670 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584553    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-host-run-netns\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.585581 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584557    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-host-run-ovn-kubernetes\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.585581 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584575    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-etc-sysctl-conf\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.585581 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584610    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-sys\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.585581 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584618    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7fb70a02-f3d7-496f-b1de-445b7165dd01-socket-dir\") pod \"aws-ebs-csi-driver-node-xtbwp\" (UID: \"7fb70a02-f3d7-496f-b1de-445b7165dd01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtbwp"
Sep 17 18:29:04.585581 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584627    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-host-run-netns\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.585581 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584635    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-host\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.585581 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584667    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-run-ovn\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.585581 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584706    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-run-ovn\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.585581 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584738    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twrhl\" (UniqueName: \"kubernetes.io/projected/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-kube-api-access-twrhl\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.585581 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584776    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-etc-kubernetes\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.585581 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584802    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42102f16-c1e3-4791-9aea-61a4b415e2ab-system-cni-dir\") pod \"multus-additional-cni-plugins-stwlh\" (UID: \"42102f16-c1e3-4791-9aea-61a4b415e2ab\") " pod="openshift-multus/multus-additional-cni-plugins-stwlh"
Sep 17 18:29:04.585581 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584842    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn6d6\" (UniqueName: \"kubernetes.io/projected/42102f16-c1e3-4791-9aea-61a4b415e2ab-kube-api-access-dn6d6\") pod \"multus-additional-cni-plugins-stwlh\" (UID: \"42102f16-c1e3-4791-9aea-61a4b415e2ab\") " pod="openshift-multus/multus-additional-cni-plugins-stwlh"
Sep 17 18:29:04.585581 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584882    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/822d9b1e-1a38-49ac-900a-6ad946e6078f-host\") pod \"node-ca-glcw2\" (UID: \"822d9b1e-1a38-49ac-900a-6ad946e6078f\") " pod="openshift-image-registry/node-ca-glcw2"
Sep 17 18:29:04.585581 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584918    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/43264d48-807c-43de-8b8f-52db0225d1c9-original-pull-secret\") pod \"global-pull-secret-syncer-4zwdx\" (UID: \"43264d48-807c-43de-8b8f-52db0225d1c9\") " pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:04.585581 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.584950    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2da5dc7d-7056-497b-a8e3-01deda7b2134-agent-certs\") pod \"konnectivity-agent-zttbn\" (UID: \"2da5dc7d-7056-497b-a8e3-01deda7b2134\") " pod="kube-system/konnectivity-agent-zttbn"
Sep 17 18:29:04.585581 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.585002    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7fb70a02-f3d7-496f-b1de-445b7165dd01-sys-fs\") pod \"aws-ebs-csi-driver-node-xtbwp\" (UID: \"7fb70a02-f3d7-496f-b1de-445b7165dd01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtbwp"
Sep 17 18:29:04.585581 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.585029    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-etc-systemd\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.586489 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.585059    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-host-run-netns\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.586489 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.585074    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bc0044f2-c713-4de4-b82e-a211c1f398c3-iptables-alerter-script\") pod \"iptables-alerter-vfv5l\" (UID: \"bc0044f2-c713-4de4-b82e-a211c1f398c3\") " pod="openshift-network-operator/iptables-alerter-vfv5l"
Sep 17 18:29:04.586489 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.585108    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-etc-kubernetes\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.586489 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.585111    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-host-run-netns\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.586489 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.585145    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-host-cni-netd\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.586489 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.585148    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-log-socket\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.586489 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.585178    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-cnibin\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.586489 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.585209    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swc2p\" (UniqueName: \"kubernetes.io/projected/db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4-kube-api-access-swc2p\") pod \"network-metrics-daemon-w67kg\" (UID: \"db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4\") " pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:29:04.586489 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.585233    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-log-socket\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.586489 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.585246    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-852t9\" (UniqueName: \"kubernetes.io/projected/5d3336f6-e38a-4183-8d58-3c99f1284f33-kube-api-access-852t9\") pod \"network-check-target-ssztx\" (UID: \"5d3336f6-e38a-4183-8d58-3c99f1284f33\") " pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:29:04.586489 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.585278    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7fb70a02-f3d7-496f-b1de-445b7165dd01-sys-fs\") pod \"aws-ebs-csi-driver-node-xtbwp\" (UID: \"7fb70a02-f3d7-496f-b1de-445b7165dd01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtbwp"
Sep 17 18:29:04.586489 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.585285    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-cnibin\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.586489 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.585321    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/43264d48-807c-43de-8b8f-52db0225d1c9-kubelet-config\") pod \"global-pull-secret-syncer-4zwdx\" (UID: \"43264d48-807c-43de-8b8f-52db0225d1c9\") " pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:04.586489 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.585364    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc0044f2-c713-4de4-b82e-a211c1f398c3-host-slash\") pod \"iptables-alerter-vfv5l\" (UID: \"bc0044f2-c713-4de4-b82e-a211c1f398c3\") " pod="openshift-network-operator/iptables-alerter-vfv5l"
Sep 17 18:29:04.586489 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.585406    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjjbj\" (UniqueName: \"kubernetes.io/projected/bc0044f2-c713-4de4-b82e-a211c1f398c3-kube-api-access-zjjbj\") pod \"iptables-alerter-vfv5l\" (UID: \"bc0044f2-c713-4de4-b82e-a211c1f398c3\") " pod="openshift-network-operator/iptables-alerter-vfv5l"
Sep 17 18:29:04.586489 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.585434    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bc0044f2-c713-4de4-b82e-a211c1f398c3-host-slash\") pod \"iptables-alerter-vfv5l\" (UID: \"bc0044f2-c713-4de4-b82e-a211c1f398c3\") " pod="openshift-network-operator/iptables-alerter-vfv5l"
Sep 17 18:29:04.586489 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.585464    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.587483 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.585600    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0eeeae49-526f-41f9-8438-3eebbee372f6-cni-binary-copy\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.587483 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.585634    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0eeeae49-526f-41f9-8438-3eebbee372f6-multus-daemon-config\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.587483 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.586190    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0eeeae49-526f-41f9-8438-3eebbee372f6-multus-daemon-config\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.587483 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.586228    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0eeeae49-526f-41f9-8438-3eebbee372f6-cni-binary-copy\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.587483 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.586230    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvt65\" (UniqueName: \"kubernetes.io/projected/0058f4db-dbaa-4b8b-942b-06ecbd1301ef-kube-api-access-kvt65\") pod \"node-resolver-mkdtq\" (UID: \"0058f4db-dbaa-4b8b-942b-06ecbd1301ef\") " pod="openshift-dns/node-resolver-mkdtq"
Sep 17 18:29:04.587483 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.586303    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-etc-sysconfig\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.587483 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.586360    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.587483 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.586364    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4-metrics-certs\") pod \"network-metrics-daemon-w67kg\" (UID: \"db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4\") " pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:29:04.587483 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.586411    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-ovnkube-config\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.587483 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.586449    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-lib-modules\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.587483 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.586485    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-env-overrides\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.587483 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.586520    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7fb70a02-f3d7-496f-b1de-445b7165dd01-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xtbwp\" (UID: \"7fb70a02-f3d7-496f-b1de-445b7165dd01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtbwp"
Sep 17 18:29:04.587483 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.586563    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-multus-cni-dir\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.587483 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.586587    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-multus-socket-dir-parent\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.587483 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.586613    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-host-run-k8s-cni-cncf-io\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.587483 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.586652    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6bkg\" (UniqueName: \"kubernetes.io/projected/653fa5e9-f667-45f8-a387-1b8a7affdb36-kube-api-access-b6bkg\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.587483 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.586691    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-system-cni-dir\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.588439 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.586715    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-host-var-lib-kubelet\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.588439 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.586739    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/653fa5e9-f667-45f8-a387-1b8a7affdb36-etc-tuned\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.588439 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.586767    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2da5dc7d-7056-497b-a8e3-01deda7b2134-konnectivity-ca\") pod \"konnectivity-agent-zttbn\" (UID: \"2da5dc7d-7056-497b-a8e3-01deda7b2134\") " pod="kube-system/konnectivity-agent-zttbn"
Sep 17 18:29:04.588439 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.586805    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-systemd-units\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.588439 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.586831    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-host-slash\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.588439 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.586856    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7fb70a02-f3d7-496f-b1de-445b7165dd01-device-dir\") pod \"aws-ebs-csi-driver-node-xtbwp\" (UID: \"7fb70a02-f3d7-496f-b1de-445b7165dd01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtbwp"
Sep 17 18:29:04.588439 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.586920    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-etc-modprobe-d\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.588439 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.586924    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-ovnkube-config\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.588439 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.586946    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-multus-conf-dir\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.588439 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.586974    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/42102f16-c1e3-4791-9aea-61a4b415e2ab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-stwlh\" (UID: \"42102f16-c1e3-4791-9aea-61a4b415e2ab\") " pod="openshift-multus/multus-additional-cni-plugins-stwlh"
Sep 17 18:29:04.588439 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.586984    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-host-run-k8s-cni-cncf-io\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.588439 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587000    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/822d9b1e-1a38-49ac-900a-6ad946e6078f-serviceca\") pod \"node-ca-glcw2\" (UID: \"822d9b1e-1a38-49ac-900a-6ad946e6078f\") " pod="openshift-image-registry/node-ca-glcw2"
Sep 17 18:29:04.588439 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587026    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-host-kubelet\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.588439 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587050    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-etc-openvswitch\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.588439 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587083    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-host-var-lib-cni-bin\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.588439 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587108    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0058f4db-dbaa-4b8b-942b-06ecbd1301ef-tmp-dir\") pod \"node-resolver-mkdtq\" (UID: \"0058f4db-dbaa-4b8b-942b-06ecbd1301ef\") " pod="openshift-dns/node-resolver-mkdtq"
Sep 17 18:29:04.588439 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587113    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7fb70a02-f3d7-496f-b1de-445b7165dd01-device-dir\") pod \"aws-ebs-csi-driver-node-xtbwp\" (UID: \"7fb70a02-f3d7-496f-b1de-445b7165dd01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtbwp"
Sep 17 18:29:04.589436 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587134    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-var-lib-kubelet\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.589436 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587166    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7fb70a02-f3d7-496f-b1de-445b7165dd01-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xtbwp\" (UID: \"7fb70a02-f3d7-496f-b1de-445b7165dd01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtbwp"
Sep 17 18:29:04.589436 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587171    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-run-systemd\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.589436 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587239    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-multus-cni-dir\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.589436 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587306    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-run-systemd\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.589436 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587309    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-system-cni-dir\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.589436 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587342    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-multus-conf-dir\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.589436 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587361    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-host-kubelet\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.589436 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587367    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-multus-socket-dir-parent\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.589436 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587382    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-host-var-lib-kubelet\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.589436 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587422    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-ovnkube-script-lib\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.589436 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587431    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-systemd-units\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.589436 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587451    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-host-run-multus-certs\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.589436 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587481    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fdjs\" (UniqueName: \"kubernetes.io/projected/0eeeae49-526f-41f9-8438-3eebbee372f6-kube-api-access-4fdjs\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.589436 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587508    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-run\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.589436 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587535    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/42102f16-c1e3-4791-9aea-61a4b415e2ab-cnibin\") pod \"multus-additional-cni-plugins-stwlh\" (UID: \"42102f16-c1e3-4791-9aea-61a4b415e2ab\") " pod="openshift-multus/multus-additional-cni-plugins-stwlh"
Sep 17 18:29:04.589436 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587484    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-host-slash\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.589436 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587603    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-etc-kubernetes\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.590303 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587631    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-var-lib-openvswitch\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.590303 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587672    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-ovn-node-metrics-cert\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.590303 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587697    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7fb70a02-f3d7-496f-b1de-445b7165dd01-registration-dir\") pod \"aws-ebs-csi-driver-node-xtbwp\" (UID: \"7fb70a02-f3d7-496f-b1de-445b7165dd01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtbwp"
Sep 17 18:29:04.590303 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587725    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-host-var-lib-cni-multus\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.590303 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587756    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/42102f16-c1e3-4791-9aea-61a4b415e2ab-cni-binary-copy\") pod \"multus-additional-cni-plugins-stwlh\" (UID: \"42102f16-c1e3-4791-9aea-61a4b415e2ab\") " pod="openshift-multus/multus-additional-cni-plugins-stwlh"
Sep 17 18:29:04.590303 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587784    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/42102f16-c1e3-4791-9aea-61a4b415e2ab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-stwlh\" (UID: \"42102f16-c1e3-4791-9aea-61a4b415e2ab\") " pod="openshift-multus/multus-additional-cni-plugins-stwlh"
Sep 17 18:29:04.590303 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587815    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/42102f16-c1e3-4791-9aea-61a4b415e2ab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-stwlh\" (UID: \"42102f16-c1e3-4791-9aea-61a4b415e2ab\") " pod="openshift-multus/multus-additional-cni-plugins-stwlh"
Sep 17 18:29:04.590303 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587842    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-run-openvswitch\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.590303 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587869    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-host-cni-bin\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.590303 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.587754    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-env-overrides\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.590303 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.588037    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-host-cni-bin\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.590303 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.588075    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-etc-openvswitch\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.590303 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.588076    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-var-lib-openvswitch\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.590303 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.588129    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-host-var-lib-cni-bin\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.590303 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.588188    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-host-run-multus-certs\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.590303 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.588245    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7fb70a02-f3d7-496f-b1de-445b7165dd01-registration-dir\") pod \"aws-ebs-csi-driver-node-xtbwp\" (UID: \"7fb70a02-f3d7-496f-b1de-445b7165dd01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtbwp"
Sep 17 18:29:04.590303 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.588301    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0eeeae49-526f-41f9-8438-3eebbee372f6-host-var-lib-cni-multus\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.591029 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.588497    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 18:29:04.591029 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.588528    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-run-openvswitch\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.591029 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.588647    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-ovnkube-script-lib\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.593549 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.593383    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-ovn-node-metrics-cert\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.595112 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.595088    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tztx2\" (UniqueName: \"kubernetes.io/projected/7fb70a02-f3d7-496f-b1de-445b7165dd01-kube-api-access-tztx2\") pod \"aws-ebs-csi-driver-node-xtbwp\" (UID: \"7fb70a02-f3d7-496f-b1de-445b7165dd01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtbwp"
Sep 17 18:29:04.596173 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.596143    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjjbj\" (UniqueName: \"kubernetes.io/projected/bc0044f2-c713-4de4-b82e-a211c1f398c3-kube-api-access-zjjbj\") pod \"iptables-alerter-vfv5l\" (UID: \"bc0044f2-c713-4de4-b82e-a211c1f398c3\") " pod="openshift-network-operator/iptables-alerter-vfv5l"
Sep 17 18:29:04.596368 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.596345    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twrhl\" (UniqueName: \"kubernetes.io/projected/9d09fbd1-01dd-4ec2-8dec-f05ee9a28709-kube-api-access-twrhl\") pod \"ovnkube-node-grgmp\" (UID: \"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709\") " pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.597586 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.597543    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fdjs\" (UniqueName: \"kubernetes.io/projected/0eeeae49-526f-41f9-8438-3eebbee372f6-kube-api-access-4fdjs\") pod \"multus-mfxhw\" (UID: \"0eeeae49-526f-41f9-8438-3eebbee372f6\") " pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.606518 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.606462    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-75.ec2.internal" event={"ID":"7ef83900f88d178fe684eab642563132","Type":"ContainerStarted","Data":"17f4be9545b46ccb5efc927cb22007c7bd9cf4db58412ecfe3dd0dd234bed55b"}
Sep 17 18:29:04.607957 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.607931    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-75.ec2.internal" event={"ID":"67e17decd2a851abd4f31ec3df7b2fb3","Type":"ContainerStarted","Data":"68b221a0cf1d12215d45a766f148b8825d2a5ed1e75e5c8b5d439e8253527f28"}
Sep 17 18:29:04.691275 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.688917    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/42102f16-c1e3-4791-9aea-61a4b415e2ab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-stwlh\" (UID: \"42102f16-c1e3-4791-9aea-61a4b415e2ab\") " pod="openshift-multus/multus-additional-cni-plugins-stwlh"
Sep 17 18:29:04.691275 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.688966    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/822d9b1e-1a38-49ac-900a-6ad946e6078f-serviceca\") pod \"node-ca-glcw2\" (UID: \"822d9b1e-1a38-49ac-900a-6ad946e6078f\") " pod="openshift-image-registry/node-ca-glcw2"
Sep 17 18:29:04.691275 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.688993    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0058f4db-dbaa-4b8b-942b-06ecbd1301ef-tmp-dir\") pod \"node-resolver-mkdtq\" (UID: \"0058f4db-dbaa-4b8b-942b-06ecbd1301ef\") " pod="openshift-dns/node-resolver-mkdtq"
Sep 17 18:29:04.691275 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.689026    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-var-lib-kubelet\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.691275 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.689061    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-run\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.691275 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.689094    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/42102f16-c1e3-4791-9aea-61a4b415e2ab-cnibin\") pod \"multus-additional-cni-plugins-stwlh\" (UID: \"42102f16-c1e3-4791-9aea-61a4b415e2ab\") " pod="openshift-multus/multus-additional-cni-plugins-stwlh"
Sep 17 18:29:04.691275 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.689127    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-etc-kubernetes\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.691275 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.689157    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/42102f16-c1e3-4791-9aea-61a4b415e2ab-cni-binary-copy\") pod \"multus-additional-cni-plugins-stwlh\" (UID: \"42102f16-c1e3-4791-9aea-61a4b415e2ab\") " pod="openshift-multus/multus-additional-cni-plugins-stwlh"
Sep 17 18:29:04.691275 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.689192    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/42102f16-c1e3-4791-9aea-61a4b415e2ab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-stwlh\" (UID: \"42102f16-c1e3-4791-9aea-61a4b415e2ab\") " pod="openshift-multus/multus-additional-cni-plugins-stwlh"
Sep 17 18:29:04.691275 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.689227    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/42102f16-c1e3-4791-9aea-61a4b415e2ab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-stwlh\" (UID: \"42102f16-c1e3-4791-9aea-61a4b415e2ab\") " pod="openshift-multus/multus-additional-cni-plugins-stwlh"
Sep 17 18:29:04.691275 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.689277    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2szjv\" (UniqueName: \"kubernetes.io/projected/822d9b1e-1a38-49ac-900a-6ad946e6078f-kube-api-access-2szjv\") pod \"node-ca-glcw2\" (UID: \"822d9b1e-1a38-49ac-900a-6ad946e6078f\") " pod="openshift-image-registry/node-ca-glcw2"
Sep 17 18:29:04.691275 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.689312    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0058f4db-dbaa-4b8b-942b-06ecbd1301ef-hosts-file\") pod \"node-resolver-mkdtq\" (UID: \"0058f4db-dbaa-4b8b-942b-06ecbd1301ef\") " pod="openshift-dns/node-resolver-mkdtq"
Sep 17 18:29:04.691275 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.689340    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-etc-sysctl-d\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.691275 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.689373    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/653fa5e9-f667-45f8-a387-1b8a7affdb36-tmp\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.691275 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.689403    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/43264d48-807c-43de-8b8f-52db0225d1c9-dbus\") pod \"global-pull-secret-syncer-4zwdx\" (UID: \"43264d48-807c-43de-8b8f-52db0225d1c9\") " pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:04.691275 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.689440    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-etc-sysctl-conf\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.691275 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.689469    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-sys\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.692306 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.689502    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-host\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.692306 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.689543    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42102f16-c1e3-4791-9aea-61a4b415e2ab-system-cni-dir\") pod \"multus-additional-cni-plugins-stwlh\" (UID: \"42102f16-c1e3-4791-9aea-61a4b415e2ab\") " pod="openshift-multus/multus-additional-cni-plugins-stwlh"
Sep 17 18:29:04.692306 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.689578    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn6d6\" (UniqueName: \"kubernetes.io/projected/42102f16-c1e3-4791-9aea-61a4b415e2ab-kube-api-access-dn6d6\") pod \"multus-additional-cni-plugins-stwlh\" (UID: \"42102f16-c1e3-4791-9aea-61a4b415e2ab\") " pod="openshift-multus/multus-additional-cni-plugins-stwlh"
Sep 17 18:29:04.692306 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.689614    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/822d9b1e-1a38-49ac-900a-6ad946e6078f-host\") pod \"node-ca-glcw2\" (UID: \"822d9b1e-1a38-49ac-900a-6ad946e6078f\") " pod="openshift-image-registry/node-ca-glcw2"
Sep 17 18:29:04.692306 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.689650    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/43264d48-807c-43de-8b8f-52db0225d1c9-original-pull-secret\") pod \"global-pull-secret-syncer-4zwdx\" (UID: \"43264d48-807c-43de-8b8f-52db0225d1c9\") " pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:04.692306 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.689691    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2da5dc7d-7056-497b-a8e3-01deda7b2134-agent-certs\") pod \"konnectivity-agent-zttbn\" (UID: \"2da5dc7d-7056-497b-a8e3-01deda7b2134\") " pod="kube-system/konnectivity-agent-zttbn"
Sep 17 18:29:04.692306 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.689732    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-etc-systemd\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.692306 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.689773    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swc2p\" (UniqueName: \"kubernetes.io/projected/db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4-kube-api-access-swc2p\") pod \"network-metrics-daemon-w67kg\" (UID: \"db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4\") " pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:29:04.692306 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.689809    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-852t9\" (UniqueName: \"kubernetes.io/projected/5d3336f6-e38a-4183-8d58-3c99f1284f33-kube-api-access-852t9\") pod \"network-check-target-ssztx\" (UID: \"5d3336f6-e38a-4183-8d58-3c99f1284f33\") " pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:29:04.692306 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.689840    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/43264d48-807c-43de-8b8f-52db0225d1c9-kubelet-config\") pod \"global-pull-secret-syncer-4zwdx\" (UID: \"43264d48-807c-43de-8b8f-52db0225d1c9\") " pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:04.692306 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.689881    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvt65\" (UniqueName: \"kubernetes.io/projected/0058f4db-dbaa-4b8b-942b-06ecbd1301ef-kube-api-access-kvt65\") pod \"node-resolver-mkdtq\" (UID: \"0058f4db-dbaa-4b8b-942b-06ecbd1301ef\") " pod="openshift-dns/node-resolver-mkdtq"
Sep 17 18:29:04.692306 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.689915    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-etc-sysconfig\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.692306 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.689946    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4-metrics-certs\") pod \"network-metrics-daemon-w67kg\" (UID: \"db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4\") " pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:29:04.692306 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.689976    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-lib-modules\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.692306 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.690014    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6bkg\" (UniqueName: \"kubernetes.io/projected/653fa5e9-f667-45f8-a387-1b8a7affdb36-kube-api-access-b6bkg\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.692306 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.690049    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/653fa5e9-f667-45f8-a387-1b8a7affdb36-etc-tuned\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.692306 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.690081    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2da5dc7d-7056-497b-a8e3-01deda7b2134-konnectivity-ca\") pod \"konnectivity-agent-zttbn\" (UID: \"2da5dc7d-7056-497b-a8e3-01deda7b2134\") " pod="kube-system/konnectivity-agent-zttbn"
Sep 17 18:29:04.693201 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.690116    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-etc-modprobe-d\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.693201 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.690307    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-etc-modprobe-d\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.693201 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:04.690792    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:04.693201 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.690802    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/43264d48-807c-43de-8b8f-52db0225d1c9-kubelet-config\") pod \"global-pull-secret-syncer-4zwdx\" (UID: \"43264d48-807c-43de-8b8f-52db0225d1c9\") " pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:04.693201 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.690904    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-etc-sysconfig\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.693201 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.690794    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-etc-kubernetes\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.693201 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:04.691023    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43264d48-807c-43de-8b8f-52db0225d1c9-original-pull-secret podName:43264d48-807c-43de-8b8f-52db0225d1c9 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:05.190951419 +0000 UTC m=+3.123152923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/43264d48-807c-43de-8b8f-52db0225d1c9-original-pull-secret") pod "global-pull-secret-syncer-4zwdx" (UID: "43264d48-807c-43de-8b8f-52db0225d1c9") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:04.693201 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.691135    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-sys\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.693201 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.691151    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-etc-sysctl-d\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.693201 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.691210    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/822d9b1e-1a38-49ac-900a-6ad946e6078f-host\") pod \"node-ca-glcw2\" (UID: \"822d9b1e-1a38-49ac-900a-6ad946e6078f\") " pod="openshift-image-registry/node-ca-glcw2"
Sep 17 18:29:04.693201 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.691231    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/42102f16-c1e3-4791-9aea-61a4b415e2ab-cnibin\") pod \"multus-additional-cni-plugins-stwlh\" (UID: \"42102f16-c1e3-4791-9aea-61a4b415e2ab\") " pod="openshift-multus/multus-additional-cni-plugins-stwlh"
Sep 17 18:29:04.693201 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.691298    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-etc-systemd\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.693201 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.690718    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-run\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.693201 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.691014    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0058f4db-dbaa-4b8b-942b-06ecbd1301ef-hosts-file\") pod \"node-resolver-mkdtq\" (UID: \"0058f4db-dbaa-4b8b-942b-06ecbd1301ef\") " pod="openshift-dns/node-resolver-mkdtq"
Sep 17 18:29:04.693201 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.691373    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0058f4db-dbaa-4b8b-942b-06ecbd1301ef-tmp-dir\") pod \"node-resolver-mkdtq\" (UID: \"0058f4db-dbaa-4b8b-942b-06ecbd1301ef\") " pod="openshift-dns/node-resolver-mkdtq"
Sep 17 18:29:04.693201 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.691389    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/42102f16-c1e3-4791-9aea-61a4b415e2ab-system-cni-dir\") pod \"multus-additional-cni-plugins-stwlh\" (UID: \"42102f16-c1e3-4791-9aea-61a4b415e2ab\") " pod="openshift-multus/multus-additional-cni-plugins-stwlh"
Sep 17 18:29:04.693201 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.691524    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-etc-sysctl-conf\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.694062 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.691558    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/42102f16-c1e3-4791-9aea-61a4b415e2ab-cni-binary-copy\") pod \"multus-additional-cni-plugins-stwlh\" (UID: \"42102f16-c1e3-4791-9aea-61a4b415e2ab\") " pod="openshift-multus/multus-additional-cni-plugins-stwlh"
Sep 17 18:29:04.694062 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.691623    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-var-lib-kubelet\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.694062 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.691782    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-lib-modules\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.694062 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:04.691893    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:04.694062 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.691972    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/822d9b1e-1a38-49ac-900a-6ad946e6078f-serviceca\") pod \"node-ca-glcw2\" (UID: \"822d9b1e-1a38-49ac-900a-6ad946e6078f\") " pod="openshift-image-registry/node-ca-glcw2"
Sep 17 18:29:04.694062 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:04.692011    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4-metrics-certs podName:db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:05.191994871 +0000 UTC m=+3.124196387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4-metrics-certs") pod "network-metrics-daemon-w67kg" (UID: "db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:04.694062 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.692069    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/653fa5e9-f667-45f8-a387-1b8a7affdb36-host\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.694062 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.692079    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/43264d48-807c-43de-8b8f-52db0225d1c9-dbus\") pod \"global-pull-secret-syncer-4zwdx\" (UID: \"43264d48-807c-43de-8b8f-52db0225d1c9\") " pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:04.694062 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.692307    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/42102f16-c1e3-4791-9aea-61a4b415e2ab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-stwlh\" (UID: \"42102f16-c1e3-4791-9aea-61a4b415e2ab\") " pod="openshift-multus/multus-additional-cni-plugins-stwlh"
Sep 17 18:29:04.694062 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.692587    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/42102f16-c1e3-4791-9aea-61a4b415e2ab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-stwlh\" (UID: \"42102f16-c1e3-4791-9aea-61a4b415e2ab\") " pod="openshift-multus/multus-additional-cni-plugins-stwlh"
Sep 17 18:29:04.694062 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.692597    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2da5dc7d-7056-497b-a8e3-01deda7b2134-konnectivity-ca\") pod \"konnectivity-agent-zttbn\" (UID: \"2da5dc7d-7056-497b-a8e3-01deda7b2134\") " pod="kube-system/konnectivity-agent-zttbn"
Sep 17 18:29:04.694062 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.693894    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/653fa5e9-f667-45f8-a387-1b8a7affdb36-tmp\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.694647 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.694365    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/653fa5e9-f667-45f8-a387-1b8a7affdb36-etc-tuned\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.694647 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.694349    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/42102f16-c1e3-4791-9aea-61a4b415e2ab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-stwlh\" (UID: \"42102f16-c1e3-4791-9aea-61a4b415e2ab\") " pod="openshift-multus/multus-additional-cni-plugins-stwlh"
Sep 17 18:29:04.695159 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.695138    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2da5dc7d-7056-497b-a8e3-01deda7b2134-agent-certs\") pod \"konnectivity-agent-zttbn\" (UID: \"2da5dc7d-7056-497b-a8e3-01deda7b2134\") " pod="kube-system/konnectivity-agent-zttbn"
Sep 17 18:29:04.701070 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.700991    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvt65\" (UniqueName: \"kubernetes.io/projected/0058f4db-dbaa-4b8b-942b-06ecbd1301ef-kube-api-access-kvt65\") pod \"node-resolver-mkdtq\" (UID: \"0058f4db-dbaa-4b8b-942b-06ecbd1301ef\") " pod="openshift-dns/node-resolver-mkdtq"
Sep 17 18:29:04.702126 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.702101    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2szjv\" (UniqueName: \"kubernetes.io/projected/822d9b1e-1a38-49ac-900a-6ad946e6078f-kube-api-access-2szjv\") pod \"node-ca-glcw2\" (UID: \"822d9b1e-1a38-49ac-900a-6ad946e6078f\") " pod="openshift-image-registry/node-ca-glcw2"
Sep 17 18:29:04.702383 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.702364    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn6d6\" (UniqueName: \"kubernetes.io/projected/42102f16-c1e3-4791-9aea-61a4b415e2ab-kube-api-access-dn6d6\") pod \"multus-additional-cni-plugins-stwlh\" (UID: \"42102f16-c1e3-4791-9aea-61a4b415e2ab\") " pod="openshift-multus/multus-additional-cni-plugins-stwlh"
Sep 17 18:29:04.702939 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.702915    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swc2p\" (UniqueName: \"kubernetes.io/projected/db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4-kube-api-access-swc2p\") pod \"network-metrics-daemon-w67kg\" (UID: \"db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4\") " pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:29:04.703184 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:04.703163    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:04.703294 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:04.703189    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:04.703294 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:04.703203    2565 projected.go:196] Error preparing data for projected volume kube-api-access-852t9 for pod openshift-network-diagnostics/network-check-target-ssztx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:04.703294 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:04.703284    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d3336f6-e38a-4183-8d58-3c99f1284f33-kube-api-access-852t9 podName:5d3336f6-e38a-4183-8d58-3c99f1284f33 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:05.203249343 +0000 UTC m=+3.135450858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-852t9" (UniqueName: "kubernetes.io/projected/5d3336f6-e38a-4183-8d58-3c99f1284f33-kube-api-access-852t9") pod "network-check-target-ssztx" (UID: "5d3336f6-e38a-4183-8d58-3c99f1284f33") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:04.705426 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.705406    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6bkg\" (UniqueName: \"kubernetes.io/projected/653fa5e9-f667-45f8-a387-1b8a7affdb36-kube-api-access-b6bkg\") pod \"tuned-pwktj\" (UID: \"653fa5e9-f667-45f8-a387-1b8a7affdb36\") " pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.780822 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.780789    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mfxhw"
Sep 17 18:29:04.789507 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.789481    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vfv5l"
Sep 17 18:29:04.810717 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.810699    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:04.817337 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.817318    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtbwp"
Sep 17 18:29:04.825167 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.825143    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mkdtq"
Sep 17 18:29:04.833030 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.833010    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pwktj"
Sep 17 18:29:04.841945 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.841929    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zttbn"
Sep 17 18:29:04.848834 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.848815    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-glcw2"
Sep 17 18:29:04.855563 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.855539    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-stwlh"
Sep 17 18:29:04.940529 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:04.940504    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:05.194266 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:05.194236    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/43264d48-807c-43de-8b8f-52db0225d1c9-original-pull-secret\") pod \"global-pull-secret-syncer-4zwdx\" (UID: \"43264d48-807c-43de-8b8f-52db0225d1c9\") " pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:05.194447 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:05.194292    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4-metrics-certs\") pod \"network-metrics-daemon-w67kg\" (UID: \"db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4\") " pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:29:05.194447 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:05.194400    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:05.194566 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:05.194471    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4-metrics-certs podName:db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:06.194451153 +0000 UTC m=+4.126652670 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4-metrics-certs") pod "network-metrics-daemon-w67kg" (UID: "db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:05.194566 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:05.194401    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:05.194660 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:05.194569    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43264d48-807c-43de-8b8f-52db0225d1c9-original-pull-secret podName:43264d48-807c-43de-8b8f-52db0225d1c9 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:06.194543144 +0000 UTC m=+4.126744647 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/43264d48-807c-43de-8b8f-52db0225d1c9-original-pull-secret") pod "global-pull-secret-syncer-4zwdx" (UID: "43264d48-807c-43de-8b8f-52db0225d1c9") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:05.294558 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:05.294520    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-852t9\" (UniqueName: \"kubernetes.io/projected/5d3336f6-e38a-4183-8d58-3c99f1284f33-kube-api-access-852t9\") pod \"network-check-target-ssztx\" (UID: \"5d3336f6-e38a-4183-8d58-3c99f1284f33\") " pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:29:05.294720 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:05.294672    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:05.294720 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:05.294698    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:05.294720 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:05.294711    2565 projected.go:196] Error preparing data for projected volume kube-api-access-852t9 for pod openshift-network-diagnostics/network-check-target-ssztx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:05.294884 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:05.294773    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d3336f6-e38a-4183-8d58-3c99f1284f33-kube-api-access-852t9 podName:5d3336f6-e38a-4183-8d58-3c99f1284f33 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:06.294754475 +0000 UTC m=+4.226955977 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-852t9" (UniqueName: "kubernetes.io/projected/5d3336f6-e38a-4183-8d58-3c99f1284f33-kube-api-access-852t9") pod "network-check-target-ssztx" (UID: "5d3336f6-e38a-4183-8d58-3c99f1284f33") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:05.531517 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:05.531434    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 18:24:05 +0000 UTC" deadline="2028-02-13 12:35:04.346274995 +0000 UTC"
Sep 17 18:29:05.531517 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:05.531471    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12330h5m58.814808273s"
Sep 17 18:29:05.601450 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:05.601421    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:29:05.601622 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:05.601544    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w67kg" podUID="db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4"
Sep 17 18:29:05.773666 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:05.773576    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653fa5e9_f667_45f8_a387_1b8a7affdb36.slice/crio-8ef688b8d65190dd9ff6da7d3b40268f1b6011671a14c8f9afd28e7325a64292 WatchSource:0}: Error finding container 8ef688b8d65190dd9ff6da7d3b40268f1b6011671a14c8f9afd28e7325a64292: Status 404 returned error can't find the container with id 8ef688b8d65190dd9ff6da7d3b40268f1b6011671a14c8f9afd28e7325a64292
Sep 17 18:29:05.774885 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:05.774854    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eeeae49_526f_41f9_8438_3eebbee372f6.slice/crio-f76581d74001baca71b58e15a50f975bb0063f4c0644cb8cfb53d877e67b1cae WatchSource:0}: Error finding container f76581d74001baca71b58e15a50f975bb0063f4c0644cb8cfb53d877e67b1cae: Status 404 returned error can't find the container with id f76581d74001baca71b58e15a50f975bb0063f4c0644cb8cfb53d877e67b1cae
Sep 17 18:29:05.779676 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:05.779650    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0058f4db_dbaa_4b8b_942b_06ecbd1301ef.slice/crio-bc5a8a5cd45c0b38c65381b54a079b3a13e44fc0a4cf405675f3b19597ad3eef WatchSource:0}: Error finding container bc5a8a5cd45c0b38c65381b54a079b3a13e44fc0a4cf405675f3b19597ad3eef: Status 404 returned error can't find the container with id bc5a8a5cd45c0b38c65381b54a079b3a13e44fc0a4cf405675f3b19597ad3eef
Sep 17 18:29:05.781463 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:05.781445    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc0044f2_c713_4de4_b82e_a211c1f398c3.slice/crio-078fdceb7484bfb80db17248cc8e302c657a7a23687c4450e1dc791fc54c27c6 WatchSource:0}: Error finding container 078fdceb7484bfb80db17248cc8e302c657a7a23687c4450e1dc791fc54c27c6: Status 404 returned error can't find the container with id 078fdceb7484bfb80db17248cc8e302c657a7a23687c4450e1dc791fc54c27c6
Sep 17 18:29:05.783344 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:05.783302    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod822d9b1e_1a38_49ac_900a_6ad946e6078f.slice/crio-1d05ae63e7de0b57b44c3c93352c3954ac903a4e27e1959a53719abeccfc9470 WatchSource:0}: Error finding container 1d05ae63e7de0b57b44c3c93352c3954ac903a4e27e1959a53719abeccfc9470: Status 404 returned error can't find the container with id 1d05ae63e7de0b57b44c3c93352c3954ac903a4e27e1959a53719abeccfc9470
Sep 17 18:29:05.786015 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:05.785874    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb70a02_f3d7_496f_b1de_445b7165dd01.slice/crio-557fec7b8c24c518f1f7001ef7e870c141d1a43e7d89727620c998dac7f739b9 WatchSource:0}: Error finding container 557fec7b8c24c518f1f7001ef7e870c141d1a43e7d89727620c998dac7f739b9: Status 404 returned error can't find the container with id 557fec7b8c24c518f1f7001ef7e870c141d1a43e7d89727620c998dac7f739b9
Sep 17 18:29:06.201984 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:06.201741    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/43264d48-807c-43de-8b8f-52db0225d1c9-original-pull-secret\") pod \"global-pull-secret-syncer-4zwdx\" (UID: \"43264d48-807c-43de-8b8f-52db0225d1c9\") " pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:06.202124 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:06.202034    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4-metrics-certs\") pod \"network-metrics-daemon-w67kg\" (UID: \"db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4\") " pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:29:06.202124 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:06.201902    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:06.202257 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:06.202145    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:06.202257 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:06.202155    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43264d48-807c-43de-8b8f-52db0225d1c9-original-pull-secret podName:43264d48-807c-43de-8b8f-52db0225d1c9 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:08.20213468 +0000 UTC m=+6.134336182 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/43264d48-807c-43de-8b8f-52db0225d1c9-original-pull-secret") pod "global-pull-secret-syncer-4zwdx" (UID: "43264d48-807c-43de-8b8f-52db0225d1c9") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:06.202257 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:06.202235    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4-metrics-certs podName:db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:08.202216297 +0000 UTC m=+6.134417814 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4-metrics-certs") pod "network-metrics-daemon-w67kg" (UID: "db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:06.303664 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:06.302946    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-852t9\" (UniqueName: \"kubernetes.io/projected/5d3336f6-e38a-4183-8d58-3c99f1284f33-kube-api-access-852t9\") pod \"network-check-target-ssztx\" (UID: \"5d3336f6-e38a-4183-8d58-3c99f1284f33\") " pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:29:06.303664 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:06.303155    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:06.303664 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:06.303181    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:06.303664 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:06.303195    2565 projected.go:196] Error preparing data for projected volume kube-api-access-852t9 for pod openshift-network-diagnostics/network-check-target-ssztx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:06.303664 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:06.303275    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d3336f6-e38a-4183-8d58-3c99f1284f33-kube-api-access-852t9 podName:5d3336f6-e38a-4183-8d58-3c99f1284f33 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:08.303253848 +0000 UTC m=+6.235455354 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-852t9" (UniqueName: "kubernetes.io/projected/5d3336f6-e38a-4183-8d58-3c99f1284f33-kube-api-access-852t9") pod "network-check-target-ssztx" (UID: "5d3336f6-e38a-4183-8d58-3c99f1284f33") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:06.532404 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:06.532358    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 18:24:05 +0000 UTC" deadline="2028-05-16 13:58:57.835880318 +0000 UTC"
Sep 17 18:29:06.532404 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:06.532401    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14563h29m51.303484057s"
Sep 17 18:29:06.602944 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:06.602383    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:29:06.602944 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:06.602522    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssztx" podUID="5d3336f6-e38a-4183-8d58-3c99f1284f33"
Sep 17 18:29:06.602944 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:06.602579    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:06.602944 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:06.602721    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zwdx" podUID="43264d48-807c-43de-8b8f-52db0225d1c9"
Sep 17 18:29:06.639711 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:06.639578    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-grgmp" event={"ID":"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709","Type":"ContainerStarted","Data":"153961cc25b422e2c54dec10c07e7b3a0703b08c07573e8c81fa4e07b65f4cce"}
Sep 17 18:29:06.643613 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:06.643532    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtbwp" event={"ID":"7fb70a02-f3d7-496f-b1de-445b7165dd01","Type":"ContainerStarted","Data":"557fec7b8c24c518f1f7001ef7e870c141d1a43e7d89727620c998dac7f739b9"}
Sep 17 18:29:06.655175 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:06.655102    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vfv5l" event={"ID":"bc0044f2-c713-4de4-b82e-a211c1f398c3","Type":"ContainerStarted","Data":"078fdceb7484bfb80db17248cc8e302c657a7a23687c4450e1dc791fc54c27c6"}
Sep 17 18:29:06.673347 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:06.673291    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stwlh" event={"ID":"42102f16-c1e3-4791-9aea-61a4b415e2ab","Type":"ContainerStarted","Data":"603333c3fba437eace17d6d19692f5980cce6f907b2d7091fb2d7f1bd0443f4e"}
Sep 17 18:29:06.676180 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:06.676120    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-glcw2" event={"ID":"822d9b1e-1a38-49ac-900a-6ad946e6078f","Type":"ContainerStarted","Data":"1d05ae63e7de0b57b44c3c93352c3954ac903a4e27e1959a53719abeccfc9470"}
Sep 17 18:29:06.682207 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:06.682144    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mkdtq" event={"ID":"0058f4db-dbaa-4b8b-942b-06ecbd1301ef","Type":"ContainerStarted","Data":"bc5a8a5cd45c0b38c65381b54a079b3a13e44fc0a4cf405675f3b19597ad3eef"}
Sep 17 18:29:06.692317 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:06.692292    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mfxhw" event={"ID":"0eeeae49-526f-41f9-8438-3eebbee372f6","Type":"ContainerStarted","Data":"f76581d74001baca71b58e15a50f975bb0063f4c0644cb8cfb53d877e67b1cae"}
Sep 17 18:29:06.717460 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:06.717426    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pwktj" event={"ID":"653fa5e9-f667-45f8-a387-1b8a7affdb36","Type":"ContainerStarted","Data":"8ef688b8d65190dd9ff6da7d3b40268f1b6011671a14c8f9afd28e7325a64292"}
Sep 17 18:29:06.738890 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:06.737943    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-75.ec2.internal" event={"ID":"7ef83900f88d178fe684eab642563132","Type":"ContainerStarted","Data":"be575afb63a5eae49769d90d60879406a41c9dcb133fba80f815013e3cc4cce4"}
Sep 17 18:29:06.750331 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:06.749696    2565 generic.go:356] "Generic (PLEG): container finished" podID="67e17decd2a851abd4f31ec3df7b2fb3" containerID="cf95a3b15b3d4ccbe362940917d62762a3ed7d04ba029cdc29f407aa6a976333" exitCode=0
Sep 17 18:29:06.750331 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:06.749952    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-75.ec2.internal" event={"ID":"67e17decd2a851abd4f31ec3df7b2fb3","Type":"ContainerDied","Data":"cf95a3b15b3d4ccbe362940917d62762a3ed7d04ba029cdc29f407aa6a976333"}
Sep 17 18:29:06.777152 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:06.776019    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-75.ec2.internal" podStartSLOduration=2.776000771 podStartE2EDuration="2.776000771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:06.757194419 +0000 UTC m=+4.689395943" watchObservedRunningTime="2026-09-17 18:29:06.776000771 +0000 UTC m=+4.708202294"
Sep 17 18:29:06.778546 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:06.778489    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zttbn" event={"ID":"2da5dc7d-7056-497b-a8e3-01deda7b2134","Type":"ContainerStarted","Data":"764eae4f373911573b022165bd34008b3bf779038e7ee844b529aba9021fcac1"}
Sep 17 18:29:07.601509 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:07.600778    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:29:07.601509 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:07.600916    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w67kg" podUID="db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4"
Sep 17 18:29:07.801614 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:07.801484    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-75.ec2.internal" event={"ID":"67e17decd2a851abd4f31ec3df7b2fb3","Type":"ContainerStarted","Data":"f72ff778d6528149430ee54f9ef0b3989eaf206f34c7be5172b16c94d9079c6d"}
Sep 17 18:29:08.218588 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:08.218553    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/43264d48-807c-43de-8b8f-52db0225d1c9-original-pull-secret\") pod \"global-pull-secret-syncer-4zwdx\" (UID: \"43264d48-807c-43de-8b8f-52db0225d1c9\") " pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:08.218793 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:08.218603    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4-metrics-certs\") pod \"network-metrics-daemon-w67kg\" (UID: \"db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4\") " pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:29:08.218873 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:08.218854    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:08.218939 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:08.218921    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4-metrics-certs podName:db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:12.218900671 +0000 UTC m=+10.151102173 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4-metrics-certs") pod "network-metrics-daemon-w67kg" (UID: "db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:08.219384 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:08.219365    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:08.219502 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:08.219420    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43264d48-807c-43de-8b8f-52db0225d1c9-original-pull-secret podName:43264d48-807c-43de-8b8f-52db0225d1c9 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:12.219404991 +0000 UTC m=+10.151606505 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/43264d48-807c-43de-8b8f-52db0225d1c9-original-pull-secret") pod "global-pull-secret-syncer-4zwdx" (UID: "43264d48-807c-43de-8b8f-52db0225d1c9") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:08.319918 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:08.319844    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-852t9\" (UniqueName: \"kubernetes.io/projected/5d3336f6-e38a-4183-8d58-3c99f1284f33-kube-api-access-852t9\") pod \"network-check-target-ssztx\" (UID: \"5d3336f6-e38a-4183-8d58-3c99f1284f33\") " pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:29:08.320097 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:08.320037    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:08.320097 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:08.320061    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:08.320097 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:08.320075    2565 projected.go:196] Error preparing data for projected volume kube-api-access-852t9 for pod openshift-network-diagnostics/network-check-target-ssztx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:08.320263 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:08.320135    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d3336f6-e38a-4183-8d58-3c99f1284f33-kube-api-access-852t9 podName:5d3336f6-e38a-4183-8d58-3c99f1284f33 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:12.320116371 +0000 UTC m=+10.252317883 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-852t9" (UniqueName: "kubernetes.io/projected/5d3336f6-e38a-4183-8d58-3c99f1284f33-kube-api-access-852t9") pod "network-check-target-ssztx" (UID: "5d3336f6-e38a-4183-8d58-3c99f1284f33") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:08.603983 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:08.603901    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:29:08.604484 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:08.604034    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssztx" podUID="5d3336f6-e38a-4183-8d58-3c99f1284f33"
Sep 17 18:29:08.604484 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:08.604111    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:08.604484 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:08.604186    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zwdx" podUID="43264d48-807c-43de-8b8f-52db0225d1c9"
Sep 17 18:29:09.601024 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:09.600996    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:29:09.601202 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:09.601103    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w67kg" podUID="db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4"
Sep 17 18:29:10.601800 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:10.601756    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:29:10.602308 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:10.601903    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssztx" podUID="5d3336f6-e38a-4183-8d58-3c99f1284f33"
Sep 17 18:29:10.602308 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:10.601757    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:10.602435 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:10.602372    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zwdx" podUID="43264d48-807c-43de-8b8f-52db0225d1c9"
Sep 17 18:29:11.601219 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:11.601186    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:29:11.601411 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:11.601349    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w67kg" podUID="db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4"
Sep 17 18:29:12.253927 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:12.253869    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4-metrics-certs\") pod \"network-metrics-daemon-w67kg\" (UID: \"db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4\") " pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:29:12.254566 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:12.253951    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/43264d48-807c-43de-8b8f-52db0225d1c9-original-pull-secret\") pod \"global-pull-secret-syncer-4zwdx\" (UID: \"43264d48-807c-43de-8b8f-52db0225d1c9\") " pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:12.254566 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:12.254087    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:12.254566 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:12.254147    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43264d48-807c-43de-8b8f-52db0225d1c9-original-pull-secret podName:43264d48-807c-43de-8b8f-52db0225d1c9 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:20.254130372 +0000 UTC m=+18.186331870 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/43264d48-807c-43de-8b8f-52db0225d1c9-original-pull-secret") pod "global-pull-secret-syncer-4zwdx" (UID: "43264d48-807c-43de-8b8f-52db0225d1c9") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:12.254566 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:12.254555    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:12.254815 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:12.254607    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4-metrics-certs podName:db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:20.254590162 +0000 UTC m=+18.186791660 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4-metrics-certs") pod "network-metrics-daemon-w67kg" (UID: "db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:12.354598 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:12.354565    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-852t9\" (UniqueName: \"kubernetes.io/projected/5d3336f6-e38a-4183-8d58-3c99f1284f33-kube-api-access-852t9\") pod \"network-check-target-ssztx\" (UID: \"5d3336f6-e38a-4183-8d58-3c99f1284f33\") " pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:29:12.355186 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:12.354752    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:12.355186 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:12.354780    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:12.355186 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:12.354796    2565 projected.go:196] Error preparing data for projected volume kube-api-access-852t9 for pod openshift-network-diagnostics/network-check-target-ssztx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:12.355186 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:12.354860    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d3336f6-e38a-4183-8d58-3c99f1284f33-kube-api-access-852t9 podName:5d3336f6-e38a-4183-8d58-3c99f1284f33 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:20.354840523 +0000 UTC m=+18.287042052 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-852t9" (UniqueName: "kubernetes.io/projected/5d3336f6-e38a-4183-8d58-3c99f1284f33-kube-api-access-852t9") pod "network-check-target-ssztx" (UID: "5d3336f6-e38a-4183-8d58-3c99f1284f33") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:12.604773 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:12.604679    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:12.604773 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:12.604693    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:29:12.604986 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:12.604797    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssztx" podUID="5d3336f6-e38a-4183-8d58-3c99f1284f33"
Sep 17 18:29:12.604986 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:12.604957    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zwdx" podUID="43264d48-807c-43de-8b8f-52db0225d1c9"
Sep 17 18:29:13.600881 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:13.600842    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:29:13.601382 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:13.600997    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w67kg" podUID="db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4"
Sep 17 18:29:14.600793 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:14.600706    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:14.601005 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:14.600847    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zwdx" podUID="43264d48-807c-43de-8b8f-52db0225d1c9"
Sep 17 18:29:14.601005 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:14.600919    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:29:14.601485 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:14.601043    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssztx" podUID="5d3336f6-e38a-4183-8d58-3c99f1284f33"
Sep 17 18:29:15.601102 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:15.601072    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:29:15.601470 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:15.601168    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w67kg" podUID="db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4"
Sep 17 18:29:16.600782 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:16.600752    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:29:16.600946 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:16.600863    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssztx" podUID="5d3336f6-e38a-4183-8d58-3c99f1284f33"
Sep 17 18:29:16.601034 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:16.600955    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:16.601084 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:16.601065    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zwdx" podUID="43264d48-807c-43de-8b8f-52db0225d1c9"
Sep 17 18:29:17.601694 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:17.601656    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:29:17.602128 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:17.601793    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w67kg" podUID="db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4"
Sep 17 18:29:18.601106 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:18.601071    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:29:18.601317 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:18.601074    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:18.601317 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:18.601195    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssztx" podUID="5d3336f6-e38a-4183-8d58-3c99f1284f33"
Sep 17 18:29:18.601317 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:18.601299    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zwdx" podUID="43264d48-807c-43de-8b8f-52db0225d1c9"
Sep 17 18:29:19.601656 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:19.601619    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:29:19.602109 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:19.601756    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w67kg" podUID="db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4"
Sep 17 18:29:20.311547 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:20.311509    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/43264d48-807c-43de-8b8f-52db0225d1c9-original-pull-secret\") pod \"global-pull-secret-syncer-4zwdx\" (UID: \"43264d48-807c-43de-8b8f-52db0225d1c9\") " pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:20.311704 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:20.311583    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4-metrics-certs\") pod \"network-metrics-daemon-w67kg\" (UID: \"db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4\") " pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:29:20.311704 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:20.311619    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:20.311704 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:20.311682    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43264d48-807c-43de-8b8f-52db0225d1c9-original-pull-secret podName:43264d48-807c-43de-8b8f-52db0225d1c9 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:36.311665527 +0000 UTC m=+34.243867040 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/43264d48-807c-43de-8b8f-52db0225d1c9-original-pull-secret") pod "global-pull-secret-syncer-4zwdx" (UID: "43264d48-807c-43de-8b8f-52db0225d1c9") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:20.311704 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:20.311693    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:20.311897 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:20.311745    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4-metrics-certs podName:db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:36.311728447 +0000 UTC m=+34.243929953 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4-metrics-certs") pod "network-metrics-daemon-w67kg" (UID: "db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:20.412714 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:20.412684    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-852t9\" (UniqueName: \"kubernetes.io/projected/5d3336f6-e38a-4183-8d58-3c99f1284f33-kube-api-access-852t9\") pod \"network-check-target-ssztx\" (UID: \"5d3336f6-e38a-4183-8d58-3c99f1284f33\") " pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:29:20.412888 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:20.412801    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:20.412888 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:20.412822    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:20.412888 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:20.412831    2565 projected.go:196] Error preparing data for projected volume kube-api-access-852t9 for pod openshift-network-diagnostics/network-check-target-ssztx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:20.412888 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:20.412877    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d3336f6-e38a-4183-8d58-3c99f1284f33-kube-api-access-852t9 podName:5d3336f6-e38a-4183-8d58-3c99f1284f33 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:36.412861171 +0000 UTC m=+34.345062673 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-852t9" (UniqueName: "kubernetes.io/projected/5d3336f6-e38a-4183-8d58-3c99f1284f33-kube-api-access-852t9") pod "network-check-target-ssztx" (UID: "5d3336f6-e38a-4183-8d58-3c99f1284f33") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:20.600773 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:20.600715    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:29:20.600773 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:20.600726    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:20.600943 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:20.600799    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssztx" podUID="5d3336f6-e38a-4183-8d58-3c99f1284f33"
Sep 17 18:29:20.600943 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:20.600920    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zwdx" podUID="43264d48-807c-43de-8b8f-52db0225d1c9"
Sep 17 18:29:21.601171 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:21.601142    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:29:21.601630 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:21.601276    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w67kg" podUID="db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4"
Sep 17 18:29:22.601765 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:22.601732    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:29:22.602177 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:22.601841    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssztx" podUID="5d3336f6-e38a-4183-8d58-3c99f1284f33"
Sep 17 18:29:22.602177 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:22.601892    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:22.602177 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:22.601971    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zwdx" podUID="43264d48-807c-43de-8b8f-52db0225d1c9"
Sep 17 18:29:23.601243 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:23.601208    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:29:23.601416 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:23.601355    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w67kg" podUID="db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4"
Sep 17 18:29:24.601607 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:24.601188    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:29:24.601941 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:24.601271    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:24.601941 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:24.601711    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssztx" podUID="5d3336f6-e38a-4183-8d58-3c99f1284f33"
Sep 17 18:29:24.601941 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:24.601827    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zwdx" podUID="43264d48-807c-43de-8b8f-52db0225d1c9"
Sep 17 18:29:24.836264 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:24.836184    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mkdtq" event={"ID":"0058f4db-dbaa-4b8b-942b-06ecbd1301ef","Type":"ContainerStarted","Data":"4fd6bcccafd725c9b9d513337e40c5687b0b42e7c9dedffbcfb5a9f954f1532b"}
Sep 17 18:29:24.837964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:24.837938    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mfxhw" event={"ID":"0eeeae49-526f-41f9-8438-3eebbee372f6","Type":"ContainerStarted","Data":"462bfb4dc6386005c1d6c2105167668a6b7af9fb42c60aaed70ad07dad2b2cf4"}
Sep 17 18:29:24.839845 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:24.839821    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pwktj" event={"ID":"653fa5e9-f667-45f8-a387-1b8a7affdb36","Type":"ContainerStarted","Data":"9f01b6f9fa8a1ae1ca2240a7d7bb887ecaeb290848e6f239fd18d5b962ff638c"}
Sep 17 18:29:24.841353 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:24.841329    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zttbn" event={"ID":"2da5dc7d-7056-497b-a8e3-01deda7b2134","Type":"ContainerStarted","Data":"4709dd9a8f28e1176a89601fc2fa80d97480acccd8cd236f6dd078c1baf9de56"}
Sep 17 18:29:24.846469 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:24.846441    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-grgmp" event={"ID":"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709","Type":"ContainerStarted","Data":"f5fb015156ffffe1e76dc9c6b6deae762c5624f7cafcb64c4b2f0925c3000ab8"}
Sep 17 18:29:24.846575 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:24.846509    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-grgmp" event={"ID":"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709","Type":"ContainerStarted","Data":"dc227479807372c9ad8fd60480be292e8c07ebbe6979445ee7cf2678808b4c0a"}
Sep 17 18:29:24.846575 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:24.846541    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-grgmp" event={"ID":"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709","Type":"ContainerStarted","Data":"2d0def9694e104f8c0e13f2bb8f743f9f9adb102cad725629133a6fdd04c8878"}
Sep 17 18:29:24.846746 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:24.846581    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-grgmp" event={"ID":"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709","Type":"ContainerStarted","Data":"382f1f45bda0c5b7b3f8177f3ea9c612d3d7f5dea3f764539806998dcdf23280"}
Sep 17 18:29:24.846746 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:24.846620    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-grgmp" event={"ID":"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709","Type":"ContainerStarted","Data":"2fc786eb5f061e912cda607a2b244a91ff9f1aea90c602ec99b3655441d7e358"}
Sep 17 18:29:24.846746 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:24.846653    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-grgmp" event={"ID":"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709","Type":"ContainerStarted","Data":"5c74a79e0242204d4f3675598aeb770d2486ff366039a6dde71c52555684da56"}
Sep 17 18:29:24.848314 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:24.848288    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtbwp" event={"ID":"7fb70a02-f3d7-496f-b1de-445b7165dd01","Type":"ContainerStarted","Data":"07e3509d9fcd651c8c83e3d1868bcb98df49111099deafccc9176395f1b78332"}
Sep 17 18:29:24.850020 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:24.849983    2565 generic.go:356] "Generic (PLEG): container finished" podID="42102f16-c1e3-4791-9aea-61a4b415e2ab" containerID="5002ac506ef6fe84d0e7bb3822b2a622cefebb86a2d0b9873a0a04e5eb952abd" exitCode=0
Sep 17 18:29:24.850127 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:24.850083    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stwlh" event={"ID":"42102f16-c1e3-4791-9aea-61a4b415e2ab","Type":"ContainerDied","Data":"5002ac506ef6fe84d0e7bb3822b2a622cefebb86a2d0b9873a0a04e5eb952abd"}
Sep 17 18:29:24.850762 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:24.850718    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mkdtq" podStartSLOduration=3.545654464 podStartE2EDuration="21.850703116s" totalImagesPullingTime="18.305048652s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:24.850129222 +0000 UTC m=+22.782330742" watchObservedRunningTime="2026-09-17 18:29:24.850703116 +0000 UTC m=+22.782904636"
Sep 17 18:29:24.850875 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:24.850812    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-75.ec2.internal" podStartSLOduration=20.850804423 podStartE2EDuration="20.850804423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:07.817219117 +0000 UTC m=+5.749420641" watchObservedRunningTime="2026-09-17 18:29:24.850804423 +0000 UTC m=+22.783005942"
Sep 17 18:29:24.852244 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:24.852193    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-glcw2" event={"ID":"822d9b1e-1a38-49ac-900a-6ad946e6078f","Type":"ContainerStarted","Data":"f52dba5bb0a8db16736248692739123d389df5e6fc230ee5dfe83d03a126a21f"}
Sep 17 18:29:24.865929 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:24.865885    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-mfxhw" podStartSLOduration=3.497293983 podStartE2EDuration="21.865869222s" totalImagesPullingTime="18.368575239s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:24.865820169 +0000 UTC m=+22.798021690" watchObservedRunningTime="2026-09-17 18:29:24.865869222 +0000 UTC m=+22.798070742"
Sep 17 18:29:24.881397 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:24.881358    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pwktj" podStartSLOduration=3.566491981 podStartE2EDuration="21.881347806s" totalImagesPullingTime="18.314855825s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:24.881097881 +0000 UTC m=+22.813299401" watchObservedRunningTime="2026-09-17 18:29:24.881347806 +0000 UTC m=+22.813549358"
Sep 17 18:29:24.897055 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:24.897020    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zttbn" podStartSLOduration=10.802105111 podStartE2EDuration="21.897010378s" totalImagesPullingTime="11.094905267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:24.896861292 +0000 UTC m=+22.829062813" watchObservedRunningTime="2026-09-17 18:29:24.897010378 +0000 UTC m=+22.829211915"
Sep 17 18:29:24.946014 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:24.945505    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-glcw2" podStartSLOduration=3.646378166 podStartE2EDuration="21.945491786s" totalImagesPullingTime="18.29911362s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:24.945296291 +0000 UTC m=+22.877497810" watchObservedRunningTime="2026-09-17 18:29:24.945491786 +0000 UTC m=+22.877693306"
Sep 17 18:29:25.564574 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:25.564551    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 18:29:25.600633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:25.600608    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:29:25.600747 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:25.600724    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w67kg" podUID="db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4"
Sep 17 18:29:25.856378 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:25.856290    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtbwp" event={"ID":"7fb70a02-f3d7-496f-b1de-445b7165dd01","Type":"ContainerStarted","Data":"3392f242acdac0f040b7bfd68a9b1576e601b73e4b40add27edce80ec34f7dc2"}
Sep 17 18:29:25.857896 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:25.857870    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vfv5l" event={"ID":"bc0044f2-c713-4de4-b82e-a211c1f398c3","Type":"ContainerStarted","Data":"769b6a2331c37a33a8f6324f085b6337a48cf7388179a283185c18b01810d4f3"}
Sep 17 18:29:25.872292 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:25.872245    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vfv5l" podStartSLOduration=4.571676443 podStartE2EDuration="22.872228656s" totalImagesPullingTime="18.300552213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:25.871805854 +0000 UTC m=+23.804007374" watchObservedRunningTime="2026-09-17 18:29:25.872228656 +0000 UTC m=+23.804430177"
Sep 17 18:29:26.548416 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:26.548326    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T18:29:25.564570374Z","UUID":"260a3bd5-7705-4d70-b936-4145b586a3d7","Handler":null,"Name":"","Endpoint":""}
Sep 17 18:29:26.550136 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:26.550103    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 18:29:26.550136 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:26.550136    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 18:29:26.600816 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:26.600769    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:26.600923 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:26.600875    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zwdx" podUID="43264d48-807c-43de-8b8f-52db0225d1c9"
Sep 17 18:29:26.600988 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:26.600939    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:29:26.601060 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:26.601040    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssztx" podUID="5d3336f6-e38a-4183-8d58-3c99f1284f33"
Sep 17 18:29:26.866029 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:26.865949    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-grgmp" event={"ID":"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709","Type":"ContainerStarted","Data":"ef20f0bfbb2a09cff232c055eace2bf421fe4c163b12ed3a97402a4b839cd65b"}
Sep 17 18:29:26.868291 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:26.868265    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtbwp" event={"ID":"7fb70a02-f3d7-496f-b1de-445b7165dd01","Type":"ContainerStarted","Data":"12ad6327c38256f1c0f8606b4ce5344791d48eea12115efb00d1a976c9d02726"}
Sep 17 18:29:26.887010 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:26.886964    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xtbwp" podStartSLOduration=3.568750451 podStartE2EDuration="23.886951272s" totalImagesPullingTime="20.318200821s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:26.886281222 +0000 UTC m=+24.818482742" watchObservedRunningTime="2026-09-17 18:29:26.886951272 +0000 UTC m=+24.819152787"
Sep 17 18:29:27.601724 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:27.601565    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:29:27.601882 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:27.601792    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w67kg" podUID="db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4"
Sep 17 18:29:28.504589 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:28.504551    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zttbn"
Sep 17 18:29:28.505218 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:28.505176    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zttbn"
Sep 17 18:29:28.601402 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:28.601369    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:29:28.601554 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:28.601482    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssztx" podUID="5d3336f6-e38a-4183-8d58-3c99f1284f33"
Sep 17 18:29:28.601609 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:28.601561    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:28.601698 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:28.601676    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zwdx" podUID="43264d48-807c-43de-8b8f-52db0225d1c9"
Sep 17 18:29:28.873946 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:28.873725    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zttbn"
Sep 17 18:29:28.874293 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:28.874273    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zttbn"
Sep 17 18:29:29.601212 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:29.601151    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:29:29.601648 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:29.601301    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w67kg" podUID="db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4"
Sep 17 18:29:29.877015 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:29.876834    2565 generic.go:356] "Generic (PLEG): container finished" podID="42102f16-c1e3-4791-9aea-61a4b415e2ab" containerID="4e727810af7cf0bd0779259be30dbb2a388a76ee0d116d727b3f12d603b0998e" exitCode=0
Sep 17 18:29:29.877113 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:29.876881    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stwlh" event={"ID":"42102f16-c1e3-4791-9aea-61a4b415e2ab","Type":"ContainerDied","Data":"4e727810af7cf0bd0779259be30dbb2a388a76ee0d116d727b3f12d603b0998e"}
Sep 17 18:29:29.880863 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:29.880838    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-grgmp" event={"ID":"9d09fbd1-01dd-4ec2-8dec-f05ee9a28709","Type":"ContainerStarted","Data":"13427909c984f6f62f2e59258779962ea5ad6fbfca174359625b12e7b296769f"}
Sep 17 18:29:29.945374 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:29.945339    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-grgmp" podStartSLOduration=8.587224379 podStartE2EDuration="26.94532713s" totalImagesPullingTime="18.358102751s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:29.944964135 +0000 UTC m=+27.877165654" watchObservedRunningTime="2026-09-17 18:29:29.94532713 +0000 UTC m=+27.877528648"
Sep 17 18:29:30.601622 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:30.601595    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:29:30.602113 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:30.601684    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:30.602113 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:30.601695    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssztx" podUID="5d3336f6-e38a-4183-8d58-3c99f1284f33"
Sep 17 18:29:30.602113 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:30.601801    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zwdx" podUID="43264d48-807c-43de-8b8f-52db0225d1c9"
Sep 17 18:29:30.883601 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:30.883531    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:30.883744 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:30.883607    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:30.883744 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:30.883644    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:30.897320 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:30.897295    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:30.897676 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:30.897657    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:29:31.308215 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:31.308179    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ssztx"]
Sep 17 18:29:31.308369 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:31.308354    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:29:31.308492 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:31.308466    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssztx" podUID="5d3336f6-e38a-4183-8d58-3c99f1284f33"
Sep 17 18:29:31.311405 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:31.311352    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4zwdx"]
Sep 17 18:29:31.311495 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:31.311455    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:31.311563 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:31.311547    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zwdx" podUID="43264d48-807c-43de-8b8f-52db0225d1c9"
Sep 17 18:29:31.312009 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:31.311979    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w67kg"]
Sep 17 18:29:31.312093 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:31.312081    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:29:31.312264 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:31.312183    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w67kg" podUID="db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4"
Sep 17 18:29:31.886393 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:31.886362    2565 generic.go:356] "Generic (PLEG): container finished" podID="42102f16-c1e3-4791-9aea-61a4b415e2ab" containerID="06fc6ba65863f81487dc91a2bee923dd2bf20ae3487734c11b1e4eb59fc5edcb" exitCode=0
Sep 17 18:29:31.886956 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:31.886452    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stwlh" event={"ID":"42102f16-c1e3-4791-9aea-61a4b415e2ab","Type":"ContainerDied","Data":"06fc6ba65863f81487dc91a2bee923dd2bf20ae3487734c11b1e4eb59fc5edcb"}
Sep 17 18:29:32.601449 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:32.601430    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:29:32.601538 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:32.601506    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssztx" podUID="5d3336f6-e38a-4183-8d58-3c99f1284f33"
Sep 17 18:29:32.601591 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:32.601579    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:32.601677 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:32.601661    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zwdx" podUID="43264d48-807c-43de-8b8f-52db0225d1c9"
Sep 17 18:29:32.891251 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:32.891222    2565 generic.go:356] "Generic (PLEG): container finished" podID="42102f16-c1e3-4791-9aea-61a4b415e2ab" containerID="40d7164b97c6a7e63fc63bace8462a376502a92bc06f6447bf4c47882ee5bf17" exitCode=0
Sep 17 18:29:32.891636 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:32.891305    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stwlh" event={"ID":"42102f16-c1e3-4791-9aea-61a4b415e2ab","Type":"ContainerDied","Data":"40d7164b97c6a7e63fc63bace8462a376502a92bc06f6447bf4c47882ee5bf17"}
Sep 17 18:29:33.601432 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:33.601242    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:29:33.601568 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:33.601548    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w67kg" podUID="db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4"
Sep 17 18:29:34.601704 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:34.601674    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:29:34.602118 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:34.601678    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:34.602118 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:34.601786    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssztx" podUID="5d3336f6-e38a-4183-8d58-3c99f1284f33"
Sep 17 18:29:34.602118 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:34.601890    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zwdx" podUID="43264d48-807c-43de-8b8f-52db0225d1c9"
Sep 17 18:29:35.601302 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:35.601269    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:29:35.601592 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:35.601388    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w67kg" podUID="db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4"
Sep 17 18:29:36.334188 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:36.334159    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/43264d48-807c-43de-8b8f-52db0225d1c9-original-pull-secret\") pod \"global-pull-secret-syncer-4zwdx\" (UID: \"43264d48-807c-43de-8b8f-52db0225d1c9\") " pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:36.334653 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:36.334223    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4-metrics-certs\") pod \"network-metrics-daemon-w67kg\" (UID: \"db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4\") " pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:29:36.334653 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:36.334320    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:36.334653 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:36.334333    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:36.334653 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:36.334385    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43264d48-807c-43de-8b8f-52db0225d1c9-original-pull-secret podName:43264d48-807c-43de-8b8f-52db0225d1c9 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:08.334367001 +0000 UTC m=+66.266568501 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/43264d48-807c-43de-8b8f-52db0225d1c9-original-pull-secret") pod "global-pull-secret-syncer-4zwdx" (UID: "43264d48-807c-43de-8b8f-52db0225d1c9") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:36.334653 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:36.334400    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4-metrics-certs podName:db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:08.334394239 +0000 UTC m=+66.266595740 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4-metrics-certs") pod "network-metrics-daemon-w67kg" (UID: "db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:36.434638 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:36.434606    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-852t9\" (UniqueName: \"kubernetes.io/projected/5d3336f6-e38a-4183-8d58-3c99f1284f33-kube-api-access-852t9\") pod \"network-check-target-ssztx\" (UID: \"5d3336f6-e38a-4183-8d58-3c99f1284f33\") " pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:29:36.434794 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:36.434741    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:36.434794 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:36.434772    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:36.434794 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:36.434785    2565 projected.go:196] Error preparing data for projected volume kube-api-access-852t9 for pod openshift-network-diagnostics/network-check-target-ssztx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:36.434920 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:36.434838    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d3336f6-e38a-4183-8d58-3c99f1284f33-kube-api-access-852t9 podName:5d3336f6-e38a-4183-8d58-3c99f1284f33 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:08.434820482 +0000 UTC m=+66.367021979 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-852t9" (UniqueName: "kubernetes.io/projected/5d3336f6-e38a-4183-8d58-3c99f1284f33-kube-api-access-852t9") pod "network-check-target-ssztx" (UID: "5d3336f6-e38a-4183-8d58-3c99f1284f33") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:36.600728 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:36.600696    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:29:36.600891 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:36.600848    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssztx" podUID="5d3336f6-e38a-4183-8d58-3c99f1284f33"
Sep 17 18:29:36.600891 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:36.600872    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:36.601016 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:36.600995    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4zwdx" podUID="43264d48-807c-43de-8b8f-52db0225d1c9"
Sep 17 18:29:36.839768 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:36.839739    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-75.ec2.internal" event="NodeReady"
Sep 17 18:29:36.839908 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:36.839864    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 18:29:36.879596 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:36.879525    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f7b5f6fff-66wm8"]
Sep 17 18:29:36.927896 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:36.927870    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:29:36.927896 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:36.927881    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-x9pxw"]
Sep 17 18:29:36.931399 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:36.931382    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 18:29:36.931544 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:36.931524    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 18:29:36.931696 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:36.931683    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pndxg\""
Sep 17 18:29:36.931960 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:36.931947    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 18:29:36.936517 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:36.936493    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 18:29:36.941646 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:36.941628    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f7b5f6fff-66wm8"]
Sep 17 18:29:36.941718 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:36.941657    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ltdzb"]
Sep 17 18:29:36.941754 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:36.941726    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x9pxw"
Sep 17 18:29:36.943876 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:36.943858    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 18:29:36.944414 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:36.944397    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 18:29:36.944485 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:36.944430    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cvvrd\""
Sep 17 18:29:36.961208 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:36.961185    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ltdzb"
Sep 17 18:29:36.961296 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:36.961194    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x9pxw"]
Sep 17 18:29:36.961296 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:36.961249    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ltdzb"]
Sep 17 18:29:36.963497 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:36.963483    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 18:29:36.964679 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:36.964662    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 18:29:36.964880 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:36.964859    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7c6q4\""
Sep 17 18:29:36.965530 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:36.965512    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 18:29:37.038967 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.038946    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-bound-sa-token\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:29:37.039048 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.038981    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8af0b1ec-975b-44eb-a6b6-fc55869a5585-image-registry-private-configuration\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:29:37.039048 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.038998    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8wk9\" (UniqueName: \"kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-kube-api-access-x8wk9\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:29:37.039121 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.039061    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3158326a-19ea-455b-8737-8f01b5d51373-cert\") pod \"ingress-canary-ltdzb\" (UID: \"3158326a-19ea-455b-8737-8f01b5d51373\") " pod="openshift-ingress-canary/ingress-canary-ltdzb"
Sep 17 18:29:37.039121 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.039107    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ba1caac4-a2a1-4917-9598-4b2003401a84-config-volume\") pod \"dns-default-x9pxw\" (UID: \"ba1caac4-a2a1-4917-9598-4b2003401a84\") " pod="openshift-dns/dns-default-x9pxw"
Sep 17 18:29:37.039181 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.039124    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8af0b1ec-975b-44eb-a6b6-fc55869a5585-installation-pull-secrets\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:29:37.039181 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.039147    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba1caac4-a2a1-4917-9598-4b2003401a84-tmp-dir\") pod \"dns-default-x9pxw\" (UID: \"ba1caac4-a2a1-4917-9598-4b2003401a84\") " pod="openshift-dns/dns-default-x9pxw"
Sep 17 18:29:37.039181 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.039162    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq5kq\" (UniqueName: \"kubernetes.io/projected/3158326a-19ea-455b-8737-8f01b5d51373-kube-api-access-hq5kq\") pod \"ingress-canary-ltdzb\" (UID: \"3158326a-19ea-455b-8737-8f01b5d51373\") " pod="openshift-ingress-canary/ingress-canary-ltdzb"
Sep 17 18:29:37.039288 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.039184    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-tls\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:29:37.039288 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.039213    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8af0b1ec-975b-44eb-a6b6-fc55869a5585-ca-trust-extracted\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:29:37.039288 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.039228    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8af0b1ec-975b-44eb-a6b6-fc55869a5585-trusted-ca\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:29:37.039288 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.039246    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba1caac4-a2a1-4917-9598-4b2003401a84-metrics-tls\") pod \"dns-default-x9pxw\" (UID: \"ba1caac4-a2a1-4917-9598-4b2003401a84\") " pod="openshift-dns/dns-default-x9pxw"
Sep 17 18:29:37.039404 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.039290    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwdfg\" (UniqueName: \"kubernetes.io/projected/ba1caac4-a2a1-4917-9598-4b2003401a84-kube-api-access-qwdfg\") pod \"dns-default-x9pxw\" (UID: \"ba1caac4-a2a1-4917-9598-4b2003401a84\") " pod="openshift-dns/dns-default-x9pxw"
Sep 17 18:29:37.039404 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.039323    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-certificates\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:29:37.139609 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.139565    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8af0b1ec-975b-44eb-a6b6-fc55869a5585-image-registry-private-configuration\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:29:37.139609 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.139590    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8wk9\" (UniqueName: \"kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-kube-api-access-x8wk9\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:29:37.139722 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.139612    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3158326a-19ea-455b-8737-8f01b5d51373-cert\") pod \"ingress-canary-ltdzb\" (UID: \"3158326a-19ea-455b-8737-8f01b5d51373\") " pod="openshift-ingress-canary/ingress-canary-ltdzb"
Sep 17 18:29:37.139722 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.139631    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ba1caac4-a2a1-4917-9598-4b2003401a84-config-volume\") pod \"dns-default-x9pxw\" (UID: \"ba1caac4-a2a1-4917-9598-4b2003401a84\") " pod="openshift-dns/dns-default-x9pxw"
Sep 17 18:29:37.139722 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.139654    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8af0b1ec-975b-44eb-a6b6-fc55869a5585-installation-pull-secrets\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:29:37.139722 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.139679    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba1caac4-a2a1-4917-9598-4b2003401a84-tmp-dir\") pod \"dns-default-x9pxw\" (UID: \"ba1caac4-a2a1-4917-9598-4b2003401a84\") " pod="openshift-dns/dns-default-x9pxw"
Sep 17 18:29:37.139722 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.139701    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq5kq\" (UniqueName: \"kubernetes.io/projected/3158326a-19ea-455b-8737-8f01b5d51373-kube-api-access-hq5kq\") pod \"ingress-canary-ltdzb\" (UID: \"3158326a-19ea-455b-8737-8f01b5d51373\") " pod="openshift-ingress-canary/ingress-canary-ltdzb"
Sep 17 18:29:37.139964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.139729    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-tls\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:29:37.139964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.139751    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8af0b1ec-975b-44eb-a6b6-fc55869a5585-ca-trust-extracted\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:29:37.139964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.139773    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8af0b1ec-975b-44eb-a6b6-fc55869a5585-trusted-ca\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:29:37.139964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.139798    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba1caac4-a2a1-4917-9598-4b2003401a84-metrics-tls\") pod \"dns-default-x9pxw\" (UID: \"ba1caac4-a2a1-4917-9598-4b2003401a84\") " pod="openshift-dns/dns-default-x9pxw"
Sep 17 18:29:37.139964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.139831    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwdfg\" (UniqueName: \"kubernetes.io/projected/ba1caac4-a2a1-4917-9598-4b2003401a84-kube-api-access-qwdfg\") pod \"dns-default-x9pxw\" (UID: \"ba1caac4-a2a1-4917-9598-4b2003401a84\") " pod="openshift-dns/dns-default-x9pxw"
Sep 17 18:29:37.139964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.139862    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-certificates\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:29:37.139964 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.139935    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-bound-sa-token\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:29:37.140328 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.140061    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba1caac4-a2a1-4917-9598-4b2003401a84-tmp-dir\") pod \"dns-default-x9pxw\" (UID: \"ba1caac4-a2a1-4917-9598-4b2003401a84\") " pod="openshift-dns/dns-default-x9pxw"
Sep 17 18:29:37.140328 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:37.140159    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:29:37.140328 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:37.140173    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f7b5f6fff-66wm8: secret "image-registry-tls" not found
Sep 17 18:29:37.140328 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:37.140237    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:29:37.140328 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.140251    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ba1caac4-a2a1-4917-9598-4b2003401a84-config-volume\") pod \"dns-default-x9pxw\" (UID: \"ba1caac4-a2a1-4917-9598-4b2003401a84\") " pod="openshift-dns/dns-default-x9pxw"
Sep 17 18:29:37.140328 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:37.140260    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-tls podName:8af0b1ec-975b-44eb-a6b6-fc55869a5585 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:37.640241564 +0000 UTC m=+35.572443072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-tls") pod "image-registry-5f7b5f6fff-66wm8" (UID: "8af0b1ec-975b-44eb-a6b6-fc55869a5585") : secret "image-registry-tls" not found
Sep 17 18:29:37.140328 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:37.140317    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba1caac4-a2a1-4917-9598-4b2003401a84-metrics-tls podName:ba1caac4-a2a1-4917-9598-4b2003401a84 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:37.640293859 +0000 UTC m=+35.572495356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ba1caac4-a2a1-4917-9598-4b2003401a84-metrics-tls") pod "dns-default-x9pxw" (UID: "ba1caac4-a2a1-4917-9598-4b2003401a84") : secret "dns-default-metrics-tls" not found
Sep 17 18:29:37.140707 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.140680    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8af0b1ec-975b-44eb-a6b6-fc55869a5585-ca-trust-extracted\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:29:37.140810 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.140775    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-certificates\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:29:37.140971 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.140946    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8af0b1ec-975b-44eb-a6b6-fc55869a5585-trusted-ca\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:29:37.143658 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.143630    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8af0b1ec-975b-44eb-a6b6-fc55869a5585-image-registry-private-configuration\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:29:37.143658 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.143640    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3158326a-19ea-455b-8737-8f01b5d51373-cert\") pod \"ingress-canary-ltdzb\" (UID: \"3158326a-19ea-455b-8737-8f01b5d51373\") " pod="openshift-ingress-canary/ingress-canary-ltdzb"
Sep 17 18:29:37.143658 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.143652    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8af0b1ec-975b-44eb-a6b6-fc55869a5585-installation-pull-secrets\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:29:37.151002 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.150975    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8wk9\" (UniqueName: \"kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-kube-api-access-x8wk9\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:29:37.152082 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.152041    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq5kq\" (UniqueName: \"kubernetes.io/projected/3158326a-19ea-455b-8737-8f01b5d51373-kube-api-access-hq5kq\") pod \"ingress-canary-ltdzb\" (UID: \"3158326a-19ea-455b-8737-8f01b5d51373\") " pod="openshift-ingress-canary/ingress-canary-ltdzb"
Sep 17 18:29:37.152250 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.152233    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-bound-sa-token\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:29:37.154488 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.154471    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwdfg\" (UniqueName: \"kubernetes.io/projected/ba1caac4-a2a1-4917-9598-4b2003401a84-kube-api-access-qwdfg\") pod \"dns-default-x9pxw\" (UID: \"ba1caac4-a2a1-4917-9598-4b2003401a84\") " pod="openshift-dns/dns-default-x9pxw"
Sep 17 18:29:37.225761 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.225736    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zttbn_2da5dc7d-7056-497b-a8e3-01deda7b2134/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:37.268877 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.268860    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ltdzb"
Sep 17 18:29:37.319107 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.319082    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zttbn_2da5dc7d-7056-497b-a8e3-01deda7b2134/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:37.593862 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.593706    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zttbn_2da5dc7d-7056-497b-a8e3-01deda7b2134/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:37.601576 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.601554    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:29:37.603981 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.603964    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 18:29:37.604090 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.604051    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-twvp2\""
Sep 17 18:29:37.619288 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.619268    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ltdzb"]
Sep 17 18:29:37.627988 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:29:37.627969    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3158326a_19ea_455b_8737_8f01b5d51373.slice/crio-9ebeca8126978a1ede14147595a3e8eadad42d81b5905b6aae5cf0dee7d25db9 WatchSource:0}: Error finding container 9ebeca8126978a1ede14147595a3e8eadad42d81b5905b6aae5cf0dee7d25db9: Status 404 returned error can't find the container with id 9ebeca8126978a1ede14147595a3e8eadad42d81b5905b6aae5cf0dee7d25db9
Sep 17 18:29:37.643887 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.643854    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-tls\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:29:37.643984 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.643923    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba1caac4-a2a1-4917-9598-4b2003401a84-metrics-tls\") pod \"dns-default-x9pxw\" (UID: \"ba1caac4-a2a1-4917-9598-4b2003401a84\") " pod="openshift-dns/dns-default-x9pxw"
Sep 17 18:29:37.643984 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:37.643949    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:29:37.643984 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:37.643968    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f7b5f6fff-66wm8: secret "image-registry-tls" not found
Sep 17 18:29:37.644098 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:37.644047    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:29:37.644098 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:37.644091    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-tls podName:8af0b1ec-975b-44eb-a6b6-fc55869a5585 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:38.644070846 +0000 UTC m=+36.576272347 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-tls") pod "image-registry-5f7b5f6fff-66wm8" (UID: "8af0b1ec-975b-44eb-a6b6-fc55869a5585") : secret "image-registry-tls" not found
Sep 17 18:29:37.644178 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:37.644111    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba1caac4-a2a1-4917-9598-4b2003401a84-metrics-tls podName:ba1caac4-a2a1-4917-9598-4b2003401a84 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:38.644101317 +0000 UTC m=+36.576302821 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ba1caac4-a2a1-4917-9598-4b2003401a84-metrics-tls") pod "dns-default-x9pxw" (UID: "ba1caac4-a2a1-4917-9598-4b2003401a84") : secret "dns-default-metrics-tls" not found
Sep 17 18:29:37.694307 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.694276    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zttbn_2da5dc7d-7056-497b-a8e3-01deda7b2134/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:37.907896 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.907874    2565 generic.go:356] "Generic (PLEG): container finished" podID="42102f16-c1e3-4791-9aea-61a4b415e2ab" containerID="06452972e832fdc031307c276f113621e8eab39e80ea018de86c80f99192d1bf" exitCode=0
Sep 17 18:29:37.908002 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.907960    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stwlh" event={"ID":"42102f16-c1e3-4791-9aea-61a4b415e2ab","Type":"ContainerDied","Data":"06452972e832fdc031307c276f113621e8eab39e80ea018de86c80f99192d1bf"}
Sep 17 18:29:37.909236 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:37.909216    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ltdzb" event={"ID":"3158326a-19ea-455b-8737-8f01b5d51373","Type":"ContainerStarted","Data":"9ebeca8126978a1ede14147595a3e8eadad42d81b5905b6aae5cf0dee7d25db9"}
Sep 17 18:29:38.601572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:38.601542    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:29:38.602006 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:38.601582    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:29:38.605934 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:38.605913    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 18:29:38.606031 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:38.605964    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 18:29:38.606649 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:38.606631    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-58cdj\""
Sep 17 18:29:38.606756 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:38.606667    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 18:29:38.652012 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:38.651989    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-tls\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:29:38.652094 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:38.652017    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba1caac4-a2a1-4917-9598-4b2003401a84-metrics-tls\") pod \"dns-default-x9pxw\" (UID: \"ba1caac4-a2a1-4917-9598-4b2003401a84\") " pod="openshift-dns/dns-default-x9pxw"
Sep 17 18:29:38.652140 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:38.652127    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:29:38.652187 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:38.652178    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba1caac4-a2a1-4917-9598-4b2003401a84-metrics-tls podName:ba1caac4-a2a1-4917-9598-4b2003401a84 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:40.652166086 +0000 UTC m=+38.584367583 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ba1caac4-a2a1-4917-9598-4b2003401a84-metrics-tls") pod "dns-default-x9pxw" (UID: "ba1caac4-a2a1-4917-9598-4b2003401a84") : secret "dns-default-metrics-tls" not found
Sep 17 18:29:38.652256 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:38.652131    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:29:38.652256 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:38.652229    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f7b5f6fff-66wm8: secret "image-registry-tls" not found
Sep 17 18:29:38.652328 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:38.652271    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-tls podName:8af0b1ec-975b-44eb-a6b6-fc55869a5585 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:40.652258389 +0000 UTC m=+38.584459889 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-tls") pod "image-registry-5f7b5f6fff-66wm8" (UID: "8af0b1ec-975b-44eb-a6b6-fc55869a5585") : secret "image-registry-tls" not found
Sep 17 18:29:38.915704 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:38.915674    2565 generic.go:356] "Generic (PLEG): container finished" podID="42102f16-c1e3-4791-9aea-61a4b415e2ab" containerID="a45eb53854c94bf24940a56e99e249b4921b88f7e8e833a3e847dad77c46e6c5" exitCode=0
Sep 17 18:29:38.915836 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:38.915726    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stwlh" event={"ID":"42102f16-c1e3-4791-9aea-61a4b415e2ab","Type":"ContainerDied","Data":"a45eb53854c94bf24940a56e99e249b4921b88f7e8e833a3e847dad77c46e6c5"}
Sep 17 18:29:39.922149 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:39.922113    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stwlh" event={"ID":"42102f16-c1e3-4791-9aea-61a4b415e2ab","Type":"ContainerStarted","Data":"e037cc2a6a7791963e351d65384e529d1225c5b7eb4ea47b388cc3154a6f8549"}
Sep 17 18:29:39.943341 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:39.943229    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-stwlh" podStartSLOduration=7.690549938 podStartE2EDuration="36.943195096s" totalImagesPullingTime="29.252645158s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:39.942510053 +0000 UTC m=+37.874711635" watchObservedRunningTime="2026-09-17 18:29:39.943195096 +0000 UTC m=+37.875396645"
Sep 17 18:29:40.063839 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:40.063806    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zttbn_2da5dc7d-7056-497b-a8e3-01deda7b2134/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:40.123960 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:40.123916    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zttbn_2da5dc7d-7056-497b-a8e3-01deda7b2134/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:40.665382 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:40.665352    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba1caac4-a2a1-4917-9598-4b2003401a84-metrics-tls\") pod \"dns-default-x9pxw\" (UID: \"ba1caac4-a2a1-4917-9598-4b2003401a84\") " pod="openshift-dns/dns-default-x9pxw"
Sep 17 18:29:40.665533 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:40.665440    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-tls\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:29:40.665570 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:40.665530    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:29:40.665570 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:40.665549    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f7b5f6fff-66wm8: secret "image-registry-tls" not found
Sep 17 18:29:40.665631 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:40.665593    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-tls podName:8af0b1ec-975b-44eb-a6b6-fc55869a5585 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:44.665580745 +0000 UTC m=+42.597782241 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-tls") pod "image-registry-5f7b5f6fff-66wm8" (UID: "8af0b1ec-975b-44eb-a6b6-fc55869a5585") : secret "image-registry-tls" not found
Sep 17 18:29:40.665631 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:40.665528    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:29:40.665701 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:40.665692    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba1caac4-a2a1-4917-9598-4b2003401a84-metrics-tls podName:ba1caac4-a2a1-4917-9598-4b2003401a84 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:44.665673127 +0000 UTC m=+42.597874631 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ba1caac4-a2a1-4917-9598-4b2003401a84-metrics-tls") pod "dns-default-x9pxw" (UID: "ba1caac4-a2a1-4917-9598-4b2003401a84") : secret "dns-default-metrics-tls" not found
Sep 17 18:29:40.925787 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:40.925760    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ltdzb" event={"ID":"3158326a-19ea-455b-8737-8f01b5d51373","Type":"ContainerStarted","Data":"2a4e69b2c20ee9a9af5408aecc4caa33ec9c488f91a673650c0537210228ba26"}
Sep 17 18:29:40.941120 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:40.941079    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ltdzb" podStartSLOduration=2.132449427 podStartE2EDuration="4.941064526s" totalImagesPullingTime="2.808615099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:40.939753915 +0000 UTC m=+38.871955433" watchObservedRunningTime="2026-09-17 18:29:40.941064526 +0000 UTC m=+38.873266089"
Sep 17 18:29:42.140633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:42.140596    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zttbn_2da5dc7d-7056-497b-a8e3-01deda7b2134/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:42.199945 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:42.199909    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zttbn_2da5dc7d-7056-497b-a8e3-01deda7b2134/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:42.635079 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:42.635046    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zttbn_2da5dc7d-7056-497b-a8e3-01deda7b2134/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:42.731520 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:42.731484    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zttbn_2da5dc7d-7056-497b-a8e3-01deda7b2134/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:44.692291 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:44.692258    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-tls\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:29:44.692291 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:44.692295    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba1caac4-a2a1-4917-9598-4b2003401a84-metrics-tls\") pod \"dns-default-x9pxw\" (UID: \"ba1caac4-a2a1-4917-9598-4b2003401a84\") " pod="openshift-dns/dns-default-x9pxw"
Sep 17 18:29:44.692716 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:44.692399    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:29:44.692716 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:44.692407    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:29:44.692716 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:44.692424    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f7b5f6fff-66wm8: secret "image-registry-tls" not found
Sep 17 18:29:44.692716 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:44.692446    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba1caac4-a2a1-4917-9598-4b2003401a84-metrics-tls podName:ba1caac4-a2a1-4917-9598-4b2003401a84 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:52.692433575 +0000 UTC m=+50.624635071 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ba1caac4-a2a1-4917-9598-4b2003401a84-metrics-tls") pod "dns-default-x9pxw" (UID: "ba1caac4-a2a1-4917-9598-4b2003401a84") : secret "dns-default-metrics-tls" not found
Sep 17 18:29:44.692716 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:44.692464    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-tls podName:8af0b1ec-975b-44eb-a6b6-fc55869a5585 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:52.692452095 +0000 UTC m=+50.624653591 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-tls") pod "image-registry-5f7b5f6fff-66wm8" (UID: "8af0b1ec-975b-44eb-a6b6-fc55869a5585") : secret "image-registry-tls" not found
Sep 17 18:29:52.743885 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:52.743853    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-tls\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:29:52.744244 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:29:52.743897    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba1caac4-a2a1-4917-9598-4b2003401a84-metrics-tls\") pod \"dns-default-x9pxw\" (UID: \"ba1caac4-a2a1-4917-9598-4b2003401a84\") " pod="openshift-dns/dns-default-x9pxw"
Sep 17 18:29:52.744244 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:52.743998    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:29:52.744244 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:52.744015    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:29:52.744244 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:52.744017    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f7b5f6fff-66wm8: secret "image-registry-tls" not found
Sep 17 18:29:52.744244 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:52.744071    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba1caac4-a2a1-4917-9598-4b2003401a84-metrics-tls podName:ba1caac4-a2a1-4917-9598-4b2003401a84 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:08.744056681 +0000 UTC m=+66.676258178 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ba1caac4-a2a1-4917-9598-4b2003401a84-metrics-tls") pod "dns-default-x9pxw" (UID: "ba1caac4-a2a1-4917-9598-4b2003401a84") : secret "dns-default-metrics-tls" not found
Sep 17 18:29:52.744244 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:29:52.744084    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-tls podName:8af0b1ec-975b-44eb-a6b6-fc55869a5585 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:08.74407862 +0000 UTC m=+66.676280117 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-tls") pod "image-registry-5f7b5f6fff-66wm8" (UID: "8af0b1ec-975b-44eb-a6b6-fc55869a5585") : secret "image-registry-tls" not found
Sep 17 18:30:01.930913 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:01.930879    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-rh6tl"]
Sep 17 18:30:01.933315 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:01.933293    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rh6tl"
Sep 17 18:30:01.935466 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:01.935444    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 18:30:01.935980 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:01.935954    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 18:30:01.936316 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:01.936296    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 18:30:01.936454 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:01.936433    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-52cqc\""
Sep 17 18:30:01.941132 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:01.941114    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:30:01.946095 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:01.946075    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-rh6tl"]
Sep 17 18:30:02.034312 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.034281    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-lnf27"]
Sep 17 18:30:02.036098 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.036083    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6b4db969fb-dbb5b"]
Sep 17 18:30:02.036227 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.036211    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-lnf27"
Sep 17 18:30:02.037868 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.037849    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:02.039098 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.039081    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-wb4wl\""
Sep 17 18:30:02.039548 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.039535    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:30:02.040475 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.040457    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 18:30:02.040559 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.040483    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 18:30:02.040887 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.040869    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 18:30:02.040973 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.040889    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 18:30:02.040973 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.040945    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 17 18:30:02.040973 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.040953    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 17 18:30:02.041131 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.041100    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vrmnj\""
Sep 17 18:30:02.041184 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.041128    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 17 18:30:02.041316 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.041185    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 17 18:30:02.042663 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.042648    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 18:30:02.060894 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.060836    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-lnf27"]
Sep 17 18:30:02.061510 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.061490    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6b4db969fb-dbb5b"]
Sep 17 18:30:02.103919 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.103896    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a9151c33-3ccc-4061-b06b-9c3b8535fea4-serving-cert\") pod \"service-ca-operator-57cdc4b768-rh6tl\" (UID: \"a9151c33-3ccc-4061-b06b-9c3b8535fea4\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rh6tl"
Sep 17 18:30:02.104006 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.103973    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a9151c33-3ccc-4061-b06b-9c3b8535fea4-config\") pod \"service-ca-operator-57cdc4b768-rh6tl\" (UID: \"a9151c33-3ccc-4061-b06b-9c3b8535fea4\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rh6tl"
Sep 17 18:30:02.104045 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.104014    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjzzk\" (UniqueName: \"kubernetes.io/projected/a9151c33-3ccc-4061-b06b-9c3b8535fea4-kube-api-access-tjzzk\") pod \"service-ca-operator-57cdc4b768-rh6tl\" (UID: \"a9151c33-3ccc-4061-b06b-9c3b8535fea4\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rh6tl"
Sep 17 18:30:02.204451 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.204401    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjzzk\" (UniqueName: \"kubernetes.io/projected/a9151c33-3ccc-4061-b06b-9c3b8535fea4-kube-api-access-tjzzk\") pod \"service-ca-operator-57cdc4b768-rh6tl\" (UID: \"a9151c33-3ccc-4061-b06b-9c3b8535fea4\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rh6tl"
Sep 17 18:30:02.204451 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.204428    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bb5g\" (UniqueName: \"kubernetes.io/projected/5e907c64-1699-4fa4-902b-b90ba5d7b8e6-kube-api-access-9bb5g\") pod \"kube-storage-version-migrator-operator-55889f7b47-lnf27\" (UID: \"5e907c64-1699-4fa4-902b-b90ba5d7b8e6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-lnf27"
Sep 17 18:30:02.204451 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.204446    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfnkg\" (UniqueName: \"kubernetes.io/projected/a9afa96b-b391-4db4-9312-20c884e90da3-kube-api-access-dfnkg\") pod \"router-default-6b4db969fb-dbb5b\" (UID: \"a9afa96b-b391-4db4-9312-20c884e90da3\") " pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:02.204592 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.204480    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a9151c33-3ccc-4061-b06b-9c3b8535fea4-serving-cert\") pod \"service-ca-operator-57cdc4b768-rh6tl\" (UID: \"a9151c33-3ccc-4061-b06b-9c3b8535fea4\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rh6tl"
Sep 17 18:30:02.204592 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.204573    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5e907c64-1699-4fa4-902b-b90ba5d7b8e6-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-lnf27\" (UID: \"5e907c64-1699-4fa4-902b-b90ba5d7b8e6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-lnf27"
Sep 17 18:30:02.204668 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.204598    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a9afa96b-b391-4db4-9312-20c884e90da3-stats-auth\") pod \"router-default-6b4db969fb-dbb5b\" (UID: \"a9afa96b-b391-4db4-9312-20c884e90da3\") " pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:02.204668 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.204625    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a9afa96b-b391-4db4-9312-20c884e90da3-default-certificate\") pod \"router-default-6b4db969fb-dbb5b\" (UID: \"a9afa96b-b391-4db4-9312-20c884e90da3\") " pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:02.204728 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.204669    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a9151c33-3ccc-4061-b06b-9c3b8535fea4-config\") pod \"service-ca-operator-57cdc4b768-rh6tl\" (UID: \"a9151c33-3ccc-4061-b06b-9c3b8535fea4\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rh6tl"
Sep 17 18:30:02.204760 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.204729    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5e907c64-1699-4fa4-902b-b90ba5d7b8e6-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-lnf27\" (UID: \"5e907c64-1699-4fa4-902b-b90ba5d7b8e6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-lnf27"
Sep 17 18:30:02.204760 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.204751    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9afa96b-b391-4db4-9312-20c884e90da3-metrics-certs\") pod \"router-default-6b4db969fb-dbb5b\" (UID: \"a9afa96b-b391-4db4-9312-20c884e90da3\") " pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:02.204824 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.204777    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9afa96b-b391-4db4-9312-20c884e90da3-service-ca-bundle\") pod \"router-default-6b4db969fb-dbb5b\" (UID: \"a9afa96b-b391-4db4-9312-20c884e90da3\") " pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:02.205113 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.205095    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a9151c33-3ccc-4061-b06b-9c3b8535fea4-config\") pod \"service-ca-operator-57cdc4b768-rh6tl\" (UID: \"a9151c33-3ccc-4061-b06b-9c3b8535fea4\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rh6tl"
Sep 17 18:30:02.207557 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.207542    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a9151c33-3ccc-4061-b06b-9c3b8535fea4-serving-cert\") pod \"service-ca-operator-57cdc4b768-rh6tl\" (UID: \"a9151c33-3ccc-4061-b06b-9c3b8535fea4\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rh6tl"
Sep 17 18:30:02.213743 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.213724    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjzzk\" (UniqueName: \"kubernetes.io/projected/a9151c33-3ccc-4061-b06b-9c3b8535fea4-kube-api-access-tjzzk\") pod \"service-ca-operator-57cdc4b768-rh6tl\" (UID: \"a9151c33-3ccc-4061-b06b-9c3b8535fea4\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rh6tl"
Sep 17 18:30:02.242802 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.242781    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rh6tl"
Sep 17 18:30:02.305689 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.305662    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a9afa96b-b391-4db4-9312-20c884e90da3-default-certificate\") pod \"router-default-6b4db969fb-dbb5b\" (UID: \"a9afa96b-b391-4db4-9312-20c884e90da3\") " pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:02.305783 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.305706    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5e907c64-1699-4fa4-902b-b90ba5d7b8e6-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-lnf27\" (UID: \"5e907c64-1699-4fa4-902b-b90ba5d7b8e6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-lnf27"
Sep 17 18:30:02.305783 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.305728    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9afa96b-b391-4db4-9312-20c884e90da3-metrics-certs\") pod \"router-default-6b4db969fb-dbb5b\" (UID: \"a9afa96b-b391-4db4-9312-20c884e90da3\") " pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:02.305783 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.305751    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9afa96b-b391-4db4-9312-20c884e90da3-service-ca-bundle\") pod \"router-default-6b4db969fb-dbb5b\" (UID: \"a9afa96b-b391-4db4-9312-20c884e90da3\") " pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:02.305783 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.305771    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bb5g\" (UniqueName: \"kubernetes.io/projected/5e907c64-1699-4fa4-902b-b90ba5d7b8e6-kube-api-access-9bb5g\") pod \"kube-storage-version-migrator-operator-55889f7b47-lnf27\" (UID: \"5e907c64-1699-4fa4-902b-b90ba5d7b8e6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-lnf27"
Sep 17 18:30:02.305947 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.305790    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfnkg\" (UniqueName: \"kubernetes.io/projected/a9afa96b-b391-4db4-9312-20c884e90da3-kube-api-access-dfnkg\") pod \"router-default-6b4db969fb-dbb5b\" (UID: \"a9afa96b-b391-4db4-9312-20c884e90da3\") " pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:02.305947 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.305847    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5e907c64-1699-4fa4-902b-b90ba5d7b8e6-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-lnf27\" (UID: \"5e907c64-1699-4fa4-902b-b90ba5d7b8e6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-lnf27"
Sep 17 18:30:02.305947 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.305863    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a9afa96b-b391-4db4-9312-20c884e90da3-stats-auth\") pod \"router-default-6b4db969fb-dbb5b\" (UID: \"a9afa96b-b391-4db4-9312-20c884e90da3\") " pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:02.306589 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.306325    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5e907c64-1699-4fa4-902b-b90ba5d7b8e6-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-lnf27\" (UID: \"5e907c64-1699-4fa4-902b-b90ba5d7b8e6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-lnf27"
Sep 17 18:30:02.306589 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:02.306356    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 18:30:02.306589 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:02.306377    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9afa96b-b391-4db4-9312-20c884e90da3-service-ca-bundle podName:a9afa96b-b391-4db4-9312-20c884e90da3 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:02.806350423 +0000 UTC m=+60.738551941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a9afa96b-b391-4db4-9312-20c884e90da3-service-ca-bundle") pod "router-default-6b4db969fb-dbb5b" (UID: "a9afa96b-b391-4db4-9312-20c884e90da3") : configmap references non-existent config key: service-ca.crt
Sep 17 18:30:02.306589 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:02.306443    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9afa96b-b391-4db4-9312-20c884e90da3-metrics-certs podName:a9afa96b-b391-4db4-9312-20c884e90da3 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:02.806424984 +0000 UTC m=+60.738626481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a9afa96b-b391-4db4-9312-20c884e90da3-metrics-certs") pod "router-default-6b4db969fb-dbb5b" (UID: "a9afa96b-b391-4db4-9312-20c884e90da3") : secret "router-metrics-certs-default" not found
Sep 17 18:30:02.308337 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.308317    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a9afa96b-b391-4db4-9312-20c884e90da3-default-certificate\") pod \"router-default-6b4db969fb-dbb5b\" (UID: \"a9afa96b-b391-4db4-9312-20c884e90da3\") " pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:02.308439 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.308392    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5e907c64-1699-4fa4-902b-b90ba5d7b8e6-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-lnf27\" (UID: \"5e907c64-1699-4fa4-902b-b90ba5d7b8e6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-lnf27"
Sep 17 18:30:02.308533 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.308512    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a9afa96b-b391-4db4-9312-20c884e90da3-stats-auth\") pod \"router-default-6b4db969fb-dbb5b\" (UID: \"a9afa96b-b391-4db4-9312-20c884e90da3\") " pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:02.319053 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.319030    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bb5g\" (UniqueName: \"kubernetes.io/projected/5e907c64-1699-4fa4-902b-b90ba5d7b8e6-kube-api-access-9bb5g\") pod \"kube-storage-version-migrator-operator-55889f7b47-lnf27\" (UID: \"5e907c64-1699-4fa4-902b-b90ba5d7b8e6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-lnf27"
Sep 17 18:30:02.319370 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.319352    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfnkg\" (UniqueName: \"kubernetes.io/projected/a9afa96b-b391-4db4-9312-20c884e90da3-kube-api-access-dfnkg\") pod \"router-default-6b4db969fb-dbb5b\" (UID: \"a9afa96b-b391-4db4-9312-20c884e90da3\") " pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:02.345718 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.345696    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-lnf27"
Sep 17 18:30:02.508234 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.508139    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-rh6tl"]
Sep 17 18:30:02.511416 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:30:02.511347    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9151c33_3ccc_4061_b06b_9c3b8535fea4.slice/crio-825e4d995b22db48862f83ac63a54531288a00c79c1907ecd2003a8fceab5e00 WatchSource:0}: Error finding container 825e4d995b22db48862f83ac63a54531288a00c79c1907ecd2003a8fceab5e00: Status 404 returned error can't find the container with id 825e4d995b22db48862f83ac63a54531288a00c79c1907ecd2003a8fceab5e00
Sep 17 18:30:02.610549 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.610517    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-lnf27"]
Sep 17 18:30:02.612709 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:30:02.612686    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e907c64_1699_4fa4_902b_b90ba5d7b8e6.slice/crio-c9d8865aec5e09fffdfb5ccbc86ba3ee0340b5021f14946d7040e2aeef71dd2b WatchSource:0}: Error finding container c9d8865aec5e09fffdfb5ccbc86ba3ee0340b5021f14946d7040e2aeef71dd2b: Status 404 returned error can't find the container with id c9d8865aec5e09fffdfb5ccbc86ba3ee0340b5021f14946d7040e2aeef71dd2b
Sep 17 18:30:02.809220 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.809131    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9afa96b-b391-4db4-9312-20c884e90da3-metrics-certs\") pod \"router-default-6b4db969fb-dbb5b\" (UID: \"a9afa96b-b391-4db4-9312-20c884e90da3\") " pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:02.809220 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.809176    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9afa96b-b391-4db4-9312-20c884e90da3-service-ca-bundle\") pod \"router-default-6b4db969fb-dbb5b\" (UID: \"a9afa96b-b391-4db4-9312-20c884e90da3\") " pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:02.809389 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:02.809310    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 18:30:02.809389 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:02.809335    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9afa96b-b391-4db4-9312-20c884e90da3-service-ca-bundle podName:a9afa96b-b391-4db4-9312-20c884e90da3 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:03.809316522 +0000 UTC m=+61.741518020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a9afa96b-b391-4db4-9312-20c884e90da3-service-ca-bundle") pod "router-default-6b4db969fb-dbb5b" (UID: "a9afa96b-b391-4db4-9312-20c884e90da3") : configmap references non-existent config key: service-ca.crt
Sep 17 18:30:02.809389 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:02.809361    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9afa96b-b391-4db4-9312-20c884e90da3-metrics-certs podName:a9afa96b-b391-4db4-9312-20c884e90da3 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:03.809347499 +0000 UTC m=+61.741549012 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a9afa96b-b391-4db4-9312-20c884e90da3-metrics-certs") pod "router-default-6b4db969fb-dbb5b" (UID: "a9afa96b-b391-4db4-9312-20c884e90da3") : secret "router-metrics-certs-default" not found
Sep 17 18:30:02.901827 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.901808    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-grgmp"
Sep 17 18:30:02.979918 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.979886    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-lnf27" event={"ID":"5e907c64-1699-4fa4-902b-b90ba5d7b8e6","Type":"ContainerStarted","Data":"c9d8865aec5e09fffdfb5ccbc86ba3ee0340b5021f14946d7040e2aeef71dd2b"}
Sep 17 18:30:02.981386 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:02.981364    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rh6tl" event={"ID":"a9151c33-3ccc-4061-b06b-9c3b8535fea4","Type":"ContainerStarted","Data":"825e4d995b22db48862f83ac63a54531288a00c79c1907ecd2003a8fceab5e00"}
Sep 17 18:30:03.816819 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:03.816781    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9afa96b-b391-4db4-9312-20c884e90da3-metrics-certs\") pod \"router-default-6b4db969fb-dbb5b\" (UID: \"a9afa96b-b391-4db4-9312-20c884e90da3\") " pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:03.817102 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:03.816842    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9afa96b-b391-4db4-9312-20c884e90da3-service-ca-bundle\") pod \"router-default-6b4db969fb-dbb5b\" (UID: \"a9afa96b-b391-4db4-9312-20c884e90da3\") " pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:03.817102 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:03.817036    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9afa96b-b391-4db4-9312-20c884e90da3-service-ca-bundle podName:a9afa96b-b391-4db4-9312-20c884e90da3 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:05.817014973 +0000 UTC m=+63.749216472 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a9afa96b-b391-4db4-9312-20c884e90da3-service-ca-bundle") pod "router-default-6b4db969fb-dbb5b" (UID: "a9afa96b-b391-4db4-9312-20c884e90da3") : configmap references non-existent config key: service-ca.crt
Sep 17 18:30:03.817404 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:03.817375    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 18:30:03.817508 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:03.817436    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9afa96b-b391-4db4-9312-20c884e90da3-metrics-certs podName:a9afa96b-b391-4db4-9312-20c884e90da3 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:05.817419497 +0000 UTC m=+63.749621008 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a9afa96b-b391-4db4-9312-20c884e90da3-metrics-certs") pod "router-default-6b4db969fb-dbb5b" (UID: "a9afa96b-b391-4db4-9312-20c884e90da3") : secret "router-metrics-certs-default" not found
Sep 17 18:30:04.987675 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:04.987644    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-lnf27" event={"ID":"5e907c64-1699-4fa4-902b-b90ba5d7b8e6","Type":"ContainerStarted","Data":"51f8e3f55c39e583fcd0a11b0070bae1bd2bfa653932f871779619b55ac11bcd"}
Sep 17 18:30:04.989165 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:04.989137    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rh6tl" event={"ID":"a9151c33-3ccc-4061-b06b-9c3b8535fea4","Type":"ContainerStarted","Data":"189f8eeef2be9c59794cf6edf2fc500e73edbc19ce4cfb436c439234b87faec9"}
Sep 17 18:30:05.004914 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:05.004870    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-lnf27" podStartSLOduration=0.767651063 podStartE2EDuration="3.004854272s" totalImagesPullingTime="2.237203209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:05.004623795 +0000 UTC m=+62.936825315" watchObservedRunningTime="2026-09-17 18:30:05.004854272 +0000 UTC m=+62.937055792"
Sep 17 18:30:05.019868 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:05.019822    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rh6tl" podStartSLOduration=1.677155751 podStartE2EDuration="4.019805402s" totalImagesPullingTime="2.342649651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:05.019016221 +0000 UTC m=+62.951217740" watchObservedRunningTime="2026-09-17 18:30:05.019805402 +0000 UTC m=+62.952006922"
Sep 17 18:30:05.832905 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:05.832871    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9afa96b-b391-4db4-9312-20c884e90da3-metrics-certs\") pod \"router-default-6b4db969fb-dbb5b\" (UID: \"a9afa96b-b391-4db4-9312-20c884e90da3\") " pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:05.833092 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:05.832922    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9afa96b-b391-4db4-9312-20c884e90da3-service-ca-bundle\") pod \"router-default-6b4db969fb-dbb5b\" (UID: \"a9afa96b-b391-4db4-9312-20c884e90da3\") " pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:05.833092 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:05.832998    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 18:30:05.833092 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:05.833057    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9afa96b-b391-4db4-9312-20c884e90da3-metrics-certs podName:a9afa96b-b391-4db4-9312-20c884e90da3 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:09.833042678 +0000 UTC m=+67.765244175 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a9afa96b-b391-4db4-9312-20c884e90da3-metrics-certs") pod "router-default-6b4db969fb-dbb5b" (UID: "a9afa96b-b391-4db4-9312-20c884e90da3") : secret "router-metrics-certs-default" not found
Sep 17 18:30:05.833092 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:05.833070    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9afa96b-b391-4db4-9312-20c884e90da3-service-ca-bundle podName:a9afa96b-b391-4db4-9312-20c884e90da3 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:09.83306384 +0000 UTC m=+67.765265336 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a9afa96b-b391-4db4-9312-20c884e90da3-service-ca-bundle") pod "router-default-6b4db969fb-dbb5b" (UID: "a9afa96b-b391-4db4-9312-20c884e90da3") : configmap references non-existent config key: service-ca.crt
Sep 17 18:30:06.306875 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:06.306837    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-cqjfb"]
Sep 17 18:30:06.310100 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:06.310079    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-cqjfb"
Sep 17 18:30:06.312431 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:06.312413    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 18:30:06.312518 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:06.312433    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-nhwh2\""
Sep 17 18:30:06.313020 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:06.313006    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 18:30:06.318341 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:06.318318    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-cqjfb"]
Sep 17 18:30:06.337040 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:06.337011    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf8bf\" (UniqueName: \"kubernetes.io/projected/3b433190-01d1-492c-af7a-2579e21e03ff-kube-api-access-qf8bf\") pod \"migrator-76794cbd6c-cqjfb\" (UID: \"3b433190-01d1-492c-af7a-2579e21e03ff\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-cqjfb"
Sep 17 18:30:06.438285 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:06.438247    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf8bf\" (UniqueName: \"kubernetes.io/projected/3b433190-01d1-492c-af7a-2579e21e03ff-kube-api-access-qf8bf\") pod \"migrator-76794cbd6c-cqjfb\" (UID: \"3b433190-01d1-492c-af7a-2579e21e03ff\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-cqjfb"
Sep 17 18:30:06.446883 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:06.446860    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf8bf\" (UniqueName: \"kubernetes.io/projected/3b433190-01d1-492c-af7a-2579e21e03ff-kube-api-access-qf8bf\") pod \"migrator-76794cbd6c-cqjfb\" (UID: \"3b433190-01d1-492c-af7a-2579e21e03ff\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-cqjfb"
Sep 17 18:30:06.619135 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:06.619078    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-cqjfb"
Sep 17 18:30:06.885348 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:06.885292    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-cqjfb"]
Sep 17 18:30:06.888506 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:30:06.888474    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b433190_01d1_492c_af7a_2579e21e03ff.slice/crio-0e4f0fdf1437758ff17d663a8c527089e883c86934530393c398c2e2c14c1025 WatchSource:0}: Error finding container 0e4f0fdf1437758ff17d663a8c527089e883c86934530393c398c2e2c14c1025: Status 404 returned error can't find the container with id 0e4f0fdf1437758ff17d663a8c527089e883c86934530393c398c2e2c14c1025
Sep 17 18:30:06.995123 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:06.995095    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-cqjfb" event={"ID":"3b433190-01d1-492c-af7a-2579e21e03ff","Type":"ContainerStarted","Data":"0e4f0fdf1437758ff17d663a8c527089e883c86934530393c398c2e2c14c1025"}
Sep 17 18:30:08.353360 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.353335    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4-metrics-certs\") pod \"network-metrics-daemon-w67kg\" (UID: \"db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4\") " pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:30:08.353690 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.353417    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/43264d48-807c-43de-8b8f-52db0225d1c9-original-pull-secret\") pod \"global-pull-secret-syncer-4zwdx\" (UID: \"43264d48-807c-43de-8b8f-52db0225d1c9\") " pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:30:08.355453 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.355434    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 18:30:08.355709 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.355694    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 18:30:08.364415 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:08.364395    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 18:30:08.364508 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:08.364458    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4-metrics-certs podName:db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4 nodeName:}" failed. No retries permitted until 2026-09-17 18:31:12.364437635 +0000 UTC m=+130.296639153 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4-metrics-certs") pod "network-metrics-daemon-w67kg" (UID: "db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4") : secret "metrics-daemon-secret" not found
Sep 17 18:30:08.366493 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.366474    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/43264d48-807c-43de-8b8f-52db0225d1c9-original-pull-secret\") pod \"global-pull-secret-syncer-4zwdx\" (UID: \"43264d48-807c-43de-8b8f-52db0225d1c9\") " pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:30:08.423115 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.423092    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-r2mgv"]
Sep 17 18:30:08.453811 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.453785    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-852t9\" (UniqueName: \"kubernetes.io/projected/5d3336f6-e38a-4183-8d58-3c99f1284f33-kube-api-access-852t9\") pod \"network-check-target-ssztx\" (UID: \"5d3336f6-e38a-4183-8d58-3c99f1284f33\") " pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:30:08.456412 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.456392    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 18:30:08.457638 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.457622    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r2mgv"]
Sep 17 18:30:08.457722 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.457715    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r2mgv"
Sep 17 18:30:08.460911 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.460891    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-h9v8g\""
Sep 17 18:30:08.461428 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.461409    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 18:30:08.461662 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.461646    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 18:30:08.461738 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.461711    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 18:30:08.462108 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.461935    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 18:30:08.466698 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.466682    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 18:30:08.476780 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.476743    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-852t9\" (UniqueName: \"kubernetes.io/projected/5d3336f6-e38a-4183-8d58-3c99f1284f33-kube-api-access-852t9\") pod \"network-check-target-ssztx\" (UID: \"5d3336f6-e38a-4183-8d58-3c99f1284f33\") " pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:30:08.554982 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.554959    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwkrs\" (UniqueName: \"kubernetes.io/projected/46404667-f42b-4c16-a119-2e4f5dfc8d41-kube-api-access-kwkrs\") pod \"insights-runtime-extractor-r2mgv\" (UID: \"46404667-f42b-4c16-a119-2e4f5dfc8d41\") " pod="openshift-insights/insights-runtime-extractor-r2mgv"
Sep 17 18:30:08.555063 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.555011    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46404667-f42b-4c16-a119-2e4f5dfc8d41-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r2mgv\" (UID: \"46404667-f42b-4c16-a119-2e4f5dfc8d41\") " pod="openshift-insights/insights-runtime-extractor-r2mgv"
Sep 17 18:30:08.555063 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.555045    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46404667-f42b-4c16-a119-2e4f5dfc8d41-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2mgv\" (UID: \"46404667-f42b-4c16-a119-2e4f5dfc8d41\") " pod="openshift-insights/insights-runtime-extractor-r2mgv"
Sep 17 18:30:08.555134 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.555106    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/46404667-f42b-4c16-a119-2e4f5dfc8d41-crio-socket\") pod \"insights-runtime-extractor-r2mgv\" (UID: \"46404667-f42b-4c16-a119-2e4f5dfc8d41\") " pod="openshift-insights/insights-runtime-extractor-r2mgv"
Sep 17 18:30:08.555134 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.555123    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/46404667-f42b-4c16-a119-2e4f5dfc8d41-data-volume\") pod \"insights-runtime-extractor-r2mgv\" (UID: \"46404667-f42b-4c16-a119-2e4f5dfc8d41\") " pod="openshift-insights/insights-runtime-extractor-r2mgv"
Sep 17 18:30:08.614255 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.614189    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-58cdj\""
Sep 17 18:30:08.617093 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.617079    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4zwdx"
Sep 17 18:30:08.622229 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.622212    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:30:08.643005 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.642964    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-66kd4"]
Sep 17 18:30:08.656102 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.655510    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46404667-f42b-4c16-a119-2e4f5dfc8d41-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r2mgv\" (UID: \"46404667-f42b-4c16-a119-2e4f5dfc8d41\") " pod="openshift-insights/insights-runtime-extractor-r2mgv"
Sep 17 18:30:08.656102 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.655551    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46404667-f42b-4c16-a119-2e4f5dfc8d41-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2mgv\" (UID: \"46404667-f42b-4c16-a119-2e4f5dfc8d41\") " pod="openshift-insights/insights-runtime-extractor-r2mgv"
Sep 17 18:30:08.656102 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.655597    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/46404667-f42b-4c16-a119-2e4f5dfc8d41-crio-socket\") pod \"insights-runtime-extractor-r2mgv\" (UID: \"46404667-f42b-4c16-a119-2e4f5dfc8d41\") " pod="openshift-insights/insights-runtime-extractor-r2mgv"
Sep 17 18:30:08.656102 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.655625    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/46404667-f42b-4c16-a119-2e4f5dfc8d41-data-volume\") pod \"insights-runtime-extractor-r2mgv\" (UID: \"46404667-f42b-4c16-a119-2e4f5dfc8d41\") " pod="openshift-insights/insights-runtime-extractor-r2mgv"
Sep 17 18:30:08.656102 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.655703    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwkrs\" (UniqueName: \"kubernetes.io/projected/46404667-f42b-4c16-a119-2e4f5dfc8d41-kube-api-access-kwkrs\") pod \"insights-runtime-extractor-r2mgv\" (UID: \"46404667-f42b-4c16-a119-2e4f5dfc8d41\") " pod="openshift-insights/insights-runtime-extractor-r2mgv"
Sep 17 18:30:08.656102 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:08.656047    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:30:08.656102 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:08.656106    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46404667-f42b-4c16-a119-2e4f5dfc8d41-kube-rbac-proxy-cm podName:46404667-f42b-4c16-a119-2e4f5dfc8d41 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:09.156085374 +0000 UTC m=+67.088286892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/46404667-f42b-4c16-a119-2e4f5dfc8d41-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r2mgv" (UID: "46404667-f42b-4c16-a119-2e4f5dfc8d41") : configmap "kube-rbac-proxy" not found
Sep 17 18:30:08.656461 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.656181    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/46404667-f42b-4c16-a119-2e4f5dfc8d41-crio-socket\") pod \"insights-runtime-extractor-r2mgv\" (UID: \"46404667-f42b-4c16-a119-2e4f5dfc8d41\") " pod="openshift-insights/insights-runtime-extractor-r2mgv"
Sep 17 18:30:08.656461 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:08.656256    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:08.656461 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:08.656318    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46404667-f42b-4c16-a119-2e4f5dfc8d41-insights-runtime-extractor-tls podName:46404667-f42b-4c16-a119-2e4f5dfc8d41 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:09.156299983 +0000 UTC m=+67.088501497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46404667-f42b-4c16-a119-2e4f5dfc8d41-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r2mgv" (UID: "46404667-f42b-4c16-a119-2e4f5dfc8d41") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:08.656461 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.656412    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/46404667-f42b-4c16-a119-2e4f5dfc8d41-data-volume\") pod \"insights-runtime-extractor-r2mgv\" (UID: \"46404667-f42b-4c16-a119-2e4f5dfc8d41\") " pod="openshift-insights/insights-runtime-extractor-r2mgv"
Sep 17 18:30:08.662857 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.662839    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-66kd4"
Sep 17 18:30:08.665823 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.665807    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-2ffnd\""
Sep 17 18:30:08.666057 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.666041    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 17 18:30:08.666297 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.666277    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 17 18:30:08.669413 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.669392    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-66kd4"]
Sep 17 18:30:08.673742 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.673695    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwkrs\" (UniqueName: \"kubernetes.io/projected/46404667-f42b-4c16-a119-2e4f5dfc8d41-kube-api-access-kwkrs\") pod \"insights-runtime-extractor-r2mgv\" (UID: \"46404667-f42b-4c16-a119-2e4f5dfc8d41\") " pod="openshift-insights/insights-runtime-extractor-r2mgv"
Sep 17 18:30:08.756548 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.756522    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6e2745b5-c8e7-4fcf-8414-94b1e315a38b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-66kd4\" (UID: \"6e2745b5-c8e7-4fcf-8414-94b1e315a38b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-66kd4"
Sep 17 18:30:08.756645 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.756577    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-tls\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:30:08.756698 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.756657    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6e2745b5-c8e7-4fcf-8414-94b1e315a38b-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-66kd4\" (UID: \"6e2745b5-c8e7-4fcf-8414-94b1e315a38b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-66kd4"
Sep 17 18:30:08.756698 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:08.756673    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:30:08.756698 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:08.756690    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f7b5f6fff-66wm8: secret "image-registry-tls" not found
Sep 17 18:30:08.756698 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.756692    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba1caac4-a2a1-4917-9598-4b2003401a84-metrics-tls\") pod \"dns-default-x9pxw\" (UID: \"ba1caac4-a2a1-4917-9598-4b2003401a84\") " pod="openshift-dns/dns-default-x9pxw"
Sep 17 18:30:08.756838 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:08.756755    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-tls podName:8af0b1ec-975b-44eb-a6b6-fc55869a5585 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:40.756731805 +0000 UTC m=+98.688933302 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-tls") pod "image-registry-5f7b5f6fff-66wm8" (UID: "8af0b1ec-975b-44eb-a6b6-fc55869a5585") : secret "image-registry-tls" not found
Sep 17 18:30:08.756838 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:08.756780    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:30:08.756838 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:08.756826    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba1caac4-a2a1-4917-9598-4b2003401a84-metrics-tls podName:ba1caac4-a2a1-4917-9598-4b2003401a84 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:40.756815327 +0000 UTC m=+98.689016825 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ba1caac4-a2a1-4917-9598-4b2003401a84-metrics-tls") pod "dns-default-x9pxw" (UID: "ba1caac4-a2a1-4917-9598-4b2003401a84") : secret "dns-default-metrics-tls" not found
Sep 17 18:30:08.857441 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.857419    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6e2745b5-c8e7-4fcf-8414-94b1e315a38b-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-66kd4\" (UID: \"6e2745b5-c8e7-4fcf-8414-94b1e315a38b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-66kd4"
Sep 17 18:30:08.857529 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.857462    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6e2745b5-c8e7-4fcf-8414-94b1e315a38b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-66kd4\" (UID: \"6e2745b5-c8e7-4fcf-8414-94b1e315a38b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-66kd4"
Sep 17 18:30:08.857577 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:08.857554    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 18:30:08.857613 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:08.857603    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e2745b5-c8e7-4fcf-8414-94b1e315a38b-networking-console-plugin-cert podName:6e2745b5-c8e7-4fcf-8414-94b1e315a38b nodeName:}" failed. No retries permitted until 2026-09-17 18:30:09.357588768 +0000 UTC m=+67.289790275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6e2745b5-c8e7-4fcf-8414-94b1e315a38b-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-66kd4" (UID: "6e2745b5-c8e7-4fcf-8414-94b1e315a38b") : secret "networking-console-plugin-cert" not found
Sep 17 18:30:08.857998 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.857981    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6e2745b5-c8e7-4fcf-8414-94b1e315a38b-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-66kd4\" (UID: \"6e2745b5-c8e7-4fcf-8414-94b1e315a38b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-66kd4"
Sep 17 18:30:08.905128 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.905079    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4zwdx"]
Sep 17 18:30:08.907704 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:30:08.907679    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43264d48_807c_43de_8b8f_52db0225d1c9.slice/crio-ddf0b76da8f80dc51ada01d9a519a6cc8ddbe21d0f51be6319d1795418ec0401 WatchSource:0}: Error finding container ddf0b76da8f80dc51ada01d9a519a6cc8ddbe21d0f51be6319d1795418ec0401: Status 404 returned error can't find the container with id ddf0b76da8f80dc51ada01d9a519a6cc8ddbe21d0f51be6319d1795418ec0401
Sep 17 18:30:08.942868 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:08.942850    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ssztx"]
Sep 17 18:30:08.946497 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:30:08.946475    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d3336f6_e38a_4183_8d58_3c99f1284f33.slice/crio-ec034904f699956a408c912ca0dcac66e1f5137a84d72cac760a9931bc30edb5 WatchSource:0}: Error finding container ec034904f699956a408c912ca0dcac66e1f5137a84d72cac760a9931bc30edb5: Status 404 returned error can't find the container with id ec034904f699956a408c912ca0dcac66e1f5137a84d72cac760a9931bc30edb5
Sep 17 18:30:09.001912 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:09.001885    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ssztx" event={"ID":"5d3336f6-e38a-4183-8d58-3c99f1284f33","Type":"ContainerStarted","Data":"ec034904f699956a408c912ca0dcac66e1f5137a84d72cac760a9931bc30edb5"}
Sep 17 18:30:09.003004 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:09.002984    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4zwdx" event={"ID":"43264d48-807c-43de-8b8f-52db0225d1c9","Type":"ContainerStarted","Data":"ddf0b76da8f80dc51ada01d9a519a6cc8ddbe21d0f51be6319d1795418ec0401"}
Sep 17 18:30:09.004589 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:09.004560    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-cqjfb" event={"ID":"3b433190-01d1-492c-af7a-2579e21e03ff","Type":"ContainerStarted","Data":"4d692dcb7ce4215e5a52eca660d8c01d05767a2e63bfc0653627595bbb0f31aa"}
Sep 17 18:30:09.004679 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:09.004611    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-cqjfb" event={"ID":"3b433190-01d1-492c-af7a-2579e21e03ff","Type":"ContainerStarted","Data":"e63c377322862ce32d0ee33911c0d34d8a339941812c296e4b5a3dff2c7ab8ab"}
Sep 17 18:30:09.024633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:09.024591    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-cqjfb" podStartSLOduration=1.564774348 podStartE2EDuration="3.024577117s" totalImagesPullingTime="1.459802769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:09.024111828 +0000 UTC m=+66.956313347" watchObservedRunningTime="2026-09-17 18:30:09.024577117 +0000 UTC m=+66.956778638"
Sep 17 18:30:09.159292 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:09.159268    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46404667-f42b-4c16-a119-2e4f5dfc8d41-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r2mgv\" (UID: \"46404667-f42b-4c16-a119-2e4f5dfc8d41\") " pod="openshift-insights/insights-runtime-extractor-r2mgv"
Sep 17 18:30:09.159407 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:09.159300    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46404667-f42b-4c16-a119-2e4f5dfc8d41-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2mgv\" (UID: \"46404667-f42b-4c16-a119-2e4f5dfc8d41\") " pod="openshift-insights/insights-runtime-extractor-r2mgv"
Sep 17 18:30:09.159407 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:09.159396    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:30:09.159511 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:09.159471    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46404667-f42b-4c16-a119-2e4f5dfc8d41-kube-rbac-proxy-cm podName:46404667-f42b-4c16-a119-2e4f5dfc8d41 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:10.159449696 +0000 UTC m=+68.091651198 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/46404667-f42b-4c16-a119-2e4f5dfc8d41-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r2mgv" (UID: "46404667-f42b-4c16-a119-2e4f5dfc8d41") : configmap "kube-rbac-proxy" not found
Sep 17 18:30:09.159511 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:09.159403    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:09.159618 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:09.159534    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46404667-f42b-4c16-a119-2e4f5dfc8d41-insights-runtime-extractor-tls podName:46404667-f42b-4c16-a119-2e4f5dfc8d41 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:10.159523158 +0000 UTC m=+68.091724654 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46404667-f42b-4c16-a119-2e4f5dfc8d41-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r2mgv" (UID: "46404667-f42b-4c16-a119-2e4f5dfc8d41") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:09.360541 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:09.360507    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6e2745b5-c8e7-4fcf-8414-94b1e315a38b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-66kd4\" (UID: \"6e2745b5-c8e7-4fcf-8414-94b1e315a38b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-66kd4"
Sep 17 18:30:09.361033 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:09.360662    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 18:30:09.361033 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:09.360734    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e2745b5-c8e7-4fcf-8414-94b1e315a38b-networking-console-plugin-cert podName:6e2745b5-c8e7-4fcf-8414-94b1e315a38b nodeName:}" failed. No retries permitted until 2026-09-17 18:30:10.360714307 +0000 UTC m=+68.292915821 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6e2745b5-c8e7-4fcf-8414-94b1e315a38b-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-66kd4" (UID: "6e2745b5-c8e7-4fcf-8414-94b1e315a38b") : secret "networking-console-plugin-cert" not found
Sep 17 18:30:09.864562 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:09.864525    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9afa96b-b391-4db4-9312-20c884e90da3-metrics-certs\") pod \"router-default-6b4db969fb-dbb5b\" (UID: \"a9afa96b-b391-4db4-9312-20c884e90da3\") " pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:09.864751 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:09.864647    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9afa96b-b391-4db4-9312-20c884e90da3-service-ca-bundle\") pod \"router-default-6b4db969fb-dbb5b\" (UID: \"a9afa96b-b391-4db4-9312-20c884e90da3\") " pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:09.864751 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:09.864696    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 18:30:09.864864 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:09.864765    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9afa96b-b391-4db4-9312-20c884e90da3-metrics-certs podName:a9afa96b-b391-4db4-9312-20c884e90da3 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:17.864747386 +0000 UTC m=+75.796948899 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a9afa96b-b391-4db4-9312-20c884e90da3-metrics-certs") pod "router-default-6b4db969fb-dbb5b" (UID: "a9afa96b-b391-4db4-9312-20c884e90da3") : secret "router-metrics-certs-default" not found
Sep 17 18:30:09.864864 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:09.864784    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9afa96b-b391-4db4-9312-20c884e90da3-service-ca-bundle podName:a9afa96b-b391-4db4-9312-20c884e90da3 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:17.864774742 +0000 UTC m=+75.796976246 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a9afa96b-b391-4db4-9312-20c884e90da3-service-ca-bundle") pod "router-default-6b4db969fb-dbb5b" (UID: "a9afa96b-b391-4db4-9312-20c884e90da3") : configmap references non-existent config key: service-ca.crt
Sep 17 18:30:10.166431 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:10.166400    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46404667-f42b-4c16-a119-2e4f5dfc8d41-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r2mgv\" (UID: \"46404667-f42b-4c16-a119-2e4f5dfc8d41\") " pod="openshift-insights/insights-runtime-extractor-r2mgv"
Sep 17 18:30:10.166603 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:10.166443    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46404667-f42b-4c16-a119-2e4f5dfc8d41-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2mgv\" (UID: \"46404667-f42b-4c16-a119-2e4f5dfc8d41\") " pod="openshift-insights/insights-runtime-extractor-r2mgv"
Sep 17 18:30:10.166603 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:10.166511    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:30:10.166603 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:10.166585    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46404667-f42b-4c16-a119-2e4f5dfc8d41-kube-rbac-proxy-cm podName:46404667-f42b-4c16-a119-2e4f5dfc8d41 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:12.166565145 +0000 UTC m=+70.098766657 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/46404667-f42b-4c16-a119-2e4f5dfc8d41-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r2mgv" (UID: "46404667-f42b-4c16-a119-2e4f5dfc8d41") : configmap "kube-rbac-proxy" not found
Sep 17 18:30:10.166770 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:10.166640    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:10.166770 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:10.166689    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46404667-f42b-4c16-a119-2e4f5dfc8d41-insights-runtime-extractor-tls podName:46404667-f42b-4c16-a119-2e4f5dfc8d41 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:12.166674516 +0000 UTC m=+70.098876031 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46404667-f42b-4c16-a119-2e4f5dfc8d41-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r2mgv" (UID: "46404667-f42b-4c16-a119-2e4f5dfc8d41") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:10.368100 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:10.368051    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6e2745b5-c8e7-4fcf-8414-94b1e315a38b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-66kd4\" (UID: \"6e2745b5-c8e7-4fcf-8414-94b1e315a38b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-66kd4"
Sep 17 18:30:10.368529 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:10.368295    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 18:30:10.368529 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:10.368362    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e2745b5-c8e7-4fcf-8414-94b1e315a38b-networking-console-plugin-cert podName:6e2745b5-c8e7-4fcf-8414-94b1e315a38b nodeName:}" failed. No retries permitted until 2026-09-17 18:30:12.368343067 +0000 UTC m=+70.300544599 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6e2745b5-c8e7-4fcf-8414-94b1e315a38b-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-66kd4" (UID: "6e2745b5-c8e7-4fcf-8414-94b1e315a38b") : secret "networking-console-plugin-cert" not found
Sep 17 18:30:10.995638 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:10.995601    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mkdtq_0058f4db-dbaa-4b8b-942b-06ecbd1301ef/dns-node-resolver/0.log"
Sep 17 18:30:11.793361 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:11.793330    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-glcw2_822d9b1e-1a38-49ac-900a-6ad946e6078f/node-ca/0.log"
Sep 17 18:30:12.183129 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:12.183092    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46404667-f42b-4c16-a119-2e4f5dfc8d41-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r2mgv\" (UID: \"46404667-f42b-4c16-a119-2e4f5dfc8d41\") " pod="openshift-insights/insights-runtime-extractor-r2mgv"
Sep 17 18:30:12.183343 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:12.183149    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46404667-f42b-4c16-a119-2e4f5dfc8d41-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2mgv\" (UID: \"46404667-f42b-4c16-a119-2e4f5dfc8d41\") " pod="openshift-insights/insights-runtime-extractor-r2mgv"
Sep 17 18:30:12.183343 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:12.183229    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:30:12.183343 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:12.183316    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46404667-f42b-4c16-a119-2e4f5dfc8d41-kube-rbac-proxy-cm podName:46404667-f42b-4c16-a119-2e4f5dfc8d41 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:16.183292723 +0000 UTC m=+74.115494240 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/46404667-f42b-4c16-a119-2e4f5dfc8d41-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r2mgv" (UID: "46404667-f42b-4c16-a119-2e4f5dfc8d41") : configmap "kube-rbac-proxy" not found
Sep 17 18:30:12.183343 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:12.183329    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:12.183567 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:12.183388    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46404667-f42b-4c16-a119-2e4f5dfc8d41-insights-runtime-extractor-tls podName:46404667-f42b-4c16-a119-2e4f5dfc8d41 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:16.183371924 +0000 UTC m=+74.115573425 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46404667-f42b-4c16-a119-2e4f5dfc8d41-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r2mgv" (UID: "46404667-f42b-4c16-a119-2e4f5dfc8d41") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:12.385394 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:12.385356    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6e2745b5-c8e7-4fcf-8414-94b1e315a38b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-66kd4\" (UID: \"6e2745b5-c8e7-4fcf-8414-94b1e315a38b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-66kd4"
Sep 17 18:30:12.385607 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:12.385503    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 18:30:12.385607 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:12.385567    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e2745b5-c8e7-4fcf-8414-94b1e315a38b-networking-console-plugin-cert podName:6e2745b5-c8e7-4fcf-8414-94b1e315a38b nodeName:}" failed. No retries permitted until 2026-09-17 18:30:16.385547952 +0000 UTC m=+74.317749452 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6e2745b5-c8e7-4fcf-8414-94b1e315a38b-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-66kd4" (UID: "6e2745b5-c8e7-4fcf-8414-94b1e315a38b") : secret "networking-console-plugin-cert" not found
Sep 17 18:30:12.595700 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:12.595615    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ltdzb_3158326a-19ea-455b-8737-8f01b5d51373/serve-healthcheck-canary/0.log"
Sep 17 18:30:12.797099 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:12.797069    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-cqjfb_3b433190-01d1-492c-af7a-2579e21e03ff/migrator/0.log"
Sep 17 18:30:12.993965 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:12.993939    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-cqjfb_3b433190-01d1-492c-af7a-2579e21e03ff/graceful-termination/0.log"
Sep 17 18:30:13.195664 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:13.195629    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-lnf27_5e907c64-1699-4fa4-902b-b90ba5d7b8e6/kube-storage-version-migrator-operator/0.log"
Sep 17 18:30:14.020849 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:14.020804    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ssztx" event={"ID":"5d3336f6-e38a-4183-8d58-3c99f1284f33","Type":"ContainerStarted","Data":"afb4af8bf68f10396ff57cd75593122810e8eb75e7d57e5a57fbd058229f69ba"}
Sep 17 18:30:14.021307 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:14.020901    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:30:14.022452 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:14.022426    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4zwdx" event={"ID":"43264d48-807c-43de-8b8f-52db0225d1c9","Type":"ContainerStarted","Data":"642ef00293d7fc436cbf773cda8a88c587d28ca837c75d9311002155df603a69"}
Sep 17 18:30:14.041477 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:14.041431    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ssztx" podStartSLOduration=66.786293629 podStartE2EDuration="1m11.041415981s" totalImagesPullingTime="4.255122352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:14.040775443 +0000 UTC m=+71.972976966" watchObservedRunningTime="2026-09-17 18:30:14.041415981 +0000 UTC m=+71.973617553"
Sep 17 18:30:14.056914 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:14.056869    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4zwdx" podStartSLOduration=66.757858415 podStartE2EDuration="1m11.056856233s" totalImagesPullingTime="4.298997818s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:14.055828708 +0000 UTC m=+71.988030228" watchObservedRunningTime="2026-09-17 18:30:14.056856233 +0000 UTC m=+71.989057751"
Sep 17 18:30:16.217579 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:16.217551    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46404667-f42b-4c16-a119-2e4f5dfc8d41-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r2mgv\" (UID: \"46404667-f42b-4c16-a119-2e4f5dfc8d41\") " pod="openshift-insights/insights-runtime-extractor-r2mgv"
Sep 17 18:30:16.217579 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:16.217583    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46404667-f42b-4c16-a119-2e4f5dfc8d41-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2mgv\" (UID: \"46404667-f42b-4c16-a119-2e4f5dfc8d41\") " pod="openshift-insights/insights-runtime-extractor-r2mgv"
Sep 17 18:30:16.218103 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:16.217682    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:30:16.218103 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:16.217752    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:16.218103 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:16.217770    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/46404667-f42b-4c16-a119-2e4f5dfc8d41-kube-rbac-proxy-cm podName:46404667-f42b-4c16-a119-2e4f5dfc8d41 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:24.217750782 +0000 UTC m=+82.149952288 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/46404667-f42b-4c16-a119-2e4f5dfc8d41-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r2mgv" (UID: "46404667-f42b-4c16-a119-2e4f5dfc8d41") : configmap "kube-rbac-proxy" not found
Sep 17 18:30:16.218103 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:16.217794    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46404667-f42b-4c16-a119-2e4f5dfc8d41-insights-runtime-extractor-tls podName:46404667-f42b-4c16-a119-2e4f5dfc8d41 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:24.217782818 +0000 UTC m=+82.149984315 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/46404667-f42b-4c16-a119-2e4f5dfc8d41-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r2mgv" (UID: "46404667-f42b-4c16-a119-2e4f5dfc8d41") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:16.419068 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:16.419036    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6e2745b5-c8e7-4fcf-8414-94b1e315a38b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-66kd4\" (UID: \"6e2745b5-c8e7-4fcf-8414-94b1e315a38b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-66kd4"
Sep 17 18:30:16.419213 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:16.419169    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 18:30:16.419332 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:16.419254    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e2745b5-c8e7-4fcf-8414-94b1e315a38b-networking-console-plugin-cert podName:6e2745b5-c8e7-4fcf-8414-94b1e315a38b nodeName:}" failed. No retries permitted until 2026-09-17 18:30:24.419240356 +0000 UTC m=+82.351441869 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6e2745b5-c8e7-4fcf-8414-94b1e315a38b-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-66kd4" (UID: "6e2745b5-c8e7-4fcf-8414-94b1e315a38b") : secret "networking-console-plugin-cert" not found
Sep 17 18:30:17.928824 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:17.928791    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9afa96b-b391-4db4-9312-20c884e90da3-metrics-certs\") pod \"router-default-6b4db969fb-dbb5b\" (UID: \"a9afa96b-b391-4db4-9312-20c884e90da3\") " pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:17.929237 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:17.928866    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9afa96b-b391-4db4-9312-20c884e90da3-service-ca-bundle\") pod \"router-default-6b4db969fb-dbb5b\" (UID: \"a9afa96b-b391-4db4-9312-20c884e90da3\") " pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:17.929237 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:30:17.928965    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a9afa96b-b391-4db4-9312-20c884e90da3-service-ca-bundle podName:a9afa96b-b391-4db4-9312-20c884e90da3 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:33.928952901 +0000 UTC m=+91.861154399 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a9afa96b-b391-4db4-9312-20c884e90da3-service-ca-bundle") pod "router-default-6b4db969fb-dbb5b" (UID: "a9afa96b-b391-4db4-9312-20c884e90da3") : configmap references non-existent config key: service-ca.crt
Sep 17 18:30:17.931153 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:17.931132    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9afa96b-b391-4db4-9312-20c884e90da3-metrics-certs\") pod \"router-default-6b4db969fb-dbb5b\" (UID: \"a9afa96b-b391-4db4-9312-20c884e90da3\") " pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:24.283469 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:24.283429    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46404667-f42b-4c16-a119-2e4f5dfc8d41-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r2mgv\" (UID: \"46404667-f42b-4c16-a119-2e4f5dfc8d41\") " pod="openshift-insights/insights-runtime-extractor-r2mgv"
Sep 17 18:30:24.283469 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:24.283467    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46404667-f42b-4c16-a119-2e4f5dfc8d41-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2mgv\" (UID: \"46404667-f42b-4c16-a119-2e4f5dfc8d41\") " pod="openshift-insights/insights-runtime-extractor-r2mgv"
Sep 17 18:30:24.284015 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:24.283994    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/46404667-f42b-4c16-a119-2e4f5dfc8d41-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r2mgv\" (UID: \"46404667-f42b-4c16-a119-2e4f5dfc8d41\") " pod="openshift-insights/insights-runtime-extractor-r2mgv"
Sep 17 18:30:24.286001 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:24.285981    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/46404667-f42b-4c16-a119-2e4f5dfc8d41-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2mgv\" (UID: \"46404667-f42b-4c16-a119-2e4f5dfc8d41\") " pod="openshift-insights/insights-runtime-extractor-r2mgv"
Sep 17 18:30:24.366302 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:24.366276    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r2mgv"
Sep 17 18:30:24.485104 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:24.485077    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6e2745b5-c8e7-4fcf-8414-94b1e315a38b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-66kd4\" (UID: \"6e2745b5-c8e7-4fcf-8414-94b1e315a38b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-66kd4"
Sep 17 18:30:24.487231 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:24.487190    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6e2745b5-c8e7-4fcf-8414-94b1e315a38b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-66kd4\" (UID: \"6e2745b5-c8e7-4fcf-8414-94b1e315a38b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-66kd4"
Sep 17 18:30:24.572789 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:24.572739    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-66kd4"
Sep 17 18:30:24.636968 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:24.636919    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r2mgv"]
Sep 17 18:30:24.640798 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:30:24.640773    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46404667_f42b_4c16_a119_2e4f5dfc8d41.slice/crio-21d61ecf64505339dede2ae62bebb89bd8eba20ecf4a058b1d2d14d8d909debb WatchSource:0}: Error finding container 21d61ecf64505339dede2ae62bebb89bd8eba20ecf4a058b1d2d14d8d909debb: Status 404 returned error can't find the container with id 21d61ecf64505339dede2ae62bebb89bd8eba20ecf4a058b1d2d14d8d909debb
Sep 17 18:30:24.849319 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:24.849251    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-66kd4"]
Sep 17 18:30:24.852177 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:30:24.852154    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e2745b5_c8e7_4fcf_8414_94b1e315a38b.slice/crio-b9ed132335ee67276f8dade0edb5a0086e3ba78b817a861fc1bf4fe85f9c33f0 WatchSource:0}: Error finding container b9ed132335ee67276f8dade0edb5a0086e3ba78b817a861fc1bf4fe85f9c33f0: Status 404 returned error can't find the container with id b9ed132335ee67276f8dade0edb5a0086e3ba78b817a861fc1bf4fe85f9c33f0
Sep 17 18:30:25.057810 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:25.057695    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r2mgv" event={"ID":"46404667-f42b-4c16-a119-2e4f5dfc8d41","Type":"ContainerStarted","Data":"6f940cf3f627f155a1893523dc430d3e54868c1d83174e256a4230e272a92edc"}
Sep 17 18:30:25.057810 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:25.057751    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r2mgv" event={"ID":"46404667-f42b-4c16-a119-2e4f5dfc8d41","Type":"ContainerStarted","Data":"21d61ecf64505339dede2ae62bebb89bd8eba20ecf4a058b1d2d14d8d909debb"}
Sep 17 18:30:25.058918 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:25.058891    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-66kd4" event={"ID":"6e2745b5-c8e7-4fcf-8414-94b1e315a38b","Type":"ContainerStarted","Data":"b9ed132335ee67276f8dade0edb5a0086e3ba78b817a861fc1bf4fe85f9c33f0"}
Sep 17 18:30:26.064544 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:26.064500    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-66kd4" event={"ID":"6e2745b5-c8e7-4fcf-8414-94b1e315a38b","Type":"ContainerStarted","Data":"f5dd3e0acf7396dba2f2e7dd5e66ca4fba3d2ddc485029ad4e8b94642c9b0147"}
Sep 17 18:30:26.066636 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:26.066614    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r2mgv" event={"ID":"46404667-f42b-4c16-a119-2e4f5dfc8d41","Type":"ContainerStarted","Data":"9c0d0e4f9cfa9b0a36fbcf299054dc2fc3a8834999cbc19b974a4c0ccba852e5"}
Sep 17 18:30:27.072294 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:27.072223    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r2mgv" event={"ID":"46404667-f42b-4c16-a119-2e4f5dfc8d41","Type":"ContainerStarted","Data":"c370b9eb331aa369d0d97fa634269d98541005d77ebfe64d266c2abeb52afe23"}
Sep 17 18:30:27.093164 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:27.093120    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-66kd4" podStartSLOduration=18.062086404 podStartE2EDuration="19.093102786s" totalImagesPullingTime="1.031016382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:26.077950493 +0000 UTC m=+84.010152013" watchObservedRunningTime="2026-09-17 18:30:27.093102786 +0000 UTC m=+85.025304305"
Sep 17 18:30:29.840697 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:29.840089    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-r2mgv" podStartSLOduration=19.852353471 podStartE2EDuration="21.840055767s" totalImagesPullingTime="1.987702296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:30:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:27.093386285 +0000 UTC m=+85.025587806" watchObservedRunningTime="2026-09-17 18:30:29.840055767 +0000 UTC m=+87.772257286"
Sep 17 18:30:29.841516 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:29.841465    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c5d49c5f-4vnfp"]
Sep 17 18:30:29.844812 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:29.844784    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c5d49c5f-4vnfp"
Sep 17 18:30:29.847347 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:29.847302    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 18:30:29.848350 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:29.848328    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 18:30:29.848463 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:29.848331    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 18:30:29.848541 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:29.848331    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 18:30:29.858649 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:29.858625    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c5d49c5f-4vnfp"]
Sep 17 18:30:29.913047 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:29.913019    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-hlwgt"]
Sep 17 18:30:29.916025 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:29.916009    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-hlwgt"
Sep 17 18:30:29.919719 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:29.919696    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 18:30:29.920000 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:29.919983    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-kjbmw\""
Sep 17 18:30:29.920067 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:29.919991    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 18:30:29.922898 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:29.922880    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8gdd\" (UniqueName: \"kubernetes.io/projected/8e8de666-5847-4d90-8f55-f9925bd148bc-kube-api-access-f8gdd\") pod \"klusterlet-addon-workmgr-7c5d49c5f-4vnfp\" (UID: \"8e8de666-5847-4d90-8f55-f9925bd148bc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c5d49c5f-4vnfp"
Sep 17 18:30:29.922982 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:29.922909    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8e8de666-5847-4d90-8f55-f9925bd148bc-klusterlet-config\") pod \"klusterlet-addon-workmgr-7c5d49c5f-4vnfp\" (UID: \"8e8de666-5847-4d90-8f55-f9925bd148bc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c5d49c5f-4vnfp"
Sep 17 18:30:29.923029 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:29.923007    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e8de666-5847-4d90-8f55-f9925bd148bc-tmp\") pod \"klusterlet-addon-workmgr-7c5d49c5f-4vnfp\" (UID: \"8e8de666-5847-4d90-8f55-f9925bd148bc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c5d49c5f-4vnfp"
Sep 17 18:30:29.929508 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:29.929485    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-hlwgt"]
Sep 17 18:30:30.023819 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:30.023790    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e8de666-5847-4d90-8f55-f9925bd148bc-tmp\") pod \"klusterlet-addon-workmgr-7c5d49c5f-4vnfp\" (UID: \"8e8de666-5847-4d90-8f55-f9925bd148bc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c5d49c5f-4vnfp"
Sep 17 18:30:30.023922 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:30.023843    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8gdd\" (UniqueName: \"kubernetes.io/projected/8e8de666-5847-4d90-8f55-f9925bd148bc-kube-api-access-f8gdd\") pod \"klusterlet-addon-workmgr-7c5d49c5f-4vnfp\" (UID: \"8e8de666-5847-4d90-8f55-f9925bd148bc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c5d49c5f-4vnfp"
Sep 17 18:30:30.023977 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:30.023954    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8e8de666-5847-4d90-8f55-f9925bd148bc-klusterlet-config\") pod \"klusterlet-addon-workmgr-7c5d49c5f-4vnfp\" (UID: \"8e8de666-5847-4d90-8f55-f9925bd148bc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c5d49c5f-4vnfp"
Sep 17 18:30:30.024069 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:30.024052    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhcqx\" (UniqueName: \"kubernetes.io/projected/172edcc2-26ce-48f5-b8a6-20c54c0537ea-kube-api-access-xhcqx\") pod \"downloads-6594f4c75-hlwgt\" (UID: \"172edcc2-26ce-48f5-b8a6-20c54c0537ea\") " pod="openshift-console/downloads-6594f4c75-hlwgt"
Sep 17 18:30:30.024223 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:30.024191    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e8de666-5847-4d90-8f55-f9925bd148bc-tmp\") pod \"klusterlet-addon-workmgr-7c5d49c5f-4vnfp\" (UID: \"8e8de666-5847-4d90-8f55-f9925bd148bc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c5d49c5f-4vnfp"
Sep 17 18:30:30.026350 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:30.026315    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8e8de666-5847-4d90-8f55-f9925bd148bc-klusterlet-config\") pod \"klusterlet-addon-workmgr-7c5d49c5f-4vnfp\" (UID: \"8e8de666-5847-4d90-8f55-f9925bd148bc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c5d49c5f-4vnfp"
Sep 17 18:30:30.039945 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:30.039926    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8gdd\" (UniqueName: \"kubernetes.io/projected/8e8de666-5847-4d90-8f55-f9925bd148bc-kube-api-access-f8gdd\") pod \"klusterlet-addon-workmgr-7c5d49c5f-4vnfp\" (UID: \"8e8de666-5847-4d90-8f55-f9925bd148bc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c5d49c5f-4vnfp"
Sep 17 18:30:30.125222 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:30.125152    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhcqx\" (UniqueName: \"kubernetes.io/projected/172edcc2-26ce-48f5-b8a6-20c54c0537ea-kube-api-access-xhcqx\") pod \"downloads-6594f4c75-hlwgt\" (UID: \"172edcc2-26ce-48f5-b8a6-20c54c0537ea\") " pod="openshift-console/downloads-6594f4c75-hlwgt"
Sep 17 18:30:30.143286 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:30.143261    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhcqx\" (UniqueName: \"kubernetes.io/projected/172edcc2-26ce-48f5-b8a6-20c54c0537ea-kube-api-access-xhcqx\") pod \"downloads-6594f4c75-hlwgt\" (UID: \"172edcc2-26ce-48f5-b8a6-20c54c0537ea\") " pod="openshift-console/downloads-6594f4c75-hlwgt"
Sep 17 18:30:30.155075 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:30.155056    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c5d49c5f-4vnfp"
Sep 17 18:30:30.224516 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:30.224490    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-hlwgt"
Sep 17 18:30:30.431348 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:30.431322    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c5d49c5f-4vnfp"]
Sep 17 18:30:30.434229 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:30:30.434190    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e8de666_5847_4d90_8f55_f9925bd148bc.slice/crio-6725b28d7272c9e6d4879f6f227f3ede20049a9281b860a5c1cf0472288b4b54 WatchSource:0}: Error finding container 6725b28d7272c9e6d4879f6f227f3ede20049a9281b860a5c1cf0472288b4b54: Status 404 returned error can't find the container with id 6725b28d7272c9e6d4879f6f227f3ede20049a9281b860a5c1cf0472288b4b54
Sep 17 18:30:30.495241 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:30.495218    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-hlwgt"]
Sep 17 18:30:30.499637 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:30:30.499615    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod172edcc2_26ce_48f5_b8a6_20c54c0537ea.slice/crio-a7ffb389ff8b68178f29ab2a14f35867bf082762cdfd37b3be83500f5e8e2ed2 WatchSource:0}: Error finding container a7ffb389ff8b68178f29ab2a14f35867bf082762cdfd37b3be83500f5e8e2ed2: Status 404 returned error can't find the container with id a7ffb389ff8b68178f29ab2a14f35867bf082762cdfd37b3be83500f5e8e2ed2
Sep 17 18:30:31.090593 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:31.090561    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-hlwgt" event={"ID":"172edcc2-26ce-48f5-b8a6-20c54c0537ea","Type":"ContainerStarted","Data":"a7ffb389ff8b68178f29ab2a14f35867bf082762cdfd37b3be83500f5e8e2ed2"}
Sep 17 18:30:31.092151 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:31.092124    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c5d49c5f-4vnfp" event={"ID":"8e8de666-5847-4d90-8f55-f9925bd148bc","Type":"ContainerStarted","Data":"6725b28d7272c9e6d4879f6f227f3ede20049a9281b860a5c1cf0472288b4b54"}
Sep 17 18:30:33.955162 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:33.955106    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9afa96b-b391-4db4-9312-20c884e90da3-service-ca-bundle\") pod \"router-default-6b4db969fb-dbb5b\" (UID: \"a9afa96b-b391-4db4-9312-20c884e90da3\") " pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:33.955845 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:33.955794    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9afa96b-b391-4db4-9312-20c884e90da3-service-ca-bundle\") pod \"router-default-6b4db969fb-dbb5b\" (UID: \"a9afa96b-b391-4db4-9312-20c884e90da3\") " pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:34.153956 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:34.153923    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vrmnj\""
Sep 17 18:30:34.162152 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:34.162115    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:34.468064 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:34.468033    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6b4db969fb-dbb5b"]
Sep 17 18:30:34.471884 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:30:34.471824    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9afa96b_b391_4db4_9312_20c884e90da3.slice/crio-1f1ca2d4a0ccedd2e26361262e26444260adc12da23ede3cf3128f14c5617c7d WatchSource:0}: Error finding container 1f1ca2d4a0ccedd2e26361262e26444260adc12da23ede3cf3128f14c5617c7d: Status 404 returned error can't find the container with id 1f1ca2d4a0ccedd2e26361262e26444260adc12da23ede3cf3128f14c5617c7d
Sep 17 18:30:35.111849 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:35.111801    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6b4db969fb-dbb5b" event={"ID":"a9afa96b-b391-4db4-9312-20c884e90da3","Type":"ContainerStarted","Data":"3ee296ade0d8f7c6079047c325ca7cdbb6c7acc20d5a3a8f1fe12c2283df9aab"}
Sep 17 18:30:35.112328 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:35.111855    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6b4db969fb-dbb5b" event={"ID":"a9afa96b-b391-4db4-9312-20c884e90da3","Type":"ContainerStarted","Data":"1f1ca2d4a0ccedd2e26361262e26444260adc12da23ede3cf3128f14c5617c7d"}
Sep 17 18:30:35.142328 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:35.141673    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6b4db969fb-dbb5b" podStartSLOduration=33.141652695 podStartE2EDuration="33.141652695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:35.140998072 +0000 UTC m=+93.073199593" watchObservedRunningTime="2026-09-17 18:30:35.141652695 +0000 UTC m=+93.073854206"
Sep 17 18:30:35.162927 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:35.162899    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:35.166023 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:35.166001    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:36.116108 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:36.116047    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:36.117413 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:36.117386    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6b4db969fb-dbb5b"
Sep 17 18:30:37.120879 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:37.120845    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c5d49c5f-4vnfp" event={"ID":"8e8de666-5847-4d90-8f55-f9925bd148bc","Type":"ContainerStarted","Data":"8f840121060f4a04b490c9dfe4d9b75000a9dd1f552fb95c5604fda69046e125"}
Sep 17 18:30:37.121341 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:37.120951    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c5d49c5f-4vnfp"
Sep 17 18:30:37.122160 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:37.122135    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c5d49c5f-4vnfp"
Sep 17 18:30:37.150922 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:37.150878    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c5d49c5f-4vnfp" podStartSLOduration=1.6058128310000002 podStartE2EDuration="8.150864783s" totalImagesPullingTime="6.545051952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:37.150371112 +0000 UTC m=+95.082572631" watchObservedRunningTime="2026-09-17 18:30:37.150864783 +0000 UTC m=+95.083066302"
Sep 17 18:30:40.810083 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:40.810034    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-tls\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:30:40.810579 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:40.810134    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba1caac4-a2a1-4917-9598-4b2003401a84-metrics-tls\") pod \"dns-default-x9pxw\" (UID: \"ba1caac4-a2a1-4917-9598-4b2003401a84\") " pod="openshift-dns/dns-default-x9pxw"
Sep 17 18:30:40.812767 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:40.812742    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ba1caac4-a2a1-4917-9598-4b2003401a84-metrics-tls\") pod \"dns-default-x9pxw\" (UID: \"ba1caac4-a2a1-4917-9598-4b2003401a84\") " pod="openshift-dns/dns-default-x9pxw"
Sep 17 18:30:40.812926 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:40.812907    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-tls\") pod \"image-registry-5f7b5f6fff-66wm8\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") " pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:30:40.840251 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:40.840228    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pndxg\""
Sep 17 18:30:40.848436 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:40.848407    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:30:40.851874 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:40.851854    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cvvrd\""
Sep 17 18:30:40.860576 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:40.860555    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x9pxw"
Sep 17 18:30:45.029549 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:45.029515    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ssztx"
Sep 17 18:30:46.914521 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:46.914487    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x9pxw"]
Sep 17 18:30:46.917751 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:30:46.917722    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba1caac4_a2a1_4917_9598_4b2003401a84.slice/crio-ca43ff1527a32882e2f2923ae65a8d99e4d8b2f14a5bdb188e2192fe1db76bf3 WatchSource:0}: Error finding container ca43ff1527a32882e2f2923ae65a8d99e4d8b2f14a5bdb188e2192fe1db76bf3: Status 404 returned error can't find the container with id ca43ff1527a32882e2f2923ae65a8d99e4d8b2f14a5bdb188e2192fe1db76bf3
Sep 17 18:30:46.922412 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:46.922369    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f7b5f6fff-66wm8"]
Sep 17 18:30:46.926116 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:30:46.926090    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8af0b1ec_975b_44eb_a6b6_fc55869a5585.slice/crio-0994928d245a307e2adc5d18b4e3882ef63c6fe7f5ad2c3b6ae9d3699c88f4f7 WatchSource:0}: Error finding container 0994928d245a307e2adc5d18b4e3882ef63c6fe7f5ad2c3b6ae9d3699c88f4f7: Status 404 returned error can't find the container with id 0994928d245a307e2adc5d18b4e3882ef63c6fe7f5ad2c3b6ae9d3699c88f4f7
Sep 17 18:30:47.162684 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:47.162644    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-hlwgt" event={"ID":"172edcc2-26ce-48f5-b8a6-20c54c0537ea","Type":"ContainerStarted","Data":"fb04ea33b897f7d647a690aa15aacc5b50372fc3b4bd4dae6fa508c1135f54fc"}
Sep 17 18:30:47.162959 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:47.162903    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-hlwgt"
Sep 17 18:30:47.164364 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:47.164327    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x9pxw" event={"ID":"ba1caac4-a2a1-4917-9598-4b2003401a84","Type":"ContainerStarted","Data":"ca43ff1527a32882e2f2923ae65a8d99e4d8b2f14a5bdb188e2192fe1db76bf3"}
Sep 17 18:30:47.166153 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:47.166099    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8" event={"ID":"8af0b1ec-975b-44eb-a6b6-fc55869a5585","Type":"ContainerStarted","Data":"d55c60b9ca71ddda7018b77f0ddfa78d25adb54070cc9b7a587166a38b9eec92"}
Sep 17 18:30:47.166153 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:47.166143    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8" event={"ID":"8af0b1ec-975b-44eb-a6b6-fc55869a5585","Type":"ContainerStarted","Data":"0994928d245a307e2adc5d18b4e3882ef63c6fe7f5ad2c3b6ae9d3699c88f4f7"}
Sep 17 18:30:47.166336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:47.166166    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:30:47.175774 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:47.175753    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-hlwgt"
Sep 17 18:30:47.196956 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:47.196906    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-hlwgt" podStartSLOduration=1.961948892 podStartE2EDuration="18.196887117s" totalImagesPullingTime="16.234938225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:47.19679177 +0000 UTC m=+105.128993290" watchObservedRunningTime="2026-09-17 18:30:47.196887117 +0000 UTC m=+105.129088639"
Sep 17 18:30:47.228708 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:47.228653    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8" podStartSLOduration=103.228638076 podStartE2EDuration="1m43.228638076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:47.227400749 +0000 UTC m=+105.159602269" watchObservedRunningTime="2026-09-17 18:30:47.228638076 +0000 UTC m=+105.160839595"
Sep 17 18:30:50.180070 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:50.180032    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x9pxw" event={"ID":"ba1caac4-a2a1-4917-9598-4b2003401a84","Type":"ContainerStarted","Data":"da7b3ea46476704c52f848555e735674f2d99283cafb857e13b8c64941fdcccf"}
Sep 17 18:30:50.180550 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:50.180091    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x9pxw" event={"ID":"ba1caac4-a2a1-4917-9598-4b2003401a84","Type":"ContainerStarted","Data":"b5df8ae0aa44a32985bfb9a7d9f412cb929b194be5016cee664c474bb2e6da59"}
Sep 17 18:30:50.180550 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:50.180127    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-x9pxw"
Sep 17 18:30:50.218072 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:50.218024    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-x9pxw" podStartSLOduration=71.82278742 podStartE2EDuration="1m14.218006686s" totalImagesPullingTime="2.395219266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:50.217268862 +0000 UTC m=+108.149470381" watchObservedRunningTime="2026-09-17 18:30:50.218006686 +0000 UTC m=+108.150208206"
Sep 17 18:30:51.835460 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:51.835427    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qhfss"]
Sep 17 18:30:51.878432 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:51.878406    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qhfss"
Sep 17 18:30:51.883071 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:51.883042    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 18:30:51.883470 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:51.883449    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 18:30:51.883756 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:51.883733    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kwtnd\""
Sep 17 18:30:51.884082 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:51.884057    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 18:30:51.884242 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:51.884138    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 18:30:51.884313 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:51.884246    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 18:30:51.884422 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:51.884389    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 18:30:52.001450 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:52.001417    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1aea236a-8063-4c52-bda5-8c22fe1a166c-node-exporter-accelerators-collector-config\") pod \"node-exporter-qhfss\" (UID: \"1aea236a-8063-4c52-bda5-8c22fe1a166c\") " pod="openshift-monitoring/node-exporter-qhfss"
Sep 17 18:30:52.001583 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:52.001467    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1aea236a-8063-4c52-bda5-8c22fe1a166c-node-exporter-wtmp\") pod \"node-exporter-qhfss\" (UID: \"1aea236a-8063-4c52-bda5-8c22fe1a166c\") " pod="openshift-monitoring/node-exporter-qhfss"
Sep 17 18:30:52.001583 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:52.001516    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1aea236a-8063-4c52-bda5-8c22fe1a166c-root\") pod \"node-exporter-qhfss\" (UID: \"1aea236a-8063-4c52-bda5-8c22fe1a166c\") " pod="openshift-monitoring/node-exporter-qhfss"
Sep 17 18:30:52.001674 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:52.001579    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29spj\" (UniqueName: \"kubernetes.io/projected/1aea236a-8063-4c52-bda5-8c22fe1a166c-kube-api-access-29spj\") pod \"node-exporter-qhfss\" (UID: \"1aea236a-8063-4c52-bda5-8c22fe1a166c\") " pod="openshift-monitoring/node-exporter-qhfss"
Sep 17 18:30:52.001674 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:52.001612    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1aea236a-8063-4c52-bda5-8c22fe1a166c-node-exporter-tls\") pod \"node-exporter-qhfss\" (UID: \"1aea236a-8063-4c52-bda5-8c22fe1a166c\") " pod="openshift-monitoring/node-exporter-qhfss"
Sep 17 18:30:52.001674 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:52.001634    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1aea236a-8063-4c52-bda5-8c22fe1a166c-metrics-client-ca\") pod \"node-exporter-qhfss\" (UID: \"1aea236a-8063-4c52-bda5-8c22fe1a166c\") " pod="openshift-monitoring/node-exporter-qhfss"
Sep 17 18:30:52.001674 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:52.001656    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1aea236a-8063-4c52-bda5-8c22fe1a166c-node-exporter-textfile\") pod \"node-exporter-qhfss\" (UID: \"1aea236a-8063-4c52-bda5-8c22fe1a166c\") " pod="openshift-monitoring/node-exporter-qhfss"
Sep 17 18:30:52.001832 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:52.001681    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1aea236a-8063-4c52-bda5-8c22fe1a166c-sys\") pod \"node-exporter-qhfss\" (UID: \"1aea236a-8063-4c52-bda5-8c22fe1a166c\") " pod="openshift-monitoring/node-exporter-qhfss"
Sep 17 18:30:52.001832 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:52.001718    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1aea236a-8063-4c52-bda5-8c22fe1a166c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qhfss\" (UID: \"1aea236a-8063-4c52-bda5-8c22fe1a166c\") " pod="openshift-monitoring/node-exporter-qhfss"
Sep 17 18:30:52.102722 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:52.102642    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1aea236a-8063-4c52-bda5-8c22fe1a166c-node-exporter-wtmp\") pod \"node-exporter-qhfss\" (UID: \"1aea236a-8063-4c52-bda5-8c22fe1a166c\") " pod="openshift-monitoring/node-exporter-qhfss"
Sep 17 18:30:52.102722 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:52.102711    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1aea236a-8063-4c52-bda5-8c22fe1a166c-root\") pod \"node-exporter-qhfss\" (UID: \"1aea236a-8063-4c52-bda5-8c22fe1a166c\") " pod="openshift-monitoring/node-exporter-qhfss"
Sep 17 18:30:52.102909 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:52.102757    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29spj\" (UniqueName: \"kubernetes.io/projected/1aea236a-8063-4c52-bda5-8c22fe1a166c-kube-api-access-29spj\") pod \"node-exporter-qhfss\" (UID: \"1aea236a-8063-4c52-bda5-8c22fe1a166c\") " pod="openshift-monitoring/node-exporter-qhfss"
Sep 17 18:30:52.102909 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:52.102789    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1aea236a-8063-4c52-bda5-8c22fe1a166c-node-exporter-tls\") pod \"node-exporter-qhfss\" (UID: \"1aea236a-8063-4c52-bda5-8c22fe1a166c\") " pod="openshift-monitoring/node-exporter-qhfss"
Sep 17 18:30:52.102909 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:52.102818    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1aea236a-8063-4c52-bda5-8c22fe1a166c-node-exporter-wtmp\") pod \"node-exporter-qhfss\" (UID: \"1aea236a-8063-4c52-bda5-8c22fe1a166c\") " pod="openshift-monitoring/node-exporter-qhfss"
Sep 17 18:30:52.102909 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:52.102836    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1aea236a-8063-4c52-bda5-8c22fe1a166c-root\") pod \"node-exporter-qhfss\" (UID: \"1aea236a-8063-4c52-bda5-8c22fe1a166c\") " pod="openshift-monitoring/node-exporter-qhfss"
Sep 17 18:30:52.103224 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:52.102952    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1aea236a-8063-4c52-bda5-8c22fe1a166c-metrics-client-ca\") pod \"node-exporter-qhfss\" (UID: \"1aea236a-8063-4c52-bda5-8c22fe1a166c\") " pod="openshift-monitoring/node-exporter-qhfss"
Sep 17 18:30:52.103224 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:52.102997    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1aea236a-8063-4c52-bda5-8c22fe1a166c-node-exporter-textfile\") pod \"node-exporter-qhfss\" (UID: \"1aea236a-8063-4c52-bda5-8c22fe1a166c\") " pod="openshift-monitoring/node-exporter-qhfss"
Sep 17 18:30:52.103224 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:52.103054    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1aea236a-8063-4c52-bda5-8c22fe1a166c-sys\") pod \"node-exporter-qhfss\" (UID: \"1aea236a-8063-4c52-bda5-8c22fe1a166c\") " pod="openshift-monitoring/node-exporter-qhfss"
Sep 17 18:30:52.103224 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:52.103094    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1aea236a-8063-4c52-bda5-8c22fe1a166c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qhfss\" (UID: \"1aea236a-8063-4c52-bda5-8c22fe1a166c\") " pod="openshift-monitoring/node-exporter-qhfss"
Sep 17 18:30:52.103224 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:52.103137    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1aea236a-8063-4c52-bda5-8c22fe1a166c-node-exporter-accelerators-collector-config\") pod \"node-exporter-qhfss\" (UID: \"1aea236a-8063-4c52-bda5-8c22fe1a166c\") " pod="openshift-monitoring/node-exporter-qhfss"
Sep 17 18:30:52.103457 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:52.103236    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1aea236a-8063-4c52-bda5-8c22fe1a166c-sys\") pod \"node-exporter-qhfss\" (UID: \"1aea236a-8063-4c52-bda5-8c22fe1a166c\") " pod="openshift-monitoring/node-exporter-qhfss"
Sep 17 18:30:52.103457 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:52.103337    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1aea236a-8063-4c52-bda5-8c22fe1a166c-node-exporter-textfile\") pod \"node-exporter-qhfss\" (UID: \"1aea236a-8063-4c52-bda5-8c22fe1a166c\") " pod="openshift-monitoring/node-exporter-qhfss"
Sep 17 18:30:52.103697 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:52.103675    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1aea236a-8063-4c52-bda5-8c22fe1a166c-metrics-client-ca\") pod \"node-exporter-qhfss\" (UID: \"1aea236a-8063-4c52-bda5-8c22fe1a166c\") " pod="openshift-monitoring/node-exporter-qhfss"
Sep 17 18:30:52.103843 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:52.103675    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1aea236a-8063-4c52-bda5-8c22fe1a166c-node-exporter-accelerators-collector-config\") pod \"node-exporter-qhfss\" (UID: \"1aea236a-8063-4c52-bda5-8c22fe1a166c\") " pod="openshift-monitoring/node-exporter-qhfss"
Sep 17 18:30:52.105354 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:52.105334    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1aea236a-8063-4c52-bda5-8c22fe1a166c-node-exporter-tls\") pod \"node-exporter-qhfss\" (UID: \"1aea236a-8063-4c52-bda5-8c22fe1a166c\") " pod="openshift-monitoring/node-exporter-qhfss"
Sep 17 18:30:52.105648 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:52.105631    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1aea236a-8063-4c52-bda5-8c22fe1a166c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qhfss\" (UID: \"1aea236a-8063-4c52-bda5-8c22fe1a166c\") " pod="openshift-monitoring/node-exporter-qhfss"
Sep 17 18:30:52.110987 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:52.110968    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29spj\" (UniqueName: \"kubernetes.io/projected/1aea236a-8063-4c52-bda5-8c22fe1a166c-kube-api-access-29spj\") pod \"node-exporter-qhfss\" (UID: \"1aea236a-8063-4c52-bda5-8c22fe1a166c\") " pod="openshift-monitoring/node-exporter-qhfss"
Sep 17 18:30:52.190306 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:52.190281    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qhfss"
Sep 17 18:30:52.220842 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:30:52.220807    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aea236a_8063_4c52_bda5_8c22fe1a166c.slice/crio-aebc654315455df015f051c7ddf4a5ca01323623dd7b24b795ca87dfcbfe5484 WatchSource:0}: Error finding container aebc654315455df015f051c7ddf4a5ca01323623dd7b24b795ca87dfcbfe5484: Status 404 returned error can't find the container with id aebc654315455df015f051c7ddf4a5ca01323623dd7b24b795ca87dfcbfe5484
Sep 17 18:30:53.193064 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:53.193018    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qhfss" event={"ID":"1aea236a-8063-4c52-bda5-8c22fe1a166c","Type":"ContainerStarted","Data":"aebc654315455df015f051c7ddf4a5ca01323623dd7b24b795ca87dfcbfe5484"}
Sep 17 18:30:53.367291 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:53.367253    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f7b5f6fff-66wm8"]
Sep 17 18:30:54.199122 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:54.199085    2565 generic.go:356] "Generic (PLEG): container finished" podID="1aea236a-8063-4c52-bda5-8c22fe1a166c" containerID="09e65ce15c5351576f28426dae23628aa0633cb79ed77e39637ab6ab6a7509cd" exitCode=0
Sep 17 18:30:54.199630 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:54.199176    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qhfss" event={"ID":"1aea236a-8063-4c52-bda5-8c22fe1a166c","Type":"ContainerDied","Data":"09e65ce15c5351576f28426dae23628aa0633cb79ed77e39637ab6ab6a7509cd"}
Sep 17 18:30:55.206228 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:55.206173    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qhfss" event={"ID":"1aea236a-8063-4c52-bda5-8c22fe1a166c","Type":"ContainerStarted","Data":"830a1743f847a5601759127e40ef70469c32a010e7d3b9d071fea66f1cf6fcfa"}
Sep 17 18:30:55.206804 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:55.206256    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qhfss" event={"ID":"1aea236a-8063-4c52-bda5-8c22fe1a166c","Type":"ContainerStarted","Data":"ef5cf903cf377a38a10cac97809bb9b8c4187f7989c51d6dbce8f7c8be4cfd11"}
Sep 17 18:30:55.233986 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:55.233941    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qhfss" podStartSLOduration=2.6302973400000003 podStartE2EDuration="4.23392661s" totalImagesPullingTime="1.528000499s" totalInitContainerRuntime="75.628771ms" isStatefulPod=true podCreationTimestamp="2026-09-17 18:30:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:55.232474714 +0000 UTC m=+113.164676247" watchObservedRunningTime="2026-09-17 18:30:55.23392661 +0000 UTC m=+113.166128129"
Sep 17 18:30:56.155270 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.155240    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-fc4f47d85-v9gh9"]
Sep 17 18:30:56.177494 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.177472    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-fc4f47d85-v9gh9"]
Sep 17 18:30:56.177610 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.177575    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-fc4f47d85-v9gh9"
Sep 17 18:30:56.180234 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.180193    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 17 18:30:56.180415 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.180394    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 18:30:56.180471 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.180460    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-z6ss9\""
Sep 17 18:30:56.180519 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.180486    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 17 18:30:56.181273 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.181259    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 17 18:30:56.181343 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.181317    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-12pm042k1q35o\""
Sep 17 18:30:56.336725 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.336701    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ae28078b-ffbf-401b-b4ff-73b634a168c5-metrics-server-audit-profiles\") pod \"metrics-server-fc4f47d85-v9gh9\" (UID: \"ae28078b-ffbf-401b-b4ff-73b634a168c5\") " pod="openshift-monitoring/metrics-server-fc4f47d85-v9gh9"
Sep 17 18:30:56.337020 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.336731    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ae28078b-ffbf-401b-b4ff-73b634a168c5-client-ca-bundle\") pod \"metrics-server-fc4f47d85-v9gh9\" (UID: \"ae28078b-ffbf-401b-b4ff-73b634a168c5\") " pod="openshift-monitoring/metrics-server-fc4f47d85-v9gh9"
Sep 17 18:30:56.337020 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.336777    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skjcp\" (UniqueName: \"kubernetes.io/projected/ae28078b-ffbf-401b-b4ff-73b634a168c5-kube-api-access-skjcp\") pod \"metrics-server-fc4f47d85-v9gh9\" (UID: \"ae28078b-ffbf-401b-b4ff-73b634a168c5\") " pod="openshift-monitoring/metrics-server-fc4f47d85-v9gh9"
Sep 17 18:30:56.337020 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.336842    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ae28078b-ffbf-401b-b4ff-73b634a168c5-secret-metrics-server-client-certs\") pod \"metrics-server-fc4f47d85-v9gh9\" (UID: \"ae28078b-ffbf-401b-b4ff-73b634a168c5\") " pod="openshift-monitoring/metrics-server-fc4f47d85-v9gh9"
Sep 17 18:30:56.337020 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.336911    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ae28078b-ffbf-401b-b4ff-73b634a168c5-audit-log\") pod \"metrics-server-fc4f47d85-v9gh9\" (UID: \"ae28078b-ffbf-401b-b4ff-73b634a168c5\") " pod="openshift-monitoring/metrics-server-fc4f47d85-v9gh9"
Sep 17 18:30:56.337020 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.336944    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ae28078b-ffbf-401b-b4ff-73b634a168c5-secret-metrics-server-tls\") pod \"metrics-server-fc4f47d85-v9gh9\" (UID: \"ae28078b-ffbf-401b-b4ff-73b634a168c5\") " pod="openshift-monitoring/metrics-server-fc4f47d85-v9gh9"
Sep 17 18:30:56.337020 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.337007    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae28078b-ffbf-401b-b4ff-73b634a168c5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-fc4f47d85-v9gh9\" (UID: \"ae28078b-ffbf-401b-b4ff-73b634a168c5\") " pod="openshift-monitoring/metrics-server-fc4f47d85-v9gh9"
Sep 17 18:30:56.437597 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.437577    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ae28078b-ffbf-401b-b4ff-73b634a168c5-metrics-server-audit-profiles\") pod \"metrics-server-fc4f47d85-v9gh9\" (UID: \"ae28078b-ffbf-401b-b4ff-73b634a168c5\") " pod="openshift-monitoring/metrics-server-fc4f47d85-v9gh9"
Sep 17 18:30:56.437716 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.437603    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ae28078b-ffbf-401b-b4ff-73b634a168c5-client-ca-bundle\") pod \"metrics-server-fc4f47d85-v9gh9\" (UID: \"ae28078b-ffbf-401b-b4ff-73b634a168c5\") " pod="openshift-monitoring/metrics-server-fc4f47d85-v9gh9"
Sep 17 18:30:56.437716 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.437630    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skjcp\" (UniqueName: \"kubernetes.io/projected/ae28078b-ffbf-401b-b4ff-73b634a168c5-kube-api-access-skjcp\") pod \"metrics-server-fc4f47d85-v9gh9\" (UID: \"ae28078b-ffbf-401b-b4ff-73b634a168c5\") " pod="openshift-monitoring/metrics-server-fc4f47d85-v9gh9"
Sep 17 18:30:56.437716 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.437660    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ae28078b-ffbf-401b-b4ff-73b634a168c5-secret-metrics-server-client-certs\") pod \"metrics-server-fc4f47d85-v9gh9\" (UID: \"ae28078b-ffbf-401b-b4ff-73b634a168c5\") " pod="openshift-monitoring/metrics-server-fc4f47d85-v9gh9"
Sep 17 18:30:56.437716 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.437699    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ae28078b-ffbf-401b-b4ff-73b634a168c5-audit-log\") pod \"metrics-server-fc4f47d85-v9gh9\" (UID: \"ae28078b-ffbf-401b-b4ff-73b634a168c5\") " pod="openshift-monitoring/metrics-server-fc4f47d85-v9gh9"
Sep 17 18:30:56.437716 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.437717    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ae28078b-ffbf-401b-b4ff-73b634a168c5-secret-metrics-server-tls\") pod \"metrics-server-fc4f47d85-v9gh9\" (UID: \"ae28078b-ffbf-401b-b4ff-73b634a168c5\") " pod="openshift-monitoring/metrics-server-fc4f47d85-v9gh9"
Sep 17 18:30:56.437971 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.437744    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae28078b-ffbf-401b-b4ff-73b634a168c5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-fc4f47d85-v9gh9\" (UID: \"ae28078b-ffbf-401b-b4ff-73b634a168c5\") " pod="openshift-monitoring/metrics-server-fc4f47d85-v9gh9"
Sep 17 18:30:56.438138 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.438115    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ae28078b-ffbf-401b-b4ff-73b634a168c5-audit-log\") pod \"metrics-server-fc4f47d85-v9gh9\" (UID: \"ae28078b-ffbf-401b-b4ff-73b634a168c5\") " pod="openshift-monitoring/metrics-server-fc4f47d85-v9gh9"
Sep 17 18:30:56.438404 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.438376    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae28078b-ffbf-401b-b4ff-73b634a168c5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-fc4f47d85-v9gh9\" (UID: \"ae28078b-ffbf-401b-b4ff-73b634a168c5\") " pod="openshift-monitoring/metrics-server-fc4f47d85-v9gh9"
Sep 17 18:30:56.438559 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.438539    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ae28078b-ffbf-401b-b4ff-73b634a168c5-metrics-server-audit-profiles\") pod \"metrics-server-fc4f47d85-v9gh9\" (UID: \"ae28078b-ffbf-401b-b4ff-73b634a168c5\") " pod="openshift-monitoring/metrics-server-fc4f47d85-v9gh9"
Sep 17 18:30:56.440149 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.440127    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ae28078b-ffbf-401b-b4ff-73b634a168c5-client-ca-bundle\") pod \"metrics-server-fc4f47d85-v9gh9\" (UID: \"ae28078b-ffbf-401b-b4ff-73b634a168c5\") " pod="openshift-monitoring/metrics-server-fc4f47d85-v9gh9"
Sep 17 18:30:56.440262 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.440176    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ae28078b-ffbf-401b-b4ff-73b634a168c5-secret-metrics-server-client-certs\") pod \"metrics-server-fc4f47d85-v9gh9\" (UID: \"ae28078b-ffbf-401b-b4ff-73b634a168c5\") " pod="openshift-monitoring/metrics-server-fc4f47d85-v9gh9"
Sep 17 18:30:56.440339 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.440318    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ae28078b-ffbf-401b-b4ff-73b634a168c5-secret-metrics-server-tls\") pod \"metrics-server-fc4f47d85-v9gh9\" (UID: \"ae28078b-ffbf-401b-b4ff-73b634a168c5\") " pod="openshift-monitoring/metrics-server-fc4f47d85-v9gh9"
Sep 17 18:30:56.447387 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.447361    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skjcp\" (UniqueName: \"kubernetes.io/projected/ae28078b-ffbf-401b-b4ff-73b634a168c5-kube-api-access-skjcp\") pod \"metrics-server-fc4f47d85-v9gh9\" (UID: \"ae28078b-ffbf-401b-b4ff-73b634a168c5\") " pod="openshift-monitoring/metrics-server-fc4f47d85-v9gh9"
Sep 17 18:30:56.487065 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.487043    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-fc4f47d85-v9gh9"
Sep 17 18:30:56.790610 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:56.790540    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-fc4f47d85-v9gh9"]
Sep 17 18:30:56.793271 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:30:56.793243    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae28078b_ffbf_401b_b4ff_73b634a168c5.slice/crio-a30dad679954fe369140e8b330c78562a424d0aa245d0f7f2499ce3bfbfeae66 WatchSource:0}: Error finding container a30dad679954fe369140e8b330c78562a424d0aa245d0f7f2499ce3bfbfeae66: Status 404 returned error can't find the container with id a30dad679954fe369140e8b330c78562a424d0aa245d0f7f2499ce3bfbfeae66
Sep 17 18:30:57.214698 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:57.214597    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-fc4f47d85-v9gh9" event={"ID":"ae28078b-ffbf-401b-b4ff-73b634a168c5","Type":"ContainerStarted","Data":"a30dad679954fe369140e8b330c78562a424d0aa245d0f7f2499ce3bfbfeae66"}
Sep 17 18:30:59.222944 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:59.222909    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-fc4f47d85-v9gh9" event={"ID":"ae28078b-ffbf-401b-b4ff-73b634a168c5","Type":"ContainerStarted","Data":"1401db4806d5147408fab7ffe323ae7a2df9f75bc14e84a366021e47fb2e1fdb"}
Sep 17 18:30:59.250859 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:30:59.250815    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-fc4f47d85-v9gh9" podStartSLOduration=1.574077994 podStartE2EDuration="3.250799918s" totalImagesPullingTime="1.676721924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:59.248967061 +0000 UTC m=+117.181168594" watchObservedRunningTime="2026-09-17 18:30:59.250799918 +0000 UTC m=+117.183001436"
Sep 17 18:31:00.185980 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:00.185948    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-x9pxw"
Sep 17 18:31:03.373505 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:03.373474    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:31:12.452761 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:12.452718    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4-metrics-certs\") pod \"network-metrics-daemon-w67kg\" (UID: \"db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4\") " pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:31:12.455050 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:12.455022    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4-metrics-certs\") pod \"network-metrics-daemon-w67kg\" (UID: \"db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4\") " pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:31:12.737940 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:12.737876    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-twvp2\""
Sep 17 18:31:12.746059 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:12.746037    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w67kg"
Sep 17 18:31:13.017091 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:13.017063    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w67kg"]
Sep 17 18:31:13.019530 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:31:13.019498    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb00c9db_dbf0_4d5b_a9e9_7708ec49dcf4.slice/crio-e2dc72b6f7bcb27202b8eb5f719f13e7972c973cb044d917912199ddd514e6a5 WatchSource:0}: Error finding container e2dc72b6f7bcb27202b8eb5f719f13e7972c973cb044d917912199ddd514e6a5: Status 404 returned error can't find the container with id e2dc72b6f7bcb27202b8eb5f719f13e7972c973cb044d917912199ddd514e6a5
Sep 17 18:31:13.272389 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:13.272304    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w67kg" event={"ID":"db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4","Type":"ContainerStarted","Data":"e2dc72b6f7bcb27202b8eb5f719f13e7972c973cb044d917912199ddd514e6a5"}
Sep 17 18:31:14.276951 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:14.276921    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w67kg" event={"ID":"db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4","Type":"ContainerStarted","Data":"b7e234d8878838d3d8245fb265009ee242ff557b4e7b9de197d3d6b452f97d35"}
Sep 17 18:31:15.282726 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:15.282692    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w67kg" event={"ID":"db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4","Type":"ContainerStarted","Data":"7e41af0ef5bb31cc60947c65dcf6a62e02b8ea5ce9d8120aa5fb7e17b2983ac5"}
Sep 17 18:31:15.303936 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:15.303889    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-w67kg" podStartSLOduration=131.203538672 podStartE2EDuration="2m12.303869418s" totalImagesPullingTime="1.100330746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:31:15.303779877 +0000 UTC m=+133.235981397" watchObservedRunningTime="2026-09-17 18:31:15.303869418 +0000 UTC m=+133.236070937"
Sep 17 18:31:16.487148 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:16.487121    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-fc4f47d85-v9gh9"
Sep 17 18:31:16.487503 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:16.487217    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-fc4f47d85-v9gh9"
Sep 17 18:31:18.390538 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:18.390487    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8" podUID="8af0b1ec-975b-44eb-a6b6-fc55869a5585" containerName="registry" containerID="cri-o://d55c60b9ca71ddda7018b77f0ddfa78d25adb54070cc9b7a587166a38b9eec92" gracePeriod=30
Sep 17 18:31:18.674251 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:18.674227    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:31:18.696242 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:18.696127    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8wk9\" (UniqueName: \"kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-kube-api-access-x8wk9\") pod \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") "
Sep 17 18:31:18.696242 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:18.696177    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8af0b1ec-975b-44eb-a6b6-fc55869a5585-ca-trust-extracted\") pod \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") "
Sep 17 18:31:18.696242 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:18.696234    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8af0b1ec-975b-44eb-a6b6-fc55869a5585-trusted-ca\") pod \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") "
Sep 17 18:31:18.697110 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:18.696269    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8af0b1ec-975b-44eb-a6b6-fc55869a5585-image-registry-private-configuration\") pod \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") "
Sep 17 18:31:18.697110 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:18.696305    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-bound-sa-token\") pod \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") "
Sep 17 18:31:18.697110 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:18.696360    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8af0b1ec-975b-44eb-a6b6-fc55869a5585-installation-pull-secrets\") pod \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") "
Sep 17 18:31:18.697110 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:18.696396    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-certificates\") pod \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") "
Sep 17 18:31:18.697110 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:18.696530    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-tls\") pod \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\" (UID: \"8af0b1ec-975b-44eb-a6b6-fc55869a5585\") "
Sep 17 18:31:18.697110 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:18.696878    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8af0b1ec-975b-44eb-a6b6-fc55869a5585" (UID: "8af0b1ec-975b-44eb-a6b6-fc55869a5585"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:31:18.697428 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:18.697396    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8af0b1ec-975b-44eb-a6b6-fc55869a5585-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8af0b1ec-975b-44eb-a6b6-fc55869a5585" (UID: "8af0b1ec-975b-44eb-a6b6-fc55869a5585"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:31:18.699427 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:18.699395    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8af0b1ec-975b-44eb-a6b6-fc55869a5585" (UID: "8af0b1ec-975b-44eb-a6b6-fc55869a5585"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:31:18.699699 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:18.699647    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-kube-api-access-x8wk9" (OuterVolumeSpecName: "kube-api-access-x8wk9") pod "8af0b1ec-975b-44eb-a6b6-fc55869a5585" (UID: "8af0b1ec-975b-44eb-a6b6-fc55869a5585"). InnerVolumeSpecName "kube-api-access-x8wk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:31:18.699816 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:18.699696    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8af0b1ec-975b-44eb-a6b6-fc55869a5585-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8af0b1ec-975b-44eb-a6b6-fc55869a5585" (UID: "8af0b1ec-975b-44eb-a6b6-fc55869a5585"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:31:18.701014 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:18.700982    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8af0b1ec-975b-44eb-a6b6-fc55869a5585-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8af0b1ec-975b-44eb-a6b6-fc55869a5585" (UID: "8af0b1ec-975b-44eb-a6b6-fc55869a5585"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:31:18.701385 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:18.701360    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8af0b1ec-975b-44eb-a6b6-fc55869a5585" (UID: "8af0b1ec-975b-44eb-a6b6-fc55869a5585"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:31:18.705992 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:18.705963    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8af0b1ec-975b-44eb-a6b6-fc55869a5585-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8af0b1ec-975b-44eb-a6b6-fc55869a5585" (UID: "8af0b1ec-975b-44eb-a6b6-fc55869a5585"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:18.797973 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:18.797947    2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8af0b1ec-975b-44eb-a6b6-fc55869a5585-installation-pull-secrets\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:31:18.797973 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:18.797970    2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-certificates\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:31:18.798107 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:18.797986    2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-registry-tls\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:31:18.798107 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:18.797998    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8wk9\" (UniqueName: \"kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-kube-api-access-x8wk9\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:31:18.798107 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:18.798011    2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8af0b1ec-975b-44eb-a6b6-fc55869a5585-ca-trust-extracted\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:31:18.798107 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:18.798022    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8af0b1ec-975b-44eb-a6b6-fc55869a5585-trusted-ca\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:31:18.798107 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:18.798036    2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8af0b1ec-975b-44eb-a6b6-fc55869a5585-image-registry-private-configuration\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:31:18.798107 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:18.798050    2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8af0b1ec-975b-44eb-a6b6-fc55869a5585-bound-sa-token\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:31:19.297281 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:19.297248    2565 generic.go:356] "Generic (PLEG): container finished" podID="8af0b1ec-975b-44eb-a6b6-fc55869a5585" containerID="d55c60b9ca71ddda7018b77f0ddfa78d25adb54070cc9b7a587166a38b9eec92" exitCode=0
Sep 17 18:31:19.297411 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:19.297308    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8"
Sep 17 18:31:19.297411 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:19.297338    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8" event={"ID":"8af0b1ec-975b-44eb-a6b6-fc55869a5585","Type":"ContainerDied","Data":"d55c60b9ca71ddda7018b77f0ddfa78d25adb54070cc9b7a587166a38b9eec92"}
Sep 17 18:31:19.297411 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:19.297400    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f7b5f6fff-66wm8" event={"ID":"8af0b1ec-975b-44eb-a6b6-fc55869a5585","Type":"ContainerDied","Data":"0994928d245a307e2adc5d18b4e3882ef63c6fe7f5ad2c3b6ae9d3699c88f4f7"}
Sep 17 18:31:19.297544 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:19.297443    2565 scope.go:117] "RemoveContainer" containerID="d55c60b9ca71ddda7018b77f0ddfa78d25adb54070cc9b7a587166a38b9eec92"
Sep 17 18:31:19.319793 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:19.319769    2565 scope.go:117] "RemoveContainer" containerID="d55c60b9ca71ddda7018b77f0ddfa78d25adb54070cc9b7a587166a38b9eec92"
Sep 17 18:31:19.319908 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:19.319876    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f7b5f6fff-66wm8"]
Sep 17 18:31:19.320145 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:31:19.320122    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d55c60b9ca71ddda7018b77f0ddfa78d25adb54070cc9b7a587166a38b9eec92\": container with ID starting with d55c60b9ca71ddda7018b77f0ddfa78d25adb54070cc9b7a587166a38b9eec92 not found: ID does not exist" containerID="d55c60b9ca71ddda7018b77f0ddfa78d25adb54070cc9b7a587166a38b9eec92"
Sep 17 18:31:19.320245 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:19.320154    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d55c60b9ca71ddda7018b77f0ddfa78d25adb54070cc9b7a587166a38b9eec92"} err="failed to get container status \"d55c60b9ca71ddda7018b77f0ddfa78d25adb54070cc9b7a587166a38b9eec92\": rpc error: code = NotFound desc = could not find container \"d55c60b9ca71ddda7018b77f0ddfa78d25adb54070cc9b7a587166a38b9eec92\": container with ID starting with d55c60b9ca71ddda7018b77f0ddfa78d25adb54070cc9b7a587166a38b9eec92 not found: ID does not exist"
Sep 17 18:31:19.322182 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:19.322157    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5f7b5f6fff-66wm8"]
Sep 17 18:31:20.605887 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:20.605850    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8af0b1ec-975b-44eb-a6b6-fc55869a5585" path="/var/lib/kubelet/pods/8af0b1ec-975b-44eb-a6b6-fc55869a5585/volumes"
Sep 17 18:31:26.326554 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:26.326521    2565 generic.go:356] "Generic (PLEG): container finished" podID="a9151c33-3ccc-4061-b06b-9c3b8535fea4" containerID="189f8eeef2be9c59794cf6edf2fc500e73edbc19ce4cfb436c439234b87faec9" exitCode=0
Sep 17 18:31:26.327077 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:26.326582    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rh6tl" event={"ID":"a9151c33-3ccc-4061-b06b-9c3b8535fea4","Type":"ContainerDied","Data":"189f8eeef2be9c59794cf6edf2fc500e73edbc19ce4cfb436c439234b87faec9"}
Sep 17 18:31:26.327077 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:26.326949    2565 scope.go:117] "RemoveContainer" containerID="189f8eeef2be9c59794cf6edf2fc500e73edbc19ce4cfb436c439234b87faec9"
Sep 17 18:31:27.331793 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:27.331762    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-rh6tl" event={"ID":"a9151c33-3ccc-4061-b06b-9c3b8535fea4","Type":"ContainerStarted","Data":"6ec66ff78b2583683e950d3197900be16310ab68e8bda5c663294c50d0d0b869"}
Sep 17 18:31:30.343181 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:30.343148    2565 generic.go:356] "Generic (PLEG): container finished" podID="5e907c64-1699-4fa4-902b-b90ba5d7b8e6" containerID="51f8e3f55c39e583fcd0a11b0070bae1bd2bfa653932f871779619b55ac11bcd" exitCode=0
Sep 17 18:31:30.343509 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:30.343220    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-lnf27" event={"ID":"5e907c64-1699-4fa4-902b-b90ba5d7b8e6","Type":"ContainerDied","Data":"51f8e3f55c39e583fcd0a11b0070bae1bd2bfa653932f871779619b55ac11bcd"}
Sep 17 18:31:30.343587 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:30.343570    2565 scope.go:117] "RemoveContainer" containerID="51f8e3f55c39e583fcd0a11b0070bae1bd2bfa653932f871779619b55ac11bcd"
Sep 17 18:31:31.349471 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:31.349431    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-lnf27" event={"ID":"5e907c64-1699-4fa4-902b-b90ba5d7b8e6","Type":"ContainerStarted","Data":"61944a37bfb0d8ede49230bc9b768ba18ae19e39891ac05cb84109cda4dea5ba"}
Sep 17 18:31:34.024054 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:34.024024    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b4db969fb-dbb5b_a9afa96b-b391-4db4-9312-20c884e90da3/router/0.log"
Sep 17 18:31:34.052399 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:34.052372    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ltdzb_3158326a-19ea-455b-8737-8f01b5d51373/serve-healthcheck-canary/0.log"
Sep 17 18:31:36.492677 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:36.492646    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-fc4f47d85-v9gh9"
Sep 17 18:31:36.496521 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:31:36.496496    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-fc4f47d85-v9gh9"
Sep 17 18:32:16.141785 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.141744    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm"]
Sep 17 18:32:16.142310 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.142008    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8af0b1ec-975b-44eb-a6b6-fc55869a5585" containerName="registry"
Sep 17 18:32:16.142310 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.142019    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8af0b1ec-975b-44eb-a6b6-fc55869a5585" containerName="registry"
Sep 17 18:32:16.142310 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.142066    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8af0b1ec-975b-44eb-a6b6-fc55869a5585" containerName="registry"
Sep 17 18:32:16.146587 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.146567    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm"
Sep 17 18:32:16.150383 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.150351    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 18:32:16.150723 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.150700    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 18:32:16.150844 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.150774    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-7mprm\""
Sep 17 18:32:16.151255 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.151233    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 18:32:16.151372 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.151256    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 18:32:16.151439 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.151373    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 18:32:16.159250 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.159182    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 18:32:16.160414 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.160387    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm"]
Sep 17 18:32:16.289111 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.289076    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8905b946-3f63-4c73-b657-cc2c937b09e3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6dcb4d49c-6fkxm\" (UID: \"8905b946-3f63-4c73-b657-cc2c937b09e3\") " pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm"
Sep 17 18:32:16.289247 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.289115    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8905b946-3f63-4c73-b657-cc2c937b09e3-federate-client-tls\") pod \"telemeter-client-6dcb4d49c-6fkxm\" (UID: \"8905b946-3f63-4c73-b657-cc2c937b09e3\") " pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm"
Sep 17 18:32:16.289247 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.289136    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8905b946-3f63-4c73-b657-cc2c937b09e3-serving-certs-ca-bundle\") pod \"telemeter-client-6dcb4d49c-6fkxm\" (UID: \"8905b946-3f63-4c73-b657-cc2c937b09e3\") " pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm"
Sep 17 18:32:16.289247 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.289152    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8905b946-3f63-4c73-b657-cc2c937b09e3-secret-telemeter-client\") pod \"telemeter-client-6dcb4d49c-6fkxm\" (UID: \"8905b946-3f63-4c73-b657-cc2c937b09e3\") " pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm"
Sep 17 18:32:16.289390 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.289279    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8905b946-3f63-4c73-b657-cc2c937b09e3-telemeter-client-tls\") pod \"telemeter-client-6dcb4d49c-6fkxm\" (UID: \"8905b946-3f63-4c73-b657-cc2c937b09e3\") " pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm"
Sep 17 18:32:16.289390 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.289320    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8905b946-3f63-4c73-b657-cc2c937b09e3-metrics-client-ca\") pod \"telemeter-client-6dcb4d49c-6fkxm\" (UID: \"8905b946-3f63-4c73-b657-cc2c937b09e3\") " pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm"
Sep 17 18:32:16.289390 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.289340    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8905b946-3f63-4c73-b657-cc2c937b09e3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6dcb4d49c-6fkxm\" (UID: \"8905b946-3f63-4c73-b657-cc2c937b09e3\") " pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm"
Sep 17 18:32:16.289390 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.289363    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9qwm\" (UniqueName: \"kubernetes.io/projected/8905b946-3f63-4c73-b657-cc2c937b09e3-kube-api-access-r9qwm\") pod \"telemeter-client-6dcb4d49c-6fkxm\" (UID: \"8905b946-3f63-4c73-b657-cc2c937b09e3\") " pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm"
Sep 17 18:32:16.390409 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.390367    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8905b946-3f63-4c73-b657-cc2c937b09e3-serving-certs-ca-bundle\") pod \"telemeter-client-6dcb4d49c-6fkxm\" (UID: \"8905b946-3f63-4c73-b657-cc2c937b09e3\") " pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm"
Sep 17 18:32:16.390528 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.390413    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8905b946-3f63-4c73-b657-cc2c937b09e3-secret-telemeter-client\") pod \"telemeter-client-6dcb4d49c-6fkxm\" (UID: \"8905b946-3f63-4c73-b657-cc2c937b09e3\") " pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm"
Sep 17 18:32:16.390528 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.390449    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8905b946-3f63-4c73-b657-cc2c937b09e3-telemeter-client-tls\") pod \"telemeter-client-6dcb4d49c-6fkxm\" (UID: \"8905b946-3f63-4c73-b657-cc2c937b09e3\") " pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm"
Sep 17 18:32:16.390528 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.390479    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8905b946-3f63-4c73-b657-cc2c937b09e3-metrics-client-ca\") pod \"telemeter-client-6dcb4d49c-6fkxm\" (UID: \"8905b946-3f63-4c73-b657-cc2c937b09e3\") " pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm"
Sep 17 18:32:16.390528 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.390506    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8905b946-3f63-4c73-b657-cc2c937b09e3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6dcb4d49c-6fkxm\" (UID: \"8905b946-3f63-4c73-b657-cc2c937b09e3\") " pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm"
Sep 17 18:32:16.390724 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.390532    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9qwm\" (UniqueName: \"kubernetes.io/projected/8905b946-3f63-4c73-b657-cc2c937b09e3-kube-api-access-r9qwm\") pod \"telemeter-client-6dcb4d49c-6fkxm\" (UID: \"8905b946-3f63-4c73-b657-cc2c937b09e3\") " pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm"
Sep 17 18:32:16.390724 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.390663    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8905b946-3f63-4c73-b657-cc2c937b09e3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6dcb4d49c-6fkxm\" (UID: \"8905b946-3f63-4c73-b657-cc2c937b09e3\") " pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm"
Sep 17 18:32:16.390724 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.390707    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8905b946-3f63-4c73-b657-cc2c937b09e3-federate-client-tls\") pod \"telemeter-client-6dcb4d49c-6fkxm\" (UID: \"8905b946-3f63-4c73-b657-cc2c937b09e3\") " pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm"
Sep 17 18:32:16.391314 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.391285    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8905b946-3f63-4c73-b657-cc2c937b09e3-serving-certs-ca-bundle\") pod \"telemeter-client-6dcb4d49c-6fkxm\" (UID: \"8905b946-3f63-4c73-b657-cc2c937b09e3\") " pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm"
Sep 17 18:32:16.391439 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.391345    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8905b946-3f63-4c73-b657-cc2c937b09e3-metrics-client-ca\") pod \"telemeter-client-6dcb4d49c-6fkxm\" (UID: \"8905b946-3f63-4c73-b657-cc2c937b09e3\") " pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm"
Sep 17 18:32:16.391676 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.391654    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8905b946-3f63-4c73-b657-cc2c937b09e3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6dcb4d49c-6fkxm\" (UID: \"8905b946-3f63-4c73-b657-cc2c937b09e3\") " pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm"
Sep 17 18:32:16.394162 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.394094    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8905b946-3f63-4c73-b657-cc2c937b09e3-secret-telemeter-client\") pod \"telemeter-client-6dcb4d49c-6fkxm\" (UID: \"8905b946-3f63-4c73-b657-cc2c937b09e3\") " pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm"
Sep 17 18:32:16.394162 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.394135    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8905b946-3f63-4c73-b657-cc2c937b09e3-telemeter-client-tls\") pod \"telemeter-client-6dcb4d49c-6fkxm\" (UID: \"8905b946-3f63-4c73-b657-cc2c937b09e3\") " pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm"
Sep 17 18:32:16.394162 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.394154    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8905b946-3f63-4c73-b657-cc2c937b09e3-federate-client-tls\") pod \"telemeter-client-6dcb4d49c-6fkxm\" (UID: \"8905b946-3f63-4c73-b657-cc2c937b09e3\") " pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm"
Sep 17 18:32:16.394347 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.394164    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8905b946-3f63-4c73-b657-cc2c937b09e3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6dcb4d49c-6fkxm\" (UID: \"8905b946-3f63-4c73-b657-cc2c937b09e3\") " pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm"
Sep 17 18:32:16.400468 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.400446    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9qwm\" (UniqueName: \"kubernetes.io/projected/8905b946-3f63-4c73-b657-cc2c937b09e3-kube-api-access-r9qwm\") pod \"telemeter-client-6dcb4d49c-6fkxm\" (UID: \"8905b946-3f63-4c73-b657-cc2c937b09e3\") " pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm"
Sep 17 18:32:16.459189 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.459166    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm"
Sep 17 18:32:16.761167 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:16.761144    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm"]
Sep 17 18:32:16.763949 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:32:16.763920    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8905b946_3f63_4c73_b657_cc2c937b09e3.slice/crio-9929c3307817e020cf4ee6e6874707651cf52ae386847302b64ccf26eee41986 WatchSource:0}: Error finding container 9929c3307817e020cf4ee6e6874707651cf52ae386847302b64ccf26eee41986: Status 404 returned error can't find the container with id 9929c3307817e020cf4ee6e6874707651cf52ae386847302b64ccf26eee41986
Sep 17 18:32:17.523901 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:17.523863    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm" event={"ID":"8905b946-3f63-4c73-b657-cc2c937b09e3","Type":"ContainerStarted","Data":"9929c3307817e020cf4ee6e6874707651cf52ae386847302b64ccf26eee41986"}
Sep 17 18:32:19.533830 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:19.533794    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm" event={"ID":"8905b946-3f63-4c73-b657-cc2c937b09e3","Type":"ContainerStarted","Data":"bccccbb4ff264a9a30ecf7238bbe87ad602fdfa6b45507d4c9029f09c78d4b8c"}
Sep 17 18:32:20.540178 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:20.540139    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm" event={"ID":"8905b946-3f63-4c73-b657-cc2c937b09e3","Type":"ContainerStarted","Data":"83028689428be99f81d089248eaa92c0774a88b2d7127b9aaceb2535809748d7"}
Sep 17 18:32:20.540677 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:20.540247    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm" event={"ID":"8905b946-3f63-4c73-b657-cc2c937b09e3","Type":"ContainerStarted","Data":"e5e51afc54f5c85371455ee75bf2689105d640c2a62814f36f87c96beb5da0ee"}
Sep 17 18:32:20.570443 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:32:20.570382    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6dcb4d49c-6fkxm" podStartSLOduration=1.694707607 podStartE2EDuration="4.570361768s" totalImagesPullingTime="2.875654161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:32:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:32:20.568447239 +0000 UTC m=+198.500648758" watchObservedRunningTime="2026-09-17 18:32:20.570361768 +0000 UTC m=+198.502563287"
Sep 17 18:33:26.788080 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:26.787982    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45"]
Sep 17 18:33:26.791502 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:26.791476    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45"
Sep 17 18:33:26.796334 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:26.796308    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 18:33:26.796448 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:26.796381    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-sd82r\""
Sep 17 18:33:26.797076 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:26.797042    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 18:33:26.812081 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:26.812056    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45"]
Sep 17 18:33:26.850974 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:26.850950    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pczcm\" (UniqueName: \"kubernetes.io/projected/3c328213-86fb-4d1d-b4a5-68c4e860e560-kube-api-access-pczcm\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45\" (UID: \"3c328213-86fb-4d1d-b4a5-68c4e860e560\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45"
Sep 17 18:33:26.851066 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:26.850986    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c328213-86fb-4d1d-b4a5-68c4e860e560-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45\" (UID: \"3c328213-86fb-4d1d-b4a5-68c4e860e560\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45"
Sep 17 18:33:26.851066 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:26.851013    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c328213-86fb-4d1d-b4a5-68c4e860e560-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45\" (UID: \"3c328213-86fb-4d1d-b4a5-68c4e860e560\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45"
Sep 17 18:33:26.951417 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:26.951396    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pczcm\" (UniqueName: \"kubernetes.io/projected/3c328213-86fb-4d1d-b4a5-68c4e860e560-kube-api-access-pczcm\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45\" (UID: \"3c328213-86fb-4d1d-b4a5-68c4e860e560\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45"
Sep 17 18:33:26.951517 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:26.951433    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c328213-86fb-4d1d-b4a5-68c4e860e560-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45\" (UID: \"3c328213-86fb-4d1d-b4a5-68c4e860e560\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45"
Sep 17 18:33:26.951517 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:26.951457    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c328213-86fb-4d1d-b4a5-68c4e860e560-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45\" (UID: \"3c328213-86fb-4d1d-b4a5-68c4e860e560\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45"
Sep 17 18:33:26.951767 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:26.951749    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c328213-86fb-4d1d-b4a5-68c4e860e560-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45\" (UID: \"3c328213-86fb-4d1d-b4a5-68c4e860e560\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45"
Sep 17 18:33:26.951838 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:26.951812    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c328213-86fb-4d1d-b4a5-68c4e860e560-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45\" (UID: \"3c328213-86fb-4d1d-b4a5-68c4e860e560\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45"
Sep 17 18:33:26.962345 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:26.962318    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pczcm\" (UniqueName: \"kubernetes.io/projected/3c328213-86fb-4d1d-b4a5-68c4e860e560-kube-api-access-pczcm\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45\" (UID: \"3c328213-86fb-4d1d-b4a5-68c4e860e560\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45"
Sep 17 18:33:27.100730 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:27.100683    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45"
Sep 17 18:33:27.376772 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:27.376700    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45"]
Sep 17 18:33:27.383552 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:33:27.383519    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c328213_86fb_4d1d_b4a5_68c4e860e560.slice/crio-576e8f5919d6e34806e71e0cec8c230aea8ca6468c177200a147334ead2d996f WatchSource:0}: Error finding container 576e8f5919d6e34806e71e0cec8c230aea8ca6468c177200a147334ead2d996f: Status 404 returned error can't find the container with id 576e8f5919d6e34806e71e0cec8c230aea8ca6468c177200a147334ead2d996f
Sep 17 18:33:27.778548 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:27.778516    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45" event={"ID":"3c328213-86fb-4d1d-b4a5-68c4e860e560","Type":"ContainerStarted","Data":"576e8f5919d6e34806e71e0cec8c230aea8ca6468c177200a147334ead2d996f"}
Sep 17 18:33:34.810374 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:34.810338    2565 generic.go:356] "Generic (PLEG): container finished" podID="3c328213-86fb-4d1d-b4a5-68c4e860e560" containerID="724023b0cf7205c5c185632721a28ed3af299e44bbfa2de92e30835913f989de" exitCode=0
Sep 17 18:33:34.810740 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:34.810426    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45" event={"ID":"3c328213-86fb-4d1d-b4a5-68c4e860e560","Type":"ContainerDied","Data":"724023b0cf7205c5c185632721a28ed3af299e44bbfa2de92e30835913f989de"}
Sep 17 18:33:36.820597 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:36.820567    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45" event={"ID":"3c328213-86fb-4d1d-b4a5-68c4e860e560","Type":"ContainerStarted","Data":"493d4df2d5e8c1130037815ce1923183ec0af7fbc4fb26b68663dd64941f534a"}
Sep 17 18:33:37.826482 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:37.826439    2565 generic.go:356] "Generic (PLEG): container finished" podID="3c328213-86fb-4d1d-b4a5-68c4e860e560" containerID="493d4df2d5e8c1130037815ce1923183ec0af7fbc4fb26b68663dd64941f534a" exitCode=0
Sep 17 18:33:37.826902 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:37.826532    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45" event={"ID":"3c328213-86fb-4d1d-b4a5-68c4e860e560","Type":"ContainerDied","Data":"493d4df2d5e8c1130037815ce1923183ec0af7fbc4fb26b68663dd64941f534a"}
Sep 17 18:33:45.858321 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:45.858279    2565 generic.go:356] "Generic (PLEG): container finished" podID="3c328213-86fb-4d1d-b4a5-68c4e860e560" containerID="510fad4cad9164557e5d33db5cb39121128873a13614d2f1d2d4095ea57a4ad0" exitCode=0
Sep 17 18:33:45.858684 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:45.858352    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45" event={"ID":"3c328213-86fb-4d1d-b4a5-68c4e860e560","Type":"ContainerDied","Data":"510fad4cad9164557e5d33db5cb39121128873a13614d2f1d2d4095ea57a4ad0"}
Sep 17 18:33:46.991076 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:46.991052    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45"
Sep 17 18:33:47.101810 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:47.101783    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pczcm\" (UniqueName: \"kubernetes.io/projected/3c328213-86fb-4d1d-b4a5-68c4e860e560-kube-api-access-pczcm\") pod \"3c328213-86fb-4d1d-b4a5-68c4e860e560\" (UID: \"3c328213-86fb-4d1d-b4a5-68c4e860e560\") "
Sep 17 18:33:47.101934 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:47.101840    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c328213-86fb-4d1d-b4a5-68c4e860e560-util\") pod \"3c328213-86fb-4d1d-b4a5-68c4e860e560\" (UID: \"3c328213-86fb-4d1d-b4a5-68c4e860e560\") "
Sep 17 18:33:47.102035 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:47.102015    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c328213-86fb-4d1d-b4a5-68c4e860e560-bundle\") pod \"3c328213-86fb-4d1d-b4a5-68c4e860e560\" (UID: \"3c328213-86fb-4d1d-b4a5-68c4e860e560\") "
Sep 17 18:33:47.102613 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:47.102586    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c328213-86fb-4d1d-b4a5-68c4e860e560-bundle" (OuterVolumeSpecName: "bundle") pod "3c328213-86fb-4d1d-b4a5-68c4e860e560" (UID: "3c328213-86fb-4d1d-b4a5-68c4e860e560"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:33:47.103926 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:47.103907    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c328213-86fb-4d1d-b4a5-68c4e860e560-kube-api-access-pczcm" (OuterVolumeSpecName: "kube-api-access-pczcm") pod "3c328213-86fb-4d1d-b4a5-68c4e860e560" (UID: "3c328213-86fb-4d1d-b4a5-68c4e860e560"). InnerVolumeSpecName "kube-api-access-pczcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:33:47.109186 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:47.109161    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c328213-86fb-4d1d-b4a5-68c4e860e560-util" (OuterVolumeSpecName: "util") pod "3c328213-86fb-4d1d-b4a5-68c4e860e560" (UID: "3c328213-86fb-4d1d-b4a5-68c4e860e560"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:33:47.202986 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:47.202955    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c328213-86fb-4d1d-b4a5-68c4e860e560-util\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:33:47.202986 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:47.202985    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c328213-86fb-4d1d-b4a5-68c4e860e560-bundle\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:33:47.202986 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:47.202994    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pczcm\" (UniqueName: \"kubernetes.io/projected/3c328213-86fb-4d1d-b4a5-68c4e860e560-kube-api-access-pczcm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:33:47.866840 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:47.866815    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45"
Sep 17 18:33:47.866958 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:47.866810    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2lt45" event={"ID":"3c328213-86fb-4d1d-b4a5-68c4e860e560","Type":"ContainerDied","Data":"576e8f5919d6e34806e71e0cec8c230aea8ca6468c177200a147334ead2d996f"}
Sep 17 18:33:47.866958 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:47.866929    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="576e8f5919d6e34806e71e0cec8c230aea8ca6468c177200a147334ead2d996f"
Sep 17 18:33:56.064979 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:56.064941    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lhg68"]
Sep 17 18:33:56.065389 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:56.065229    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c328213-86fb-4d1d-b4a5-68c4e860e560" containerName="util"
Sep 17 18:33:56.065389 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:56.065244    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c328213-86fb-4d1d-b4a5-68c4e860e560" containerName="util"
Sep 17 18:33:56.065389 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:56.065252    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c328213-86fb-4d1d-b4a5-68c4e860e560" containerName="pull"
Sep 17 18:33:56.065389 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:56.065259    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c328213-86fb-4d1d-b4a5-68c4e860e560" containerName="pull"
Sep 17 18:33:56.065389 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:56.065270    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c328213-86fb-4d1d-b4a5-68c4e860e560" containerName="extract"
Sep 17 18:33:56.065389 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:56.065277    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c328213-86fb-4d1d-b4a5-68c4e860e560" containerName="extract"
Sep 17 18:33:56.065389 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:56.065322    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c328213-86fb-4d1d-b4a5-68c4e860e560" containerName="extract"
Sep 17 18:33:56.068097 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:56.068080    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lhg68"
Sep 17 18:33:56.071157 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:56.071134    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 18:33:56.071274 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:56.071185    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 18:33:56.071322 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:56.071290    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-tt2f9\""
Sep 17 18:33:56.071361 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:56.071329    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 18:33:56.083611 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:56.083586    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lhg68"]
Sep 17 18:33:56.164914 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:56.164892    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lpt2\" (UniqueName: \"kubernetes.io/projected/8876b6ad-9ede-4b24-892e-e3210f9317da-kube-api-access-4lpt2\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-lhg68\" (UID: \"8876b6ad-9ede-4b24-892e-e3210f9317da\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lhg68"
Sep 17 18:33:56.165018 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:56.164934    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/8876b6ad-9ede-4b24-892e-e3210f9317da-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-lhg68\" (UID: \"8876b6ad-9ede-4b24-892e-e3210f9317da\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lhg68"
Sep 17 18:33:56.265716 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:56.265692    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lpt2\" (UniqueName: \"kubernetes.io/projected/8876b6ad-9ede-4b24-892e-e3210f9317da-kube-api-access-4lpt2\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-lhg68\" (UID: \"8876b6ad-9ede-4b24-892e-e3210f9317da\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lhg68"
Sep 17 18:33:56.265791 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:56.265731    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/8876b6ad-9ede-4b24-892e-e3210f9317da-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-lhg68\" (UID: \"8876b6ad-9ede-4b24-892e-e3210f9317da\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lhg68"
Sep 17 18:33:56.267882 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:56.267864    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/8876b6ad-9ede-4b24-892e-e3210f9317da-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-lhg68\" (UID: \"8876b6ad-9ede-4b24-892e-e3210f9317da\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lhg68"
Sep 17 18:33:56.277905 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:56.277875    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lpt2\" (UniqueName: \"kubernetes.io/projected/8876b6ad-9ede-4b24-892e-e3210f9317da-kube-api-access-4lpt2\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-lhg68\" (UID: \"8876b6ad-9ede-4b24-892e-e3210f9317da\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lhg68"
Sep 17 18:33:56.378023 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:56.377962    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lhg68"
Sep 17 18:33:56.734278 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:56.734241    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lhg68"]
Sep 17 18:33:56.746906 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:33:56.746877    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8876b6ad_9ede_4b24_892e_e3210f9317da.slice/crio-f3b81b5ab1c838847d6073b4ec5ca3871146bf9c8d9c81de124429e679a52b7a WatchSource:0}: Error finding container f3b81b5ab1c838847d6073b4ec5ca3871146bf9c8d9c81de124429e679a52b7a: Status 404 returned error can't find the container with id f3b81b5ab1c838847d6073b4ec5ca3871146bf9c8d9c81de124429e679a52b7a
Sep 17 18:33:56.904212 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:33:56.904179    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lhg68" event={"ID":"8876b6ad-9ede-4b24-892e-e3210f9317da","Type":"ContainerStarted","Data":"f3b81b5ab1c838847d6073b4ec5ca3871146bf9c8d9c81de124429e679a52b7a"}
Sep 17 18:34:00.779538 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:00.779504    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-dwnbk"]
Sep 17 18:34:00.782747 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:00.782730    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-dwnbk"
Sep 17 18:34:00.785148 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:00.785120    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 17 18:34:00.785291 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:00.785253    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-gnm97\""
Sep 17 18:34:00.785291 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:00.785253    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 18:34:00.801543 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:00.801518    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-dwnbk"]
Sep 17 18:34:00.900817 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:00.900716    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/26f55e2f-455c-4cb7-a14a-ed14ef9761d7-certificates\") pod \"keda-operator-579c8958f7-dwnbk\" (UID: \"26f55e2f-455c-4cb7-a14a-ed14ef9761d7\") " pod="openshift-keda/keda-operator-579c8958f7-dwnbk"
Sep 17 18:34:00.901012 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:00.900868    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/26f55e2f-455c-4cb7-a14a-ed14ef9761d7-cabundle0\") pod \"keda-operator-579c8958f7-dwnbk\" (UID: \"26f55e2f-455c-4cb7-a14a-ed14ef9761d7\") " pod="openshift-keda/keda-operator-579c8958f7-dwnbk"
Sep 17 18:34:00.901012 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:00.900900    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd2ls\" (UniqueName: \"kubernetes.io/projected/26f55e2f-455c-4cb7-a14a-ed14ef9761d7-kube-api-access-zd2ls\") pod \"keda-operator-579c8958f7-dwnbk\" (UID: \"26f55e2f-455c-4cb7-a14a-ed14ef9761d7\") " pod="openshift-keda/keda-operator-579c8958f7-dwnbk"
Sep 17 18:34:00.923952 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:00.923926    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lhg68" event={"ID":"8876b6ad-9ede-4b24-892e-e3210f9317da","Type":"ContainerStarted","Data":"80326a04a0423306619ebf2e71deee9e6fd76c0d2155eaf1df7596c506ce0764"}
Sep 17 18:34:00.924073 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:00.924059    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lhg68"
Sep 17 18:34:00.956865 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:00.956820    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lhg68" podStartSLOduration=1.8152031690000001 podStartE2EDuration="4.956806042s" totalImagesPullingTime="3.141602873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:34:00.955540802 +0000 UTC m=+298.887742322" watchObservedRunningTime="2026-09-17 18:34:00.956806042 +0000 UTC m=+298.889007561"
Sep 17 18:34:01.001979 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.001953    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/26f55e2f-455c-4cb7-a14a-ed14ef9761d7-cabundle0\") pod \"keda-operator-579c8958f7-dwnbk\" (UID: \"26f55e2f-455c-4cb7-a14a-ed14ef9761d7\") " pod="openshift-keda/keda-operator-579c8958f7-dwnbk"
Sep 17 18:34:01.001979 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.001979    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd2ls\" (UniqueName: \"kubernetes.io/projected/26f55e2f-455c-4cb7-a14a-ed14ef9761d7-kube-api-access-zd2ls\") pod \"keda-operator-579c8958f7-dwnbk\" (UID: \"26f55e2f-455c-4cb7-a14a-ed14ef9761d7\") " pod="openshift-keda/keda-operator-579c8958f7-dwnbk"
Sep 17 18:34:01.002146 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.002040    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/26f55e2f-455c-4cb7-a14a-ed14ef9761d7-certificates\") pod \"keda-operator-579c8958f7-dwnbk\" (UID: \"26f55e2f-455c-4cb7-a14a-ed14ef9761d7\") " pod="openshift-keda/keda-operator-579c8958f7-dwnbk"
Sep 17 18:34:01.002236 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:34:01.002150    2565 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Sep 17 18:34:01.002236 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:34:01.002169    2565 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-dwnbk: secret "keda-operator-certs" not found
Sep 17 18:34:01.002349 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:34:01.002252    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26f55e2f-455c-4cb7-a14a-ed14ef9761d7-certificates podName:26f55e2f-455c-4cb7-a14a-ed14ef9761d7 nodeName:}" failed. No retries permitted until 2026-09-17 18:34:01.502231608 +0000 UTC m=+299.434433125 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/26f55e2f-455c-4cb7-a14a-ed14ef9761d7-certificates") pod "keda-operator-579c8958f7-dwnbk" (UID: "26f55e2f-455c-4cb7-a14a-ed14ef9761d7") : secret "keda-operator-certs" not found
Sep 17 18:34:01.002678 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.002659    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/26f55e2f-455c-4cb7-a14a-ed14ef9761d7-cabundle0\") pod \"keda-operator-579c8958f7-dwnbk\" (UID: \"26f55e2f-455c-4cb7-a14a-ed14ef9761d7\") " pod="openshift-keda/keda-operator-579c8958f7-dwnbk"
Sep 17 18:34:01.020511 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.020490    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd2ls\" (UniqueName: \"kubernetes.io/projected/26f55e2f-455c-4cb7-a14a-ed14ef9761d7-kube-api-access-zd2ls\") pod \"keda-operator-579c8958f7-dwnbk\" (UID: \"26f55e2f-455c-4cb7-a14a-ed14ef9761d7\") " pod="openshift-keda/keda-operator-579c8958f7-dwnbk"
Sep 17 18:34:01.067361 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.067304    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-q7srz"]
Sep 17 18:34:01.070391 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.070376    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-q7srz"
Sep 17 18:34:01.075898 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.075870    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 17 18:34:01.108957 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.108935    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-q7srz"]
Sep 17 18:34:01.203856 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.203830    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/84a079be-626c-46b5-8eeb-1794b8010ce4-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-q7srz\" (UID: \"84a079be-626c-46b5-8eeb-1794b8010ce4\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-q7srz"
Sep 17 18:34:01.203971 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.203861    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgf86\" (UniqueName: \"kubernetes.io/projected/84a079be-626c-46b5-8eeb-1794b8010ce4-kube-api-access-mgf86\") pod \"keda-metrics-apiserver-77857b7b98-q7srz\" (UID: \"84a079be-626c-46b5-8eeb-1794b8010ce4\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-q7srz"
Sep 17 18:34:01.203971 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.203950    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/84a079be-626c-46b5-8eeb-1794b8010ce4-certificates\") pod \"keda-metrics-apiserver-77857b7b98-q7srz\" (UID: \"84a079be-626c-46b5-8eeb-1794b8010ce4\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-q7srz"
Sep 17 18:34:01.305169 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.305142    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/84a079be-626c-46b5-8eeb-1794b8010ce4-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-q7srz\" (UID: \"84a079be-626c-46b5-8eeb-1794b8010ce4\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-q7srz"
Sep 17 18:34:01.305331 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.305178    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgf86\" (UniqueName: \"kubernetes.io/projected/84a079be-626c-46b5-8eeb-1794b8010ce4-kube-api-access-mgf86\") pod \"keda-metrics-apiserver-77857b7b98-q7srz\" (UID: \"84a079be-626c-46b5-8eeb-1794b8010ce4\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-q7srz"
Sep 17 18:34:01.305331 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.305254    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/84a079be-626c-46b5-8eeb-1794b8010ce4-certificates\") pod \"keda-metrics-apiserver-77857b7b98-q7srz\" (UID: \"84a079be-626c-46b5-8eeb-1794b8010ce4\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-q7srz"
Sep 17 18:34:01.305502 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.305484    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/84a079be-626c-46b5-8eeb-1794b8010ce4-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-q7srz\" (UID: \"84a079be-626c-46b5-8eeb-1794b8010ce4\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-q7srz"
Sep 17 18:34:01.307663 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.307640    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/84a079be-626c-46b5-8eeb-1794b8010ce4-certificates\") pod \"keda-metrics-apiserver-77857b7b98-q7srz\" (UID: \"84a079be-626c-46b5-8eeb-1794b8010ce4\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-q7srz"
Sep 17 18:34:01.322151 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.322096    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgf86\" (UniqueName: \"kubernetes.io/projected/84a079be-626c-46b5-8eeb-1794b8010ce4-kube-api-access-mgf86\") pod \"keda-metrics-apiserver-77857b7b98-q7srz\" (UID: \"84a079be-626c-46b5-8eeb-1794b8010ce4\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-q7srz"
Sep 17 18:34:01.344521 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.344498    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-6gmg2"]
Sep 17 18:34:01.347632 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.347618    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-6gmg2"
Sep 17 18:34:01.351026 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.351006    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 18:34:01.366604 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.366578    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-6gmg2"]
Sep 17 18:34:01.386607 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.386582    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-q7srz"
Sep 17 18:34:01.406543 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.406518    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9eec280d-0780-45ef-832a-091c401c234c-certificates\") pod \"keda-admission-6c4d864b86-6gmg2\" (UID: \"9eec280d-0780-45ef-832a-091c401c234c\") " pod="openshift-keda/keda-admission-6c4d864b86-6gmg2"
Sep 17 18:34:01.406649 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.406572    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z48jp\" (UniqueName: \"kubernetes.io/projected/9eec280d-0780-45ef-832a-091c401c234c-kube-api-access-z48jp\") pod \"keda-admission-6c4d864b86-6gmg2\" (UID: \"9eec280d-0780-45ef-832a-091c401c234c\") " pod="openshift-keda/keda-admission-6c4d864b86-6gmg2"
Sep 17 18:34:01.507151 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.507117    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/26f55e2f-455c-4cb7-a14a-ed14ef9761d7-certificates\") pod \"keda-operator-579c8958f7-dwnbk\" (UID: \"26f55e2f-455c-4cb7-a14a-ed14ef9761d7\") " pod="openshift-keda/keda-operator-579c8958f7-dwnbk"
Sep 17 18:34:01.507151 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.507156    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9eec280d-0780-45ef-832a-091c401c234c-certificates\") pod \"keda-admission-6c4d864b86-6gmg2\" (UID: \"9eec280d-0780-45ef-832a-091c401c234c\") " pod="openshift-keda/keda-admission-6c4d864b86-6gmg2"
Sep 17 18:34:01.507357 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:34:01.507278    2565 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 17 18:34:01.507357 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.507279    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z48jp\" (UniqueName: \"kubernetes.io/projected/9eec280d-0780-45ef-832a-091c401c234c-kube-api-access-z48jp\") pod \"keda-admission-6c4d864b86-6gmg2\" (UID: \"9eec280d-0780-45ef-832a-091c401c234c\") " pod="openshift-keda/keda-admission-6c4d864b86-6gmg2"
Sep 17 18:34:01.507357 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:34:01.507300    2565 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-6gmg2: secret "keda-admission-webhooks-certs" not found
Sep 17 18:34:01.507482 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:34:01.507364    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9eec280d-0780-45ef-832a-091c401c234c-certificates podName:9eec280d-0780-45ef-832a-091c401c234c nodeName:}" failed. No retries permitted until 2026-09-17 18:34:02.007343723 +0000 UTC m=+299.939545242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/9eec280d-0780-45ef-832a-091c401c234c-certificates") pod "keda-admission-6c4d864b86-6gmg2" (UID: "9eec280d-0780-45ef-832a-091c401c234c") : secret "keda-admission-webhooks-certs" not found
Sep 17 18:34:01.509495 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.509474    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/26f55e2f-455c-4cb7-a14a-ed14ef9761d7-certificates\") pod \"keda-operator-579c8958f7-dwnbk\" (UID: \"26f55e2f-455c-4cb7-a14a-ed14ef9761d7\") " pod="openshift-keda/keda-operator-579c8958f7-dwnbk"
Sep 17 18:34:01.524181 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.524157    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z48jp\" (UniqueName: \"kubernetes.io/projected/9eec280d-0780-45ef-832a-091c401c234c-kube-api-access-z48jp\") pod \"keda-admission-6c4d864b86-6gmg2\" (UID: \"9eec280d-0780-45ef-832a-091c401c234c\") " pod="openshift-keda/keda-admission-6c4d864b86-6gmg2"
Sep 17 18:34:01.681775 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.681745    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-q7srz"]
Sep 17 18:34:01.685746 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:34:01.685707    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a079be_626c_46b5_8eeb_1794b8010ce4.slice/crio-7fe3f33b41e7f12eae9794cb351a114277e8ed6b8adafdface41dc1af45c42ce WatchSource:0}: Error finding container 7fe3f33b41e7f12eae9794cb351a114277e8ed6b8adafdface41dc1af45c42ce: Status 404 returned error can't find the container with id 7fe3f33b41e7f12eae9794cb351a114277e8ed6b8adafdface41dc1af45c42ce
Sep 17 18:34:01.692641 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.692617    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-dwnbk"
Sep 17 18:34:01.929493 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.929458    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-q7srz" event={"ID":"84a079be-626c-46b5-8eeb-1794b8010ce4","Type":"ContainerStarted","Data":"7fe3f33b41e7f12eae9794cb351a114277e8ed6b8adafdface41dc1af45c42ce"}
Sep 17 18:34:01.967558 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:01.967486    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-dwnbk"]
Sep 17 18:34:01.970259 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:34:01.970233    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f55e2f_455c_4cb7_a14a_ed14ef9761d7.slice/crio-d54dd19c36a53ea351907e39466c6792e753922226670b47b4b8a3718c94f54a WatchSource:0}: Error finding container d54dd19c36a53ea351907e39466c6792e753922226670b47b4b8a3718c94f54a: Status 404 returned error can't find the container with id d54dd19c36a53ea351907e39466c6792e753922226670b47b4b8a3718c94f54a
Sep 17 18:34:02.012738 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:02.012711    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9eec280d-0780-45ef-832a-091c401c234c-certificates\") pod \"keda-admission-6c4d864b86-6gmg2\" (UID: \"9eec280d-0780-45ef-832a-091c401c234c\") " pod="openshift-keda/keda-admission-6c4d864b86-6gmg2"
Sep 17 18:34:02.015032 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:02.015014    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9eec280d-0780-45ef-832a-091c401c234c-certificates\") pod \"keda-admission-6c4d864b86-6gmg2\" (UID: \"9eec280d-0780-45ef-832a-091c401c234c\") " pod="openshift-keda/keda-admission-6c4d864b86-6gmg2"
Sep 17 18:34:02.259493 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:02.259426    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-6gmg2"
Sep 17 18:34:02.493813 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:02.493776    2565 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 18:34:02.530485 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:02.530307    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-6gmg2"]
Sep 17 18:34:02.546041 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:34:02.532823    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eec280d_0780_45ef_832a_091c401c234c.slice/crio-1fdcd01038ec69e8fbf3fd77d38775c62d461d3ad6a0ffa546fe684ab629859f WatchSource:0}: Error finding container 1fdcd01038ec69e8fbf3fd77d38775c62d461d3ad6a0ffa546fe684ab629859f: Status 404 returned error can't find the container with id 1fdcd01038ec69e8fbf3fd77d38775c62d461d3ad6a0ffa546fe684ab629859f
Sep 17 18:34:02.936039 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:02.936005    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-6gmg2" event={"ID":"9eec280d-0780-45ef-832a-091c401c234c","Type":"ContainerStarted","Data":"1fdcd01038ec69e8fbf3fd77d38775c62d461d3ad6a0ffa546fe684ab629859f"}
Sep 17 18:34:02.937719 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:02.937672    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-dwnbk" event={"ID":"26f55e2f-455c-4cb7-a14a-ed14ef9761d7","Type":"ContainerStarted","Data":"d54dd19c36a53ea351907e39466c6792e753922226670b47b4b8a3718c94f54a"}
Sep 17 18:34:04.950215 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:04.950160    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-6gmg2" event={"ID":"9eec280d-0780-45ef-832a-091c401c234c","Type":"ContainerStarted","Data":"b3c06d8d722cf0cf1beab58a7ba70a707701f213fd7d48ed3edb048e0422d681"}
Sep 17 18:34:04.950690 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:04.950376    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-6gmg2"
Sep 17 18:34:06.965056 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:06.965019    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-dwnbk" event={"ID":"26f55e2f-455c-4cb7-a14a-ed14ef9761d7","Type":"ContainerStarted","Data":"c3740bc3cc18a43a24904c10f491a4d694d3ad39d90b2e8022a8db3f039ae366"}
Sep 17 18:34:06.965521 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:06.965087    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-dwnbk"
Sep 17 18:34:06.966708 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:06.966678    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-q7srz" event={"ID":"84a079be-626c-46b5-8eeb-1794b8010ce4","Type":"ContainerStarted","Data":"dae64d40145a814dd88af76d478c6d2e8c835234aa25d202f58011ad8d450e02"}
Sep 17 18:34:06.966865 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:06.966838    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-q7srz"
Sep 17 18:34:07.005610 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:07.005565    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-6gmg2" podStartSLOduration=4.147329176 podStartE2EDuration="6.005550446s" totalImagesPullingTime="1.85822127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:34:04.977617875 +0000 UTC m=+302.909819394" watchObservedRunningTime="2026-09-17 18:34:07.005550446 +0000 UTC m=+304.937751959"
Sep 17 18:34:07.006668 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:07.006633    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-dwnbk" podStartSLOduration=2.415582241 podStartE2EDuration="7.006624342s" totalImagesPullingTime="4.591042101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:34:07.005516633 +0000 UTC m=+304.937718152" watchObservedRunningTime="2026-09-17 18:34:07.006624342 +0000 UTC m=+304.938825861"
Sep 17 18:34:07.046219 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:07.046165    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-q7srz" podStartSLOduration=1.1746642760000001 podStartE2EDuration="6.046152487s" totalImagesPullingTime="4.871488211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:34:07.041658057 +0000 UTC m=+304.973859577" watchObservedRunningTime="2026-09-17 18:34:07.046152487 +0000 UTC m=+304.978354006"
Sep 17 18:34:17.975969 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:17.975936    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-q7srz"
Sep 17 18:34:21.931714 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:21.931682    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lhg68"
Sep 17 18:34:25.960483 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:25.960453    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-6gmg2"
Sep 17 18:34:27.974049 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:27.974024    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-dwnbk"
Sep 17 18:34:56.347088 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:56.347003    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr"]
Sep 17 18:34:56.350303 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:56.350278    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr"
Sep 17 18:34:56.353298 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:56.353273    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 18:34:56.353439 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:56.353399    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 18:34:56.354249 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:56.354222    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-sd82r\""
Sep 17 18:34:56.360564 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:56.360543    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr"]
Sep 17 18:34:56.386983 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:56.386955    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/374a7d29-b736-463e-a881-485b2fa5c6ca-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr\" (UID: \"374a7d29-b736-463e-a881-485b2fa5c6ca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr"
Sep 17 18:34:56.387096 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:56.386986    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/374a7d29-b736-463e-a881-485b2fa5c6ca-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr\" (UID: \"374a7d29-b736-463e-a881-485b2fa5c6ca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr"
Sep 17 18:34:56.387096 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:56.387025    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmv2q\" (UniqueName: \"kubernetes.io/projected/374a7d29-b736-463e-a881-485b2fa5c6ca-kube-api-access-gmv2q\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr\" (UID: \"374a7d29-b736-463e-a881-485b2fa5c6ca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr"
Sep 17 18:34:56.487569 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:56.487539    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/374a7d29-b736-463e-a881-485b2fa5c6ca-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr\" (UID: \"374a7d29-b736-463e-a881-485b2fa5c6ca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr"
Sep 17 18:34:56.487649 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:56.487572    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/374a7d29-b736-463e-a881-485b2fa5c6ca-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr\" (UID: \"374a7d29-b736-463e-a881-485b2fa5c6ca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr"
Sep 17 18:34:56.487649 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:56.487613    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmv2q\" (UniqueName: \"kubernetes.io/projected/374a7d29-b736-463e-a881-485b2fa5c6ca-kube-api-access-gmv2q\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr\" (UID: \"374a7d29-b736-463e-a881-485b2fa5c6ca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr"
Sep 17 18:34:56.487911 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:56.487893    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/374a7d29-b736-463e-a881-485b2fa5c6ca-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr\" (UID: \"374a7d29-b736-463e-a881-485b2fa5c6ca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr"
Sep 17 18:34:56.487990 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:56.487970    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/374a7d29-b736-463e-a881-485b2fa5c6ca-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr\" (UID: \"374a7d29-b736-463e-a881-485b2fa5c6ca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr"
Sep 17 18:34:56.496353 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:56.496330    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmv2q\" (UniqueName: \"kubernetes.io/projected/374a7d29-b736-463e-a881-485b2fa5c6ca-kube-api-access-gmv2q\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr\" (UID: \"374a7d29-b736-463e-a881-485b2fa5c6ca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr"
Sep 17 18:34:56.659674 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:56.659656    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr"
Sep 17 18:34:56.934979 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:56.934776    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr"]
Sep 17 18:34:56.937863 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:34:56.937832    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod374a7d29_b736_463e_a881_485b2fa5c6ca.slice/crio-efdebd64beb73f218342e25f21014cab69a42ade6ddc3c64f6a8f3f00b546dd6 WatchSource:0}: Error finding container efdebd64beb73f218342e25f21014cab69a42ade6ddc3c64f6a8f3f00b546dd6: Status 404 returned error can't find the container with id efdebd64beb73f218342e25f21014cab69a42ade6ddc3c64f6a8f3f00b546dd6
Sep 17 18:34:57.183632 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:57.183597    2565 generic.go:356] "Generic (PLEG): container finished" podID="374a7d29-b736-463e-a881-485b2fa5c6ca" containerID="9a64ec98271d438aa9c01799ba89540e5fdb1c943c32a04532859b1d77c191ea" exitCode=0
Sep 17 18:34:57.183750 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:57.183653    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr" event={"ID":"374a7d29-b736-463e-a881-485b2fa5c6ca","Type":"ContainerDied","Data":"9a64ec98271d438aa9c01799ba89540e5fdb1c943c32a04532859b1d77c191ea"}
Sep 17 18:34:57.183750 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:57.183695    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr" event={"ID":"374a7d29-b736-463e-a881-485b2fa5c6ca","Type":"ContainerStarted","Data":"efdebd64beb73f218342e25f21014cab69a42ade6ddc3c64f6a8f3f00b546dd6"}
Sep 17 18:34:57.184658 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:57.184640    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:34:59.196847 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:34:59.196817    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr" event={"ID":"374a7d29-b736-463e-a881-485b2fa5c6ca","Type":"ContainerStarted","Data":"e2111ca4c986d4051ce1044382a769d5019dc78b3ddb7e2c9d74a518c8c073b3"}
Sep 17 18:35:00.203491 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:00.203458    2565 generic.go:356] "Generic (PLEG): container finished" podID="374a7d29-b736-463e-a881-485b2fa5c6ca" containerID="e2111ca4c986d4051ce1044382a769d5019dc78b3ddb7e2c9d74a518c8c073b3" exitCode=0
Sep 17 18:35:00.203856 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:00.203500    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr" event={"ID":"374a7d29-b736-463e-a881-485b2fa5c6ca","Type":"ContainerDied","Data":"e2111ca4c986d4051ce1044382a769d5019dc78b3ddb7e2c9d74a518c8c073b3"}
Sep 17 18:35:01.210275 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:01.210234    2565 generic.go:356] "Generic (PLEG): container finished" podID="374a7d29-b736-463e-a881-485b2fa5c6ca" containerID="aa926883aafe860c09ea47e1e83583c6cd8d25112c812e1c366c02144f4dadd1" exitCode=0
Sep 17 18:35:01.210727 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:01.210281    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr" event={"ID":"374a7d29-b736-463e-a881-485b2fa5c6ca","Type":"ContainerDied","Data":"aa926883aafe860c09ea47e1e83583c6cd8d25112c812e1c366c02144f4dadd1"}
Sep 17 18:35:02.338160 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:02.338138    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr"
Sep 17 18:35:02.431986 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:02.431960    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/374a7d29-b736-463e-a881-485b2fa5c6ca-util\") pod \"374a7d29-b736-463e-a881-485b2fa5c6ca\" (UID: \"374a7d29-b736-463e-a881-485b2fa5c6ca\") "
Sep 17 18:35:02.432125 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:02.432007    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmv2q\" (UniqueName: \"kubernetes.io/projected/374a7d29-b736-463e-a881-485b2fa5c6ca-kube-api-access-gmv2q\") pod \"374a7d29-b736-463e-a881-485b2fa5c6ca\" (UID: \"374a7d29-b736-463e-a881-485b2fa5c6ca\") "
Sep 17 18:35:02.432125 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:02.432087    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/374a7d29-b736-463e-a881-485b2fa5c6ca-bundle\") pod \"374a7d29-b736-463e-a881-485b2fa5c6ca\" (UID: \"374a7d29-b736-463e-a881-485b2fa5c6ca\") "
Sep 17 18:35:02.432812 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:02.432782    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/374a7d29-b736-463e-a881-485b2fa5c6ca-bundle" (OuterVolumeSpecName: "bundle") pod "374a7d29-b736-463e-a881-485b2fa5c6ca" (UID: "374a7d29-b736-463e-a881-485b2fa5c6ca"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:35:02.434021 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:02.433998    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/374a7d29-b736-463e-a881-485b2fa5c6ca-kube-api-access-gmv2q" (OuterVolumeSpecName: "kube-api-access-gmv2q") pod "374a7d29-b736-463e-a881-485b2fa5c6ca" (UID: "374a7d29-b736-463e-a881-485b2fa5c6ca"). InnerVolumeSpecName "kube-api-access-gmv2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:35:02.439036 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:02.439012    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/374a7d29-b736-463e-a881-485b2fa5c6ca-util" (OuterVolumeSpecName: "util") pod "374a7d29-b736-463e-a881-485b2fa5c6ca" (UID: "374a7d29-b736-463e-a881-485b2fa5c6ca"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:35:02.533445 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:02.533381    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/374a7d29-b736-463e-a881-485b2fa5c6ca-bundle\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:35:02.533445 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:02.533404    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/374a7d29-b736-463e-a881-485b2fa5c6ca-util\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:35:02.533445 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:02.533417    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmv2q\" (UniqueName: \"kubernetes.io/projected/374a7d29-b736-463e-a881-485b2fa5c6ca-kube-api-access-gmv2q\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:35:03.220334 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:03.220295    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr" event={"ID":"374a7d29-b736-463e-a881-485b2fa5c6ca","Type":"ContainerDied","Data":"efdebd64beb73f218342e25f21014cab69a42ade6ddc3c64f6a8f3f00b546dd6"}
Sep 17 18:35:03.220484 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:03.220359    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efdebd64beb73f218342e25f21014cab69a42ade6ddc3c64f6a8f3f00b546dd6"
Sep 17 18:35:03.220484 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:03.220307    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404rmtr"
Sep 17 18:35:10.233700 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:10.233658    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-8tv22"]
Sep 17 18:35:10.234112 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:10.233963    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="374a7d29-b736-463e-a881-485b2fa5c6ca" containerName="util"
Sep 17 18:35:10.234112 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:10.233976    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="374a7d29-b736-463e-a881-485b2fa5c6ca" containerName="util"
Sep 17 18:35:10.234112 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:10.233986    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="374a7d29-b736-463e-a881-485b2fa5c6ca" containerName="extract"
Sep 17 18:35:10.234112 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:10.233992    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="374a7d29-b736-463e-a881-485b2fa5c6ca" containerName="extract"
Sep 17 18:35:10.234112 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:10.234003    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="374a7d29-b736-463e-a881-485b2fa5c6ca" containerName="pull"
Sep 17 18:35:10.234112 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:10.234009    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="374a7d29-b736-463e-a881-485b2fa5c6ca" containerName="pull"
Sep 17 18:35:10.234112 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:10.234060    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="374a7d29-b736-463e-a881-485b2fa5c6ca" containerName="extract"
Sep 17 18:35:10.238160 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:10.238138    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-8tv22"
Sep 17 18:35:10.241018 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:10.240996    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-8dxr5\""
Sep 17 18:35:10.241222 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:10.241188    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:35:10.241291 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:10.241224    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 17 18:35:10.257417 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:10.257390    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-8tv22"]
Sep 17 18:35:10.289811 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:10.289783    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wwgt\" (UniqueName: \"kubernetes.io/projected/022f1f36-1318-4986-af24-c5e4b2e007b9-kube-api-access-4wwgt\") pod \"cert-manager-operator-controller-manager-684f79c9b9-8tv22\" (UID: \"022f1f36-1318-4986-af24-c5e4b2e007b9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-8tv22"
Sep 17 18:35:10.289899 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:10.289830    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/022f1f36-1318-4986-af24-c5e4b2e007b9-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-8tv22\" (UID: \"022f1f36-1318-4986-af24-c5e4b2e007b9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-8tv22"
Sep 17 18:35:10.390635 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:10.390613    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/022f1f36-1318-4986-af24-c5e4b2e007b9-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-8tv22\" (UID: \"022f1f36-1318-4986-af24-c5e4b2e007b9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-8tv22"
Sep 17 18:35:10.390746 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:10.390685    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wwgt\" (UniqueName: \"kubernetes.io/projected/022f1f36-1318-4986-af24-c5e4b2e007b9-kube-api-access-4wwgt\") pod \"cert-manager-operator-controller-manager-684f79c9b9-8tv22\" (UID: \"022f1f36-1318-4986-af24-c5e4b2e007b9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-8tv22"
Sep 17 18:35:10.390957 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:10.390936    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/022f1f36-1318-4986-af24-c5e4b2e007b9-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-8tv22\" (UID: \"022f1f36-1318-4986-af24-c5e4b2e007b9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-8tv22"
Sep 17 18:35:10.403006 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:10.402981    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wwgt\" (UniqueName: \"kubernetes.io/projected/022f1f36-1318-4986-af24-c5e4b2e007b9-kube-api-access-4wwgt\") pod \"cert-manager-operator-controller-manager-684f79c9b9-8tv22\" (UID: \"022f1f36-1318-4986-af24-c5e4b2e007b9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-8tv22"
Sep 17 18:35:10.548296 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:10.548239    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-8tv22"
Sep 17 18:35:10.835537 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:10.835511    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-8tv22"]
Sep 17 18:35:10.836814 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:35:10.836789    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod022f1f36_1318_4986_af24_c5e4b2e007b9.slice/crio-4877de732e1e775b72be1100af0b4ffb9e59601f93590b9b7356fe8d09dc8f0e WatchSource:0}: Error finding container 4877de732e1e775b72be1100af0b4ffb9e59601f93590b9b7356fe8d09dc8f0e: Status 404 returned error can't find the container with id 4877de732e1e775b72be1100af0b4ffb9e59601f93590b9b7356fe8d09dc8f0e
Sep 17 18:35:11.257134 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:11.257105    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-8tv22" event={"ID":"022f1f36-1318-4986-af24-c5e4b2e007b9","Type":"ContainerStarted","Data":"4877de732e1e775b72be1100af0b4ffb9e59601f93590b9b7356fe8d09dc8f0e"}
Sep 17 18:35:22.306548 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:22.306509    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-8tv22" event={"ID":"022f1f36-1318-4986-af24-c5e4b2e007b9","Type":"ContainerStarted","Data":"b4e81f68e65c13767c556b6c4562d0206fe2127a7fe0bb5de3a9f3f3490ae78e"}
Sep 17 18:35:22.333458 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:22.333363    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-8tv22" podStartSLOduration=1.109446184 podStartE2EDuration="12.333345458s" totalImagesPullingTime="11.223899274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:35:22.330053843 +0000 UTC m=+380.262255368" watchObservedRunningTime="2026-09-17 18:35:22.333345458 +0000 UTC m=+380.265546978"
Sep 17 18:35:29.013162 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:29.013118    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzplm2"]
Sep 17 18:35:29.016642 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:29.016620    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzplm2"
Sep 17 18:35:29.019159 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:29.019126    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-sd82r\""
Sep 17 18:35:29.019311 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:29.019191    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 18:35:29.019311 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:29.019263    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 18:35:29.024922 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:29.024904    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzplm2"]
Sep 17 18:35:29.122136 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:29.122103    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f496c38f-3c25-4caa-ad34-9ebd75c91c1a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzplm2\" (UID: \"f496c38f-3c25-4caa-ad34-9ebd75c91c1a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzplm2"
Sep 17 18:35:29.122321 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:29.122165    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f496c38f-3c25-4caa-ad34-9ebd75c91c1a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzplm2\" (UID: \"f496c38f-3c25-4caa-ad34-9ebd75c91c1a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzplm2"
Sep 17 18:35:29.122321 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:29.122231    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99rbl\" (UniqueName: \"kubernetes.io/projected/f496c38f-3c25-4caa-ad34-9ebd75c91c1a-kube-api-access-99rbl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzplm2\" (UID: \"f496c38f-3c25-4caa-ad34-9ebd75c91c1a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzplm2"
Sep 17 18:35:29.223114 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:29.223086    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f496c38f-3c25-4caa-ad34-9ebd75c91c1a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzplm2\" (UID: \"f496c38f-3c25-4caa-ad34-9ebd75c91c1a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzplm2"
Sep 17 18:35:29.223239 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:29.223118    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99rbl\" (UniqueName: \"kubernetes.io/projected/f496c38f-3c25-4caa-ad34-9ebd75c91c1a-kube-api-access-99rbl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzplm2\" (UID: \"f496c38f-3c25-4caa-ad34-9ebd75c91c1a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzplm2"
Sep 17 18:35:29.223239 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:29.223142    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f496c38f-3c25-4caa-ad34-9ebd75c91c1a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzplm2\" (UID: \"f496c38f-3c25-4caa-ad34-9ebd75c91c1a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzplm2"
Sep 17 18:35:29.223502 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:29.223479    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f496c38f-3c25-4caa-ad34-9ebd75c91c1a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzplm2\" (UID: \"f496c38f-3c25-4caa-ad34-9ebd75c91c1a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzplm2"
Sep 17 18:35:29.223554 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:29.223490    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f496c38f-3c25-4caa-ad34-9ebd75c91c1a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzplm2\" (UID: \"f496c38f-3c25-4caa-ad34-9ebd75c91c1a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzplm2"
Sep 17 18:35:29.232012 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:29.231988    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99rbl\" (UniqueName: \"kubernetes.io/projected/f496c38f-3c25-4caa-ad34-9ebd75c91c1a-kube-api-access-99rbl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzplm2\" (UID: \"f496c38f-3c25-4caa-ad34-9ebd75c91c1a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzplm2"
Sep 17 18:35:29.328083 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:29.328024    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzplm2"
Sep 17 18:35:29.600888 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:29.600859    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzplm2"]
Sep 17 18:35:29.603737 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:35:29.603698    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf496c38f_3c25_4caa_ad34_9ebd75c91c1a.slice/crio-521125b194e3edb75f4a99c00467e98c5c00c6000332d078bfc51675137dbaa7 WatchSource:0}: Error finding container 521125b194e3edb75f4a99c00467e98c5c00c6000332d078bfc51675137dbaa7: Status 404 returned error can't find the container with id 521125b194e3edb75f4a99c00467e98c5c00c6000332d078bfc51675137dbaa7
Sep 17 18:35:30.341725 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:30.341696    2565 generic.go:356] "Generic (PLEG): container finished" podID="f496c38f-3c25-4caa-ad34-9ebd75c91c1a" containerID="e2b8db863b25f6ef348cdc4721a02f03d431ebd7adf31edc76d0d92ca0c4be04" exitCode=0
Sep 17 18:35:30.342067 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:30.341738    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzplm2" event={"ID":"f496c38f-3c25-4caa-ad34-9ebd75c91c1a","Type":"ContainerDied","Data":"e2b8db863b25f6ef348cdc4721a02f03d431ebd7adf31edc76d0d92ca0c4be04"}
Sep 17 18:35:30.342067 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:30.341771    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzplm2" event={"ID":"f496c38f-3c25-4caa-ad34-9ebd75c91c1a","Type":"ContainerStarted","Data":"521125b194e3edb75f4a99c00467e98c5c00c6000332d078bfc51675137dbaa7"}
Sep 17 18:35:33.357300 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:33.357265    2565 generic.go:356] "Generic (PLEG): container finished" podID="f496c38f-3c25-4caa-ad34-9ebd75c91c1a" containerID="3d51e3cb471f4749a4a22edd357477cce6b690461d4470870cb80d0298138705" exitCode=0
Sep 17 18:35:33.357761 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:33.357328    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzplm2" event={"ID":"f496c38f-3c25-4caa-ad34-9ebd75c91c1a","Type":"ContainerDied","Data":"3d51e3cb471f4749a4a22edd357477cce6b690461d4470870cb80d0298138705"}
Sep 17 18:35:34.364263 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:34.364232    2565 generic.go:356] "Generic (PLEG): container finished" podID="f496c38f-3c25-4caa-ad34-9ebd75c91c1a" containerID="4627741a3704c1dbe7c83052cbe593319f100f146932dd75966d38dada3d3221" exitCode=0
Sep 17 18:35:34.364635 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:34.364293    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzplm2" event={"ID":"f496c38f-3c25-4caa-ad34-9ebd75c91c1a","Type":"ContainerDied","Data":"4627741a3704c1dbe7c83052cbe593319f100f146932dd75966d38dada3d3221"}
Sep 17 18:35:34.400459 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:34.400427    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-md8cw"]
Sep 17 18:35:34.405222 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:34.404033    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-md8cw"
Sep 17 18:35:34.407132 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:34.407104    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 17 18:35:34.407282 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:34.407139    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 17 18:35:34.407340 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:34.407288    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-xdh4d\""
Sep 17 18:35:34.415116 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:34.415094    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-md8cw"]
Sep 17 18:35:34.463886 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:34.463861    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk6t5\" (UniqueName: \"kubernetes.io/projected/edfbc9f3-a864-4ba6-9561-609845dc9e10-kube-api-access-xk6t5\") pod \"cert-manager-55b4d954d5-md8cw\" (UID: \"edfbc9f3-a864-4ba6-9561-609845dc9e10\") " pod="cert-manager/cert-manager-55b4d954d5-md8cw"
Sep 17 18:35:34.463975 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:34.463902    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/edfbc9f3-a864-4ba6-9561-609845dc9e10-bound-sa-token\") pod \"cert-manager-55b4d954d5-md8cw\" (UID: \"edfbc9f3-a864-4ba6-9561-609845dc9e10\") " pod="cert-manager/cert-manager-55b4d954d5-md8cw"
Sep 17 18:35:34.564740 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:34.564717    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/edfbc9f3-a864-4ba6-9561-609845dc9e10-bound-sa-token\") pod \"cert-manager-55b4d954d5-md8cw\" (UID: \"edfbc9f3-a864-4ba6-9561-609845dc9e10\") " pod="cert-manager/cert-manager-55b4d954d5-md8cw"
Sep 17 18:35:34.564958 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:34.564794    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk6t5\" (UniqueName: \"kubernetes.io/projected/edfbc9f3-a864-4ba6-9561-609845dc9e10-kube-api-access-xk6t5\") pod \"cert-manager-55b4d954d5-md8cw\" (UID: \"edfbc9f3-a864-4ba6-9561-609845dc9e10\") " pod="cert-manager/cert-manager-55b4d954d5-md8cw"
Sep 17 18:35:34.572759 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:34.572733    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/edfbc9f3-a864-4ba6-9561-609845dc9e10-bound-sa-token\") pod \"cert-manager-55b4d954d5-md8cw\" (UID: \"edfbc9f3-a864-4ba6-9561-609845dc9e10\") " pod="cert-manager/cert-manager-55b4d954d5-md8cw"
Sep 17 18:35:34.572759 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:34.572752    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk6t5\" (UniqueName: \"kubernetes.io/projected/edfbc9f3-a864-4ba6-9561-609845dc9e10-kube-api-access-xk6t5\") pod \"cert-manager-55b4d954d5-md8cw\" (UID: \"edfbc9f3-a864-4ba6-9561-609845dc9e10\") " pod="cert-manager/cert-manager-55b4d954d5-md8cw"
Sep 17 18:35:34.729664 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:34.729640    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-md8cw"
Sep 17 18:35:35.001188 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:35.001157    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-md8cw"]
Sep 17 18:35:35.003814 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:35:35.003773    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podedfbc9f3_a864_4ba6_9561_609845dc9e10.slice/crio-204019fc8462c9a14b24ce0019d3f3ef5c6b2c5baa3393389aec2ccc5484ebcc WatchSource:0}: Error finding container 204019fc8462c9a14b24ce0019d3f3ef5c6b2c5baa3393389aec2ccc5484ebcc: Status 404 returned error can't find the container with id 204019fc8462c9a14b24ce0019d3f3ef5c6b2c5baa3393389aec2ccc5484ebcc
Sep 17 18:35:35.370881 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:35.370798    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-md8cw" event={"ID":"edfbc9f3-a864-4ba6-9561-609845dc9e10","Type":"ContainerStarted","Data":"204019fc8462c9a14b24ce0019d3f3ef5c6b2c5baa3393389aec2ccc5484ebcc"}
Sep 17 18:35:35.484833 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:35.484812    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzplm2"
Sep 17 18:35:35.571829 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:35.571800    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f496c38f-3c25-4caa-ad34-9ebd75c91c1a-util\") pod \"f496c38f-3c25-4caa-ad34-9ebd75c91c1a\" (UID: \"f496c38f-3c25-4caa-ad34-9ebd75c91c1a\") "
Sep 17 18:35:35.571937 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:35.571837    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f496c38f-3c25-4caa-ad34-9ebd75c91c1a-bundle\") pod \"f496c38f-3c25-4caa-ad34-9ebd75c91c1a\" (UID: \"f496c38f-3c25-4caa-ad34-9ebd75c91c1a\") "
Sep 17 18:35:35.571937 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:35.571884    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99rbl\" (UniqueName: \"kubernetes.io/projected/f496c38f-3c25-4caa-ad34-9ebd75c91c1a-kube-api-access-99rbl\") pod \"f496c38f-3c25-4caa-ad34-9ebd75c91c1a\" (UID: \"f496c38f-3c25-4caa-ad34-9ebd75c91c1a\") "
Sep 17 18:35:35.572326 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:35.572293    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f496c38f-3c25-4caa-ad34-9ebd75c91c1a-bundle" (OuterVolumeSpecName: "bundle") pod "f496c38f-3c25-4caa-ad34-9ebd75c91c1a" (UID: "f496c38f-3c25-4caa-ad34-9ebd75c91c1a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:35:35.573886 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:35.573861    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f496c38f-3c25-4caa-ad34-9ebd75c91c1a-kube-api-access-99rbl" (OuterVolumeSpecName: "kube-api-access-99rbl") pod "f496c38f-3c25-4caa-ad34-9ebd75c91c1a" (UID: "f496c38f-3c25-4caa-ad34-9ebd75c91c1a"). InnerVolumeSpecName "kube-api-access-99rbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:35:35.576180 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:35.576149    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f496c38f-3c25-4caa-ad34-9ebd75c91c1a-util" (OuterVolumeSpecName: "util") pod "f496c38f-3c25-4caa-ad34-9ebd75c91c1a" (UID: "f496c38f-3c25-4caa-ad34-9ebd75c91c1a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:35:35.673410 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:35.673378    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f496c38f-3c25-4caa-ad34-9ebd75c91c1a-util\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:35:35.673498 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:35.673413    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f496c38f-3c25-4caa-ad34-9ebd75c91c1a-bundle\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:35:35.673498 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:35.673426    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99rbl\" (UniqueName: \"kubernetes.io/projected/f496c38f-3c25-4caa-ad34-9ebd75c91c1a-kube-api-access-99rbl\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:35:36.376865 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:36.376830    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzplm2" event={"ID":"f496c38f-3c25-4caa-ad34-9ebd75c91c1a","Type":"ContainerDied","Data":"521125b194e3edb75f4a99c00467e98c5c00c6000332d078bfc51675137dbaa7"}
Sep 17 18:35:36.376865 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:36.376863    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzplm2"
Sep 17 18:35:36.377294 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:36.376881    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="521125b194e3edb75f4a99c00467e98c5c00c6000332d078bfc51675137dbaa7"
Sep 17 18:35:38.388614 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:38.388582    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-md8cw" event={"ID":"edfbc9f3-a864-4ba6-9561-609845dc9e10","Type":"ContainerStarted","Data":"436a4c41bf7b3f00e43334d7c84c191019f688a1c622f736fe4ee7aeb604db35"}
Sep 17 18:35:38.409819 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:38.407278    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-md8cw" podStartSLOduration=1.184221056 podStartE2EDuration="4.40725186s" totalImagesPullingTime="3.223030804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:35:38.40402785 +0000 UTC m=+396.336229369" watchObservedRunningTime="2026-09-17 18:35:38.40725186 +0000 UTC m=+396.339453382"
Sep 17 18:35:56.451012 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.450972    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"]
Sep 17 18:35:56.451496 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.451284    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f496c38f-3c25-4caa-ad34-9ebd75c91c1a" containerName="util"
Sep 17 18:35:56.451496 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.451296    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f496c38f-3c25-4caa-ad34-9ebd75c91c1a" containerName="util"
Sep 17 18:35:56.451496 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.451308    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f496c38f-3c25-4caa-ad34-9ebd75c91c1a" containerName="extract"
Sep 17 18:35:56.451496 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.451313    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f496c38f-3c25-4caa-ad34-9ebd75c91c1a" containerName="extract"
Sep 17 18:35:56.451496 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.451327    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f496c38f-3c25-4caa-ad34-9ebd75c91c1a" containerName="pull"
Sep 17 18:35:56.451496 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.451332    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f496c38f-3c25-4caa-ad34-9ebd75c91c1a" containerName="pull"
Sep 17 18:35:56.451496 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.451380    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f496c38f-3c25-4caa-ad34-9ebd75c91c1a" containerName="extract"
Sep 17 18:35:56.454246 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.454229    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"
Sep 17 18:35:56.456742 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.456693    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 17 18:35:56.457110 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.457080    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 18:35:56.457256 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.457110    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 17 18:35:56.457256 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.457244    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 17 18:35:56.457385 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.457366    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gw57w\""
Sep 17 18:35:56.468859 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.468839    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"]
Sep 17 18:35:56.624379 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.624350    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snvzs\" (UniqueName: \"kubernetes.io/projected/f7462b94-47fc-4939-873a-b93525e013b3-kube-api-access-snvzs\") pod \"istiod-openshift-gateway-7b7995d5cd-6jfbz\" (UID: \"f7462b94-47fc-4939-873a-b93525e013b3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"
Sep 17 18:35:56.624521 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.624392    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7462b94-47fc-4939-873a-b93525e013b3-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6jfbz\" (UID: \"f7462b94-47fc-4939-873a-b93525e013b3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"
Sep 17 18:35:56.624521 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.624415    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7462b94-47fc-4939-873a-b93525e013b3-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6jfbz\" (UID: \"f7462b94-47fc-4939-873a-b93525e013b3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"
Sep 17 18:35:56.624521 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.624498    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f7462b94-47fc-4939-873a-b93525e013b3-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6jfbz\" (UID: \"f7462b94-47fc-4939-873a-b93525e013b3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"
Sep 17 18:35:56.624631 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.624539    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f7462b94-47fc-4939-873a-b93525e013b3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6jfbz\" (UID: \"f7462b94-47fc-4939-873a-b93525e013b3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"
Sep 17 18:35:56.624631 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.624559    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f7462b94-47fc-4939-873a-b93525e013b3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6jfbz\" (UID: \"f7462b94-47fc-4939-873a-b93525e013b3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"
Sep 17 18:35:56.624631 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.624580    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7462b94-47fc-4939-873a-b93525e013b3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6jfbz\" (UID: \"f7462b94-47fc-4939-873a-b93525e013b3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"
Sep 17 18:35:56.725662 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.725602    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f7462b94-47fc-4939-873a-b93525e013b3-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6jfbz\" (UID: \"f7462b94-47fc-4939-873a-b93525e013b3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"
Sep 17 18:35:56.725662 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.725634    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f7462b94-47fc-4939-873a-b93525e013b3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6jfbz\" (UID: \"f7462b94-47fc-4939-873a-b93525e013b3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"
Sep 17 18:35:56.725662 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.725652    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f7462b94-47fc-4939-873a-b93525e013b3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6jfbz\" (UID: \"f7462b94-47fc-4939-873a-b93525e013b3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"
Sep 17 18:35:56.725864 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.725841    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7462b94-47fc-4939-873a-b93525e013b3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6jfbz\" (UID: \"f7462b94-47fc-4939-873a-b93525e013b3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"
Sep 17 18:35:56.725933 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.725917    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snvzs\" (UniqueName: \"kubernetes.io/projected/f7462b94-47fc-4939-873a-b93525e013b3-kube-api-access-snvzs\") pod \"istiod-openshift-gateway-7b7995d5cd-6jfbz\" (UID: \"f7462b94-47fc-4939-873a-b93525e013b3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"
Sep 17 18:35:56.726002 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.725978    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7462b94-47fc-4939-873a-b93525e013b3-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6jfbz\" (UID: \"f7462b94-47fc-4939-873a-b93525e013b3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"
Sep 17 18:35:56.726150 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.726020    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7462b94-47fc-4939-873a-b93525e013b3-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6jfbz\" (UID: \"f7462b94-47fc-4939-873a-b93525e013b3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"
Sep 17 18:35:56.726266 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.726238    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f7462b94-47fc-4939-873a-b93525e013b3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6jfbz\" (UID: \"f7462b94-47fc-4939-873a-b93525e013b3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"
Sep 17 18:35:56.728256 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.728224    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7462b94-47fc-4939-873a-b93525e013b3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6jfbz\" (UID: \"f7462b94-47fc-4939-873a-b93525e013b3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"
Sep 17 18:35:56.728256 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.728241    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f7462b94-47fc-4939-873a-b93525e013b3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6jfbz\" (UID: \"f7462b94-47fc-4939-873a-b93525e013b3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"
Sep 17 18:35:56.728256 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.728230    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7462b94-47fc-4939-873a-b93525e013b3-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6jfbz\" (UID: \"f7462b94-47fc-4939-873a-b93525e013b3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"
Sep 17 18:35:56.728468 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.728326    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f7462b94-47fc-4939-873a-b93525e013b3-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6jfbz\" (UID: \"f7462b94-47fc-4939-873a-b93525e013b3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"
Sep 17 18:35:56.741247 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.741225    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snvzs\" (UniqueName: \"kubernetes.io/projected/f7462b94-47fc-4939-873a-b93525e013b3-kube-api-access-snvzs\") pod \"istiod-openshift-gateway-7b7995d5cd-6jfbz\" (UID: \"f7462b94-47fc-4939-873a-b93525e013b3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"
Sep 17 18:35:56.741353 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.741251    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7462b94-47fc-4939-873a-b93525e013b3-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6jfbz\" (UID: \"f7462b94-47fc-4939-873a-b93525e013b3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"
Sep 17 18:35:56.763929 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:56.763906    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"
Sep 17 18:35:57.040916 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:57.040888    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"]
Sep 17 18:35:57.042947 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:35:57.042921    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7462b94_47fc_4939_873a_b93525e013b3.slice/crio-14236c3599d335917682cb456ed10aaa3f495d4e1e8ca4c40a7c818d3d43fde9 WatchSource:0}: Error finding container 14236c3599d335917682cb456ed10aaa3f495d4e1e8ca4c40a7c818d3d43fde9: Status 404 returned error can't find the container with id 14236c3599d335917682cb456ed10aaa3f495d4e1e8ca4c40a7c818d3d43fde9
Sep 17 18:35:57.484712 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:57.484674    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz" event={"ID":"f7462b94-47fc-4939-873a-b93525e013b3","Type":"ContainerStarted","Data":"14236c3599d335917682cb456ed10aaa3f495d4e1e8ca4c40a7c818d3d43fde9"}
Sep 17 18:35:59.687641 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:59.687593    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 17 18:35:59.687901 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:35:59.687676    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 17 18:36:00.501407 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:00.501365    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz" event={"ID":"f7462b94-47fc-4939-873a-b93525e013b3","Type":"ContainerStarted","Data":"cf3231c1d8c8d90a68ce515cc2f52c561b42d1ac2a90990a507c9d2f4490307e"}
Sep 17 18:36:00.501581 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:00.501558    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"
Sep 17 18:36:00.503315 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:00.503285    2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-6jfbz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 18:36:00.503454 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:00.503343    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz" podUID="f7462b94-47fc-4939-873a-b93525e013b3" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:36:00.538000 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:00.537953    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz" podStartSLOduration=1.895725724 podStartE2EDuration="4.537941215s" totalImagesPullingTime="2.642215491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:36:00.534798383 +0000 UTC m=+418.466999901" watchObservedRunningTime="2026-09-17 18:36:00.537941215 +0000 UTC m=+418.470142734"
Sep 17 18:36:01.505841 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:01.505810    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"
Sep 17 18:36:14.168959 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.168921    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arfw7s"]
Sep 17 18:36:14.172759 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.172737    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arfw7s"
Sep 17 18:36:14.175311 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.175282    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 18:36:14.175311 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.175293    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 18:36:14.176060 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.176038    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-sd82r\""
Sep 17 18:36:14.180797 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.180424    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arfw7s"]
Sep 17 18:36:14.253060 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.253036    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5slr\" (UniqueName: \"kubernetes.io/projected/4afebe82-6ed2-4020-9891-e215044d49b2-kube-api-access-m5slr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arfw7s\" (UID: \"4afebe82-6ed2-4020-9891-e215044d49b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arfw7s"
Sep 17 18:36:14.253166 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.253081    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4afebe82-6ed2-4020-9891-e215044d49b2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arfw7s\" (UID: \"4afebe82-6ed2-4020-9891-e215044d49b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arfw7s"
Sep 17 18:36:14.253166 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.253123    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4afebe82-6ed2-4020-9891-e215044d49b2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arfw7s\" (UID: \"4afebe82-6ed2-4020-9891-e215044d49b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arfw7s"
Sep 17 18:36:14.264902 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.264880    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7"]
Sep 17 18:36:14.268134 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.268118    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7"
Sep 17 18:36:14.276758 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.276735    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7"]
Sep 17 18:36:14.354072 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.354049    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4afebe82-6ed2-4020-9891-e215044d49b2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arfw7s\" (UID: \"4afebe82-6ed2-4020-9891-e215044d49b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arfw7s"
Sep 17 18:36:14.354178 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.354087    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02845402-c51d-4752-8752-bce172067f50-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7\" (UID: \"02845402-c51d-4752-8752-bce172067f50\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7"
Sep 17 18:36:14.354178 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.354127    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02845402-c51d-4752-8752-bce172067f50-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7\" (UID: \"02845402-c51d-4752-8752-bce172067f50\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7"
Sep 17 18:36:14.354178 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.354152    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5slr\" (UniqueName: \"kubernetes.io/projected/4afebe82-6ed2-4020-9891-e215044d49b2-kube-api-access-m5slr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arfw7s\" (UID: \"4afebe82-6ed2-4020-9891-e215044d49b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arfw7s"
Sep 17 18:36:14.354330 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.354189    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4afebe82-6ed2-4020-9891-e215044d49b2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arfw7s\" (UID: \"4afebe82-6ed2-4020-9891-e215044d49b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arfw7s"
Sep 17 18:36:14.354330 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.354237    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdvj6\" (UniqueName: \"kubernetes.io/projected/02845402-c51d-4752-8752-bce172067f50-kube-api-access-vdvj6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7\" (UID: \"02845402-c51d-4752-8752-bce172067f50\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7"
Sep 17 18:36:14.354418 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.354406    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4afebe82-6ed2-4020-9891-e215044d49b2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arfw7s\" (UID: \"4afebe82-6ed2-4020-9891-e215044d49b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arfw7s"
Sep 17 18:36:14.354457 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.354436    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4afebe82-6ed2-4020-9891-e215044d49b2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arfw7s\" (UID: \"4afebe82-6ed2-4020-9891-e215044d49b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arfw7s"
Sep 17 18:36:14.362978 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.362951    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5slr\" (UniqueName: \"kubernetes.io/projected/4afebe82-6ed2-4020-9891-e215044d49b2-kube-api-access-m5slr\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arfw7s\" (UID: \"4afebe82-6ed2-4020-9891-e215044d49b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arfw7s"
Sep 17 18:36:14.370458 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.370439    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf4tbq"]
Sep 17 18:36:14.373767 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.373752    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf4tbq"
Sep 17 18:36:14.381807 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.381781    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf4tbq"]
Sep 17 18:36:14.455333 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.455280    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdvj6\" (UniqueName: \"kubernetes.io/projected/02845402-c51d-4752-8752-bce172067f50-kube-api-access-vdvj6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7\" (UID: \"02845402-c51d-4752-8752-bce172067f50\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7"
Sep 17 18:36:14.455333 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.455324    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f73d52e-a6fd-4708-9b85-fe4bb931e851-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf4tbq\" (UID: \"1f73d52e-a6fd-4708-9b85-fe4bb931e851\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf4tbq"
Sep 17 18:36:14.455491 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.455343    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02845402-c51d-4752-8752-bce172067f50-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7\" (UID: \"02845402-c51d-4752-8752-bce172067f50\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7"
Sep 17 18:36:14.455491 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.455361    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts472\" (UniqueName: \"kubernetes.io/projected/1f73d52e-a6fd-4708-9b85-fe4bb931e851-kube-api-access-ts472\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf4tbq\" (UID: \"1f73d52e-a6fd-4708-9b85-fe4bb931e851\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf4tbq"
Sep 17 18:36:14.455601 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.455485    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02845402-c51d-4752-8752-bce172067f50-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7\" (UID: \"02845402-c51d-4752-8752-bce172067f50\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7"
Sep 17 18:36:14.455601 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.455527    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f73d52e-a6fd-4708-9b85-fe4bb931e851-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf4tbq\" (UID: \"1f73d52e-a6fd-4708-9b85-fe4bb931e851\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf4tbq"
Sep 17 18:36:14.455669 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.455641    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02845402-c51d-4752-8752-bce172067f50-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7\" (UID: \"02845402-c51d-4752-8752-bce172067f50\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7"
Sep 17 18:36:14.455776 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.455760    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02845402-c51d-4752-8752-bce172067f50-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7\" (UID: \"02845402-c51d-4752-8752-bce172067f50\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7"
Sep 17 18:36:14.471442 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.471419    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdvj6\" (UniqueName: \"kubernetes.io/projected/02845402-c51d-4752-8752-bce172067f50-kube-api-access-vdvj6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7\" (UID: \"02845402-c51d-4752-8752-bce172067f50\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7"
Sep 17 18:36:14.483215 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.483181    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arfw7s"
Sep 17 18:36:14.531610 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.531581    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kbgw2"]
Sep 17 18:36:14.536030 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.536007    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kbgw2"
Sep 17 18:36:14.549991 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.549939    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kbgw2"]
Sep 17 18:36:14.556047 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.556017    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5fbf6c7-9b5a-4999-979d-2c495c60eae4-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kbgw2\" (UID: \"b5fbf6c7-9b5a-4999-979d-2c495c60eae4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kbgw2"
Sep 17 18:36:14.556336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.556097    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f73d52e-a6fd-4708-9b85-fe4bb931e851-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf4tbq\" (UID: \"1f73d52e-a6fd-4708-9b85-fe4bb931e851\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf4tbq"
Sep 17 18:36:14.556336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.556146    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5fbf6c7-9b5a-4999-979d-2c495c60eae4-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kbgw2\" (UID: \"b5fbf6c7-9b5a-4999-979d-2c495c60eae4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kbgw2"
Sep 17 18:36:14.556336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.556281    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f73d52e-a6fd-4708-9b85-fe4bb931e851-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf4tbq\" (UID: \"1f73d52e-a6fd-4708-9b85-fe4bb931e851\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf4tbq"
Sep 17 18:36:14.556336 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.556330    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6797p\" (UniqueName: \"kubernetes.io/projected/b5fbf6c7-9b5a-4999-979d-2c495c60eae4-kube-api-access-6797p\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kbgw2\" (UID: \"b5fbf6c7-9b5a-4999-979d-2c495c60eae4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kbgw2"
Sep 17 18:36:14.556564 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.556365    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts472\" (UniqueName: \"kubernetes.io/projected/1f73d52e-a6fd-4708-9b85-fe4bb931e851-kube-api-access-ts472\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf4tbq\" (UID: \"1f73d52e-a6fd-4708-9b85-fe4bb931e851\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf4tbq"
Sep 17 18:36:14.556603 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.556562    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f73d52e-a6fd-4708-9b85-fe4bb931e851-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf4tbq\" (UID: \"1f73d52e-a6fd-4708-9b85-fe4bb931e851\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf4tbq"
Sep 17 18:36:14.556635 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.556602    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f73d52e-a6fd-4708-9b85-fe4bb931e851-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf4tbq\" (UID: \"1f73d52e-a6fd-4708-9b85-fe4bb931e851\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf4tbq"
Sep 17 18:36:14.566129 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.566106    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts472\" (UniqueName: \"kubernetes.io/projected/1f73d52e-a6fd-4708-9b85-fe4bb931e851-kube-api-access-ts472\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf4tbq\" (UID: \"1f73d52e-a6fd-4708-9b85-fe4bb931e851\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf4tbq"
Sep 17 18:36:14.577820 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.577801    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7"
Sep 17 18:36:14.657364 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.657336    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5fbf6c7-9b5a-4999-979d-2c495c60eae4-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kbgw2\" (UID: \"b5fbf6c7-9b5a-4999-979d-2c495c60eae4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kbgw2"
Sep 17 18:36:14.657491 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.657389    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6797p\" (UniqueName: \"kubernetes.io/projected/b5fbf6c7-9b5a-4999-979d-2c495c60eae4-kube-api-access-6797p\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kbgw2\" (UID: \"b5fbf6c7-9b5a-4999-979d-2c495c60eae4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kbgw2"
Sep 17 18:36:14.657491 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.657418    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5fbf6c7-9b5a-4999-979d-2c495c60eae4-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kbgw2\" (UID: \"b5fbf6c7-9b5a-4999-979d-2c495c60eae4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kbgw2"
Sep 17 18:36:14.657899 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.657878    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5fbf6c7-9b5a-4999-979d-2c495c60eae4-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kbgw2\" (UID: \"b5fbf6c7-9b5a-4999-979d-2c495c60eae4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kbgw2"
Sep 17 18:36:14.657957 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.657878    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5fbf6c7-9b5a-4999-979d-2c495c60eae4-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kbgw2\" (UID: \"b5fbf6c7-9b5a-4999-979d-2c495c60eae4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kbgw2"
Sep 17 18:36:14.665845 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.665820    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6797p\" (UniqueName: \"kubernetes.io/projected/b5fbf6c7-9b5a-4999-979d-2c495c60eae4-kube-api-access-6797p\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kbgw2\" (UID: \"b5fbf6c7-9b5a-4999-979d-2c495c60eae4\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kbgw2"
Sep 17 18:36:14.683104 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.683085    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf4tbq"
Sep 17 18:36:14.756261 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.756232    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arfw7s"]
Sep 17 18:36:14.757893 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:36:14.757866    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4afebe82_6ed2_4020_9891_e215044d49b2.slice/crio-25e6d3f9e2172b4d20fac2281148424c4d78b9b9d9df75ed0a608266e72e1058 WatchSource:0}: Error finding container 25e6d3f9e2172b4d20fac2281148424c4d78b9b9d9df75ed0a608266e72e1058: Status 404 returned error can't find the container with id 25e6d3f9e2172b4d20fac2281148424c4d78b9b9d9df75ed0a608266e72e1058
Sep 17 18:36:14.848401 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.848378    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7"]
Sep 17 18:36:14.851086 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:36:14.851063    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02845402_c51d_4752_8752_bce172067f50.slice/crio-ae3810bafe928afed043cca98ae394c676879cd8e3824efb69a8e558a88a0791 WatchSource:0}: Error finding container ae3810bafe928afed043cca98ae394c676879cd8e3824efb69a8e558a88a0791: Status 404 returned error can't find the container with id ae3810bafe928afed043cca98ae394c676879cd8e3824efb69a8e558a88a0791
Sep 17 18:36:14.852657 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.852634    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kbgw2"
Sep 17 18:36:14.968267 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:14.968235    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf4tbq"]
Sep 17 18:36:14.969536 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:36:14.969508    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f73d52e_a6fd_4708_9b85_fe4bb931e851.slice/crio-21455274438f6918f875d91949a1a3033baccb4d1782fc2014c676480858954d WatchSource:0}: Error finding container 21455274438f6918f875d91949a1a3033baccb4d1782fc2014c676480858954d: Status 404 returned error can't find the container with id 21455274438f6918f875d91949a1a3033baccb4d1782fc2014c676480858954d
Sep 17 18:36:15.126985 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:15.126959    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kbgw2"]
Sep 17 18:36:15.146933 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:36:15.146909    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5fbf6c7_9b5a_4999_979d_2c495c60eae4.slice/crio-f325d0422d7960c8c5dfa41cdc383cff56df9f721f7464538fb75b2933dedd8b WatchSource:0}: Error finding container f325d0422d7960c8c5dfa41cdc383cff56df9f721f7464538fb75b2933dedd8b: Status 404 returned error can't find the container with id f325d0422d7960c8c5dfa41cdc383cff56df9f721f7464538fb75b2933dedd8b
Sep 17 18:36:15.574249 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:15.574154    2565 generic.go:356] "Generic (PLEG): container finished" podID="b5fbf6c7-9b5a-4999-979d-2c495c60eae4" containerID="dec899b01916955dc7fa00dcdf5438aac2bfa81393976fe8f42a1fc3a1c34b2e" exitCode=0
Sep 17 18:36:15.574249 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:15.574230    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kbgw2" event={"ID":"b5fbf6c7-9b5a-4999-979d-2c495c60eae4","Type":"ContainerDied","Data":"dec899b01916955dc7fa00dcdf5438aac2bfa81393976fe8f42a1fc3a1c34b2e"}
Sep 17 18:36:15.574734 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:15.574285    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kbgw2" event={"ID":"b5fbf6c7-9b5a-4999-979d-2c495c60eae4","Type":"ContainerStarted","Data":"f325d0422d7960c8c5dfa41cdc383cff56df9f721f7464538fb75b2933dedd8b"}
Sep 17 18:36:15.575944 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:15.575918    2565 generic.go:356] "Generic (PLEG): container finished" podID="4afebe82-6ed2-4020-9891-e215044d49b2" containerID="2293b34c00c707e645c9f991646a28edd7a82d9d5ddb83cebd2c770004ad94bb" exitCode=0
Sep 17 18:36:15.576089 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:15.576069    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arfw7s" event={"ID":"4afebe82-6ed2-4020-9891-e215044d49b2","Type":"ContainerDied","Data":"2293b34c00c707e645c9f991646a28edd7a82d9d5ddb83cebd2c770004ad94bb"}
Sep 17 18:36:15.576144 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:15.576112    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arfw7s" event={"ID":"4afebe82-6ed2-4020-9891-e215044d49b2","Type":"ContainerStarted","Data":"25e6d3f9e2172b4d20fac2281148424c4d78b9b9d9df75ed0a608266e72e1058"}
Sep 17 18:36:15.578050 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:15.577954    2565 generic.go:356] "Generic (PLEG): container finished" podID="02845402-c51d-4752-8752-bce172067f50" containerID="a2558c253f0f3750cb3ec00743ee97304f1921898ad6023f7bf47076ef8d5e72" exitCode=0
Sep 17 18:36:15.578050 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:15.577980    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7" event={"ID":"02845402-c51d-4752-8752-bce172067f50","Type":"ContainerDied","Data":"a2558c253f0f3750cb3ec00743ee97304f1921898ad6023f7bf47076ef8d5e72"}
Sep 17 18:36:15.578050 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:15.578024    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7" event={"ID":"02845402-c51d-4752-8752-bce172067f50","Type":"ContainerStarted","Data":"ae3810bafe928afed043cca98ae394c676879cd8e3824efb69a8e558a88a0791"}
Sep 17 18:36:15.579776 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:15.579759    2565 generic.go:356] "Generic (PLEG): container finished" podID="1f73d52e-a6fd-4708-9b85-fe4bb931e851" containerID="58523d91e6475d25fe442bd129e84c48083c0c73c06960822c828c91077124e7" exitCode=0
Sep 17 18:36:15.579868 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:15.579800    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf4tbq" event={"ID":"1f73d52e-a6fd-4708-9b85-fe4bb931e851","Type":"ContainerDied","Data":"58523d91e6475d25fe442bd129e84c48083c0c73c06960822c828c91077124e7"}
Sep 17 18:36:15.580267 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:15.580240    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf4tbq" event={"ID":"1f73d52e-a6fd-4708-9b85-fe4bb931e851","Type":"ContainerStarted","Data":"21455274438f6918f875d91949a1a3033baccb4d1782fc2014c676480858954d"}
Sep 17 18:36:16.586251 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:16.586213    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7" event={"ID":"02845402-c51d-4752-8752-bce172067f50","Type":"ContainerStarted","Data":"f45411cbda903ca1266b0ab5e970e23c410e2fc50318ede19fd4d93d82335ef2"}
Sep 17 18:36:16.588240 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:16.588192    2565 generic.go:356] "Generic (PLEG): container finished" podID="b5fbf6c7-9b5a-4999-979d-2c495c60eae4" containerID="ce5ec2b9083b20a969cfd9a52bd63a9f0679c6131887ccf48a3bb82f0b15faef" exitCode=0
Sep 17 18:36:16.588352 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:16.588240    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kbgw2" event={"ID":"b5fbf6c7-9b5a-4999-979d-2c495c60eae4","Type":"ContainerDied","Data":"ce5ec2b9083b20a969cfd9a52bd63a9f0679c6131887ccf48a3bb82f0b15faef"}
Sep 17 18:36:16.590525 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:16.590496    2565 generic.go:356] "Generic (PLEG): container finished" podID="4afebe82-6ed2-4020-9891-e215044d49b2" containerID="c371d5f7562cbbf261736703364e8246feedfca42874cc360867e4b118631bde" exitCode=0
Sep 17 18:36:16.590643 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:16.590550    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arfw7s" event={"ID":"4afebe82-6ed2-4020-9891-e215044d49b2","Type":"ContainerDied","Data":"c371d5f7562cbbf261736703364e8246feedfca42874cc360867e4b118631bde"}
Sep 17 18:36:17.600487 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:17.600445    2565 generic.go:356] "Generic (PLEG): container finished" podID="02845402-c51d-4752-8752-bce172067f50" containerID="f45411cbda903ca1266b0ab5e970e23c410e2fc50318ede19fd4d93d82335ef2" exitCode=0
Sep 17 18:36:17.600912 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:17.600535    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7" event={"ID":"02845402-c51d-4752-8752-bce172067f50","Type":"ContainerDied","Data":"f45411cbda903ca1266b0ab5e970e23c410e2fc50318ede19fd4d93d82335ef2"}
Sep 17 18:36:17.602798 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:17.602769    2565 generic.go:356] "Generic (PLEG): container finished" podID="1f73d52e-a6fd-4708-9b85-fe4bb931e851" containerID="b008e8bfc82dbacd50c9a6fae8abe854c7482f152e2c7df1f8b522877349a09e" exitCode=0
Sep 17 18:36:17.602928 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:17.602860    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf4tbq" event={"ID":"1f73d52e-a6fd-4708-9b85-fe4bb931e851","Type":"ContainerDied","Data":"b008e8bfc82dbacd50c9a6fae8abe854c7482f152e2c7df1f8b522877349a09e"}
Sep 17 18:36:17.605602 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:17.605579    2565 generic.go:356] "Generic (PLEG): container finished" podID="b5fbf6c7-9b5a-4999-979d-2c495c60eae4" containerID="052ad5a5a7759dcb7d20677883e5c78a0dd4cd27b2e034e2cdb6f14559d199c6" exitCode=0
Sep 17 18:36:17.605696 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:17.605669    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kbgw2" event={"ID":"b5fbf6c7-9b5a-4999-979d-2c495c60eae4","Type":"ContainerDied","Data":"052ad5a5a7759dcb7d20677883e5c78a0dd4cd27b2e034e2cdb6f14559d199c6"}
Sep 17 18:36:17.608267 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:17.608233    2565 generic.go:356] "Generic (PLEG): container finished" podID="4afebe82-6ed2-4020-9891-e215044d49b2" containerID="9ace075aa15c48ca9a16e3e1b8e88c6ed7235ba04e66958d88a7ab664fdb2f05" exitCode=0
Sep 17 18:36:17.608409 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:17.608291    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arfw7s" event={"ID":"4afebe82-6ed2-4020-9891-e215044d49b2","Type":"ContainerDied","Data":"9ace075aa15c48ca9a16e3e1b8e88c6ed7235ba04e66958d88a7ab664fdb2f05"}
Sep 17 18:36:18.615518 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:18.615480    2565 generic.go:356] "Generic (PLEG): container finished" podID="02845402-c51d-4752-8752-bce172067f50" containerID="e07ff210d0158d02eec2bf885120965e1ff1e9f633fc3e7480f2f7fe023f27ed" exitCode=0
Sep 17 18:36:18.615976 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:18.615579    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7" event={"ID":"02845402-c51d-4752-8752-bce172067f50","Type":"ContainerDied","Data":"e07ff210d0158d02eec2bf885120965e1ff1e9f633fc3e7480f2f7fe023f27ed"}
Sep 17 18:36:18.623861 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:18.623826    2565 generic.go:356] "Generic (PLEG): container finished" podID="1f73d52e-a6fd-4708-9b85-fe4bb931e851" containerID="df12c1e8c1328966862a83a06e04c9b83869d953395b5d35d2bcb5beae31ae6c" exitCode=0
Sep 17 18:36:18.623963 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:18.623883    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf4tbq" event={"ID":"1f73d52e-a6fd-4708-9b85-fe4bb931e851","Type":"ContainerDied","Data":"df12c1e8c1328966862a83a06e04c9b83869d953395b5d35d2bcb5beae31ae6c"}
Sep 17 18:36:18.751908 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:18.751848    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arfw7s"
Sep 17 18:36:18.784545 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:18.784522    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kbgw2"
Sep 17 18:36:18.791401 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:18.791381    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5slr\" (UniqueName: \"kubernetes.io/projected/4afebe82-6ed2-4020-9891-e215044d49b2-kube-api-access-m5slr\") pod \"4afebe82-6ed2-4020-9891-e215044d49b2\" (UID: \"4afebe82-6ed2-4020-9891-e215044d49b2\") "
Sep 17 18:36:18.791486 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:18.791453    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4afebe82-6ed2-4020-9891-e215044d49b2-util\") pod \"4afebe82-6ed2-4020-9891-e215044d49b2\" (UID: \"4afebe82-6ed2-4020-9891-e215044d49b2\") "
Sep 17 18:36:18.791525 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:18.791501    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4afebe82-6ed2-4020-9891-e215044d49b2-bundle\") pod \"4afebe82-6ed2-4020-9891-e215044d49b2\" (UID: \"4afebe82-6ed2-4020-9891-e215044d49b2\") "
Sep 17 18:36:18.792098 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:18.792075    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4afebe82-6ed2-4020-9891-e215044d49b2-bundle" (OuterVolumeSpecName: "bundle") pod "4afebe82-6ed2-4020-9891-e215044d49b2" (UID: "4afebe82-6ed2-4020-9891-e215044d49b2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:36:18.793463 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:18.793431    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4afebe82-6ed2-4020-9891-e215044d49b2-kube-api-access-m5slr" (OuterVolumeSpecName: "kube-api-access-m5slr") pod "4afebe82-6ed2-4020-9891-e215044d49b2" (UID: "4afebe82-6ed2-4020-9891-e215044d49b2"). InnerVolumeSpecName "kube-api-access-m5slr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:36:18.796916 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:18.796895    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4afebe82-6ed2-4020-9891-e215044d49b2-util" (OuterVolumeSpecName: "util") pod "4afebe82-6ed2-4020-9891-e215044d49b2" (UID: "4afebe82-6ed2-4020-9891-e215044d49b2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:36:18.892554 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:18.892499    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6797p\" (UniqueName: \"kubernetes.io/projected/b5fbf6c7-9b5a-4999-979d-2c495c60eae4-kube-api-access-6797p\") pod \"b5fbf6c7-9b5a-4999-979d-2c495c60eae4\" (UID: \"b5fbf6c7-9b5a-4999-979d-2c495c60eae4\") "
Sep 17 18:36:18.892655 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:18.892558    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5fbf6c7-9b5a-4999-979d-2c495c60eae4-bundle\") pod \"b5fbf6c7-9b5a-4999-979d-2c495c60eae4\" (UID: \"b5fbf6c7-9b5a-4999-979d-2c495c60eae4\") "
Sep 17 18:36:18.892655 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:18.892604    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5fbf6c7-9b5a-4999-979d-2c495c60eae4-util\") pod \"b5fbf6c7-9b5a-4999-979d-2c495c60eae4\" (UID: \"b5fbf6c7-9b5a-4999-979d-2c495c60eae4\") "
Sep 17 18:36:18.892742 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:18.892719    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4afebe82-6ed2-4020-9891-e215044d49b2-bundle\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:36:18.892742 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:18.892731    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5slr\" (UniqueName: \"kubernetes.io/projected/4afebe82-6ed2-4020-9891-e215044d49b2-kube-api-access-m5slr\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:36:18.892742 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:18.892740    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4afebe82-6ed2-4020-9891-e215044d49b2-util\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:36:18.893126 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:18.893100    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5fbf6c7-9b5a-4999-979d-2c495c60eae4-bundle" (OuterVolumeSpecName: "bundle") pod "b5fbf6c7-9b5a-4999-979d-2c495c60eae4" (UID: "b5fbf6c7-9b5a-4999-979d-2c495c60eae4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:36:18.894441 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:18.894420    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5fbf6c7-9b5a-4999-979d-2c495c60eae4-kube-api-access-6797p" (OuterVolumeSpecName: "kube-api-access-6797p") pod "b5fbf6c7-9b5a-4999-979d-2c495c60eae4" (UID: "b5fbf6c7-9b5a-4999-979d-2c495c60eae4"). InnerVolumeSpecName "kube-api-access-6797p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:36:18.898117 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:18.898095    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5fbf6c7-9b5a-4999-979d-2c495c60eae4-util" (OuterVolumeSpecName: "util") pod "b5fbf6c7-9b5a-4999-979d-2c495c60eae4" (UID: "b5fbf6c7-9b5a-4999-979d-2c495c60eae4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:36:18.993190 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:18.993167    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5fbf6c7-9b5a-4999-979d-2c495c60eae4-bundle\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:36:18.993190 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:18.993187    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5fbf6c7-9b5a-4999-979d-2c495c60eae4-util\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:36:18.993306 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:18.993210    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6797p\" (UniqueName: \"kubernetes.io/projected/b5fbf6c7-9b5a-4999-979d-2c495c60eae4-kube-api-access-6797p\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:36:19.630469 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:19.630434    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kbgw2"
Sep 17 18:36:19.630909 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:19.630433    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kbgw2" event={"ID":"b5fbf6c7-9b5a-4999-979d-2c495c60eae4","Type":"ContainerDied","Data":"f325d0422d7960c8c5dfa41cdc383cff56df9f721f7464538fb75b2933dedd8b"}
Sep 17 18:36:19.630909 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:19.630582    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f325d0422d7960c8c5dfa41cdc383cff56df9f721f7464538fb75b2933dedd8b"
Sep 17 18:36:19.632641 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:19.632606    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arfw7s" event={"ID":"4afebe82-6ed2-4020-9891-e215044d49b2","Type":"ContainerDied","Data":"25e6d3f9e2172b4d20fac2281148424c4d78b9b9d9df75ed0a608266e72e1058"}
Sep 17 18:36:19.632761 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:19.632650    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arfw7s"
Sep 17 18:36:19.632761 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:19.632655    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25e6d3f9e2172b4d20fac2281148424c4d78b9b9d9df75ed0a608266e72e1058"
Sep 17 18:36:19.787252 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:19.787232    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf4tbq"
Sep 17 18:36:19.789319 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:19.789301    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7"
Sep 17 18:36:19.798739 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:19.798719    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f73d52e-a6fd-4708-9b85-fe4bb931e851-util\") pod \"1f73d52e-a6fd-4708-9b85-fe4bb931e851\" (UID: \"1f73d52e-a6fd-4708-9b85-fe4bb931e851\") "
Sep 17 18:36:19.798838 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:19.798753    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ts472\" (UniqueName: \"kubernetes.io/projected/1f73d52e-a6fd-4708-9b85-fe4bb931e851-kube-api-access-ts472\") pod \"1f73d52e-a6fd-4708-9b85-fe4bb931e851\" (UID: \"1f73d52e-a6fd-4708-9b85-fe4bb931e851\") "
Sep 17 18:36:19.798838 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:19.798812    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f73d52e-a6fd-4708-9b85-fe4bb931e851-bundle\") pod \"1f73d52e-a6fd-4708-9b85-fe4bb931e851\" (UID: \"1f73d52e-a6fd-4708-9b85-fe4bb931e851\") "
Sep 17 18:36:19.799368 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:19.799342    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f73d52e-a6fd-4708-9b85-fe4bb931e851-bundle" (OuterVolumeSpecName: "bundle") pod "1f73d52e-a6fd-4708-9b85-fe4bb931e851" (UID: "1f73d52e-a6fd-4708-9b85-fe4bb931e851"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:36:19.800777 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:19.800744    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f73d52e-a6fd-4708-9b85-fe4bb931e851-kube-api-access-ts472" (OuterVolumeSpecName: "kube-api-access-ts472") pod "1f73d52e-a6fd-4708-9b85-fe4bb931e851" (UID: "1f73d52e-a6fd-4708-9b85-fe4bb931e851"). InnerVolumeSpecName "kube-api-access-ts472". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:36:19.804083 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:19.804050    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f73d52e-a6fd-4708-9b85-fe4bb931e851-util" (OuterVolumeSpecName: "util") pod "1f73d52e-a6fd-4708-9b85-fe4bb931e851" (UID: "1f73d52e-a6fd-4708-9b85-fe4bb931e851"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:36:19.899457 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:19.899396    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02845402-c51d-4752-8752-bce172067f50-bundle\") pod \"02845402-c51d-4752-8752-bce172067f50\" (UID: \"02845402-c51d-4752-8752-bce172067f50\") "
Sep 17 18:36:19.899457 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:19.899445    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02845402-c51d-4752-8752-bce172067f50-util\") pod \"02845402-c51d-4752-8752-bce172067f50\" (UID: \"02845402-c51d-4752-8752-bce172067f50\") "
Sep 17 18:36:19.899627 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:19.899467    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdvj6\" (UniqueName: \"kubernetes.io/projected/02845402-c51d-4752-8752-bce172067f50-kube-api-access-vdvj6\") pod \"02845402-c51d-4752-8752-bce172067f50\" (UID: \"02845402-c51d-4752-8752-bce172067f50\") "
Sep 17 18:36:19.899627 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:19.899585    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f73d52e-a6fd-4708-9b85-fe4bb931e851-bundle\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:36:19.899627 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:19.899594    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f73d52e-a6fd-4708-9b85-fe4bb931e851-util\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:36:19.899627 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:19.899603    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts472\" (UniqueName: \"kubernetes.io/projected/1f73d52e-a6fd-4708-9b85-fe4bb931e851-kube-api-access-ts472\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:36:19.899924 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:19.899894    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02845402-c51d-4752-8752-bce172067f50-bundle" (OuterVolumeSpecName: "bundle") pod "02845402-c51d-4752-8752-bce172067f50" (UID: "02845402-c51d-4752-8752-bce172067f50"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:36:19.901460 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:19.901439    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02845402-c51d-4752-8752-bce172067f50-kube-api-access-vdvj6" (OuterVolumeSpecName: "kube-api-access-vdvj6") pod "02845402-c51d-4752-8752-bce172067f50" (UID: "02845402-c51d-4752-8752-bce172067f50"). InnerVolumeSpecName "kube-api-access-vdvj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:36:19.907699 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:19.907673    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02845402-c51d-4752-8752-bce172067f50-util" (OuterVolumeSpecName: "util") pod "02845402-c51d-4752-8752-bce172067f50" (UID: "02845402-c51d-4752-8752-bce172067f50"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:36:20.000885 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:20.000859    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02845402-c51d-4752-8752-bce172067f50-util\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:36:20.000885 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:20.000884    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdvj6\" (UniqueName: \"kubernetes.io/projected/02845402-c51d-4752-8752-bce172067f50-kube-api-access-vdvj6\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:36:20.001004 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:20.000899    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02845402-c51d-4752-8752-bce172067f50-bundle\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:36:20.639754 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:20.639725    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7"
Sep 17 18:36:20.640064 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:20.639720    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6n99t7" event={"ID":"02845402-c51d-4752-8752-bce172067f50","Type":"ContainerDied","Data":"ae3810bafe928afed043cca98ae394c676879cd8e3824efb69a8e558a88a0791"}
Sep 17 18:36:20.640064 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:20.639864    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae3810bafe928afed043cca98ae394c676879cd8e3824efb69a8e558a88a0791"
Sep 17 18:36:20.642046 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:20.642023    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf4tbq" event={"ID":"1f73d52e-a6fd-4708-9b85-fe4bb931e851","Type":"ContainerDied","Data":"21455274438f6918f875d91949a1a3033baccb4d1782fc2014c676480858954d"}
Sep 17 18:36:20.642144 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:20.642061    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21455274438f6918f875d91949a1a3033baccb4d1782fc2014c676480858954d"
Sep 17 18:36:20.642144 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:20.642026    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cf4tbq"
Sep 17 18:36:27.979133 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.979054    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-zhm82"]
Sep 17 18:36:27.979572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.979363    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f73d52e-a6fd-4708-9b85-fe4bb931e851" containerName="pull"
Sep 17 18:36:27.979572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.979376    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f73d52e-a6fd-4708-9b85-fe4bb931e851" containerName="pull"
Sep 17 18:36:27.979572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.979387    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4afebe82-6ed2-4020-9891-e215044d49b2" containerName="util"
Sep 17 18:36:27.979572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.979392    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4afebe82-6ed2-4020-9891-e215044d49b2" containerName="util"
Sep 17 18:36:27.979572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.979399    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4afebe82-6ed2-4020-9891-e215044d49b2" containerName="pull"
Sep 17 18:36:27.979572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.979404    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4afebe82-6ed2-4020-9891-e215044d49b2" containerName="pull"
Sep 17 18:36:27.979572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.979410    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5fbf6c7-9b5a-4999-979d-2c495c60eae4" containerName="pull"
Sep 17 18:36:27.979572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.979415    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5fbf6c7-9b5a-4999-979d-2c495c60eae4" containerName="pull"
Sep 17 18:36:27.979572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.979427    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02845402-c51d-4752-8752-bce172067f50" containerName="pull"
Sep 17 18:36:27.979572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.979433    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="02845402-c51d-4752-8752-bce172067f50" containerName="pull"
Sep 17 18:36:27.979572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.979440    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5fbf6c7-9b5a-4999-979d-2c495c60eae4" containerName="util"
Sep 17 18:36:27.979572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.979446    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5fbf6c7-9b5a-4999-979d-2c495c60eae4" containerName="util"
Sep 17 18:36:27.979572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.979452    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4afebe82-6ed2-4020-9891-e215044d49b2" containerName="extract"
Sep 17 18:36:27.979572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.979458    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4afebe82-6ed2-4020-9891-e215044d49b2" containerName="extract"
Sep 17 18:36:27.979572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.979468    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f73d52e-a6fd-4708-9b85-fe4bb931e851" containerName="util"
Sep 17 18:36:27.979572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.979473    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f73d52e-a6fd-4708-9b85-fe4bb931e851" containerName="util"
Sep 17 18:36:27.979572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.979481    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f73d52e-a6fd-4708-9b85-fe4bb931e851" containerName="extract"
Sep 17 18:36:27.979572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.979486    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f73d52e-a6fd-4708-9b85-fe4bb931e851" containerName="extract"
Sep 17 18:36:27.979572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.979492    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02845402-c51d-4752-8752-bce172067f50" containerName="extract"
Sep 17 18:36:27.979572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.979497    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="02845402-c51d-4752-8752-bce172067f50" containerName="extract"
Sep 17 18:36:27.979572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.979503    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5fbf6c7-9b5a-4999-979d-2c495c60eae4" containerName="extract"
Sep 17 18:36:27.979572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.979508    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5fbf6c7-9b5a-4999-979d-2c495c60eae4" containerName="extract"
Sep 17 18:36:27.979572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.979514    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02845402-c51d-4752-8752-bce172067f50" containerName="util"
Sep 17 18:36:27.979572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.979519    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="02845402-c51d-4752-8752-bce172067f50" containerName="util"
Sep 17 18:36:27.979572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.979564    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f73d52e-a6fd-4708-9b85-fe4bb931e851" containerName="extract"
Sep 17 18:36:27.979572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.979572    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5fbf6c7-9b5a-4999-979d-2c495c60eae4" containerName="extract"
Sep 17 18:36:27.979572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.979579    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4afebe82-6ed2-4020-9891-e215044d49b2" containerName="extract"
Sep 17 18:36:27.979572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.979584    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="02845402-c51d-4752-8752-bce172067f50" containerName="extract"
Sep 17 18:36:27.982148 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.982130    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-zhm82"
Sep 17 18:36:27.986536 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.986516    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 18:36:27.986992 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.986962    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 18:36:27.987384 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.987361    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-d44gp\""
Sep 17 18:36:27.996074 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:27.996052    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-zhm82"]
Sep 17 18:36:28.058988 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:28.058966    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs6g4\" (UniqueName: \"kubernetes.io/projected/f74c2e58-f318-4563-8c9a-d6d695ad70cf-kube-api-access-hs6g4\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-zhm82\" (UID: \"f74c2e58-f318-4563-8c9a-d6d695ad70cf\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-zhm82"
Sep 17 18:36:28.160158 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:28.160134    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs6g4\" (UniqueName: \"kubernetes.io/projected/f74c2e58-f318-4563-8c9a-d6d695ad70cf-kube-api-access-hs6g4\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-zhm82\" (UID: \"f74c2e58-f318-4563-8c9a-d6d695ad70cf\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-zhm82"
Sep 17 18:36:28.171599 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:28.171579    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs6g4\" (UniqueName: \"kubernetes.io/projected/f74c2e58-f318-4563-8c9a-d6d695ad70cf-kube-api-access-hs6g4\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-zhm82\" (UID: \"f74c2e58-f318-4563-8c9a-d6d695ad70cf\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-zhm82"
Sep 17 18:36:28.292528 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:28.292472    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-zhm82"
Sep 17 18:36:28.568443 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:28.568419    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-zhm82"]
Sep 17 18:36:28.570378 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:36:28.570344    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf74c2e58_f318_4563_8c9a_d6d695ad70cf.slice/crio-8e32602ba072372fec6f2e248a53c0a03e8720bb294e5aef4cf9c1eebcde7b35 WatchSource:0}: Error finding container 8e32602ba072372fec6f2e248a53c0a03e8720bb294e5aef4cf9c1eebcde7b35: Status 404 returned error can't find the container with id 8e32602ba072372fec6f2e248a53c0a03e8720bb294e5aef4cf9c1eebcde7b35
Sep 17 18:36:28.687146 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:28.687118    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-zhm82" event={"ID":"f74c2e58-f318-4563-8c9a-d6d695ad70cf","Type":"ContainerStarted","Data":"8e32602ba072372fec6f2e248a53c0a03e8720bb294e5aef4cf9c1eebcde7b35"}
Sep 17 18:36:30.699313 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:30.699280    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-zhm82" event={"ID":"f74c2e58-f318-4563-8c9a-d6d695ad70cf","Type":"ContainerStarted","Data":"0e0c7732f7e84f2f04cdc991e77d6cb4e5b03b12d5207ad00c7b8543dc3f9f81"}
Sep 17 18:36:30.699695 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:30.699370    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-zhm82"
Sep 17 18:36:30.760143 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:30.760095    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-zhm82" podStartSLOduration=1.967352976 podStartE2EDuration="3.760081467s" totalImagesPullingTime="1.792728491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:36:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:36:30.759373009 +0000 UTC m=+448.691574529" watchObservedRunningTime="2026-09-17 18:36:30.760081467 +0000 UTC m=+448.692282985"
Sep 17 18:36:36.582871 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:36.582836    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-m6nk9"]
Sep 17 18:36:36.592007 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:36.591985    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-m6nk9"
Sep 17 18:36:36.597169 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:36.597144    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-l24wl\""
Sep 17 18:36:36.607267 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:36.607244    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-m6nk9"]
Sep 17 18:36:36.623410 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:36.623383    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pkpd\" (UniqueName: \"kubernetes.io/projected/19ce5dd0-8789-4072-b3b1-b934e727585f-kube-api-access-7pkpd\") pod \"authorino-operator-6fc8f864f4-m6nk9\" (UID: \"19ce5dd0-8789-4072-b3b1-b934e727585f\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-m6nk9"
Sep 17 18:36:36.723717 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:36.723690    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pkpd\" (UniqueName: \"kubernetes.io/projected/19ce5dd0-8789-4072-b3b1-b934e727585f-kube-api-access-7pkpd\") pod \"authorino-operator-6fc8f864f4-m6nk9\" (UID: \"19ce5dd0-8789-4072-b3b1-b934e727585f\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-m6nk9"
Sep 17 18:36:36.737210 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:36.737172    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pkpd\" (UniqueName: \"kubernetes.io/projected/19ce5dd0-8789-4072-b3b1-b934e727585f-kube-api-access-7pkpd\") pod \"authorino-operator-6fc8f864f4-m6nk9\" (UID: \"19ce5dd0-8789-4072-b3b1-b934e727585f\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-m6nk9"
Sep 17 18:36:36.902248 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:36.902176    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-m6nk9"
Sep 17 18:36:37.179551 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:37.179519    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-m6nk9"]
Sep 17 18:36:37.180897 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:36:37.180873    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ce5dd0_8789_4072_b3b1_b934e727585f.slice/crio-b271915771499f26c1b408fc0946976337716489bc0bd86126c3eb0b199022c7 WatchSource:0}: Error finding container b271915771499f26c1b408fc0946976337716489bc0bd86126c3eb0b199022c7: Status 404 returned error can't find the container with id b271915771499f26c1b408fc0946976337716489bc0bd86126c3eb0b199022c7
Sep 17 18:36:37.736438 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:37.736403    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-m6nk9" event={"ID":"19ce5dd0-8789-4072-b3b1-b934e727585f","Type":"ContainerStarted","Data":"b271915771499f26c1b408fc0946976337716489bc0bd86126c3eb0b199022c7"}
Sep 17 18:36:39.748608 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:39.748560    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-m6nk9" event={"ID":"19ce5dd0-8789-4072-b3b1-b934e727585f","Type":"ContainerStarted","Data":"a2dd41daeedeb90853c93f146fd272ecb8c9bcd9f7c4259a6612d1bf32c0efca"}
Sep 17 18:36:39.749022 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:39.748655    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-m6nk9"
Sep 17 18:36:39.768047 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:39.768000    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6fc8f864f4-m6nk9" podStartSLOduration=1.962025744 podStartE2EDuration="3.767986503s" totalImagesPullingTime="1.805960759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:36:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:36:39.766997544 +0000 UTC m=+457.699199064" watchObservedRunningTime="2026-09-17 18:36:39.767986503 +0000 UTC m=+457.700188021"
Sep 17 18:36:41.706607 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:41.706575    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-zhm82"
Sep 17 18:36:50.756352 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:36:50.756319    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-m6nk9"
Sep 17 18:37:21.291788 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:37:21.291750    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-njxjg"]
Sep 17 18:37:21.297731 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:37:21.297705    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-njxjg"
Sep 17 18:37:21.300266 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:37:21.300244    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kl6rs\""
Sep 17 18:37:21.300358 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:37:21.300245    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 17 18:37:21.312209 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:37:21.312173    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-njxjg"]
Sep 17 18:37:21.345291 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:37:21.345265    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-njxjg"]
Sep 17 18:37:21.456614 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:37:21.456589    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6ml7\" (UniqueName: \"kubernetes.io/projected/f483ed7b-343f-4d14-9ed4-0b9c138621b9-kube-api-access-q6ml7\") pod \"limitador-limitador-84994f7ffc-njxjg\" (UID: \"f483ed7b-343f-4d14-9ed4-0b9c138621b9\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-njxjg"
Sep 17 18:37:21.456712 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:37:21.456620    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f483ed7b-343f-4d14-9ed4-0b9c138621b9-config-file\") pod \"limitador-limitador-84994f7ffc-njxjg\" (UID: \"f483ed7b-343f-4d14-9ed4-0b9c138621b9\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-njxjg"
Sep 17 18:37:21.557948 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:37:21.557892    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6ml7\" (UniqueName: \"kubernetes.io/projected/f483ed7b-343f-4d14-9ed4-0b9c138621b9-kube-api-access-q6ml7\") pod \"limitador-limitador-84994f7ffc-njxjg\" (UID: \"f483ed7b-343f-4d14-9ed4-0b9c138621b9\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-njxjg"
Sep 17 18:37:21.557948 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:37:21.557923    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f483ed7b-343f-4d14-9ed4-0b9c138621b9-config-file\") pod \"limitador-limitador-84994f7ffc-njxjg\" (UID: \"f483ed7b-343f-4d14-9ed4-0b9c138621b9\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-njxjg"
Sep 17 18:37:21.558546 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:37:21.558529    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f483ed7b-343f-4d14-9ed4-0b9c138621b9-config-file\") pod \"limitador-limitador-84994f7ffc-njxjg\" (UID: \"f483ed7b-343f-4d14-9ed4-0b9c138621b9\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-njxjg"
Sep 17 18:37:21.566029 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:37:21.566000    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6ml7\" (UniqueName: \"kubernetes.io/projected/f483ed7b-343f-4d14-9ed4-0b9c138621b9-kube-api-access-q6ml7\") pod \"limitador-limitador-84994f7ffc-njxjg\" (UID: \"f483ed7b-343f-4d14-9ed4-0b9c138621b9\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-njxjg"
Sep 17 18:37:21.607098 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:37:21.607073    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-njxjg"
Sep 17 18:37:21.884838 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:37:21.884815    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-njxjg"]
Sep 17 18:37:21.886512 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:37:21.886481    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf483ed7b_343f_4d14_9ed4_0b9c138621b9.slice/crio-54565ae1f778a2db1dc042071169fb1b6b1a993d440d7b209960344ff28a98f6 WatchSource:0}: Error finding container 54565ae1f778a2db1dc042071169fb1b6b1a993d440d7b209960344ff28a98f6: Status 404 returned error can't find the container with id 54565ae1f778a2db1dc042071169fb1b6b1a993d440d7b209960344ff28a98f6
Sep 17 18:37:21.950565 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:37:21.950534    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-njxjg" event={"ID":"f483ed7b-343f-4d14-9ed4-0b9c138621b9","Type":"ContainerStarted","Data":"54565ae1f778a2db1dc042071169fb1b6b1a993d440d7b209960344ff28a98f6"}
Sep 17 18:37:23.962320 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:37:23.962287    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-njxjg" event={"ID":"f483ed7b-343f-4d14-9ed4-0b9c138621b9","Type":"ContainerStarted","Data":"1b3b54814e752fcdb2ab2bad02e0024047849db7504e808d4622561d6f91e775"}
Sep 17 18:37:23.962730 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:37:23.962418    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-njxjg"
Sep 17 18:37:23.982185 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:37:23.982138    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-njxjg" podStartSLOduration=1.899605397 podStartE2EDuration="2.982124245s" totalImagesPullingTime="1.082518848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:37:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:37:23.980306092 +0000 UTC m=+501.912507615" watchObservedRunningTime="2026-09-17 18:37:23.982124245 +0000 UTC m=+501.914325799"
Sep 17 18:37:34.968698 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:37:34.968662    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-njxjg"
Sep 17 18:38:15.402826 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:15.402786    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"]
Sep 17 18:38:15.403413 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:15.403030    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz" podUID="f7462b94-47fc-4939-873a-b93525e013b3" containerName="discovery" containerID="cri-o://cf3231c1d8c8d90a68ce515cc2f52c561b42d1ac2a90990a507c9d2f4490307e" gracePeriod=30
Sep 17 18:38:15.647282 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:15.647257    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"
Sep 17 18:38:15.750212 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:15.750180    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snvzs\" (UniqueName: \"kubernetes.io/projected/f7462b94-47fc-4939-873a-b93525e013b3-kube-api-access-snvzs\") pod \"f7462b94-47fc-4939-873a-b93525e013b3\" (UID: \"f7462b94-47fc-4939-873a-b93525e013b3\") "
Sep 17 18:38:15.750343 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:15.750227    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f7462b94-47fc-4939-873a-b93525e013b3-istio-csr-ca-configmap\") pod \"f7462b94-47fc-4939-873a-b93525e013b3\" (UID: \"f7462b94-47fc-4939-873a-b93525e013b3\") "
Sep 17 18:38:15.750428 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:15.750409    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7462b94-47fc-4939-873a-b93525e013b3-istio-token\") pod \"f7462b94-47fc-4939-873a-b93525e013b3\" (UID: \"f7462b94-47fc-4939-873a-b93525e013b3\") "
Sep 17 18:38:15.750467 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:15.750452    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f7462b94-47fc-4939-873a-b93525e013b3-cacerts\") pod \"f7462b94-47fc-4939-873a-b93525e013b3\" (UID: \"f7462b94-47fc-4939-873a-b93525e013b3\") "
Sep 17 18:38:15.750507 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:15.750474    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7462b94-47fc-4939-873a-b93525e013b3-local-certs\") pod \"f7462b94-47fc-4939-873a-b93525e013b3\" (UID: \"f7462b94-47fc-4939-873a-b93525e013b3\") "
Sep 17 18:38:15.750571 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:15.750551    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7462b94-47fc-4939-873a-b93525e013b3-istio-kubeconfig\") pod \"f7462b94-47fc-4939-873a-b93525e013b3\" (UID: \"f7462b94-47fc-4939-873a-b93525e013b3\") "
Sep 17 18:38:15.750637 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:15.750580    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7462b94-47fc-4939-873a-b93525e013b3-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "f7462b94-47fc-4939-873a-b93525e013b3" (UID: "f7462b94-47fc-4939-873a-b93525e013b3"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:38:15.750637 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:15.750596    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f7462b94-47fc-4939-873a-b93525e013b3-istio-csr-dns-cert\") pod \"f7462b94-47fc-4939-873a-b93525e013b3\" (UID: \"f7462b94-47fc-4939-873a-b93525e013b3\") "
Sep 17 18:38:15.750912 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:15.750888    2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f7462b94-47fc-4939-873a-b93525e013b3-istio-csr-ca-configmap\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:38:15.752728 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:15.752707    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7462b94-47fc-4939-873a-b93525e013b3-local-certs" (OuterVolumeSpecName: "local-certs") pod "f7462b94-47fc-4939-873a-b93525e013b3" (UID: "f7462b94-47fc-4939-873a-b93525e013b3"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:38:15.752844 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:15.752764    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7462b94-47fc-4939-873a-b93525e013b3-istio-token" (OuterVolumeSpecName: "istio-token") pod "f7462b94-47fc-4939-873a-b93525e013b3" (UID: "f7462b94-47fc-4939-873a-b93525e013b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:38:15.752909 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:15.752859    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7462b94-47fc-4939-873a-b93525e013b3-kube-api-access-snvzs" (OuterVolumeSpecName: "kube-api-access-snvzs") pod "f7462b94-47fc-4939-873a-b93525e013b3" (UID: "f7462b94-47fc-4939-873a-b93525e013b3"). InnerVolumeSpecName "kube-api-access-snvzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:38:15.752969 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:15.752923    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7462b94-47fc-4939-873a-b93525e013b3-cacerts" (OuterVolumeSpecName: "cacerts") pod "f7462b94-47fc-4939-873a-b93525e013b3" (UID: "f7462b94-47fc-4939-873a-b93525e013b3"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:38:15.752969 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:15.752952    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7462b94-47fc-4939-873a-b93525e013b3-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "f7462b94-47fc-4939-873a-b93525e013b3" (UID: "f7462b94-47fc-4939-873a-b93525e013b3"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:38:15.753040 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:15.753009    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7462b94-47fc-4939-873a-b93525e013b3-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "f7462b94-47fc-4939-873a-b93525e013b3" (UID: "f7462b94-47fc-4939-873a-b93525e013b3"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:38:15.851385 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:15.851349    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7462b94-47fc-4939-873a-b93525e013b3-istio-token\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:38:15.851505 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:15.851398    2565 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f7462b94-47fc-4939-873a-b93525e013b3-cacerts\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:38:15.851505 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:15.851412    2565 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7462b94-47fc-4939-873a-b93525e013b3-local-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:38:15.851505 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:15.851425    2565 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7462b94-47fc-4939-873a-b93525e013b3-istio-kubeconfig\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:38:15.851505 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:15.851437    2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f7462b94-47fc-4939-873a-b93525e013b3-istio-csr-dns-cert\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:38:15.851505 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:15.851452    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snvzs\" (UniqueName: \"kubernetes.io/projected/f7462b94-47fc-4939-873a-b93525e013b3-kube-api-access-snvzs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:38:16.225985 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.225953    2565 generic.go:356] "Generic (PLEG): container finished" podID="f7462b94-47fc-4939-873a-b93525e013b3" containerID="cf3231c1d8c8d90a68ce515cc2f52c561b42d1ac2a90990a507c9d2f4490307e" exitCode=0
Sep 17 18:38:16.226145 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.226002    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"
Sep 17 18:38:16.226145 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.226033    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz" event={"ID":"f7462b94-47fc-4939-873a-b93525e013b3","Type":"ContainerDied","Data":"cf3231c1d8c8d90a68ce515cc2f52c561b42d1ac2a90990a507c9d2f4490307e"}
Sep 17 18:38:16.226145 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.226082    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz" event={"ID":"f7462b94-47fc-4939-873a-b93525e013b3","Type":"ContainerDied","Data":"14236c3599d335917682cb456ed10aaa3f495d4e1e8ca4c40a7c818d3d43fde9"}
Sep 17 18:38:16.226145 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.226103    2565 scope.go:117] "RemoveContainer" containerID="cf3231c1d8c8d90a68ce515cc2f52c561b42d1ac2a90990a507c9d2f4490307e"
Sep 17 18:38:16.235240 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.235225    2565 scope.go:117] "RemoveContainer" containerID="cf3231c1d8c8d90a68ce515cc2f52c561b42d1ac2a90990a507c9d2f4490307e"
Sep 17 18:38:16.235502 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:38:16.235480    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf3231c1d8c8d90a68ce515cc2f52c561b42d1ac2a90990a507c9d2f4490307e\": container with ID starting with cf3231c1d8c8d90a68ce515cc2f52c561b42d1ac2a90990a507c9d2f4490307e not found: ID does not exist" containerID="cf3231c1d8c8d90a68ce515cc2f52c561b42d1ac2a90990a507c9d2f4490307e"
Sep 17 18:38:16.235545 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.235511    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf3231c1d8c8d90a68ce515cc2f52c561b42d1ac2a90990a507c9d2f4490307e"} err="failed to get container status \"cf3231c1d8c8d90a68ce515cc2f52c561b42d1ac2a90990a507c9d2f4490307e\": rpc error: code = NotFound desc = could not find container \"cf3231c1d8c8d90a68ce515cc2f52c561b42d1ac2a90990a507c9d2f4490307e\": container with ID starting with cf3231c1d8c8d90a68ce515cc2f52c561b42d1ac2a90990a507c9d2f4490307e not found: ID does not exist"
Sep 17 18:38:16.249059 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.249038    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"]
Sep 17 18:38:16.257564 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.257540    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6jfbz"]
Sep 17 18:38:16.518184 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.518109    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-69d59d7cdd-5dh64"]
Sep 17 18:38:16.518555 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.518455    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7462b94-47fc-4939-873a-b93525e013b3" containerName="discovery"
Sep 17 18:38:16.518555 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.518467    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7462b94-47fc-4939-873a-b93525e013b3" containerName="discovery"
Sep 17 18:38:16.518555 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.518521    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7462b94-47fc-4939-873a-b93525e013b3" containerName="discovery"
Sep 17 18:38:16.523473 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.523453    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-69d59d7cdd-5dh64"
Sep 17 18:38:16.525950 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.525923    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 17 18:38:16.526080 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.526000    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 18:38:16.526080 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.526030    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-gl68m\""
Sep 17 18:38:16.526178 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.526096    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 18:38:16.533212 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.533178    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-69d59d7cdd-5dh64"]
Sep 17 18:38:16.541172 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.541148    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-5c49f86564-cdnq9"]
Sep 17 18:38:16.544711 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.544692    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-5c49f86564-cdnq9"
Sep 17 18:38:16.546892 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.546868    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-z7gqr\""
Sep 17 18:38:16.546985 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.546868    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 17 18:38:16.562262 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.562232    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-5c49f86564-cdnq9"]
Sep 17 18:38:16.576902 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.576881    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-47bvz"]
Sep 17 18:38:16.580385 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.580364    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-47bvz"
Sep 17 18:38:16.582747 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.582724    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-rdzvv\""
Sep 17 18:38:16.582875 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.582858    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 18:38:16.587745 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.587727    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-47bvz"]
Sep 17 18:38:16.606877 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.606847    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7462b94-47fc-4939-873a-b93525e013b3" path="/var/lib/kubelet/pods/f7462b94-47fc-4939-873a-b93525e013b3/volumes"
Sep 17 18:38:16.658394 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.658367    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c68s\" (UniqueName: \"kubernetes.io/projected/84f28508-030d-404b-a15f-a693c52537d4-kube-api-access-8c68s\") pod \"seaweedfs-64568bcd49-47bvz\" (UID: \"84f28508-030d-404b-a15f-a693c52537d4\") " pod="kserve/seaweedfs-64568bcd49-47bvz"
Sep 17 18:38:16.658495 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.658412    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4672w\" (UniqueName: \"kubernetes.io/projected/908a08ac-5056-40dd-8ff4-1a601e23fe35-kube-api-access-4672w\") pod \"kserve-controller-manager-69d59d7cdd-5dh64\" (UID: \"908a08ac-5056-40dd-8ff4-1a601e23fe35\") " pod="kserve/kserve-controller-manager-69d59d7cdd-5dh64"
Sep 17 18:38:16.658495 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.658452    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/908a08ac-5056-40dd-8ff4-1a601e23fe35-cert\") pod \"kserve-controller-manager-69d59d7cdd-5dh64\" (UID: \"908a08ac-5056-40dd-8ff4-1a601e23fe35\") " pod="kserve/kserve-controller-manager-69d59d7cdd-5dh64"
Sep 17 18:38:16.658495 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.658478    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrmvm\" (UniqueName: \"kubernetes.io/projected/b520ade1-0e10-41c3-813e-c079b247e5eb-kube-api-access-lrmvm\") pod \"llmisvc-controller-manager-5c49f86564-cdnq9\" (UID: \"b520ade1-0e10-41c3-813e-c079b247e5eb\") " pod="kserve/llmisvc-controller-manager-5c49f86564-cdnq9"
Sep 17 18:38:16.658656 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.658563    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b520ade1-0e10-41c3-813e-c079b247e5eb-cert\") pod \"llmisvc-controller-manager-5c49f86564-cdnq9\" (UID: \"b520ade1-0e10-41c3-813e-c079b247e5eb\") " pod="kserve/llmisvc-controller-manager-5c49f86564-cdnq9"
Sep 17 18:38:16.658704 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.658689    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/84f28508-030d-404b-a15f-a693c52537d4-data\") pod \"seaweedfs-64568bcd49-47bvz\" (UID: \"84f28508-030d-404b-a15f-a693c52537d4\") " pod="kserve/seaweedfs-64568bcd49-47bvz"
Sep 17 18:38:16.759434 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.759399    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4672w\" (UniqueName: \"kubernetes.io/projected/908a08ac-5056-40dd-8ff4-1a601e23fe35-kube-api-access-4672w\") pod \"kserve-controller-manager-69d59d7cdd-5dh64\" (UID: \"908a08ac-5056-40dd-8ff4-1a601e23fe35\") " pod="kserve/kserve-controller-manager-69d59d7cdd-5dh64"
Sep 17 18:38:16.759434 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.759440    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/908a08ac-5056-40dd-8ff4-1a601e23fe35-cert\") pod \"kserve-controller-manager-69d59d7cdd-5dh64\" (UID: \"908a08ac-5056-40dd-8ff4-1a601e23fe35\") " pod="kserve/kserve-controller-manager-69d59d7cdd-5dh64"
Sep 17 18:38:16.759566 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.759459    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrmvm\" (UniqueName: \"kubernetes.io/projected/b520ade1-0e10-41c3-813e-c079b247e5eb-kube-api-access-lrmvm\") pod \"llmisvc-controller-manager-5c49f86564-cdnq9\" (UID: \"b520ade1-0e10-41c3-813e-c079b247e5eb\") " pod="kserve/llmisvc-controller-manager-5c49f86564-cdnq9"
Sep 17 18:38:16.759566 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.759496    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b520ade1-0e10-41c3-813e-c079b247e5eb-cert\") pod \"llmisvc-controller-manager-5c49f86564-cdnq9\" (UID: \"b520ade1-0e10-41c3-813e-c079b247e5eb\") " pod="kserve/llmisvc-controller-manager-5c49f86564-cdnq9"
Sep 17 18:38:16.759566 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.759519    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/84f28508-030d-404b-a15f-a693c52537d4-data\") pod \"seaweedfs-64568bcd49-47bvz\" (UID: \"84f28508-030d-404b-a15f-a693c52537d4\") " pod="kserve/seaweedfs-64568bcd49-47bvz"
Sep 17 18:38:16.759708 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.759568    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c68s\" (UniqueName: \"kubernetes.io/projected/84f28508-030d-404b-a15f-a693c52537d4-kube-api-access-8c68s\") pod \"seaweedfs-64568bcd49-47bvz\" (UID: \"84f28508-030d-404b-a15f-a693c52537d4\") " pod="kserve/seaweedfs-64568bcd49-47bvz"
Sep 17 18:38:16.759921 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.759900    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/84f28508-030d-404b-a15f-a693c52537d4-data\") pod \"seaweedfs-64568bcd49-47bvz\" (UID: \"84f28508-030d-404b-a15f-a693c52537d4\") " pod="kserve/seaweedfs-64568bcd49-47bvz"
Sep 17 18:38:16.761905 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.761877    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/908a08ac-5056-40dd-8ff4-1a601e23fe35-cert\") pod \"kserve-controller-manager-69d59d7cdd-5dh64\" (UID: \"908a08ac-5056-40dd-8ff4-1a601e23fe35\") " pod="kserve/kserve-controller-manager-69d59d7cdd-5dh64"
Sep 17 18:38:16.761985 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.761919    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b520ade1-0e10-41c3-813e-c079b247e5eb-cert\") pod \"llmisvc-controller-manager-5c49f86564-cdnq9\" (UID: \"b520ade1-0e10-41c3-813e-c079b247e5eb\") " pod="kserve/llmisvc-controller-manager-5c49f86564-cdnq9"
Sep 17 18:38:16.768586 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.768530    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c68s\" (UniqueName: \"kubernetes.io/projected/84f28508-030d-404b-a15f-a693c52537d4-kube-api-access-8c68s\") pod \"seaweedfs-64568bcd49-47bvz\" (UID: \"84f28508-030d-404b-a15f-a693c52537d4\") " pod="kserve/seaweedfs-64568bcd49-47bvz"
Sep 17 18:38:16.768837 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.768810    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrmvm\" (UniqueName: \"kubernetes.io/projected/b520ade1-0e10-41c3-813e-c079b247e5eb-kube-api-access-lrmvm\") pod \"llmisvc-controller-manager-5c49f86564-cdnq9\" (UID: \"b520ade1-0e10-41c3-813e-c079b247e5eb\") " pod="kserve/llmisvc-controller-manager-5c49f86564-cdnq9"
Sep 17 18:38:16.769419 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.769399    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4672w\" (UniqueName: \"kubernetes.io/projected/908a08ac-5056-40dd-8ff4-1a601e23fe35-kube-api-access-4672w\") pod \"kserve-controller-manager-69d59d7cdd-5dh64\" (UID: \"908a08ac-5056-40dd-8ff4-1a601e23fe35\") " pod="kserve/kserve-controller-manager-69d59d7cdd-5dh64"
Sep 17 18:38:16.834364 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.834342    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-69d59d7cdd-5dh64"
Sep 17 18:38:16.853934 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.853911    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-5c49f86564-cdnq9"
Sep 17 18:38:16.889367 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:16.889086    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-47bvz"
Sep 17 18:38:17.117417 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:17.117388    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-69d59d7cdd-5dh64"]
Sep 17 18:38:17.119746 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:38:17.119716    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908a08ac_5056_40dd_8ff4_1a601e23fe35.slice/crio-75fa907eff61703523eb8e987fc95547054504c85acb3f2a22c36c759c87dff2 WatchSource:0}: Error finding container 75fa907eff61703523eb8e987fc95547054504c85acb3f2a22c36c759c87dff2: Status 404 returned error can't find the container with id 75fa907eff61703523eb8e987fc95547054504c85acb3f2a22c36c759c87dff2
Sep 17 18:38:17.148339 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:17.148317    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-5c49f86564-cdnq9"]
Sep 17 18:38:17.149574 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:38:17.149550    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb520ade1_0e10_41c3_813e_c079b247e5eb.slice/crio-6eee8ab4a0eca9e27c5e549b9d7b011a17076cbb8fc028fa5192f90b6918c638 WatchSource:0}: Error finding container 6eee8ab4a0eca9e27c5e549b9d7b011a17076cbb8fc028fa5192f90b6918c638: Status 404 returned error can't find the container with id 6eee8ab4a0eca9e27c5e549b9d7b011a17076cbb8fc028fa5192f90b6918c638
Sep 17 18:38:17.178403 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:17.178380    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-47bvz"]
Sep 17 18:38:17.179535 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:38:17.179515    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f28508_030d_404b_a15f_a693c52537d4.slice/crio-0a828a3ec36a0dbc158dab17fe5b41743d843a4143c21b14f2d7e15f1727d64e WatchSource:0}: Error finding container 0a828a3ec36a0dbc158dab17fe5b41743d843a4143c21b14f2d7e15f1727d64e: Status 404 returned error can't find the container with id 0a828a3ec36a0dbc158dab17fe5b41743d843a4143c21b14f2d7e15f1727d64e
Sep 17 18:38:17.233123 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:17.233095    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-47bvz" event={"ID":"84f28508-030d-404b-a15f-a693c52537d4","Type":"ContainerStarted","Data":"0a828a3ec36a0dbc158dab17fe5b41743d843a4143c21b14f2d7e15f1727d64e"}
Sep 17 18:38:17.234292 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:17.234269    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-5c49f86564-cdnq9" event={"ID":"b520ade1-0e10-41c3-813e-c079b247e5eb","Type":"ContainerStarted","Data":"6eee8ab4a0eca9e27c5e549b9d7b011a17076cbb8fc028fa5192f90b6918c638"}
Sep 17 18:38:17.235459 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:17.235441    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-69d59d7cdd-5dh64" event={"ID":"908a08ac-5056-40dd-8ff4-1a601e23fe35","Type":"ContainerStarted","Data":"75fa907eff61703523eb8e987fc95547054504c85acb3f2a22c36c759c87dff2"}
Sep 17 18:38:22.269092 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:22.269060    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-47bvz" event={"ID":"84f28508-030d-404b-a15f-a693c52537d4","Type":"ContainerStarted","Data":"b4d501af9b2cf87ad58e68687107a5b05c64e5301323076de7efebbfc54bf1f6"}
Sep 17 18:38:22.269573 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:22.269128    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-47bvz"
Sep 17 18:38:22.270732 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:22.270708    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-5c49f86564-cdnq9" event={"ID":"b520ade1-0e10-41c3-813e-c079b247e5eb","Type":"ContainerStarted","Data":"f4656ab2f06e87faa7d4b8b23f9b5d7af40bbe8976a89105a1b1480417d45f8c"}
Sep 17 18:38:22.270838 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:22.270795    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-5c49f86564-cdnq9"
Sep 17 18:38:22.272180 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:22.272162    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-69d59d7cdd-5dh64" event={"ID":"908a08ac-5056-40dd-8ff4-1a601e23fe35","Type":"ContainerStarted","Data":"8a3cadf0ecad1480e32f5f9b58eaf4ea79a1d6793ecf0efe18cb78c1f8ef0a84"}
Sep 17 18:38:22.272297 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:22.272234    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-69d59d7cdd-5dh64"
Sep 17 18:38:22.287530 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:22.287445    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-47bvz" podStartSLOduration=1.8931185 podStartE2EDuration="6.287426894s" totalImagesPullingTime="4.394308394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:38:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:38:22.28644756 +0000 UTC m=+560.218649080" watchObservedRunningTime="2026-09-17 18:38:22.287426894 +0000 UTC m=+560.219628414"
Sep 17 18:38:22.306948 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:22.306904    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-69d59d7cdd-5dh64" podStartSLOduration=1.984095529 podStartE2EDuration="6.306891739s" totalImagesPullingTime="4.32279621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:38:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:38:22.304994145 +0000 UTC m=+560.237195667" watchObservedRunningTime="2026-09-17 18:38:22.306891739 +0000 UTC m=+560.239093258"
Sep 17 18:38:22.325929 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:22.325888    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-5c49f86564-cdnq9" podStartSLOduration=1.994100967 podStartE2EDuration="6.325874746s" totalImagesPullingTime="4.331773779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:38:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:38:22.323377036 +0000 UTC m=+560.255578547" watchObservedRunningTime="2026-09-17 18:38:22.325874746 +0000 UTC m=+560.258076265"
Sep 17 18:38:28.279755 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:28.279724    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-47bvz"
Sep 17 18:38:53.279219 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:53.279174    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-5c49f86564-cdnq9"
Sep 17 18:38:53.282156 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:38:53.282135    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-69d59d7cdd-5dh64"
Sep 17 18:39:01.046908 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:01.046870    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-69d59d7cdd-5dh64"]
Sep 17 18:39:01.047355 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:01.047066    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-69d59d7cdd-5dh64" podUID="908a08ac-5056-40dd-8ff4-1a601e23fe35" containerName="manager" containerID="cri-o://8a3cadf0ecad1480e32f5f9b58eaf4ea79a1d6793ecf0efe18cb78c1f8ef0a84" gracePeriod=10
Sep 17 18:39:01.072090 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:01.072064    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-69d59d7cdd-5hq8f"]
Sep 17 18:39:01.075358 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:01.075340    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-69d59d7cdd-5hq8f"
Sep 17 18:39:01.083412 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:01.083388    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-69d59d7cdd-5hq8f"]
Sep 17 18:39:01.194688 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:01.194656    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh6pt\" (UniqueName: \"kubernetes.io/projected/c12116ee-ac6f-4448-ab40-3643615402da-kube-api-access-qh6pt\") pod \"kserve-controller-manager-69d59d7cdd-5hq8f\" (UID: \"c12116ee-ac6f-4448-ab40-3643615402da\") " pod="kserve/kserve-controller-manager-69d59d7cdd-5hq8f"
Sep 17 18:39:01.194845 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:01.194741    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c12116ee-ac6f-4448-ab40-3643615402da-cert\") pod \"kserve-controller-manager-69d59d7cdd-5hq8f\" (UID: \"c12116ee-ac6f-4448-ab40-3643615402da\") " pod="kserve/kserve-controller-manager-69d59d7cdd-5hq8f"
Sep 17 18:39:01.289815 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:01.289791    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-69d59d7cdd-5dh64"
Sep 17 18:39:01.295512 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:01.295480    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c12116ee-ac6f-4448-ab40-3643615402da-cert\") pod \"kserve-controller-manager-69d59d7cdd-5hq8f\" (UID: \"c12116ee-ac6f-4448-ab40-3643615402da\") " pod="kserve/kserve-controller-manager-69d59d7cdd-5hq8f"
Sep 17 18:39:01.295571 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:01.295559    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh6pt\" (UniqueName: \"kubernetes.io/projected/c12116ee-ac6f-4448-ab40-3643615402da-kube-api-access-qh6pt\") pod \"kserve-controller-manager-69d59d7cdd-5hq8f\" (UID: \"c12116ee-ac6f-4448-ab40-3643615402da\") " pod="kserve/kserve-controller-manager-69d59d7cdd-5hq8f"
Sep 17 18:39:01.297622 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:01.297573    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c12116ee-ac6f-4448-ab40-3643615402da-cert\") pod \"kserve-controller-manager-69d59d7cdd-5hq8f\" (UID: \"c12116ee-ac6f-4448-ab40-3643615402da\") " pod="kserve/kserve-controller-manager-69d59d7cdd-5hq8f"
Sep 17 18:39:01.303864 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:01.303839    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh6pt\" (UniqueName: \"kubernetes.io/projected/c12116ee-ac6f-4448-ab40-3643615402da-kube-api-access-qh6pt\") pod \"kserve-controller-manager-69d59d7cdd-5hq8f\" (UID: \"c12116ee-ac6f-4448-ab40-3643615402da\") " pod="kserve/kserve-controller-manager-69d59d7cdd-5hq8f"
Sep 17 18:39:01.396164 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:01.396138    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/908a08ac-5056-40dd-8ff4-1a601e23fe35-cert\") pod \"908a08ac-5056-40dd-8ff4-1a601e23fe35\" (UID: \"908a08ac-5056-40dd-8ff4-1a601e23fe35\") "
Sep 17 18:39:01.396333 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:01.396182    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4672w\" (UniqueName: \"kubernetes.io/projected/908a08ac-5056-40dd-8ff4-1a601e23fe35-kube-api-access-4672w\") pod \"908a08ac-5056-40dd-8ff4-1a601e23fe35\" (UID: \"908a08ac-5056-40dd-8ff4-1a601e23fe35\") "
Sep 17 18:39:01.398237 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:01.398193    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908a08ac-5056-40dd-8ff4-1a601e23fe35-kube-api-access-4672w" (OuterVolumeSpecName: "kube-api-access-4672w") pod "908a08ac-5056-40dd-8ff4-1a601e23fe35" (UID: "908a08ac-5056-40dd-8ff4-1a601e23fe35"). InnerVolumeSpecName "kube-api-access-4672w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:39:01.398350 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:01.398213    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/908a08ac-5056-40dd-8ff4-1a601e23fe35-cert" (OuterVolumeSpecName: "cert") pod "908a08ac-5056-40dd-8ff4-1a601e23fe35" (UID: "908a08ac-5056-40dd-8ff4-1a601e23fe35"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:39:01.431595 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:01.431578    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-69d59d7cdd-5hq8f"
Sep 17 18:39:01.477839 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:01.477575    2565 generic.go:356] "Generic (PLEG): container finished" podID="908a08ac-5056-40dd-8ff4-1a601e23fe35" containerID="8a3cadf0ecad1480e32f5f9b58eaf4ea79a1d6793ecf0efe18cb78c1f8ef0a84" exitCode=0
Sep 17 18:39:01.477839 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:01.477731    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-69d59d7cdd-5dh64" event={"ID":"908a08ac-5056-40dd-8ff4-1a601e23fe35","Type":"ContainerDied","Data":"8a3cadf0ecad1480e32f5f9b58eaf4ea79a1d6793ecf0efe18cb78c1f8ef0a84"}
Sep 17 18:39:01.477839 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:01.477816    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-69d59d7cdd-5dh64" event={"ID":"908a08ac-5056-40dd-8ff4-1a601e23fe35","Type":"ContainerDied","Data":"75fa907eff61703523eb8e987fc95547054504c85acb3f2a22c36c759c87dff2"}
Sep 17 18:39:01.478029 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:01.477877    2565 scope.go:117] "RemoveContainer" containerID="8a3cadf0ecad1480e32f5f9b58eaf4ea79a1d6793ecf0efe18cb78c1f8ef0a84"
Sep 17 18:39:01.478029 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:01.477886    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-69d59d7cdd-5dh64"
Sep 17 18:39:01.496471 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:01.496449    2565 scope.go:117] "RemoveContainer" containerID="8a3cadf0ecad1480e32f5f9b58eaf4ea79a1d6793ecf0efe18cb78c1f8ef0a84"
Sep 17 18:39:01.496789 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:39:01.496758    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a3cadf0ecad1480e32f5f9b58eaf4ea79a1d6793ecf0efe18cb78c1f8ef0a84\": container with ID starting with 8a3cadf0ecad1480e32f5f9b58eaf4ea79a1d6793ecf0efe18cb78c1f8ef0a84 not found: ID does not exist" containerID="8a3cadf0ecad1480e32f5f9b58eaf4ea79a1d6793ecf0efe18cb78c1f8ef0a84"
Sep 17 18:39:01.496789 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:01.496781    2565 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/908a08ac-5056-40dd-8ff4-1a601e23fe35-cert\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:39:01.496902 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:01.496800    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4672w\" (UniqueName: \"kubernetes.io/projected/908a08ac-5056-40dd-8ff4-1a601e23fe35-kube-api-access-4672w\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:39:01.496902 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:01.496791    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a3cadf0ecad1480e32f5f9b58eaf4ea79a1d6793ecf0efe18cb78c1f8ef0a84"} err="failed to get container status \"8a3cadf0ecad1480e32f5f9b58eaf4ea79a1d6793ecf0efe18cb78c1f8ef0a84\": rpc error: code = NotFound desc = could not find container \"8a3cadf0ecad1480e32f5f9b58eaf4ea79a1d6793ecf0efe18cb78c1f8ef0a84\": container with ID starting with 8a3cadf0ecad1480e32f5f9b58eaf4ea79a1d6793ecf0efe18cb78c1f8ef0a84 not found: ID does not exist"
Sep 17 18:39:01.505642 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:01.505616    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-69d59d7cdd-5dh64"]
Sep 17 18:39:01.508905 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:01.508885    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-69d59d7cdd-5dh64"]
Sep 17 18:39:01.700672 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:01.700651    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-69d59d7cdd-5hq8f"]
Sep 17 18:39:01.702429 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:39:01.702403    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc12116ee_ac6f_4448_ab40_3643615402da.slice/crio-323cbafe536d15bd321a9efd44e50646c7d78e3eb1a42d39e79ac1e6f77425e3 WatchSource:0}: Error finding container 323cbafe536d15bd321a9efd44e50646c7d78e3eb1a42d39e79ac1e6f77425e3: Status 404 returned error can't find the container with id 323cbafe536d15bd321a9efd44e50646c7d78e3eb1a42d39e79ac1e6f77425e3
Sep 17 18:39:02.484722 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:02.484685    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-69d59d7cdd-5hq8f" event={"ID":"c12116ee-ac6f-4448-ab40-3643615402da","Type":"ContainerStarted","Data":"8f4a0c515c6d4238c2b1798e0e4d94b6154c7fe548d3d8cc0a842e9f4da7272c"}
Sep 17 18:39:02.485184 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:02.484751    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-69d59d7cdd-5hq8f"
Sep 17 18:39:02.485184 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:02.484779    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-69d59d7cdd-5hq8f" event={"ID":"c12116ee-ac6f-4448-ab40-3643615402da","Type":"ContainerStarted","Data":"323cbafe536d15bd321a9efd44e50646c7d78e3eb1a42d39e79ac1e6f77425e3"}
Sep 17 18:39:02.507158 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:02.507108    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-69d59d7cdd-5hq8f" podStartSLOduration=1.119928942 podStartE2EDuration="1.507088939s" totalImagesPullingTime="387.159997ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:02.503747126 +0000 UTC m=+600.435948647" watchObservedRunningTime="2026-09-17 18:39:02.507088939 +0000 UTC m=+600.439290465"
Sep 17 18:39:02.606455 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:02.606430    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="908a08ac-5056-40dd-8ff4-1a601e23fe35" path="/var/lib/kubelet/pods/908a08ac-5056-40dd-8ff4-1a601e23fe35/volumes"
Sep 17 18:39:33.497154 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:33.497067    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-69d59d7cdd-5hq8f"
Sep 17 18:39:34.786826 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:34.786795    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-f7d8g"]
Sep 17 18:39:34.787190 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:34.787143    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="908a08ac-5056-40dd-8ff4-1a601e23fe35" containerName="manager"
Sep 17 18:39:34.787190 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:34.787155    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="908a08ac-5056-40dd-8ff4-1a601e23fe35" containerName="manager"
Sep 17 18:39:34.787292 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:34.787241    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="908a08ac-5056-40dd-8ff4-1a601e23fe35" containerName="manager"
Sep 17 18:39:34.790346 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:34.790323    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-f7d8g"
Sep 17 18:39:34.793697 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:34.793677    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-68tfm\""
Sep 17 18:39:34.793781 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:34.793680    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 18:39:34.804017 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:34.803992    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-f7d8g"]
Sep 17 18:39:34.805941 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:34.805918    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-zgn4g"]
Sep 17 18:39:34.809133 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:34.809118    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-zgn4g"
Sep 17 18:39:34.813110 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:34.813090    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 18:39:34.813276 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:34.813253    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-zh7kq\""
Sep 17 18:39:34.813357 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:34.813307    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 18:39:34.821179 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:34.821157    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-zgn4g"]
Sep 17 18:39:34.830583 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:34.830557    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50a1c0f1-dd09-416b-be08-875f33721b5e-cert\") pod \"odh-model-controller-7b966d7984-zgn4g\" (UID: \"50a1c0f1-dd09-416b-be08-875f33721b5e\") " pod="kserve/odh-model-controller-7b966d7984-zgn4g"
Sep 17 18:39:34.830680 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:34.830591    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfzj6\" (UniqueName: \"kubernetes.io/projected/50a1c0f1-dd09-416b-be08-875f33721b5e-kube-api-access-zfzj6\") pod \"odh-model-controller-7b966d7984-zgn4g\" (UID: \"50a1c0f1-dd09-416b-be08-875f33721b5e\") " pod="kserve/odh-model-controller-7b966d7984-zgn4g"
Sep 17 18:39:34.830680 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:34.830641    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50a1c0f1-dd09-416b-be08-875f33721b5e-metrics-certs\") pod \"odh-model-controller-7b966d7984-zgn4g\" (UID: \"50a1c0f1-dd09-416b-be08-875f33721b5e\") " pod="kserve/odh-model-controller-7b966d7984-zgn4g"
Sep 17 18:39:34.830795 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:34.830679    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x568\" (UniqueName: \"kubernetes.io/projected/b48875aa-e67f-4c77-8e0d-aa2051d34499-kube-api-access-4x568\") pod \"model-serving-api-7b954ffc95-f7d8g\" (UID: \"b48875aa-e67f-4c77-8e0d-aa2051d34499\") " pod="kserve/model-serving-api-7b954ffc95-f7d8g"
Sep 17 18:39:34.830795 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:34.830699    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b48875aa-e67f-4c77-8e0d-aa2051d34499-tls-certs\") pod \"model-serving-api-7b954ffc95-f7d8g\" (UID: \"b48875aa-e67f-4c77-8e0d-aa2051d34499\") " pod="kserve/model-serving-api-7b954ffc95-f7d8g"
Sep 17 18:39:34.931492 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:34.931463    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50a1c0f1-dd09-416b-be08-875f33721b5e-metrics-certs\") pod \"odh-model-controller-7b966d7984-zgn4g\" (UID: \"50a1c0f1-dd09-416b-be08-875f33721b5e\") " pod="kserve/odh-model-controller-7b966d7984-zgn4g"
Sep 17 18:39:34.931618 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:34.931505    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x568\" (UniqueName: \"kubernetes.io/projected/b48875aa-e67f-4c77-8e0d-aa2051d34499-kube-api-access-4x568\") pod \"model-serving-api-7b954ffc95-f7d8g\" (UID: \"b48875aa-e67f-4c77-8e0d-aa2051d34499\") " pod="kserve/model-serving-api-7b954ffc95-f7d8g"
Sep 17 18:39:34.931618 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:34.931540    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b48875aa-e67f-4c77-8e0d-aa2051d34499-tls-certs\") pod \"model-serving-api-7b954ffc95-f7d8g\" (UID: \"b48875aa-e67f-4c77-8e0d-aa2051d34499\") " pod="kserve/model-serving-api-7b954ffc95-f7d8g"
Sep 17 18:39:34.931618 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:34.931581    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50a1c0f1-dd09-416b-be08-875f33721b5e-cert\") pod \"odh-model-controller-7b966d7984-zgn4g\" (UID: \"50a1c0f1-dd09-416b-be08-875f33721b5e\") " pod="kserve/odh-model-controller-7b966d7984-zgn4g"
Sep 17 18:39:34.931618 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:39:34.931598    2565 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 17 18:39:34.931851 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:39:34.931668    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50a1c0f1-dd09-416b-be08-875f33721b5e-metrics-certs podName:50a1c0f1-dd09-416b-be08-875f33721b5e nodeName:}" failed. No retries permitted until 2026-09-17 18:39:35.431653236 +0000 UTC m=+633.363854733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50a1c0f1-dd09-416b-be08-875f33721b5e-metrics-certs") pod "odh-model-controller-7b966d7984-zgn4g" (UID: "50a1c0f1-dd09-416b-be08-875f33721b5e") : secret "odh-model-controller-metrics-tls" not found
Sep 17 18:39:34.931851 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:39:34.931670    2565 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 17 18:39:34.931851 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:34.931603    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfzj6\" (UniqueName: \"kubernetes.io/projected/50a1c0f1-dd09-416b-be08-875f33721b5e-kube-api-access-zfzj6\") pod \"odh-model-controller-7b966d7984-zgn4g\" (UID: \"50a1c0f1-dd09-416b-be08-875f33721b5e\") " pod="kserve/odh-model-controller-7b966d7984-zgn4g"
Sep 17 18:39:34.931851 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:39:34.931669    2565 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 17 18:39:34.931851 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:39:34.931739    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b48875aa-e67f-4c77-8e0d-aa2051d34499-tls-certs podName:b48875aa-e67f-4c77-8e0d-aa2051d34499 nodeName:}" failed. No retries permitted until 2026-09-17 18:39:35.43172237 +0000 UTC m=+633.363923885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b48875aa-e67f-4c77-8e0d-aa2051d34499-tls-certs") pod "model-serving-api-7b954ffc95-f7d8g" (UID: "b48875aa-e67f-4c77-8e0d-aa2051d34499") : secret "model-serving-api-tls" not found
Sep 17 18:39:34.931851 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:39:34.931767    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50a1c0f1-dd09-416b-be08-875f33721b5e-cert podName:50a1c0f1-dd09-416b-be08-875f33721b5e nodeName:}" failed. No retries permitted until 2026-09-17 18:39:35.431750774 +0000 UTC m=+633.363952343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/50a1c0f1-dd09-416b-be08-875f33721b5e-cert") pod "odh-model-controller-7b966d7984-zgn4g" (UID: "50a1c0f1-dd09-416b-be08-875f33721b5e") : secret "odh-model-controller-webhook-cert" not found
Sep 17 18:39:34.943059 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:34.943036    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x568\" (UniqueName: \"kubernetes.io/projected/b48875aa-e67f-4c77-8e0d-aa2051d34499-kube-api-access-4x568\") pod \"model-serving-api-7b954ffc95-f7d8g\" (UID: \"b48875aa-e67f-4c77-8e0d-aa2051d34499\") " pod="kserve/model-serving-api-7b954ffc95-f7d8g"
Sep 17 18:39:34.945630 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:34.945608    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfzj6\" (UniqueName: \"kubernetes.io/projected/50a1c0f1-dd09-416b-be08-875f33721b5e-kube-api-access-zfzj6\") pod \"odh-model-controller-7b966d7984-zgn4g\" (UID: \"50a1c0f1-dd09-416b-be08-875f33721b5e\") " pod="kserve/odh-model-controller-7b966d7984-zgn4g"
Sep 17 18:39:35.434397 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:35.434365    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50a1c0f1-dd09-416b-be08-875f33721b5e-cert\") pod \"odh-model-controller-7b966d7984-zgn4g\" (UID: \"50a1c0f1-dd09-416b-be08-875f33721b5e\") " pod="kserve/odh-model-controller-7b966d7984-zgn4g"
Sep 17 18:39:35.434588 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:35.434417    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50a1c0f1-dd09-416b-be08-875f33721b5e-metrics-certs\") pod \"odh-model-controller-7b966d7984-zgn4g\" (UID: \"50a1c0f1-dd09-416b-be08-875f33721b5e\") " pod="kserve/odh-model-controller-7b966d7984-zgn4g"
Sep 17 18:39:35.434588 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:35.434532    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b48875aa-e67f-4c77-8e0d-aa2051d34499-tls-certs\") pod \"model-serving-api-7b954ffc95-f7d8g\" (UID: \"b48875aa-e67f-4c77-8e0d-aa2051d34499\") " pod="kserve/model-serving-api-7b954ffc95-f7d8g"
Sep 17 18:39:35.436831 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:35.436801    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50a1c0f1-dd09-416b-be08-875f33721b5e-cert\") pod \"odh-model-controller-7b966d7984-zgn4g\" (UID: \"50a1c0f1-dd09-416b-be08-875f33721b5e\") " pod="kserve/odh-model-controller-7b966d7984-zgn4g"
Sep 17 18:39:35.436935 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:35.436854    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b48875aa-e67f-4c77-8e0d-aa2051d34499-tls-certs\") pod \"model-serving-api-7b954ffc95-f7d8g\" (UID: \"b48875aa-e67f-4c77-8e0d-aa2051d34499\") " pod="kserve/model-serving-api-7b954ffc95-f7d8g"
Sep 17 18:39:35.436935 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:35.436910    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50a1c0f1-dd09-416b-be08-875f33721b5e-metrics-certs\") pod \"odh-model-controller-7b966d7984-zgn4g\" (UID: \"50a1c0f1-dd09-416b-be08-875f33721b5e\") " pod="kserve/odh-model-controller-7b966d7984-zgn4g"
Sep 17 18:39:35.700468 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:35.700407    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-f7d8g"
Sep 17 18:39:35.718370 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:35.718344    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-zgn4g"
Sep 17 18:39:35.978082 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:35.978052    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-f7d8g"]
Sep 17 18:39:35.979469 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:39:35.979446    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb48875aa_e67f_4c77_8e0d_aa2051d34499.slice/crio-f39fce71560bfa34ccd3744e0a008b9905b8d47062d712664b55ae108506f0a5 WatchSource:0}: Error finding container f39fce71560bfa34ccd3744e0a008b9905b8d47062d712664b55ae108506f0a5: Status 404 returned error can't find the container with id f39fce71560bfa34ccd3744e0a008b9905b8d47062d712664b55ae108506f0a5
Sep 17 18:39:35.997998 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:35.997976    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-zgn4g"]
Sep 17 18:39:35.999427 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:39:35.999405    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a1c0f1_dd09_416b_be08_875f33721b5e.slice/crio-1a91e5b077470ab0e98f81f94b6e922dc9076109f5496a91db2af3d865e93c91 WatchSource:0}: Error finding container 1a91e5b077470ab0e98f81f94b6e922dc9076109f5496a91db2af3d865e93c91: Status 404 returned error can't find the container with id 1a91e5b077470ab0e98f81f94b6e922dc9076109f5496a91db2af3d865e93c91
Sep 17 18:39:36.661684 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:36.661639    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-zgn4g" event={"ID":"50a1c0f1-dd09-416b-be08-875f33721b5e","Type":"ContainerStarted","Data":"1a91e5b077470ab0e98f81f94b6e922dc9076109f5496a91db2af3d865e93c91"}
Sep 17 18:39:36.663295 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:36.663222    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-f7d8g" event={"ID":"b48875aa-e67f-4c77-8e0d-aa2051d34499","Type":"ContainerStarted","Data":"f39fce71560bfa34ccd3744e0a008b9905b8d47062d712664b55ae108506f0a5"}
Sep 17 18:39:40.688619 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:40.688574    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-zgn4g" event={"ID":"50a1c0f1-dd09-416b-be08-875f33721b5e","Type":"ContainerStarted","Data":"a13e0e924560bb72a6ea522c9b8cc79cc449e846dbbc8876f2275b1048eba526"}
Sep 17 18:39:40.689071 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:40.688689    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-zgn4g"
Sep 17 18:39:40.690344 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:40.690318    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-f7d8g" event={"ID":"b48875aa-e67f-4c77-8e0d-aa2051d34499","Type":"ContainerStarted","Data":"2f179c928c3f1f7ce79ea82f5f42c1211b8f1c4ac2f694334ac5af659199e19e"}
Sep 17 18:39:40.690476 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:40.690367    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-f7d8g"
Sep 17 18:39:40.710279 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:40.710230    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-zgn4g" podStartSLOduration=3.079480913 podStartE2EDuration="6.710214627s" totalImagesPullingTime="3.630733714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:40.709126779 +0000 UTC m=+638.641328299" watchObservedRunningTime="2026-09-17 18:39:40.710214627 +0000 UTC m=+638.642416137"
Sep 17 18:39:40.743134 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:40.743089    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-f7d8g" podStartSLOduration=3.145725402 podStartE2EDuration="6.743071616s" totalImagesPullingTime="3.597346214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:39:40.741382717 +0000 UTC m=+638.673584239" watchObservedRunningTime="2026-09-17 18:39:40.743071616 +0000 UTC m=+638.675273133"
Sep 17 18:39:51.697970 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:51.697939    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-zgn4g"
Sep 17 18:39:51.699700 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:51.699678    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-f7d8g"
Sep 17 18:39:52.531550 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:52.531515    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-bp7px"]
Sep 17 18:39:52.535262 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:52.535236    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bp7px"
Sep 17 18:39:52.537853 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:52.537830    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 18:39:52.550091 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:52.550064    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-bp7px"]
Sep 17 18:39:52.672936 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:52.672906    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38583224-6640-45bf-9a02-1dc45f709cb1-model-cache\") pod \"s3-init-bp7px\" (UID: \"38583224-6640-45bf-9a02-1dc45f709cb1\") " pod="kserve/s3-init-bp7px"
Sep 17 18:39:52.673086 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:52.672949    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rnnf\" (UniqueName: \"kubernetes.io/projected/38583224-6640-45bf-9a02-1dc45f709cb1-kube-api-access-2rnnf\") pod \"s3-init-bp7px\" (UID: \"38583224-6640-45bf-9a02-1dc45f709cb1\") " pod="kserve/s3-init-bp7px"
Sep 17 18:39:52.773631 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:52.773604    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38583224-6640-45bf-9a02-1dc45f709cb1-model-cache\") pod \"s3-init-bp7px\" (UID: \"38583224-6640-45bf-9a02-1dc45f709cb1\") " pod="kserve/s3-init-bp7px"
Sep 17 18:39:52.773955 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:52.773646    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rnnf\" (UniqueName: \"kubernetes.io/projected/38583224-6640-45bf-9a02-1dc45f709cb1-kube-api-access-2rnnf\") pod \"s3-init-bp7px\" (UID: \"38583224-6640-45bf-9a02-1dc45f709cb1\") " pod="kserve/s3-init-bp7px"
Sep 17 18:39:52.773955 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:52.773936    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38583224-6640-45bf-9a02-1dc45f709cb1-model-cache\") pod \"s3-init-bp7px\" (UID: \"38583224-6640-45bf-9a02-1dc45f709cb1\") " pod="kserve/s3-init-bp7px"
Sep 17 18:39:52.782801 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:52.782755    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rnnf\" (UniqueName: \"kubernetes.io/projected/38583224-6640-45bf-9a02-1dc45f709cb1-kube-api-access-2rnnf\") pod \"s3-init-bp7px\" (UID: \"38583224-6640-45bf-9a02-1dc45f709cb1\") " pod="kserve/s3-init-bp7px"
Sep 17 18:39:52.844483 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:52.844456    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bp7px"
Sep 17 18:39:53.118450 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:53.118424    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-bp7px"]
Sep 17 18:39:53.120243 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:39:53.120213    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38583224_6640_45bf_9a02_1dc45f709cb1.slice/crio-49d38f18274f551b9647b0e76468588d263916bba8ce95e592fc589ecabb6e8a WatchSource:0}: Error finding container 49d38f18274f551b9647b0e76468588d263916bba8ce95e592fc589ecabb6e8a: Status 404 returned error can't find the container with id 49d38f18274f551b9647b0e76468588d263916bba8ce95e592fc589ecabb6e8a
Sep 17 18:39:53.757447 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:53.757413    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bp7px" event={"ID":"38583224-6640-45bf-9a02-1dc45f709cb1","Type":"ContainerStarted","Data":"49d38f18274f551b9647b0e76468588d263916bba8ce95e592fc589ecabb6e8a"}
Sep 17 18:39:56.775498 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:56.775465    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bp7px" event={"ID":"38583224-6640-45bf-9a02-1dc45f709cb1","Type":"ContainerStarted","Data":"73f77eb1dcb29c37d2384c607a21d32034e1925a7dfda128a51cd8f581b40d4b"}
Sep 17 18:39:58.793170 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:58.793134    2565 generic.go:356] "Generic (PLEG): container finished" podID="38583224-6640-45bf-9a02-1dc45f709cb1" containerID="73f77eb1dcb29c37d2384c607a21d32034e1925a7dfda128a51cd8f581b40d4b" exitCode=0
Sep 17 18:39:58.793615 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:58.793179    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bp7px" event={"ID":"38583224-6640-45bf-9a02-1dc45f709cb1","Type":"ContainerDied","Data":"73f77eb1dcb29c37d2384c607a21d32034e1925a7dfda128a51cd8f581b40d4b"}
Sep 17 18:39:58.794587 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:39:58.794260    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:40:03.826209 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:40:03.826164    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bp7px" event={"ID":"38583224-6640-45bf-9a02-1dc45f709cb1","Type":"ContainerStarted","Data":"f2e000399d9c86f7ed3ec9ac557803b9678fe6100d587fcf6e796ada72fac4d0"}
Sep 17 18:40:03.850341 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:40:03.850290    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-bp7px" podStartSLOduration=2.639773265 podStartE2EDuration="11.850274409s" totalImagesPullingTime="7.420162495s" totalInitContainerRuntime="1.790338649s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:03.848547079 +0000 UTC m=+661.780748597" watchObservedRunningTime="2026-09-17 18:40:03.850274409 +0000 UTC m=+661.782475928"
Sep 17 18:41:08.164721 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:08.164689    2565 generic.go:356] "Generic (PLEG): container finished" podID="38583224-6640-45bf-9a02-1dc45f709cb1" containerID="f2e000399d9c86f7ed3ec9ac557803b9678fe6100d587fcf6e796ada72fac4d0" exitCode=0
Sep 17 18:41:08.165070 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:08.164758    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bp7px" event={"ID":"38583224-6640-45bf-9a02-1dc45f709cb1","Type":"ContainerDied","Data":"f2e000399d9c86f7ed3ec9ac557803b9678fe6100d587fcf6e796ada72fac4d0"}
Sep 17 18:41:09.313264 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:09.313241    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bp7px"
Sep 17 18:41:09.340301 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:09.340275    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38583224-6640-45bf-9a02-1dc45f709cb1-model-cache\") pod \"38583224-6640-45bf-9a02-1dc45f709cb1\" (UID: \"38583224-6640-45bf-9a02-1dc45f709cb1\") "
Sep 17 18:41:09.340417 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:09.340350    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rnnf\" (UniqueName: \"kubernetes.io/projected/38583224-6640-45bf-9a02-1dc45f709cb1-kube-api-access-2rnnf\") pod \"38583224-6640-45bf-9a02-1dc45f709cb1\" (UID: \"38583224-6640-45bf-9a02-1dc45f709cb1\") "
Sep 17 18:41:09.342335 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:09.342304    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38583224-6640-45bf-9a02-1dc45f709cb1-kube-api-access-2rnnf" (OuterVolumeSpecName: "kube-api-access-2rnnf") pod "38583224-6640-45bf-9a02-1dc45f709cb1" (UID: "38583224-6640-45bf-9a02-1dc45f709cb1"). InnerVolumeSpecName "kube-api-access-2rnnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:41:09.360128 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:09.360100    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38583224-6640-45bf-9a02-1dc45f709cb1-model-cache" (OuterVolumeSpecName: "model-cache") pod "38583224-6640-45bf-9a02-1dc45f709cb1" (UID: "38583224-6640-45bf-9a02-1dc45f709cb1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:09.441577 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:09.441525    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rnnf\" (UniqueName: \"kubernetes.io/projected/38583224-6640-45bf-9a02-1dc45f709cb1-kube-api-access-2rnnf\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:41:09.441577 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:09.441551    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38583224-6640-45bf-9a02-1dc45f709cb1-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:41:10.177033 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:10.177009    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bp7px"
Sep 17 18:41:10.177221 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:10.176999    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bp7px" event={"ID":"38583224-6640-45bf-9a02-1dc45f709cb1","Type":"ContainerDied","Data":"49d38f18274f551b9647b0e76468588d263916bba8ce95e592fc589ecabb6e8a"}
Sep 17 18:41:10.177221 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:10.177128    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49d38f18274f551b9647b0e76468588d263916bba8ce95e592fc589ecabb6e8a"
Sep 17 18:41:31.401012 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.400966    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"]
Sep 17 18:41:31.401607 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.401508    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38583224-6640-45bf-9a02-1dc45f709cb1" containerName="download-hf-model"
Sep 17 18:41:31.401607 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.401530    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="38583224-6640-45bf-9a02-1dc45f709cb1" containerName="download-hf-model"
Sep 17 18:41:31.401607 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.401568    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38583224-6640-45bf-9a02-1dc45f709cb1" containerName="s3-init"
Sep 17 18:41:31.401607 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.401577    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="38583224-6640-45bf-9a02-1dc45f709cb1" containerName="s3-init"
Sep 17 18:41:31.401819 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.401659    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="38583224-6640-45bf-9a02-1dc45f709cb1" containerName="s3-init"
Sep 17 18:41:31.404868 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.404848    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"
Sep 17 18:41:31.408181 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.408135    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 17 18:41:31.408321 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.408238    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 17 18:41:31.408321 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.408254    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 17 18:41:31.408321 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.408284    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-dgs77\""
Sep 17 18:41:31.419486 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.419465    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"]
Sep 17 18:41:31.494508 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.494483    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61f4f17b-1cde-4c4a-9cea-7915ff41c348-dshm\") pod \"fc-smoke-test-kserve-7f4687897-555qt\" (UID: \"61f4f17b-1cde-4c4a-9cea-7915ff41c348\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"
Sep 17 18:41:31.494614 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.494533    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61f4f17b-1cde-4c4a-9cea-7915ff41c348-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-555qt\" (UID: \"61f4f17b-1cde-4c4a-9cea-7915ff41c348\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"
Sep 17 18:41:31.494614 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.494604    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61f4f17b-1cde-4c4a-9cea-7915ff41c348-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-555qt\" (UID: \"61f4f17b-1cde-4c4a-9cea-7915ff41c348\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"
Sep 17 18:41:31.494706 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.494633    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpm2v\" (UniqueName: \"kubernetes.io/projected/61f4f17b-1cde-4c4a-9cea-7915ff41c348-kube-api-access-bpm2v\") pod \"fc-smoke-test-kserve-7f4687897-555qt\" (UID: \"61f4f17b-1cde-4c4a-9cea-7915ff41c348\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"
Sep 17 18:41:31.494706 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.494661    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61f4f17b-1cde-4c4a-9cea-7915ff41c348-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-555qt\" (UID: \"61f4f17b-1cde-4c4a-9cea-7915ff41c348\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"
Sep 17 18:41:31.494706 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.494679    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61f4f17b-1cde-4c4a-9cea-7915ff41c348-home\") pod \"fc-smoke-test-kserve-7f4687897-555qt\" (UID: \"61f4f17b-1cde-4c4a-9cea-7915ff41c348\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"
Sep 17 18:41:31.595960 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.595938    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61f4f17b-1cde-4c4a-9cea-7915ff41c348-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-555qt\" (UID: \"61f4f17b-1cde-4c4a-9cea-7915ff41c348\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"
Sep 17 18:41:31.596071 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.595965    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpm2v\" (UniqueName: \"kubernetes.io/projected/61f4f17b-1cde-4c4a-9cea-7915ff41c348-kube-api-access-bpm2v\") pod \"fc-smoke-test-kserve-7f4687897-555qt\" (UID: \"61f4f17b-1cde-4c4a-9cea-7915ff41c348\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"
Sep 17 18:41:31.596071 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.595990    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61f4f17b-1cde-4c4a-9cea-7915ff41c348-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-555qt\" (UID: \"61f4f17b-1cde-4c4a-9cea-7915ff41c348\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"
Sep 17 18:41:31.596071 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.596008    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61f4f17b-1cde-4c4a-9cea-7915ff41c348-home\") pod \"fc-smoke-test-kserve-7f4687897-555qt\" (UID: \"61f4f17b-1cde-4c4a-9cea-7915ff41c348\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"
Sep 17 18:41:31.596071 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.596038    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61f4f17b-1cde-4c4a-9cea-7915ff41c348-dshm\") pod \"fc-smoke-test-kserve-7f4687897-555qt\" (UID: \"61f4f17b-1cde-4c4a-9cea-7915ff41c348\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"
Sep 17 18:41:31.596347 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.596085    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61f4f17b-1cde-4c4a-9cea-7915ff41c348-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-555qt\" (UID: \"61f4f17b-1cde-4c4a-9cea-7915ff41c348\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"
Sep 17 18:41:31.596419 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.596396    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61f4f17b-1cde-4c4a-9cea-7915ff41c348-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-555qt\" (UID: \"61f4f17b-1cde-4c4a-9cea-7915ff41c348\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"
Sep 17 18:41:31.596477 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.596406    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61f4f17b-1cde-4c4a-9cea-7915ff41c348-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-555qt\" (UID: \"61f4f17b-1cde-4c4a-9cea-7915ff41c348\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"
Sep 17 18:41:31.596477 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.596462    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61f4f17b-1cde-4c4a-9cea-7915ff41c348-home\") pod \"fc-smoke-test-kserve-7f4687897-555qt\" (UID: \"61f4f17b-1cde-4c4a-9cea-7915ff41c348\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"
Sep 17 18:41:31.598393 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.598372    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61f4f17b-1cde-4c4a-9cea-7915ff41c348-dshm\") pod \"fc-smoke-test-kserve-7f4687897-555qt\" (UID: \"61f4f17b-1cde-4c4a-9cea-7915ff41c348\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"
Sep 17 18:41:31.598549 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.598532    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61f4f17b-1cde-4c4a-9cea-7915ff41c348-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-555qt\" (UID: \"61f4f17b-1cde-4c4a-9cea-7915ff41c348\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"
Sep 17 18:41:31.606502 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.606482    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpm2v\" (UniqueName: \"kubernetes.io/projected/61f4f17b-1cde-4c4a-9cea-7915ff41c348-kube-api-access-bpm2v\") pod \"fc-smoke-test-kserve-7f4687897-555qt\" (UID: \"61f4f17b-1cde-4c4a-9cea-7915ff41c348\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"
Sep 17 18:41:31.714963 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.714944    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"
Sep 17 18:41:31.990580 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:31.990554    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"]
Sep 17 18:41:31.992630 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:41:31.992604    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f4f17b_1cde_4c4a_9cea_7915ff41c348.slice/crio-9220386c1f5384d85bf993f4c448596c1efb5595f9818b65eb9280f7aa3dd8b4 WatchSource:0}: Error finding container 9220386c1f5384d85bf993f4c448596c1efb5595f9818b65eb9280f7aa3dd8b4: Status 404 returned error can't find the container with id 9220386c1f5384d85bf993f4c448596c1efb5595f9818b65eb9280f7aa3dd8b4
Sep 17 18:41:32.290677 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:32.290596    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt" event={"ID":"61f4f17b-1cde-4c4a-9cea-7915ff41c348","Type":"ContainerStarted","Data":"9220386c1f5384d85bf993f4c448596c1efb5595f9818b65eb9280f7aa3dd8b4"}
Sep 17 18:41:34.304038 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:34.304000    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt" event={"ID":"61f4f17b-1cde-4c4a-9cea-7915ff41c348","Type":"ContainerStarted","Data":"0c9c1143226ebc27d596e6f5da143df3004adafb9211815aa93ad2a3312df29e"}
Sep 17 18:41:34.328670 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:34.328625    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt" podStartSLOduration=1.949942215 podStartE2EDuration="3.328609436s" totalImagesPullingTime="1.378667221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:41:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:34.325971288 +0000 UTC m=+752.258172806" watchObservedRunningTime="2026-09-17 18:41:34.328609436 +0000 UTC m=+752.260810955"
Sep 17 18:41:34.715258 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:34.715231    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"
Sep 17 18:41:41.715735 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:41.715704    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"
Sep 17 18:41:41.728088 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:41.728065    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"
Sep 17 18:41:42.356982 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:42.356950    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"
Sep 17 18:41:53.814855 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:53.814817    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"]
Sep 17 18:41:53.818891 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:53.818866    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"
Sep 17 18:41:53.822510 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:53.822490    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 17 18:41:53.822626 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:53.822608    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 17 18:41:53.822681 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:53.822610    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-pcrwh\""
Sep 17 18:41:53.822736 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:53.822617    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 17 18:41:53.828686 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:53.828665    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"]
Sep 17 18:41:53.953781 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:53.953750    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm7fd\" (UniqueName: \"kubernetes.io/projected/17483d91-34ec-496b-ab8e-c453c81ef401-kube-api-access-zm7fd\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4\" (UID: \"17483d91-34ec-496b-ab8e-c453c81ef401\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"
Sep 17 18:41:53.953926 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:53.953788    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17483d91-34ec-496b-ab8e-c453c81ef401-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4\" (UID: \"17483d91-34ec-496b-ab8e-c453c81ef401\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"
Sep 17 18:41:53.953926 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:53.953809    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17483d91-34ec-496b-ab8e-c453c81ef401-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4\" (UID: \"17483d91-34ec-496b-ab8e-c453c81ef401\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"
Sep 17 18:41:53.953926 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:53.953885    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17483d91-34ec-496b-ab8e-c453c81ef401-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4\" (UID: \"17483d91-34ec-496b-ab8e-c453c81ef401\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"
Sep 17 18:41:53.953926 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:53.953908    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17483d91-34ec-496b-ab8e-c453c81ef401-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4\" (UID: \"17483d91-34ec-496b-ab8e-c453c81ef401\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"
Sep 17 18:41:53.954084 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:53.953930    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17483d91-34ec-496b-ab8e-c453c81ef401-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4\" (UID: \"17483d91-34ec-496b-ab8e-c453c81ef401\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"
Sep 17 18:41:54.054465 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:54.054425    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17483d91-34ec-496b-ab8e-c453c81ef401-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4\" (UID: \"17483d91-34ec-496b-ab8e-c453c81ef401\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"
Sep 17 18:41:54.054595 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:54.054476    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17483d91-34ec-496b-ab8e-c453c81ef401-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4\" (UID: \"17483d91-34ec-496b-ab8e-c453c81ef401\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"
Sep 17 18:41:54.054595 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:54.054533    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm7fd\" (UniqueName: \"kubernetes.io/projected/17483d91-34ec-496b-ab8e-c453c81ef401-kube-api-access-zm7fd\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4\" (UID: \"17483d91-34ec-496b-ab8e-c453c81ef401\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"
Sep 17 18:41:54.054595 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:54.054565    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17483d91-34ec-496b-ab8e-c453c81ef401-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4\" (UID: \"17483d91-34ec-496b-ab8e-c453c81ef401\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"
Sep 17 18:41:54.054761 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:54.054643    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17483d91-34ec-496b-ab8e-c453c81ef401-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4\" (UID: \"17483d91-34ec-496b-ab8e-c453c81ef401\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"
Sep 17 18:41:54.054761 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:54.054718    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17483d91-34ec-496b-ab8e-c453c81ef401-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4\" (UID: \"17483d91-34ec-496b-ab8e-c453c81ef401\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"
Sep 17 18:41:54.055255 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:54.055172    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17483d91-34ec-496b-ab8e-c453c81ef401-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4\" (UID: \"17483d91-34ec-496b-ab8e-c453c81ef401\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"
Sep 17 18:41:54.055255 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:54.055195    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17483d91-34ec-496b-ab8e-c453c81ef401-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4\" (UID: \"17483d91-34ec-496b-ab8e-c453c81ef401\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"
Sep 17 18:41:54.055458 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:54.055319    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17483d91-34ec-496b-ab8e-c453c81ef401-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4\" (UID: \"17483d91-34ec-496b-ab8e-c453c81ef401\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"
Sep 17 18:41:54.057476 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:54.057452    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17483d91-34ec-496b-ab8e-c453c81ef401-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4\" (UID: \"17483d91-34ec-496b-ab8e-c453c81ef401\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"
Sep 17 18:41:54.057640 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:54.057578    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17483d91-34ec-496b-ab8e-c453c81ef401-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4\" (UID: \"17483d91-34ec-496b-ab8e-c453c81ef401\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"
Sep 17 18:41:54.064304 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:54.064271    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm7fd\" (UniqueName: \"kubernetes.io/projected/17483d91-34ec-496b-ab8e-c453c81ef401-kube-api-access-zm7fd\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4\" (UID: \"17483d91-34ec-496b-ab8e-c453c81ef401\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"
Sep 17 18:41:54.080430 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:54.080368    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq"]
Sep 17 18:41:54.084017 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:54.083997    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq"
Sep 17 18:41:54.086521 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:54.086500    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-6t2zd\""
Sep 17 18:41:54.095801 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:54.095780    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq"]
Sep 17 18:41:54.129510 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:54.129484    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"
Sep 17 18:41:54.155960 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:54.155934    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq\" (UID: \"eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq"
Sep 17 18:41:54.156062 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:54.156030    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjh9j\" (UniqueName: \"kubernetes.io/projected/eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5-kube-api-access-cjh9j\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq\" (UID: \"eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq"
Sep 17 18:41:54.256598 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:54.256567    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq\" (UID: \"eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq"
Sep 17 18:41:54.256738 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:54.256633    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjh9j\" (UniqueName: \"kubernetes.io/projected/eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5-kube-api-access-cjh9j\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq\" (UID: \"eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq"
Sep 17 18:41:54.258864 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:54.258838    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq\" (UID: \"eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq"
Sep 17 18:41:54.264765 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:54.264734    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjh9j\" (UniqueName: \"kubernetes.io/projected/eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5-kube-api-access-cjh9j\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq\" (UID: \"eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq"
Sep 17 18:41:54.394793 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:54.394724    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq"
Sep 17 18:41:54.403873 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:54.403853    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"]
Sep 17 18:41:54.405347 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:41:54.405319    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17483d91_34ec_496b_ab8e_c453c81ef401.slice/crio-5dacdf43b91fdb150705474c4262839f774af9685cc5d686f4d3280bd7aafe34 WatchSource:0}: Error finding container 5dacdf43b91fdb150705474c4262839f774af9685cc5d686f4d3280bd7aafe34: Status 404 returned error can't find the container with id 5dacdf43b91fdb150705474c4262839f774af9685cc5d686f4d3280bd7aafe34
Sep 17 18:41:54.416771 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:54.416744    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4" event={"ID":"17483d91-34ec-496b-ab8e-c453c81ef401","Type":"ContainerStarted","Data":"5dacdf43b91fdb150705474c4262839f774af9685cc5d686f4d3280bd7aafe34"}
Sep 17 18:41:54.679433 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:54.679408    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq"]
Sep 17 18:41:54.681070 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:41:54.681047    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb97e32f_d4a9_47b1_b1dc_4f407c85a6f5.slice/crio-5ba364752d07c6b9fef6dc6dae78b1846ea86e06ed3ad24713eb9070129c41b5 WatchSource:0}: Error finding container 5ba364752d07c6b9fef6dc6dae78b1846ea86e06ed3ad24713eb9070129c41b5: Status 404 returned error can't find the container with id 5ba364752d07c6b9fef6dc6dae78b1846ea86e06ed3ad24713eb9070129c41b5
Sep 17 18:41:55.426343 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:55.426305    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4" event={"ID":"17483d91-34ec-496b-ab8e-c453c81ef401","Type":"ContainerStarted","Data":"872a4a283fdb356a5a704dcc7eee0c67b276f28be1461cfd0ececee617760404"}
Sep 17 18:41:55.428444 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:55.428383    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq" event={"ID":"eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5","Type":"ContainerStarted","Data":"5ba364752d07c6b9fef6dc6dae78b1846ea86e06ed3ad24713eb9070129c41b5"}
Sep 17 18:41:55.447775 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:55.447725    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4" podStartSLOduration=2.447705079 podStartE2EDuration="2.447705079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:41:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:55.445985063 +0000 UTC m=+773.378186582" watchObservedRunningTime="2026-09-17 18:41:55.447705079 +0000 UTC m=+773.379906599"
Sep 17 18:41:56.130505 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:56.130470    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"
Sep 17 18:41:57.445107 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:57.445062    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq" event={"ID":"eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5","Type":"ContainerStarted","Data":"d98757921fbcaaf7719455aed932d61118357a909cfb2678cbc9d62122fac0b6"}
Sep 17 18:41:57.445733 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:57.445707    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq"
Sep 17 18:41:57.470659 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:41:57.470615    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq" podStartSLOduration=1.225083161 podStartE2EDuration="3.470602333s" totalImagesPullingTime="2.245519172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:41:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:57.469603374 +0000 UTC m=+775.401804894" watchObservedRunningTime="2026-09-17 18:41:57.470602333 +0000 UTC m=+775.402803852"
Sep 17 18:42:04.130270 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:04.130232    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"
Sep 17 18:42:04.142844 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:04.142815    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"
Sep 17 18:42:04.491905 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:04.491873    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"
Sep 17 18:42:28.362010 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:28.361929    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"]
Sep 17 18:42:28.371187 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:42:28.371155    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:42:28.371187 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:42:28.371187 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:42:28.371187 ip-10-0-139-75 kubenswrapper[2565]:  > containerID="0c9c1143226ebc27d596e6f5da143df3004adafb9211815aa93ad2a3312df29e" cmd=["/bin/sleep","15"]
Sep 17 18:42:28.371423 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:28.371221    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt" message="" err=<
Sep 17 18:42:28.371423 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:42:28.371423 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:42:28.371423 ip-10-0-139-75 kubenswrapper[2565]:  >
Sep 17 18:42:28.371423 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:42:28.371233    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:42:28.371423 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:42:28.371423 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:42:28.371423 ip-10-0-139-75 kubenswrapper[2565]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt" podUID="61f4f17b-1cde-4c4a-9cea-7915ff41c348" containerName="main" containerID="cri-o://0c9c1143226ebc27d596e6f5da143df3004adafb9211815aa93ad2a3312df29e"
Sep 17 18:42:28.371423 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:28.371273    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt" podUID="61f4f17b-1cde-4c4a-9cea-7915ff41c348" containerName="main" containerID="cri-o://0c9c1143226ebc27d596e6f5da143df3004adafb9211815aa93ad2a3312df29e" gracePeriod=60
Sep 17 18:42:28.451364 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:28.451326    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq" podUID="eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:42:28.611509 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:28.611474    2565 generic.go:356] "Generic (PLEG): container finished" podID="61f4f17b-1cde-4c4a-9cea-7915ff41c348" containerID="0c9c1143226ebc27d596e6f5da143df3004adafb9211815aa93ad2a3312df29e" exitCode=0
Sep 17 18:42:28.611673 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:28.611529    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt" event={"ID":"61f4f17b-1cde-4c4a-9cea-7915ff41c348","Type":"ContainerDied","Data":"0c9c1143226ebc27d596e6f5da143df3004adafb9211815aa93ad2a3312df29e"}
Sep 17 18:42:28.722349 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:28.722326    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"
Sep 17 18:42:28.818696 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:28.818669    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpm2v\" (UniqueName: \"kubernetes.io/projected/61f4f17b-1cde-4c4a-9cea-7915ff41c348-kube-api-access-bpm2v\") pod \"61f4f17b-1cde-4c4a-9cea-7915ff41c348\" (UID: \"61f4f17b-1cde-4c4a-9cea-7915ff41c348\") "
Sep 17 18:42:28.818831 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:28.818703    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61f4f17b-1cde-4c4a-9cea-7915ff41c348-home\") pod \"61f4f17b-1cde-4c4a-9cea-7915ff41c348\" (UID: \"61f4f17b-1cde-4c4a-9cea-7915ff41c348\") "
Sep 17 18:42:28.818831 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:28.818765    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61f4f17b-1cde-4c4a-9cea-7915ff41c348-model-cache\") pod \"61f4f17b-1cde-4c4a-9cea-7915ff41c348\" (UID: \"61f4f17b-1cde-4c4a-9cea-7915ff41c348\") "
Sep 17 18:42:28.818831 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:28.818800    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61f4f17b-1cde-4c4a-9cea-7915ff41c348-dshm\") pod \"61f4f17b-1cde-4c4a-9cea-7915ff41c348\" (UID: \"61f4f17b-1cde-4c4a-9cea-7915ff41c348\") "
Sep 17 18:42:28.818831 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:28.818828    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61f4f17b-1cde-4c4a-9cea-7915ff41c348-tls-certs\") pod \"61f4f17b-1cde-4c4a-9cea-7915ff41c348\" (UID: \"61f4f17b-1cde-4c4a-9cea-7915ff41c348\") "
Sep 17 18:42:28.819023 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:28.818888    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61f4f17b-1cde-4c4a-9cea-7915ff41c348-tmp-dir\") pod \"61f4f17b-1cde-4c4a-9cea-7915ff41c348\" (UID: \"61f4f17b-1cde-4c4a-9cea-7915ff41c348\") "
Sep 17 18:42:28.819088 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:28.818997    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f4f17b-1cde-4c4a-9cea-7915ff41c348-home" (OuterVolumeSpecName: "home") pod "61f4f17b-1cde-4c4a-9cea-7915ff41c348" (UID: "61f4f17b-1cde-4c4a-9cea-7915ff41c348"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:42:28.819088 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:28.819070    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f4f17b-1cde-4c4a-9cea-7915ff41c348-model-cache" (OuterVolumeSpecName: "model-cache") pod "61f4f17b-1cde-4c4a-9cea-7915ff41c348" (UID: "61f4f17b-1cde-4c4a-9cea-7915ff41c348"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:42:28.819222 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:28.819163    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f4f17b-1cde-4c4a-9cea-7915ff41c348-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "61f4f17b-1cde-4c4a-9cea-7915ff41c348" (UID: "61f4f17b-1cde-4c4a-9cea-7915ff41c348"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:42:28.819313 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:28.819285    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61f4f17b-1cde-4c4a-9cea-7915ff41c348-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:42:28.819356 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:28.819321    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61f4f17b-1cde-4c4a-9cea-7915ff41c348-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:42:28.819356 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:28.819333    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61f4f17b-1cde-4c4a-9cea-7915ff41c348-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:42:28.820930 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:28.820897    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f4f17b-1cde-4c4a-9cea-7915ff41c348-dshm" (OuterVolumeSpecName: "dshm") pod "61f4f17b-1cde-4c4a-9cea-7915ff41c348" (UID: "61f4f17b-1cde-4c4a-9cea-7915ff41c348"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:42:28.820930 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:28.820910    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61f4f17b-1cde-4c4a-9cea-7915ff41c348-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "61f4f17b-1cde-4c4a-9cea-7915ff41c348" (UID: "61f4f17b-1cde-4c4a-9cea-7915ff41c348"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:42:28.820930 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:28.820905    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f4f17b-1cde-4c4a-9cea-7915ff41c348-kube-api-access-bpm2v" (OuterVolumeSpecName: "kube-api-access-bpm2v") pod "61f4f17b-1cde-4c4a-9cea-7915ff41c348" (UID: "61f4f17b-1cde-4c4a-9cea-7915ff41c348"). InnerVolumeSpecName "kube-api-access-bpm2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:42:28.919649 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:28.919622    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpm2v\" (UniqueName: \"kubernetes.io/projected/61f4f17b-1cde-4c4a-9cea-7915ff41c348-kube-api-access-bpm2v\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:42:28.919649 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:28.919646    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61f4f17b-1cde-4c4a-9cea-7915ff41c348-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:42:28.919788 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:28.919656    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61f4f17b-1cde-4c4a-9cea-7915ff41c348-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:42:29.130792 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.130756    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"]
Sep 17 18:42:29.140286 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:42:29.140249    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:42:29.140286 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:42:29.140286 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:42:29.140286 ip-10-0-139-75 kubenswrapper[2565]:  > containerID="872a4a283fdb356a5a704dcc7eee0c67b276f28be1461cfd0ececee617760404" cmd=["/bin/sleep","15"]
Sep 17 18:42:29.140506 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.140303    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4" message="" err=<
Sep 17 18:42:29.140506 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:42:29.140506 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:42:29.140506 ip-10-0-139-75 kubenswrapper[2565]:  >
Sep 17 18:42:29.140506 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:42:29.140315    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:42:29.140506 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:42:29.140506 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:42:29.140506 ip-10-0-139-75 kubenswrapper[2565]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4" podUID="17483d91-34ec-496b-ab8e-c453c81ef401" containerName="main" containerID="cri-o://872a4a283fdb356a5a704dcc7eee0c67b276f28be1461cfd0ececee617760404"
Sep 17 18:42:29.140506 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.140371    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4" podUID="17483d91-34ec-496b-ab8e-c453c81ef401" containerName="main" containerID="cri-o://872a4a283fdb356a5a704dcc7eee0c67b276f28be1461cfd0ececee617760404" gracePeriod=60
Sep 17 18:42:29.143878 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.143850    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq"]
Sep 17 18:42:29.145371 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.145335    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq" podUID="eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:42:29.492738 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.492717    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"
Sep 17 18:42:29.618331 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.618250    2565 generic.go:356] "Generic (PLEG): container finished" podID="17483d91-34ec-496b-ab8e-c453c81ef401" containerID="872a4a283fdb356a5a704dcc7eee0c67b276f28be1461cfd0ececee617760404" exitCode=0
Sep 17 18:42:29.618331 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.618300    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"
Sep 17 18:42:29.618515 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.618340    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4" event={"ID":"17483d91-34ec-496b-ab8e-c453c81ef401","Type":"ContainerDied","Data":"872a4a283fdb356a5a704dcc7eee0c67b276f28be1461cfd0ececee617760404"}
Sep 17 18:42:29.618515 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.618388    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4" event={"ID":"17483d91-34ec-496b-ab8e-c453c81ef401","Type":"ContainerDied","Data":"5dacdf43b91fdb150705474c4262839f774af9685cc5d686f4d3280bd7aafe34"}
Sep 17 18:42:29.618515 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.618418    2565 scope.go:117] "RemoveContainer" containerID="872a4a283fdb356a5a704dcc7eee0c67b276f28be1461cfd0ececee617760404"
Sep 17 18:42:29.619911 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.619890    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt" event={"ID":"61f4f17b-1cde-4c4a-9cea-7915ff41c348","Type":"ContainerDied","Data":"9220386c1f5384d85bf993f4c448596c1efb5595f9818b65eb9280f7aa3dd8b4"}
Sep 17 18:42:29.619911 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.619902    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"
Sep 17 18:42:29.624350 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.624329    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17483d91-34ec-496b-ab8e-c453c81ef401-dshm\") pod \"17483d91-34ec-496b-ab8e-c453c81ef401\" (UID: \"17483d91-34ec-496b-ab8e-c453c81ef401\") "
Sep 17 18:42:29.624451 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.624361    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zm7fd\" (UniqueName: \"kubernetes.io/projected/17483d91-34ec-496b-ab8e-c453c81ef401-kube-api-access-zm7fd\") pod \"17483d91-34ec-496b-ab8e-c453c81ef401\" (UID: \"17483d91-34ec-496b-ab8e-c453c81ef401\") "
Sep 17 18:42:29.624451 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.624378    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17483d91-34ec-496b-ab8e-c453c81ef401-model-cache\") pod \"17483d91-34ec-496b-ab8e-c453c81ef401\" (UID: \"17483d91-34ec-496b-ab8e-c453c81ef401\") "
Sep 17 18:42:29.624451 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.624402    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17483d91-34ec-496b-ab8e-c453c81ef401-tls-certs\") pod \"17483d91-34ec-496b-ab8e-c453c81ef401\" (UID: \"17483d91-34ec-496b-ab8e-c453c81ef401\") "
Sep 17 18:42:29.624630 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.624558    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17483d91-34ec-496b-ab8e-c453c81ef401-tmp-dir\") pod \"17483d91-34ec-496b-ab8e-c453c81ef401\" (UID: \"17483d91-34ec-496b-ab8e-c453c81ef401\") "
Sep 17 18:42:29.624759 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.624682    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17483d91-34ec-496b-ab8e-c453c81ef401-model-cache" (OuterVolumeSpecName: "model-cache") pod "17483d91-34ec-496b-ab8e-c453c81ef401" (UID: "17483d91-34ec-496b-ab8e-c453c81ef401"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:42:29.624831 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.624794    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17483d91-34ec-496b-ab8e-c453c81ef401-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "17483d91-34ec-496b-ab8e-c453c81ef401" (UID: "17483d91-34ec-496b-ab8e-c453c81ef401"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:42:29.624892 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.624871    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17483d91-34ec-496b-ab8e-c453c81ef401-home\") pod \"17483d91-34ec-496b-ab8e-c453c81ef401\" (UID: \"17483d91-34ec-496b-ab8e-c453c81ef401\") "
Sep 17 18:42:29.625161 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.625137    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17483d91-34ec-496b-ab8e-c453c81ef401-home" (OuterVolumeSpecName: "home") pod "17483d91-34ec-496b-ab8e-c453c81ef401" (UID: "17483d91-34ec-496b-ab8e-c453c81ef401"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:42:29.625282 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.625175    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17483d91-34ec-496b-ab8e-c453c81ef401-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:42:29.625282 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.625193    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17483d91-34ec-496b-ab8e-c453c81ef401-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:42:29.627000 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.626931    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17483d91-34ec-496b-ab8e-c453c81ef401-dshm" (OuterVolumeSpecName: "dshm") pod "17483d91-34ec-496b-ab8e-c453c81ef401" (UID: "17483d91-34ec-496b-ab8e-c453c81ef401"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:42:29.627097 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.627033    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17483d91-34ec-496b-ab8e-c453c81ef401-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "17483d91-34ec-496b-ab8e-c453c81ef401" (UID: "17483d91-34ec-496b-ab8e-c453c81ef401"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:42:29.627097 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.627044    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17483d91-34ec-496b-ab8e-c453c81ef401-kube-api-access-zm7fd" (OuterVolumeSpecName: "kube-api-access-zm7fd") pod "17483d91-34ec-496b-ab8e-c453c81ef401" (UID: "17483d91-34ec-496b-ab8e-c453c81ef401"). InnerVolumeSpecName "kube-api-access-zm7fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:42:29.629000 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.628985    2565 scope.go:117] "RemoveContainer" containerID="872a4a283fdb356a5a704dcc7eee0c67b276f28be1461cfd0ececee617760404"
Sep 17 18:42:29.629305 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:42:29.629281    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"872a4a283fdb356a5a704dcc7eee0c67b276f28be1461cfd0ececee617760404\": container with ID starting with 872a4a283fdb356a5a704dcc7eee0c67b276f28be1461cfd0ececee617760404 not found: ID does not exist" containerID="872a4a283fdb356a5a704dcc7eee0c67b276f28be1461cfd0ececee617760404"
Sep 17 18:42:29.629393 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.629311    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"872a4a283fdb356a5a704dcc7eee0c67b276f28be1461cfd0ececee617760404"} err="failed to get container status \"872a4a283fdb356a5a704dcc7eee0c67b276f28be1461cfd0ececee617760404\": rpc error: code = NotFound desc = could not find container \"872a4a283fdb356a5a704dcc7eee0c67b276f28be1461cfd0ececee617760404\": container with ID starting with 872a4a283fdb356a5a704dcc7eee0c67b276f28be1461cfd0ececee617760404 not found: ID does not exist"
Sep 17 18:42:29.629393 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.629328    2565 scope.go:117] "RemoveContainer" containerID="0c9c1143226ebc27d596e6f5da143df3004adafb9211815aa93ad2a3312df29e"
Sep 17 18:42:29.655478 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.655454    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"]
Sep 17 18:42:29.659559 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.659541    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-555qt"]
Sep 17 18:42:29.726340 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.726315    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17483d91-34ec-496b-ab8e-c453c81ef401-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:42:29.726340 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.726336    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17483d91-34ec-496b-ab8e-c453c81ef401-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:42:29.726471 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.726345    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm7fd\" (UniqueName: \"kubernetes.io/projected/17483d91-34ec-496b-ab8e-c453c81ef401-kube-api-access-zm7fd\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:42:29.726471 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.726354    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17483d91-34ec-496b-ab8e-c453c81ef401-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:42:29.948995 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.948970    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"]
Sep 17 18:42:29.955308 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:29.955285    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4"]
Sep 17 18:42:30.481854 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:30.481821    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2zmd4" podUID="17483d91-34ec-496b-ab8e-c453c81ef401" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": context deadline exceeded"
Sep 17 18:42:30.607090 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:30.607054    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17483d91-34ec-496b-ab8e-c453c81ef401" path="/var/lib/kubelet/pods/17483d91-34ec-496b-ab8e-c453c81ef401/volumes"
Sep 17 18:42:30.607617 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:30.607595    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61f4f17b-1cde-4c4a-9cea-7915ff41c348" path="/var/lib/kubelet/pods/61f4f17b-1cde-4c4a-9cea-7915ff41c348/volumes"
Sep 17 18:42:39.146446 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:39.146409    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq" podUID="eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:42:44.162478 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:44.162421    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq" podUID="eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5" containerName="main" containerID="cri-o://d98757921fbcaaf7719455aed932d61118357a909cfb2678cbc9d62122fac0b6" gracePeriod=45
Sep 17 18:42:49.145946 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:49.145904    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq" podUID="eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:42:59.145919 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:42:59.145878    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq" podUID="eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:43:09.146158 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:09.146110    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq" podUID="eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:43:14.402122 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:14.402097    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq"
Sep 17 18:43:14.441983 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:14.441923    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5-tls-certs\") pod \"eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5\" (UID: \"eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5\") "
Sep 17 18:43:14.441983 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:14.441972    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjh9j\" (UniqueName: \"kubernetes.io/projected/eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5-kube-api-access-cjh9j\") pod \"eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5\" (UID: \"eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5\") "
Sep 17 18:43:14.443956 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:14.443934    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5-kube-api-access-cjh9j" (OuterVolumeSpecName: "kube-api-access-cjh9j") pod "eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5" (UID: "eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5"). InnerVolumeSpecName "kube-api-access-cjh9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:43:14.444069 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:14.443993    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5" (UID: "eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:43:14.542882 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:14.542859    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:43:14.542882 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:14.542883    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjh9j\" (UniqueName: \"kubernetes.io/projected/eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5-kube-api-access-cjh9j\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:43:14.860584 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:14.860518    2565 generic.go:356] "Generic (PLEG): container finished" podID="eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5" containerID="d98757921fbcaaf7719455aed932d61118357a909cfb2678cbc9d62122fac0b6" exitCode=0
Sep 17 18:43:14.860584 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:14.860560    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq"
Sep 17 18:43:14.860742 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:14.860608    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq" event={"ID":"eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5","Type":"ContainerDied","Data":"d98757921fbcaaf7719455aed932d61118357a909cfb2678cbc9d62122fac0b6"}
Sep 17 18:43:14.860742 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:14.860655    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq" event={"ID":"eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5","Type":"ContainerDied","Data":"5ba364752d07c6b9fef6dc6dae78b1846ea86e06ed3ad24713eb9070129c41b5"}
Sep 17 18:43:14.860742 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:14.860680    2565 scope.go:117] "RemoveContainer" containerID="d98757921fbcaaf7719455aed932d61118357a909cfb2678cbc9d62122fac0b6"
Sep 17 18:43:14.869428 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:14.869410    2565 scope.go:117] "RemoveContainer" containerID="d98757921fbcaaf7719455aed932d61118357a909cfb2678cbc9d62122fac0b6"
Sep 17 18:43:14.869675 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:43:14.869656    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d98757921fbcaaf7719455aed932d61118357a909cfb2678cbc9d62122fac0b6\": container with ID starting with d98757921fbcaaf7719455aed932d61118357a909cfb2678cbc9d62122fac0b6 not found: ID does not exist" containerID="d98757921fbcaaf7719455aed932d61118357a909cfb2678cbc9d62122fac0b6"
Sep 17 18:43:14.869726 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:14.869683    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d98757921fbcaaf7719455aed932d61118357a909cfb2678cbc9d62122fac0b6"} err="failed to get container status \"d98757921fbcaaf7719455aed932d61118357a909cfb2678cbc9d62122fac0b6\": rpc error: code = NotFound desc = could not find container \"d98757921fbcaaf7719455aed932d61118357a909cfb2678cbc9d62122fac0b6\": container with ID starting with d98757921fbcaaf7719455aed932d61118357a909cfb2678cbc9d62122fac0b6 not found: ID does not exist"
Sep 17 18:43:14.878733 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:14.878712    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq"]
Sep 17 18:43:14.882604 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:14.882584    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5ftscq"]
Sep 17 18:43:16.606787 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:16.606751    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5" path="/var/lib/kubelet/pods/eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5/volumes"
Sep 17 18:43:22.010938 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.010895    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"]
Sep 17 18:43:22.011446 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.011426    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61f4f17b-1cde-4c4a-9cea-7915ff41c348" containerName="main"
Sep 17 18:43:22.011525 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.011449    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="61f4f17b-1cde-4c4a-9cea-7915ff41c348" containerName="main"
Sep 17 18:43:22.011525 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.011467    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17483d91-34ec-496b-ab8e-c453c81ef401" containerName="main"
Sep 17 18:43:22.011525 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.011475    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="17483d91-34ec-496b-ab8e-c453c81ef401" containerName="main"
Sep 17 18:43:22.011525 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.011502    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5" containerName="main"
Sep 17 18:43:22.011525 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.011512    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5" containerName="main"
Sep 17 18:43:22.011774 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.011612    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="61f4f17b-1cde-4c4a-9cea-7915ff41c348" containerName="main"
Sep 17 18:43:22.011774 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.011630    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb97e32f-d4a9-47b1-b1dc-4f407c85a6f5" containerName="main"
Sep 17 18:43:22.011774 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.011641    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="17483d91-34ec-496b-ab8e-c453c81ef401" containerName="main"
Sep 17 18:43:22.016087 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.016067    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"
Sep 17 18:43:22.018689 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.018668    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-sn2kx\""
Sep 17 18:43:22.018878 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.018859    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 17 18:43:22.019535 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.019512    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 17 18:43:22.019650 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.019634    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 17 18:43:22.024308 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.024286    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"]
Sep 17 18:43:22.095572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.095543    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/265e2f82-7f90-481c-ad2c-8179d648c8dd-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-69js9\" (UID: \"265e2f82-7f90-481c-ad2c-8179d648c8dd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"
Sep 17 18:43:22.095695 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.095578    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/265e2f82-7f90-481c-ad2c-8179d648c8dd-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-69js9\" (UID: \"265e2f82-7f90-481c-ad2c-8179d648c8dd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"
Sep 17 18:43:22.095695 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.095596    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdszp\" (UniqueName: \"kubernetes.io/projected/265e2f82-7f90-481c-ad2c-8179d648c8dd-kube-api-access-fdszp\") pod \"fc-concurrency-test-kserve-b5d44467c-69js9\" (UID: \"265e2f82-7f90-481c-ad2c-8179d648c8dd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"
Sep 17 18:43:22.095695 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.095613    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/265e2f82-7f90-481c-ad2c-8179d648c8dd-home\") pod \"fc-concurrency-test-kserve-b5d44467c-69js9\" (UID: \"265e2f82-7f90-481c-ad2c-8179d648c8dd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"
Sep 17 18:43:22.095695 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.095683    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/265e2f82-7f90-481c-ad2c-8179d648c8dd-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-69js9\" (UID: \"265e2f82-7f90-481c-ad2c-8179d648c8dd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"
Sep 17 18:43:22.095839 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.095752    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/265e2f82-7f90-481c-ad2c-8179d648c8dd-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-69js9\" (UID: \"265e2f82-7f90-481c-ad2c-8179d648c8dd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"
Sep 17 18:43:22.196755 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.196732    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/265e2f82-7f90-481c-ad2c-8179d648c8dd-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-69js9\" (UID: \"265e2f82-7f90-481c-ad2c-8179d648c8dd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"
Sep 17 18:43:22.196881 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.196763    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/265e2f82-7f90-481c-ad2c-8179d648c8dd-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-69js9\" (UID: \"265e2f82-7f90-481c-ad2c-8179d648c8dd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"
Sep 17 18:43:22.196881 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.196781    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdszp\" (UniqueName: \"kubernetes.io/projected/265e2f82-7f90-481c-ad2c-8179d648c8dd-kube-api-access-fdszp\") pod \"fc-concurrency-test-kserve-b5d44467c-69js9\" (UID: \"265e2f82-7f90-481c-ad2c-8179d648c8dd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"
Sep 17 18:43:22.196881 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.196798    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/265e2f82-7f90-481c-ad2c-8179d648c8dd-home\") pod \"fc-concurrency-test-kserve-b5d44467c-69js9\" (UID: \"265e2f82-7f90-481c-ad2c-8179d648c8dd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"
Sep 17 18:43:22.196881 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.196827    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/265e2f82-7f90-481c-ad2c-8179d648c8dd-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-69js9\" (UID: \"265e2f82-7f90-481c-ad2c-8179d648c8dd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"
Sep 17 18:43:22.197116 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.196883    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/265e2f82-7f90-481c-ad2c-8179d648c8dd-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-69js9\" (UID: \"265e2f82-7f90-481c-ad2c-8179d648c8dd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"
Sep 17 18:43:22.197219 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.197175    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/265e2f82-7f90-481c-ad2c-8179d648c8dd-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-69js9\" (UID: \"265e2f82-7f90-481c-ad2c-8179d648c8dd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"
Sep 17 18:43:22.197344 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.197312    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/265e2f82-7f90-481c-ad2c-8179d648c8dd-home\") pod \"fc-concurrency-test-kserve-b5d44467c-69js9\" (UID: \"265e2f82-7f90-481c-ad2c-8179d648c8dd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"
Sep 17 18:43:22.197397 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.197344    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/265e2f82-7f90-481c-ad2c-8179d648c8dd-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-69js9\" (UID: \"265e2f82-7f90-481c-ad2c-8179d648c8dd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"
Sep 17 18:43:22.198996 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.198977    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/265e2f82-7f90-481c-ad2c-8179d648c8dd-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-69js9\" (UID: \"265e2f82-7f90-481c-ad2c-8179d648c8dd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"
Sep 17 18:43:22.199361 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.199342    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/265e2f82-7f90-481c-ad2c-8179d648c8dd-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-69js9\" (UID: \"265e2f82-7f90-481c-ad2c-8179d648c8dd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"
Sep 17 18:43:22.207950 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.207929    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdszp\" (UniqueName: \"kubernetes.io/projected/265e2f82-7f90-481c-ad2c-8179d648c8dd-kube-api-access-fdszp\") pod \"fc-concurrency-test-kserve-b5d44467c-69js9\" (UID: \"265e2f82-7f90-481c-ad2c-8179d648c8dd\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"
Sep 17 18:43:22.328337 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.328275    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"
Sep 17 18:43:22.606344 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:43:22.606301    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod265e2f82_7f90_481c_ad2c_8179d648c8dd.slice/crio-d5641bf925a1b217c773f706bcbe3d330dd1f17b933cf3fe1270a7c875c26a65 WatchSource:0}: Error finding container d5641bf925a1b217c773f706bcbe3d330dd1f17b933cf3fe1270a7c875c26a65: Status 404 returned error can't find the container with id d5641bf925a1b217c773f706bcbe3d330dd1f17b933cf3fe1270a7c875c26a65
Sep 17 18:43:22.608506 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.608484    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"]
Sep 17 18:43:22.912621 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.912588    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9" event={"ID":"265e2f82-7f90-481c-ad2c-8179d648c8dd","Type":"ContainerStarted","Data":"29a1312a53b745b465bddc7d15d0b3ca4da728adfc2f1c62c25027126030c00c"}
Sep 17 18:43:22.912767 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.912666    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9" event={"ID":"265e2f82-7f90-481c-ad2c-8179d648c8dd","Type":"ContainerStarted","Data":"d5641bf925a1b217c773f706bcbe3d330dd1f17b933cf3fe1270a7c875c26a65"}
Sep 17 18:43:22.935920 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:22.935874    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9" podStartSLOduration=1.935855917 podStartE2EDuration="1.935855917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:43:22.935605996 +0000 UTC m=+860.867807514" watchObservedRunningTime="2026-09-17 18:43:22.935855917 +0000 UTC m=+860.868057436"
Sep 17 18:43:23.328882 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:23.328814    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"
Sep 17 18:43:32.329380 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:32.329342    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"
Sep 17 18:43:32.341557 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:32.341532    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"
Sep 17 18:43:32.976241 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:32.976210    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"
Sep 17 18:43:41.818673 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:41.818639    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"]
Sep 17 18:43:41.822164 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:41.822143    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"
Sep 17 18:43:41.825537 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:41.825512    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 17 18:43:41.825662 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:41.825551    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 17 18:43:41.825662 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:41.825518    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 17 18:43:41.825662 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:41.825512    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-fv6gz\""
Sep 17 18:43:41.834600 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:41.834578    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"]
Sep 17 18:43:41.941462 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:41.941440    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hnnj6\" (UID: \"0b221c52-fe4b-4e79-9a24-6ccd56bd784d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"
Sep 17 18:43:41.941580 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:41.941474    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q4k2\" (UniqueName: \"kubernetes.io/projected/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-kube-api-access-5q4k2\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hnnj6\" (UID: \"0b221c52-fe4b-4e79-9a24-6ccd56bd784d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"
Sep 17 18:43:41.941580 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:41.941496    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hnnj6\" (UID: \"0b221c52-fe4b-4e79-9a24-6ccd56bd784d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"
Sep 17 18:43:41.941580 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:41.941514    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hnnj6\" (UID: \"0b221c52-fe4b-4e79-9a24-6ccd56bd784d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"
Sep 17 18:43:41.941580 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:41.941549    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hnnj6\" (UID: \"0b221c52-fe4b-4e79-9a24-6ccd56bd784d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"
Sep 17 18:43:41.941712 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:41.941577    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hnnj6\" (UID: \"0b221c52-fe4b-4e79-9a24-6ccd56bd784d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"
Sep 17 18:43:42.042680 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:42.042652    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hnnj6\" (UID: \"0b221c52-fe4b-4e79-9a24-6ccd56bd784d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"
Sep 17 18:43:42.042828 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:42.042711    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q4k2\" (UniqueName: \"kubernetes.io/projected/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-kube-api-access-5q4k2\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hnnj6\" (UID: \"0b221c52-fe4b-4e79-9a24-6ccd56bd784d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"
Sep 17 18:43:42.042828 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:42.042751    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hnnj6\" (UID: \"0b221c52-fe4b-4e79-9a24-6ccd56bd784d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"
Sep 17 18:43:42.042828 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:42.042782    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hnnj6\" (UID: \"0b221c52-fe4b-4e79-9a24-6ccd56bd784d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"
Sep 17 18:43:42.042828 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:42.042808    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hnnj6\" (UID: \"0b221c52-fe4b-4e79-9a24-6ccd56bd784d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"
Sep 17 18:43:42.043048 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:42.042834    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hnnj6\" (UID: \"0b221c52-fe4b-4e79-9a24-6ccd56bd784d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"
Sep 17 18:43:42.043377 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:42.043353    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hnnj6\" (UID: \"0b221c52-fe4b-4e79-9a24-6ccd56bd784d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"
Sep 17 18:43:42.043476 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:42.043369    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hnnj6\" (UID: \"0b221c52-fe4b-4e79-9a24-6ccd56bd784d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"
Sep 17 18:43:42.043539 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:42.043494    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hnnj6\" (UID: \"0b221c52-fe4b-4e79-9a24-6ccd56bd784d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"
Sep 17 18:43:42.045276 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:42.045249    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hnnj6\" (UID: \"0b221c52-fe4b-4e79-9a24-6ccd56bd784d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"
Sep 17 18:43:42.045866 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:42.045842    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hnnj6\" (UID: \"0b221c52-fe4b-4e79-9a24-6ccd56bd784d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"
Sep 17 18:43:42.052631 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:42.052587    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q4k2\" (UniqueName: \"kubernetes.io/projected/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-kube-api-access-5q4k2\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-hnnj6\" (UID: \"0b221c52-fe4b-4e79-9a24-6ccd56bd784d\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"
Sep 17 18:43:42.134640 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:42.134586    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"
Sep 17 18:43:42.410262 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:42.410235    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"]
Sep 17 18:43:42.412062 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:43:42.412037    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b221c52_fe4b_4e79_9a24_6ccd56bd784d.slice/crio-a286811c6cb552963b9da9c1aad4fefccc611475410ab216d0f973f71869a460 WatchSource:0}: Error finding container a286811c6cb552963b9da9c1aad4fefccc611475410ab216d0f973f71869a460: Status 404 returned error can't find the container with id a286811c6cb552963b9da9c1aad4fefccc611475410ab216d0f973f71869a460
Sep 17 18:43:43.017557 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:43.017516    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6" event={"ID":"0b221c52-fe4b-4e79-9a24-6ccd56bd784d","Type":"ContainerStarted","Data":"f2f95248e00e328cf8693f786b2d46258c70ed6328d944eb23d368a29ba8b233"}
Sep 17 18:43:43.017982 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:43.017570    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6" event={"ID":"0b221c52-fe4b-4e79-9a24-6ccd56bd784d","Type":"ContainerStarted","Data":"a286811c6cb552963b9da9c1aad4fefccc611475410ab216d0f973f71869a460"}
Sep 17 18:43:43.036899 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:43.036841    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6" podStartSLOduration=2.036826178 podStartE2EDuration="2.036826178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:43:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:43:43.034414784 +0000 UTC m=+880.966616294" watchObservedRunningTime="2026-09-17 18:43:43.036826178 +0000 UTC m=+880.969027694"
Sep 17 18:43:43.135319 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:43.135293    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"
Sep 17 18:43:52.135784 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:52.135745    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"
Sep 17 18:43:52.148255 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:52.148229    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"
Sep 17 18:43:53.080806 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:43:53.080776    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"
Sep 17 18:44:00.905135 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:00.905054    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"]
Sep 17 18:44:00.924270 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:44:00.924238    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:44:00.924270 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:44:00.924270 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:44:00.924270 ip-10-0-139-75 kubenswrapper[2565]:  > containerID="29a1312a53b745b465bddc7d15d0b3ca4da728adfc2f1c62c25027126030c00c" cmd=["/bin/sleep","15"]
Sep 17 18:44:00.924549 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:00.924287    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9" message="" err=<
Sep 17 18:44:00.924549 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:44:00.924549 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:44:00.924549 ip-10-0-139-75 kubenswrapper[2565]:  >
Sep 17 18:44:00.924549 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:44:00.924305    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:44:00.924549 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:44:00.924549 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:44:00.924549 ip-10-0-139-75 kubenswrapper[2565]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9" podUID="265e2f82-7f90-481c-ad2c-8179d648c8dd" containerName="main" containerID="cri-o://29a1312a53b745b465bddc7d15d0b3ca4da728adfc2f1c62c25027126030c00c"
Sep 17 18:44:00.924549 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:00.924342    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9" podUID="265e2f82-7f90-481c-ad2c-8179d648c8dd" containerName="main" containerID="cri-o://29a1312a53b745b465bddc7d15d0b3ca4da728adfc2f1c62c25027126030c00c" gracePeriod=60
Sep 17 18:44:00.966353 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:00.966314    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9" podUID="265e2f82-7f90-481c-ad2c-8179d648c8dd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 17 18:44:01.270178 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:01.270151    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"
Sep 17 18:44:01.396357 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:01.396328    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/265e2f82-7f90-481c-ad2c-8179d648c8dd-tls-certs\") pod \"265e2f82-7f90-481c-ad2c-8179d648c8dd\" (UID: \"265e2f82-7f90-481c-ad2c-8179d648c8dd\") "
Sep 17 18:44:01.396504 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:01.396366    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/265e2f82-7f90-481c-ad2c-8179d648c8dd-dshm\") pod \"265e2f82-7f90-481c-ad2c-8179d648c8dd\" (UID: \"265e2f82-7f90-481c-ad2c-8179d648c8dd\") "
Sep 17 18:44:01.396504 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:01.396398    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/265e2f82-7f90-481c-ad2c-8179d648c8dd-tmp-dir\") pod \"265e2f82-7f90-481c-ad2c-8179d648c8dd\" (UID: \"265e2f82-7f90-481c-ad2c-8179d648c8dd\") "
Sep 17 18:44:01.396504 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:01.396443    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/265e2f82-7f90-481c-ad2c-8179d648c8dd-model-cache\") pod \"265e2f82-7f90-481c-ad2c-8179d648c8dd\" (UID: \"265e2f82-7f90-481c-ad2c-8179d648c8dd\") "
Sep 17 18:44:01.396504 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:01.396465    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdszp\" (UniqueName: \"kubernetes.io/projected/265e2f82-7f90-481c-ad2c-8179d648c8dd-kube-api-access-fdszp\") pod \"265e2f82-7f90-481c-ad2c-8179d648c8dd\" (UID: \"265e2f82-7f90-481c-ad2c-8179d648c8dd\") "
Sep 17 18:44:01.396723 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:01.396508    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/265e2f82-7f90-481c-ad2c-8179d648c8dd-home\") pod \"265e2f82-7f90-481c-ad2c-8179d648c8dd\" (UID: \"265e2f82-7f90-481c-ad2c-8179d648c8dd\") "
Sep 17 18:44:01.396723 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:01.396691    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265e2f82-7f90-481c-ad2c-8179d648c8dd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "265e2f82-7f90-481c-ad2c-8179d648c8dd" (UID: "265e2f82-7f90-481c-ad2c-8179d648c8dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:44:01.396819 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:01.396707    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265e2f82-7f90-481c-ad2c-8179d648c8dd-model-cache" (OuterVolumeSpecName: "model-cache") pod "265e2f82-7f90-481c-ad2c-8179d648c8dd" (UID: "265e2f82-7f90-481c-ad2c-8179d648c8dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:44:01.396899 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:01.396877    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265e2f82-7f90-481c-ad2c-8179d648c8dd-home" (OuterVolumeSpecName: "home") pod "265e2f82-7f90-481c-ad2c-8179d648c8dd" (UID: "265e2f82-7f90-481c-ad2c-8179d648c8dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:44:01.398602 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:01.398577    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/265e2f82-7f90-481c-ad2c-8179d648c8dd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "265e2f82-7f90-481c-ad2c-8179d648c8dd" (UID: "265e2f82-7f90-481c-ad2c-8179d648c8dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:44:01.398602 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:01.398584    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265e2f82-7f90-481c-ad2c-8179d648c8dd-dshm" (OuterVolumeSpecName: "dshm") pod "265e2f82-7f90-481c-ad2c-8179d648c8dd" (UID: "265e2f82-7f90-481c-ad2c-8179d648c8dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:44:01.398747 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:01.398637    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/265e2f82-7f90-481c-ad2c-8179d648c8dd-kube-api-access-fdszp" (OuterVolumeSpecName: "kube-api-access-fdszp") pod "265e2f82-7f90-481c-ad2c-8179d648c8dd" (UID: "265e2f82-7f90-481c-ad2c-8179d648c8dd"). InnerVolumeSpecName "kube-api-access-fdszp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:44:01.497449 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:01.497388    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/265e2f82-7f90-481c-ad2c-8179d648c8dd-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:44:01.497449 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:01.497410    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/265e2f82-7f90-481c-ad2c-8179d648c8dd-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:44:01.497449 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:01.497420    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/265e2f82-7f90-481c-ad2c-8179d648c8dd-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:44:01.497449 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:01.497428    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/265e2f82-7f90-481c-ad2c-8179d648c8dd-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:44:01.497449 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:01.497436    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/265e2f82-7f90-481c-ad2c-8179d648c8dd-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:44:01.497449 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:01.497444    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdszp\" (UniqueName: \"kubernetes.io/projected/265e2f82-7f90-481c-ad2c-8179d648c8dd-kube-api-access-fdszp\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:44:02.118897 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:02.118863    2565 generic.go:356] "Generic (PLEG): container finished" podID="265e2f82-7f90-481c-ad2c-8179d648c8dd" containerID="29a1312a53b745b465bddc7d15d0b3ca4da728adfc2f1c62c25027126030c00c" exitCode=0
Sep 17 18:44:02.119332 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:02.118922    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"
Sep 17 18:44:02.119332 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:02.118931    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9" event={"ID":"265e2f82-7f90-481c-ad2c-8179d648c8dd","Type":"ContainerDied","Data":"29a1312a53b745b465bddc7d15d0b3ca4da728adfc2f1c62c25027126030c00c"}
Sep 17 18:44:02.119332 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:02.118971    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9" event={"ID":"265e2f82-7f90-481c-ad2c-8179d648c8dd","Type":"ContainerDied","Data":"d5641bf925a1b217c773f706bcbe3d330dd1f17b933cf3fe1270a7c875c26a65"}
Sep 17 18:44:02.119332 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:02.118993    2565 scope.go:117] "RemoveContainer" containerID="29a1312a53b745b465bddc7d15d0b3ca4da728adfc2f1c62c25027126030c00c"
Sep 17 18:44:02.128910 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:02.128891    2565 scope.go:117] "RemoveContainer" containerID="29a1312a53b745b465bddc7d15d0b3ca4da728adfc2f1c62c25027126030c00c"
Sep 17 18:44:02.129165 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:44:02.129145    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29a1312a53b745b465bddc7d15d0b3ca4da728adfc2f1c62c25027126030c00c\": container with ID starting with 29a1312a53b745b465bddc7d15d0b3ca4da728adfc2f1c62c25027126030c00c not found: ID does not exist" containerID="29a1312a53b745b465bddc7d15d0b3ca4da728adfc2f1c62c25027126030c00c"
Sep 17 18:44:02.129264 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:02.129176    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29a1312a53b745b465bddc7d15d0b3ca4da728adfc2f1c62c25027126030c00c"} err="failed to get container status \"29a1312a53b745b465bddc7d15d0b3ca4da728adfc2f1c62c25027126030c00c\": rpc error: code = NotFound desc = could not find container \"29a1312a53b745b465bddc7d15d0b3ca4da728adfc2f1c62c25027126030c00c\": container with ID starting with 29a1312a53b745b465bddc7d15d0b3ca4da728adfc2f1c62c25027126030c00c not found: ID does not exist"
Sep 17 18:44:02.141481 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:02.141455    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"]
Sep 17 18:44:02.145581 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:02.145559    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-69js9"]
Sep 17 18:44:02.606786 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:02.606759    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="265e2f82-7f90-481c-ad2c-8179d648c8dd" path="/var/lib/kubelet/pods/265e2f82-7f90-481c-ad2c-8179d648c8dd/volumes"
Sep 17 18:44:15.568576 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:15.568540    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"]
Sep 17 18:44:15.578334 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:44:15.578284    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:44:15.578334 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:44:15.578334 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:44:15.578334 ip-10-0-139-75 kubenswrapper[2565]:  > containerID="f2f95248e00e328cf8693f786b2d46258c70ed6328d944eb23d368a29ba8b233" cmd=["/bin/sleep","15"]
Sep 17 18:44:15.578588 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:15.578364    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6" message="" err=<
Sep 17 18:44:15.578588 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:44:15.578588 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:44:15.578588 ip-10-0-139-75 kubenswrapper[2565]:  >
Sep 17 18:44:15.578588 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:44:15.578381    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:44:15.578588 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:44:15.578588 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:44:15.578588 ip-10-0-139-75 kubenswrapper[2565]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6" podUID="0b221c52-fe4b-4e79-9a24-6ccd56bd784d" containerName="main" containerID="cri-o://f2f95248e00e328cf8693f786b2d46258c70ed6328d944eb23d368a29ba8b233"
Sep 17 18:44:15.578588 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:15.578447    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6" podUID="0b221c52-fe4b-4e79-9a24-6ccd56bd784d" containerName="main" containerID="cri-o://f2f95248e00e328cf8693f786b2d46258c70ed6328d944eb23d368a29ba8b233" gracePeriod=60
Sep 17 18:44:15.925302 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:15.925272    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"
Sep 17 18:44:16.003153 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:16.003129    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-home\") pod \"0b221c52-fe4b-4e79-9a24-6ccd56bd784d\" (UID: \"0b221c52-fe4b-4e79-9a24-6ccd56bd784d\") "
Sep 17 18:44:16.003309 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:16.003159    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-tmp-dir\") pod \"0b221c52-fe4b-4e79-9a24-6ccd56bd784d\" (UID: \"0b221c52-fe4b-4e79-9a24-6ccd56bd784d\") "
Sep 17 18:44:16.003309 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:16.003188    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5q4k2\" (UniqueName: \"kubernetes.io/projected/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-kube-api-access-5q4k2\") pod \"0b221c52-fe4b-4e79-9a24-6ccd56bd784d\" (UID: \"0b221c52-fe4b-4e79-9a24-6ccd56bd784d\") "
Sep 17 18:44:16.003309 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:16.003243    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-dshm\") pod \"0b221c52-fe4b-4e79-9a24-6ccd56bd784d\" (UID: \"0b221c52-fe4b-4e79-9a24-6ccd56bd784d\") "
Sep 17 18:44:16.003309 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:16.003269    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-tls-certs\") pod \"0b221c52-fe4b-4e79-9a24-6ccd56bd784d\" (UID: \"0b221c52-fe4b-4e79-9a24-6ccd56bd784d\") "
Sep 17 18:44:16.003309 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:16.003297    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-model-cache\") pod \"0b221c52-fe4b-4e79-9a24-6ccd56bd784d\" (UID: \"0b221c52-fe4b-4e79-9a24-6ccd56bd784d\") "
Sep 17 18:44:16.003583 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:16.003399    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-home" (OuterVolumeSpecName: "home") pod "0b221c52-fe4b-4e79-9a24-6ccd56bd784d" (UID: "0b221c52-fe4b-4e79-9a24-6ccd56bd784d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:44:16.003583 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:16.003437    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0b221c52-fe4b-4e79-9a24-6ccd56bd784d" (UID: "0b221c52-fe4b-4e79-9a24-6ccd56bd784d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:44:16.003583 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:16.003558    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:44:16.003583 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:16.003578    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:44:16.003740 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:16.003619    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-model-cache" (OuterVolumeSpecName: "model-cache") pod "0b221c52-fe4b-4e79-9a24-6ccd56bd784d" (UID: "0b221c52-fe4b-4e79-9a24-6ccd56bd784d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:44:16.005444 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:16.005412    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0b221c52-fe4b-4e79-9a24-6ccd56bd784d" (UID: "0b221c52-fe4b-4e79-9a24-6ccd56bd784d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:44:16.005556 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:16.005461    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-dshm" (OuterVolumeSpecName: "dshm") pod "0b221c52-fe4b-4e79-9a24-6ccd56bd784d" (UID: "0b221c52-fe4b-4e79-9a24-6ccd56bd784d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:44:16.005556 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:16.005494    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-kube-api-access-5q4k2" (OuterVolumeSpecName: "kube-api-access-5q4k2") pod "0b221c52-fe4b-4e79-9a24-6ccd56bd784d" (UID: "0b221c52-fe4b-4e79-9a24-6ccd56bd784d"). InnerVolumeSpecName "kube-api-access-5q4k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:44:16.104368 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:16.104312    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q4k2\" (UniqueName: \"kubernetes.io/projected/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-kube-api-access-5q4k2\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:44:16.104368 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:16.104334    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:44:16.104368 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:16.104343    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:44:16.104368 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:16.104352    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b221c52-fe4b-4e79-9a24-6ccd56bd784d-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:44:16.194225 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:16.194188    2565 generic.go:356] "Generic (PLEG): container finished" podID="0b221c52-fe4b-4e79-9a24-6ccd56bd784d" containerID="f2f95248e00e328cf8693f786b2d46258c70ed6328d944eb23d368a29ba8b233" exitCode=0
Sep 17 18:44:16.194341 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:16.194229    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6" event={"ID":"0b221c52-fe4b-4e79-9a24-6ccd56bd784d","Type":"ContainerDied","Data":"f2f95248e00e328cf8693f786b2d46258c70ed6328d944eb23d368a29ba8b233"}
Sep 17 18:44:16.194341 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:16.194263    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"
Sep 17 18:44:16.194341 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:16.194293    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6" event={"ID":"0b221c52-fe4b-4e79-9a24-6ccd56bd784d","Type":"ContainerDied","Data":"a286811c6cb552963b9da9c1aad4fefccc611475410ab216d0f973f71869a460"}
Sep 17 18:44:16.194341 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:16.194317    2565 scope.go:117] "RemoveContainer" containerID="f2f95248e00e328cf8693f786b2d46258c70ed6328d944eb23d368a29ba8b233"
Sep 17 18:44:16.203549 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:16.203532    2565 scope.go:117] "RemoveContainer" containerID="f2f95248e00e328cf8693f786b2d46258c70ed6328d944eb23d368a29ba8b233"
Sep 17 18:44:16.203809 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:44:16.203793    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2f95248e00e328cf8693f786b2d46258c70ed6328d944eb23d368a29ba8b233\": container with ID starting with f2f95248e00e328cf8693f786b2d46258c70ed6328d944eb23d368a29ba8b233 not found: ID does not exist" containerID="f2f95248e00e328cf8693f786b2d46258c70ed6328d944eb23d368a29ba8b233"
Sep 17 18:44:16.203867 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:16.203818    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2f95248e00e328cf8693f786b2d46258c70ed6328d944eb23d368a29ba8b233"} err="failed to get container status \"f2f95248e00e328cf8693f786b2d46258c70ed6328d944eb23d368a29ba8b233\": rpc error: code = NotFound desc = could not find container \"f2f95248e00e328cf8693f786b2d46258c70ed6328d944eb23d368a29ba8b233\": container with ID starting with f2f95248e00e328cf8693f786b2d46258c70ed6328d944eb23d368a29ba8b233 not found: ID does not exist"
Sep 17 18:44:16.217044 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:16.217023    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"]
Sep 17 18:44:16.220426 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:16.220405    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-hnnj6"]
Sep 17 18:44:16.607409 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:16.607375    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b221c52-fe4b-4e79-9a24-6ccd56bd784d" path="/var/lib/kubelet/pods/0b221c52-fe4b-4e79-9a24-6ccd56bd784d/volumes"
Sep 17 18:44:51.285809 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.285776    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"]
Sep 17 18:44:51.286244 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.286128    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b221c52-fe4b-4e79-9a24-6ccd56bd784d" containerName="main"
Sep 17 18:44:51.286244 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.286139    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b221c52-fe4b-4e79-9a24-6ccd56bd784d" containerName="main"
Sep 17 18:44:51.286244 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.286149    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="265e2f82-7f90-481c-ad2c-8179d648c8dd" containerName="main"
Sep 17 18:44:51.286244 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.286155    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="265e2f82-7f90-481c-ad2c-8179d648c8dd" containerName="main"
Sep 17 18:44:51.286244 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.286244    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="265e2f82-7f90-481c-ad2c-8179d648c8dd" containerName="main"
Sep 17 18:44:51.286442 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.286253    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b221c52-fe4b-4e79-9a24-6ccd56bd784d" containerName="main"
Sep 17 18:44:51.289467 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.289445    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:44:51.296066 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.296039    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 17 18:44:51.297056 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.296627    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 17 18:44:51.297056 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.296992    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-ghvhg\""
Sep 17 18:44:51.315146 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.315126    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 17 18:44:51.322399 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.322377    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"]
Sep 17 18:44:51.357051 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.357028    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93d1937b-f039-43e9-8609-c272baac54e3-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-cqfdl\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:44:51.357149 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.357058    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d1937b-f039-43e9-8609-c272baac54e3-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-cqfdl\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:44:51.357149 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.357079    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93d1937b-f039-43e9-8609-c272baac54e3-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-cqfdl\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:44:51.357149 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.357108    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93d1937b-f039-43e9-8609-c272baac54e3-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-cqfdl\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:44:51.357149 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.357132    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93d1937b-f039-43e9-8609-c272baac54e3-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-cqfdl\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:44:51.357149 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.357147    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vclt7\" (UniqueName: \"kubernetes.io/projected/93d1937b-f039-43e9-8609-c272baac54e3-kube-api-access-vclt7\") pod \"router-gateway-1-openshift-default-5b4d9458bd-cqfdl\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:44:51.357405 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.357237    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d1937b-f039-43e9-8609-c272baac54e3-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-cqfdl\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:44:51.357405 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.357319    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93d1937b-f039-43e9-8609-c272baac54e3-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-cqfdl\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:44:51.357405 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.357344    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93d1937b-f039-43e9-8609-c272baac54e3-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-cqfdl\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:44:51.458165 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.458138    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93d1937b-f039-43e9-8609-c272baac54e3-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-cqfdl\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:44:51.458275 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.458169    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d1937b-f039-43e9-8609-c272baac54e3-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-cqfdl\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:44:51.458275 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.458191    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93d1937b-f039-43e9-8609-c272baac54e3-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-cqfdl\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:44:51.458275 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.458238    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93d1937b-f039-43e9-8609-c272baac54e3-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-cqfdl\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:44:51.458433 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.458407    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93d1937b-f039-43e9-8609-c272baac54e3-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-cqfdl\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:44:51.458481 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.458456    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vclt7\" (UniqueName: \"kubernetes.io/projected/93d1937b-f039-43e9-8609-c272baac54e3-kube-api-access-vclt7\") pod \"router-gateway-1-openshift-default-5b4d9458bd-cqfdl\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:44:51.458524 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.458511    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d1937b-f039-43e9-8609-c272baac54e3-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-cqfdl\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:44:51.458572 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.458556    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d1937b-f039-43e9-8609-c272baac54e3-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-cqfdl\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:44:51.458611 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.458584    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93d1937b-f039-43e9-8609-c272baac54e3-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-cqfdl\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:44:51.458655 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.458612    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93d1937b-f039-43e9-8609-c272baac54e3-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-cqfdl\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:44:51.458655 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.458617    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93d1937b-f039-43e9-8609-c272baac54e3-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-cqfdl\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:44:51.458836 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.458814    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93d1937b-f039-43e9-8609-c272baac54e3-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-cqfdl\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:44:51.458908 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.458845    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d1937b-f039-43e9-8609-c272baac54e3-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-cqfdl\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:44:51.459003 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.458971    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93d1937b-f039-43e9-8609-c272baac54e3-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-cqfdl\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:44:51.460861 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.460846    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93d1937b-f039-43e9-8609-c272baac54e3-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-cqfdl\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:44:51.461023 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.461006    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93d1937b-f039-43e9-8609-c272baac54e3-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-cqfdl\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:44:51.480860 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.480839    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93d1937b-f039-43e9-8609-c272baac54e3-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-cqfdl\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:44:51.490746 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.490720    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vclt7\" (UniqueName: \"kubernetes.io/projected/93d1937b-f039-43e9-8609-c272baac54e3-kube-api-access-vclt7\") pod \"router-gateway-1-openshift-default-5b4d9458bd-cqfdl\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:44:51.601393 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.601336    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:44:51.895463 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:51.895434    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"]
Sep 17 18:44:51.896476 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:44:51.896437    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93d1937b_f039_43e9_8609_c272baac54e3.slice/crio-9cdcf5c59056131ceb8d4a1f08260d9c696a1393c1591d70a3f90abfdcf15e08 WatchSource:0}: Error finding container 9cdcf5c59056131ceb8d4a1f08260d9c696a1393c1591d70a3f90abfdcf15e08: Status 404 returned error can't find the container with id 9cdcf5c59056131ceb8d4a1f08260d9c696a1393c1591d70a3f90abfdcf15e08
Sep 17 18:44:52.398499 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:44:52.398462    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl" event={"ID":"93d1937b-f039-43e9-8609-c272baac54e3","Type":"ContainerStarted","Data":"9cdcf5c59056131ceb8d4a1f08260d9c696a1393c1591d70a3f90abfdcf15e08"}
Sep 17 18:45:01.694382 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:01.694311    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 17 18:45:01.694660 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:01.694422    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 17 18:45:01.694660 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:01.694451    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 17 18:45:02.453241 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:02.453188    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl" event={"ID":"93d1937b-f039-43e9-8609-c272baac54e3","Type":"ContainerStarted","Data":"ea9db6d8f3d140e6cb958736fce3a9299e92d0ed9d36783d011baa40303d5f15"}
Sep 17 18:45:02.475746 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:02.475692    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl" podStartSLOduration=1.6801607349999998 podStartE2EDuration="11.475677836s" totalImagesPullingTime="9.795517101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:44:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:45:02.473756776 +0000 UTC m=+960.405958295" watchObservedRunningTime="2026-09-17 18:45:02.475677836 +0000 UTC m=+960.407879354"
Sep 17 18:45:02.608424 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:02.608401    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:45:02.608530 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:02.608486    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:45:03.458089 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:03.458057    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:45:03.459127 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:03.459105    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:45:07.796170 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:07.796129    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q"]
Sep 17 18:45:07.801809 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:07.801780    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q"
Sep 17 18:45:07.805582 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:07.805562    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-sxjtf\""
Sep 17 18:45:07.806559 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:07.806539    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Sep 17 18:45:07.811191 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:07.811166    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q"]
Sep 17 18:45:07.882356 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:07.882333    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b9a2503-061b-45a0-ab68-fb9b860aa90c-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q\" (UID: \"5b9a2503-061b-45a0-ab68-fb9b860aa90c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q"
Sep 17 18:45:07.882482 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:07.882362    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b9a2503-061b-45a0-ab68-fb9b860aa90c-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q\" (UID: \"5b9a2503-061b-45a0-ab68-fb9b860aa90c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q"
Sep 17 18:45:07.882482 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:07.882390    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcvlw\" (UniqueName: \"kubernetes.io/projected/5b9a2503-061b-45a0-ab68-fb9b860aa90c-kube-api-access-qcvlw\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q\" (UID: \"5b9a2503-061b-45a0-ab68-fb9b860aa90c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q"
Sep 17 18:45:07.882482 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:07.882419    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b9a2503-061b-45a0-ab68-fb9b860aa90c-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q\" (UID: \"5b9a2503-061b-45a0-ab68-fb9b860aa90c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q"
Sep 17 18:45:07.882482 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:07.882448    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b9a2503-061b-45a0-ab68-fb9b860aa90c-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q\" (UID: \"5b9a2503-061b-45a0-ab68-fb9b860aa90c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q"
Sep 17 18:45:07.882622 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:07.882497    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b9a2503-061b-45a0-ab68-fb9b860aa90c-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q\" (UID: \"5b9a2503-061b-45a0-ab68-fb9b860aa90c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q"
Sep 17 18:45:07.982789 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:07.982767    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b9a2503-061b-45a0-ab68-fb9b860aa90c-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q\" (UID: \"5b9a2503-061b-45a0-ab68-fb9b860aa90c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q"
Sep 17 18:45:07.982902 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:07.982802    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b9a2503-061b-45a0-ab68-fb9b860aa90c-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q\" (UID: \"5b9a2503-061b-45a0-ab68-fb9b860aa90c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q"
Sep 17 18:45:07.982902 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:07.982819    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b9a2503-061b-45a0-ab68-fb9b860aa90c-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q\" (UID: \"5b9a2503-061b-45a0-ab68-fb9b860aa90c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q"
Sep 17 18:45:07.982902 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:07.982842    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcvlw\" (UniqueName: \"kubernetes.io/projected/5b9a2503-061b-45a0-ab68-fb9b860aa90c-kube-api-access-qcvlw\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q\" (UID: \"5b9a2503-061b-45a0-ab68-fb9b860aa90c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q"
Sep 17 18:45:07.983071 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:07.982964    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b9a2503-061b-45a0-ab68-fb9b860aa90c-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q\" (UID: \"5b9a2503-061b-45a0-ab68-fb9b860aa90c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q"
Sep 17 18:45:07.983071 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:07.983033    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b9a2503-061b-45a0-ab68-fb9b860aa90c-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q\" (UID: \"5b9a2503-061b-45a0-ab68-fb9b860aa90c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q"
Sep 17 18:45:07.983168 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:07.983139    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b9a2503-061b-45a0-ab68-fb9b860aa90c-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q\" (UID: \"5b9a2503-061b-45a0-ab68-fb9b860aa90c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q"
Sep 17 18:45:07.983232 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:07.983191    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b9a2503-061b-45a0-ab68-fb9b860aa90c-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q\" (UID: \"5b9a2503-061b-45a0-ab68-fb9b860aa90c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q"
Sep 17 18:45:07.983368 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:07.983336    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b9a2503-061b-45a0-ab68-fb9b860aa90c-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q\" (UID: \"5b9a2503-061b-45a0-ab68-fb9b860aa90c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q"
Sep 17 18:45:07.985071 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:07.985042    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b9a2503-061b-45a0-ab68-fb9b860aa90c-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q\" (UID: \"5b9a2503-061b-45a0-ab68-fb9b860aa90c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q"
Sep 17 18:45:07.985425 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:07.985404    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b9a2503-061b-45a0-ab68-fb9b860aa90c-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q\" (UID: \"5b9a2503-061b-45a0-ab68-fb9b860aa90c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q"
Sep 17 18:45:07.993835 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:07.993816    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcvlw\" (UniqueName: \"kubernetes.io/projected/5b9a2503-061b-45a0-ab68-fb9b860aa90c-kube-api-access-qcvlw\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q\" (UID: \"5b9a2503-061b-45a0-ab68-fb9b860aa90c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q"
Sep 17 18:45:08.113471 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:08.113411    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q"
Sep 17 18:45:08.387270 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:08.387242    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q"]
Sep 17 18:45:08.389338 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:45:08.389310    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b9a2503_061b_45a0_ab68_fb9b860aa90c.slice/crio-4274980cce7c0a066a213268f6576e3451c77aec34eeb83978462579d4817f2e WatchSource:0}: Error finding container 4274980cce7c0a066a213268f6576e3451c77aec34eeb83978462579d4817f2e: Status 404 returned error can't find the container with id 4274980cce7c0a066a213268f6576e3451c77aec34eeb83978462579d4817f2e
Sep 17 18:45:08.484755 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:08.484727    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q" event={"ID":"5b9a2503-061b-45a0-ab68-fb9b860aa90c","Type":"ContainerStarted","Data":"3e995ab6c1efc63b685bed1de7f1c1ff7224571306f164b5ad9f3b5542e2958a"}
Sep 17 18:45:08.484848 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:08.484776    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q" event={"ID":"5b9a2503-061b-45a0-ab68-fb9b860aa90c","Type":"ContainerStarted","Data":"4274980cce7c0a066a213268f6576e3451c77aec34eeb83978462579d4817f2e"}
Sep 17 18:45:08.525961 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:08.525916    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q" podStartSLOduration=1.525900815 podStartE2EDuration="1.525900815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:45:08.524451699 +0000 UTC m=+966.456653219" watchObservedRunningTime="2026-09-17 18:45:08.525900815 +0000 UTC m=+966.458102334"
Sep 17 18:45:09.113734 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:09.113693    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q"
Sep 17 18:45:16.020695 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.020647    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q"]
Sep 17 18:45:16.025947 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.025916    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"]
Sep 17 18:45:16.026178 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.026133    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl" podUID="93d1937b-f039-43e9-8609-c272baac54e3" containerName="istio-proxy" containerID="cri-o://ea9db6d8f3d140e6cb958736fce3a9299e92d0ed9d36783d011baa40303d5f15" gracePeriod=30
Sep 17 18:45:16.031175 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:45:16.031131    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:45:16.031175 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:45:16.031175 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:45:16.031175 ip-10-0-139-75 kubenswrapper[2565]:  > containerID="3e995ab6c1efc63b685bed1de7f1c1ff7224571306f164b5ad9f3b5542e2958a" cmd=["/bin/sleep","15"]
Sep 17 18:45:16.031387 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.031193    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q" message="" err=<
Sep 17 18:45:16.031387 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:45:16.031387 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:45:16.031387 ip-10-0-139-75 kubenswrapper[2565]:  >
Sep 17 18:45:16.031387 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:45:16.031237    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:45:16.031387 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:45:16.031387 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:45:16.031387 ip-10-0-139-75 kubenswrapper[2565]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q" podUID="5b9a2503-061b-45a0-ab68-fb9b860aa90c" containerName="main" containerID="cri-o://3e995ab6c1efc63b685bed1de7f1c1ff7224571306f164b5ad9f3b5542e2958a"
Sep 17 18:45:16.031387 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.031271    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q" podUID="5b9a2503-061b-45a0-ab68-fb9b860aa90c" containerName="main" containerID="cri-o://3e995ab6c1efc63b685bed1de7f1c1ff7224571306f164b5ad9f3b5542e2958a" gracePeriod=60
Sep 17 18:45:16.298753 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.298724    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q"
Sep 17 18:45:16.455346 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.455318    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b9a2503-061b-45a0-ab68-fb9b860aa90c-home\") pod \"5b9a2503-061b-45a0-ab68-fb9b860aa90c\" (UID: \"5b9a2503-061b-45a0-ab68-fb9b860aa90c\") "
Sep 17 18:45:16.455520 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.455379    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcvlw\" (UniqueName: \"kubernetes.io/projected/5b9a2503-061b-45a0-ab68-fb9b860aa90c-kube-api-access-qcvlw\") pod \"5b9a2503-061b-45a0-ab68-fb9b860aa90c\" (UID: \"5b9a2503-061b-45a0-ab68-fb9b860aa90c\") "
Sep 17 18:45:16.455520 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.455397    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b9a2503-061b-45a0-ab68-fb9b860aa90c-model-cache\") pod \"5b9a2503-061b-45a0-ab68-fb9b860aa90c\" (UID: \"5b9a2503-061b-45a0-ab68-fb9b860aa90c\") "
Sep 17 18:45:16.455520 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.455415    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b9a2503-061b-45a0-ab68-fb9b860aa90c-tmp-dir\") pod \"5b9a2503-061b-45a0-ab68-fb9b860aa90c\" (UID: \"5b9a2503-061b-45a0-ab68-fb9b860aa90c\") "
Sep 17 18:45:16.455520 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.455440    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b9a2503-061b-45a0-ab68-fb9b860aa90c-dshm\") pod \"5b9a2503-061b-45a0-ab68-fb9b860aa90c\" (UID: \"5b9a2503-061b-45a0-ab68-fb9b860aa90c\") "
Sep 17 18:45:16.455520 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.455476    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b9a2503-061b-45a0-ab68-fb9b860aa90c-tls-certs\") pod \"5b9a2503-061b-45a0-ab68-fb9b860aa90c\" (UID: \"5b9a2503-061b-45a0-ab68-fb9b860aa90c\") "
Sep 17 18:45:16.455803 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.455682    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9a2503-061b-45a0-ab68-fb9b860aa90c-model-cache" (OuterVolumeSpecName: "model-cache") pod "5b9a2503-061b-45a0-ab68-fb9b860aa90c" (UID: "5b9a2503-061b-45a0-ab68-fb9b860aa90c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:16.455803 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.455697    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9a2503-061b-45a0-ab68-fb9b860aa90c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b9a2503-061b-45a0-ab68-fb9b860aa90c" (UID: "5b9a2503-061b-45a0-ab68-fb9b860aa90c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:16.455947 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.455611    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9a2503-061b-45a0-ab68-fb9b860aa90c-home" (OuterVolumeSpecName: "home") pod "5b9a2503-061b-45a0-ab68-fb9b860aa90c" (UID: "5b9a2503-061b-45a0-ab68-fb9b860aa90c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:16.457719 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.457690    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b9a2503-061b-45a0-ab68-fb9b860aa90c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b9a2503-061b-45a0-ab68-fb9b860aa90c" (UID: "5b9a2503-061b-45a0-ab68-fb9b860aa90c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:45:16.457719 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.457710    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b9a2503-061b-45a0-ab68-fb9b860aa90c-dshm" (OuterVolumeSpecName: "dshm") pod "5b9a2503-061b-45a0-ab68-fb9b860aa90c" (UID: "5b9a2503-061b-45a0-ab68-fb9b860aa90c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:16.457839 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.457716    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b9a2503-061b-45a0-ab68-fb9b860aa90c-kube-api-access-qcvlw" (OuterVolumeSpecName: "kube-api-access-qcvlw") pod "5b9a2503-061b-45a0-ab68-fb9b860aa90c" (UID: "5b9a2503-061b-45a0-ab68-fb9b860aa90c"). InnerVolumeSpecName "kube-api-access-qcvlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:45:16.529669 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.529614    2565 generic.go:356] "Generic (PLEG): container finished" podID="5b9a2503-061b-45a0-ab68-fb9b860aa90c" containerID="3e995ab6c1efc63b685bed1de7f1c1ff7224571306f164b5ad9f3b5542e2958a" exitCode=0
Sep 17 18:45:16.529765 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.529673    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q"
Sep 17 18:45:16.529765 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.529704    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q" event={"ID":"5b9a2503-061b-45a0-ab68-fb9b860aa90c","Type":"ContainerDied","Data":"3e995ab6c1efc63b685bed1de7f1c1ff7224571306f164b5ad9f3b5542e2958a"}
Sep 17 18:45:16.529765 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.529753    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q" event={"ID":"5b9a2503-061b-45a0-ab68-fb9b860aa90c","Type":"ContainerDied","Data":"4274980cce7c0a066a213268f6576e3451c77aec34eeb83978462579d4817f2e"}
Sep 17 18:45:16.529881 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.529774    2565 scope.go:117] "RemoveContainer" containerID="3e995ab6c1efc63b685bed1de7f1c1ff7224571306f164b5ad9f3b5542e2958a"
Sep 17 18:45:16.538443 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.538424    2565 scope.go:117] "RemoveContainer" containerID="3e995ab6c1efc63b685bed1de7f1c1ff7224571306f164b5ad9f3b5542e2958a"
Sep 17 18:45:16.538735 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:45:16.538716    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e995ab6c1efc63b685bed1de7f1c1ff7224571306f164b5ad9f3b5542e2958a\": container with ID starting with 3e995ab6c1efc63b685bed1de7f1c1ff7224571306f164b5ad9f3b5542e2958a not found: ID does not exist" containerID="3e995ab6c1efc63b685bed1de7f1c1ff7224571306f164b5ad9f3b5542e2958a"
Sep 17 18:45:16.538779 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.538742    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e995ab6c1efc63b685bed1de7f1c1ff7224571306f164b5ad9f3b5542e2958a"} err="failed to get container status \"3e995ab6c1efc63b685bed1de7f1c1ff7224571306f164b5ad9f3b5542e2958a\": rpc error: code = NotFound desc = could not find container \"3e995ab6c1efc63b685bed1de7f1c1ff7224571306f164b5ad9f3b5542e2958a\": container with ID starting with 3e995ab6c1efc63b685bed1de7f1c1ff7224571306f164b5ad9f3b5542e2958a not found: ID does not exist"
Sep 17 18:45:16.556636 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.556616    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b9a2503-061b-45a0-ab68-fb9b860aa90c-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:45:16.556636 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.556637    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b9a2503-061b-45a0-ab68-fb9b860aa90c-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:45:16.556775 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.556648    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcvlw\" (UniqueName: \"kubernetes.io/projected/5b9a2503-061b-45a0-ab68-fb9b860aa90c-kube-api-access-qcvlw\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:45:16.556775 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.556658    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b9a2503-061b-45a0-ab68-fb9b860aa90c-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:45:16.556775 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.556666    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b9a2503-061b-45a0-ab68-fb9b860aa90c-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:45:16.556775 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.556673    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b9a2503-061b-45a0-ab68-fb9b860aa90c-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:45:16.557747 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.557729    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q"]
Sep 17 18:45:16.563287 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.563268    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-tvj5q"]
Sep 17 18:45:16.607978 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:16.607956    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b9a2503-061b-45a0-ab68-fb9b860aa90c" path="/var/lib/kubelet/pods/5b9a2503-061b-45a0-ab68-fb9b860aa90c/volumes"
Sep 17 18:45:17.870978 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:17.870956    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:45:17.968108 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:17.968080    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vclt7\" (UniqueName: \"kubernetes.io/projected/93d1937b-f039-43e9-8609-c272baac54e3-kube-api-access-vclt7\") pod \"93d1937b-f039-43e9-8609-c272baac54e3\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") "
Sep 17 18:45:17.968249 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:17.968118    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93d1937b-f039-43e9-8609-c272baac54e3-istio-token\") pod \"93d1937b-f039-43e9-8609-c272baac54e3\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") "
Sep 17 18:45:17.968249 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:17.968136    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93d1937b-f039-43e9-8609-c272baac54e3-istio-podinfo\") pod \"93d1937b-f039-43e9-8609-c272baac54e3\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") "
Sep 17 18:45:17.968249 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:17.968155    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93d1937b-f039-43e9-8609-c272baac54e3-workload-certs\") pod \"93d1937b-f039-43e9-8609-c272baac54e3\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") "
Sep 17 18:45:17.968249 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:17.968190    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93d1937b-f039-43e9-8609-c272baac54e3-istiod-ca-cert\") pod \"93d1937b-f039-43e9-8609-c272baac54e3\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") "
Sep 17 18:45:17.968497 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:17.968465    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93d1937b-f039-43e9-8609-c272baac54e3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "93d1937b-f039-43e9-8609-c272baac54e3" (UID: "93d1937b-f039-43e9-8609-c272baac54e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:17.968576 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:17.968552    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d1937b-f039-43e9-8609-c272baac54e3-credential-socket\") pod \"93d1937b-f039-43e9-8609-c272baac54e3\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") "
Sep 17 18:45:17.968643 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:17.968600    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d1937b-f039-43e9-8609-c272baac54e3-workload-socket\") pod \"93d1937b-f039-43e9-8609-c272baac54e3\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") "
Sep 17 18:45:17.968643 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:17.968629    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93d1937b-f039-43e9-8609-c272baac54e3-istio-data\") pod \"93d1937b-f039-43e9-8609-c272baac54e3\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") "
Sep 17 18:45:17.968740 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:17.968659    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93d1937b-f039-43e9-8609-c272baac54e3-istio-envoy\") pod \"93d1937b-f039-43e9-8609-c272baac54e3\" (UID: \"93d1937b-f039-43e9-8609-c272baac54e3\") "
Sep 17 18:45:17.968909 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:17.968886    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93d1937b-f039-43e9-8609-c272baac54e3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "93d1937b-f039-43e9-8609-c272baac54e3" (UID: "93d1937b-f039-43e9-8609-c272baac54e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:45:17.968909 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:17.968895    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93d1937b-f039-43e9-8609-c272baac54e3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "93d1937b-f039-43e9-8609-c272baac54e3" (UID: "93d1937b-f039-43e9-8609-c272baac54e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:17.969022 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:17.968967    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93d1937b-f039-43e9-8609-c272baac54e3-workload-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:45:17.969022 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:17.968988    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93d1937b-f039-43e9-8609-c272baac54e3-istiod-ca-cert\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:45:17.969022 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:17.969001    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d1937b-f039-43e9-8609-c272baac54e3-workload-socket\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:45:17.969185 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:17.969086    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93d1937b-f039-43e9-8609-c272baac54e3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "93d1937b-f039-43e9-8609-c272baac54e3" (UID: "93d1937b-f039-43e9-8609-c272baac54e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:17.969185 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:17.969096    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93d1937b-f039-43e9-8609-c272baac54e3-istio-data" (OuterVolumeSpecName: "istio-data") pod "93d1937b-f039-43e9-8609-c272baac54e3" (UID: "93d1937b-f039-43e9-8609-c272baac54e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:17.970520 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:17.970492    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93d1937b-f039-43e9-8609-c272baac54e3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "93d1937b-f039-43e9-8609-c272baac54e3" (UID: "93d1937b-f039-43e9-8609-c272baac54e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 18:45:17.970614 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:17.970518    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93d1937b-f039-43e9-8609-c272baac54e3-istio-token" (OuterVolumeSpecName: "istio-token") pod "93d1937b-f039-43e9-8609-c272baac54e3" (UID: "93d1937b-f039-43e9-8609-c272baac54e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:45:17.970769 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:17.970747    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93d1937b-f039-43e9-8609-c272baac54e3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "93d1937b-f039-43e9-8609-c272baac54e3" (UID: "93d1937b-f039-43e9-8609-c272baac54e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:45:17.970897 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:17.970880    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93d1937b-f039-43e9-8609-c272baac54e3-kube-api-access-vclt7" (OuterVolumeSpecName: "kube-api-access-vclt7") pod "93d1937b-f039-43e9-8609-c272baac54e3" (UID: "93d1937b-f039-43e9-8609-c272baac54e3"). InnerVolumeSpecName "kube-api-access-vclt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:45:18.069635 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:18.069611    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vclt7\" (UniqueName: \"kubernetes.io/projected/93d1937b-f039-43e9-8609-c272baac54e3-kube-api-access-vclt7\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:45:18.069635 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:18.069635    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93d1937b-f039-43e9-8609-c272baac54e3-istio-token\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:45:18.069772 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:18.069645    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93d1937b-f039-43e9-8609-c272baac54e3-istio-podinfo\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:45:18.069772 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:18.069653    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93d1937b-f039-43e9-8609-c272baac54e3-credential-socket\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:45:18.069772 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:18.069661    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93d1937b-f039-43e9-8609-c272baac54e3-istio-data\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:45:18.069772 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:18.069669    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93d1937b-f039-43e9-8609-c272baac54e3-istio-envoy\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:45:18.542375 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:18.542347    2565 generic.go:356] "Generic (PLEG): container finished" podID="93d1937b-f039-43e9-8609-c272baac54e3" containerID="ea9db6d8f3d140e6cb958736fce3a9299e92d0ed9d36783d011baa40303d5f15" exitCode=0
Sep 17 18:45:18.542494 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:18.542416    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"
Sep 17 18:45:18.542494 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:18.542429    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl" event={"ID":"93d1937b-f039-43e9-8609-c272baac54e3","Type":"ContainerDied","Data":"ea9db6d8f3d140e6cb958736fce3a9299e92d0ed9d36783d011baa40303d5f15"}
Sep 17 18:45:18.542598 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:18.542511    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl" event={"ID":"93d1937b-f039-43e9-8609-c272baac54e3","Type":"ContainerDied","Data":"9cdcf5c59056131ceb8d4a1f08260d9c696a1393c1591d70a3f90abfdcf15e08"}
Sep 17 18:45:18.542598 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:18.542536    2565 scope.go:117] "RemoveContainer" containerID="ea9db6d8f3d140e6cb958736fce3a9299e92d0ed9d36783d011baa40303d5f15"
Sep 17 18:45:18.552411 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:18.552390    2565 scope.go:117] "RemoveContainer" containerID="ea9db6d8f3d140e6cb958736fce3a9299e92d0ed9d36783d011baa40303d5f15"
Sep 17 18:45:18.552675 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:45:18.552655    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea9db6d8f3d140e6cb958736fce3a9299e92d0ed9d36783d011baa40303d5f15\": container with ID starting with ea9db6d8f3d140e6cb958736fce3a9299e92d0ed9d36783d011baa40303d5f15 not found: ID does not exist" containerID="ea9db6d8f3d140e6cb958736fce3a9299e92d0ed9d36783d011baa40303d5f15"
Sep 17 18:45:18.552740 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:18.552686    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea9db6d8f3d140e6cb958736fce3a9299e92d0ed9d36783d011baa40303d5f15"} err="failed to get container status \"ea9db6d8f3d140e6cb958736fce3a9299e92d0ed9d36783d011baa40303d5f15\": rpc error: code = NotFound desc = could not find container \"ea9db6d8f3d140e6cb958736fce3a9299e92d0ed9d36783d011baa40303d5f15\": container with ID starting with ea9db6d8f3d140e6cb958736fce3a9299e92d0ed9d36783d011baa40303d5f15 not found: ID does not exist"
Sep 17 18:45:18.564839 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:18.564817    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"]
Sep 17 18:45:18.568721 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:18.568702    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-cqfdl"]
Sep 17 18:45:18.606471 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:18.606449    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93d1937b-f039-43e9-8609-c272baac54e3" path="/var/lib/kubelet/pods/93d1937b-f039-43e9-8609-c272baac54e3/volumes"
Sep 17 18:45:51.872869 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:51.872837    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"]
Sep 17 18:45:51.873335 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:51.873183    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b9a2503-061b-45a0-ab68-fb9b860aa90c" containerName="main"
Sep 17 18:45:51.873335 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:51.873195    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b9a2503-061b-45a0-ab68-fb9b860aa90c" containerName="main"
Sep 17 18:45:51.873335 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:51.873218    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93d1937b-f039-43e9-8609-c272baac54e3" containerName="istio-proxy"
Sep 17 18:45:51.873335 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:51.873224    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="93d1937b-f039-43e9-8609-c272baac54e3" containerName="istio-proxy"
Sep 17 18:45:51.873335 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:51.873275    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b9a2503-061b-45a0-ab68-fb9b860aa90c" containerName="main"
Sep 17 18:45:51.873335 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:51.873288    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="93d1937b-f039-43e9-8609-c272baac54e3" containerName="istio-proxy"
Sep 17 18:45:51.876425 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:51.876405    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"
Sep 17 18:45:51.880730 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:51.880708    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 17 18:45:51.880846 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:51.880825    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-txtkm\""
Sep 17 18:45:51.881611 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:51.881592    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 17 18:45:51.881727 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:51.881592    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 17 18:45:51.881727 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:51.881622    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-wrxbl\""
Sep 17 18:45:51.896179 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:51.896152    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"]
Sep 17 18:45:51.913977 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:51.913949    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19181c95-7c8a-4419-89a2-55d370e1249c-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s\" (UID: \"19181c95-7c8a-4419-89a2-55d370e1249c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"
Sep 17 18:45:51.914080 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:51.913993    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19181c95-7c8a-4419-89a2-55d370e1249c-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s\" (UID: \"19181c95-7c8a-4419-89a2-55d370e1249c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"
Sep 17 18:45:51.914080 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:51.914032    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19181c95-7c8a-4419-89a2-55d370e1249c-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s\" (UID: \"19181c95-7c8a-4419-89a2-55d370e1249c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"
Sep 17 18:45:51.914080 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:51.914054    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpczk\" (UniqueName: \"kubernetes.io/projected/19181c95-7c8a-4419-89a2-55d370e1249c-kube-api-access-mpczk\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s\" (UID: \"19181c95-7c8a-4419-89a2-55d370e1249c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"
Sep 17 18:45:51.914221 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:51.914081    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19181c95-7c8a-4419-89a2-55d370e1249c-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s\" (UID: \"19181c95-7c8a-4419-89a2-55d370e1249c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"
Sep 17 18:45:51.914221 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:51.914113    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19181c95-7c8a-4419-89a2-55d370e1249c-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s\" (UID: \"19181c95-7c8a-4419-89a2-55d370e1249c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"
Sep 17 18:45:52.014815 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:52.014781    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19181c95-7c8a-4419-89a2-55d370e1249c-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s\" (UID: \"19181c95-7c8a-4419-89a2-55d370e1249c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"
Sep 17 18:45:52.014969 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:52.014832    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19181c95-7c8a-4419-89a2-55d370e1249c-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s\" (UID: \"19181c95-7c8a-4419-89a2-55d370e1249c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"
Sep 17 18:45:52.014969 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:52.014856    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19181c95-7c8a-4419-89a2-55d370e1249c-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s\" (UID: \"19181c95-7c8a-4419-89a2-55d370e1249c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"
Sep 17 18:45:52.014969 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:52.014886    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpczk\" (UniqueName: \"kubernetes.io/projected/19181c95-7c8a-4419-89a2-55d370e1249c-kube-api-access-mpczk\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s\" (UID: \"19181c95-7c8a-4419-89a2-55d370e1249c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"
Sep 17 18:45:52.014969 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:52.014918    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19181c95-7c8a-4419-89a2-55d370e1249c-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s\" (UID: \"19181c95-7c8a-4419-89a2-55d370e1249c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"
Sep 17 18:45:52.014969 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:52.014957    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19181c95-7c8a-4419-89a2-55d370e1249c-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s\" (UID: \"19181c95-7c8a-4419-89a2-55d370e1249c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"
Sep 17 18:45:52.015307 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:52.015245    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19181c95-7c8a-4419-89a2-55d370e1249c-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s\" (UID: \"19181c95-7c8a-4419-89a2-55d370e1249c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"
Sep 17 18:45:52.015370 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:52.015344    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19181c95-7c8a-4419-89a2-55d370e1249c-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s\" (UID: \"19181c95-7c8a-4419-89a2-55d370e1249c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"
Sep 17 18:45:52.015418 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:52.015396    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19181c95-7c8a-4419-89a2-55d370e1249c-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s\" (UID: \"19181c95-7c8a-4419-89a2-55d370e1249c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"
Sep 17 18:45:52.017162 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:52.017134    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19181c95-7c8a-4419-89a2-55d370e1249c-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s\" (UID: \"19181c95-7c8a-4419-89a2-55d370e1249c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"
Sep 17 18:45:52.017451 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:52.017429    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19181c95-7c8a-4419-89a2-55d370e1249c-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s\" (UID: \"19181c95-7c8a-4419-89a2-55d370e1249c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"
Sep 17 18:45:52.024823 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:52.024803    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpczk\" (UniqueName: \"kubernetes.io/projected/19181c95-7c8a-4419-89a2-55d370e1249c-kube-api-access-mpczk\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s\" (UID: \"19181c95-7c8a-4419-89a2-55d370e1249c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"
Sep 17 18:45:52.186005 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:52.185974    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"
Sep 17 18:45:52.460447 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:52.460420    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"]
Sep 17 18:45:52.462421 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:45:52.462390    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19181c95_7c8a_4419_89a2_55d370e1249c.slice/crio-f549f5c88868a18b87e07ddb651792e37f72d3aa61399539528a77cdce202c38 WatchSource:0}: Error finding container f549f5c88868a18b87e07ddb651792e37f72d3aa61399539528a77cdce202c38: Status 404 returned error can't find the container with id f549f5c88868a18b87e07ddb651792e37f72d3aa61399539528a77cdce202c38
Sep 17 18:45:52.464502 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:52.464481    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:45:52.722843 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:52.722768    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s" event={"ID":"19181c95-7c8a-4419-89a2-55d370e1249c","Type":"ContainerStarted","Data":"f549f5c88868a18b87e07ddb651792e37f72d3aa61399539528a77cdce202c38"}
Sep 17 18:45:54.735742 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:54.735706    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s" event={"ID":"19181c95-7c8a-4419-89a2-55d370e1249c","Type":"ContainerStarted","Data":"42515b7a1bf8e60d697ffc32dba19fec8249e38af471515a26ab02082f673d5b"}
Sep 17 18:45:54.736231 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:54.735780    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"
Sep 17 18:45:55.742135 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:55.742099    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s" event={"ID":"19181c95-7c8a-4419-89a2-55d370e1249c","Type":"ContainerStarted","Data":"fbd7903a64d569b561f3d46721a5ae69314ab6bf7425659d9b9d9f530e6512f4"}
Sep 17 18:45:55.763755 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:55.763709    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s" podStartSLOduration=3.379288915 podStartE2EDuration="4.763692723s" totalImagesPullingTime="1.384403808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:45:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:45:55.76229999 +0000 UTC m=+1013.694501511" watchObservedRunningTime="2026-09-17 18:45:55.763692723 +0000 UTC m=+1013.695894242"
Sep 17 18:45:56.186924 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:56.186893    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"
Sep 17 18:45:59.521864 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:59.521830    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd"]
Sep 17 18:45:59.525495 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:59.525476    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd"
Sep 17 18:45:59.532455 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:59.532431    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 17 18:45:59.532455 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:59.532445    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-mwbhw\""
Sep 17 18:45:59.532651 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:59.532466    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Sep 17 18:45:59.532651 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:59.532481    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 17 18:45:59.539859 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:59.539836    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd"]
Sep 17 18:45:59.571159 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:59.571133    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a5e6b4-5a29-4082-809f-4126c1f71589-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd\" (UID: \"71a5e6b4-5a29-4082-809f-4126c1f71589\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd"
Sep 17 18:45:59.571268 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:59.571173    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a5e6b4-5a29-4082-809f-4126c1f71589-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd\" (UID: \"71a5e6b4-5a29-4082-809f-4126c1f71589\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd"
Sep 17 18:45:59.571268 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:59.571256    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s8p9\" (UniqueName: \"kubernetes.io/projected/71a5e6b4-5a29-4082-809f-4126c1f71589-kube-api-access-4s8p9\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd\" (UID: \"71a5e6b4-5a29-4082-809f-4126c1f71589\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd"
Sep 17 18:45:59.571390 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:59.571314    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a5e6b4-5a29-4082-809f-4126c1f71589-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd\" (UID: \"71a5e6b4-5a29-4082-809f-4126c1f71589\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd"
Sep 17 18:45:59.571390 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:59.571376    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a5e6b4-5a29-4082-809f-4126c1f71589-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd\" (UID: \"71a5e6b4-5a29-4082-809f-4126c1f71589\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd"
Sep 17 18:45:59.571492 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:59.571407    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a5e6b4-5a29-4082-809f-4126c1f71589-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd\" (UID: \"71a5e6b4-5a29-4082-809f-4126c1f71589\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd"
Sep 17 18:45:59.672160 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:59.672136    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s8p9\" (UniqueName: \"kubernetes.io/projected/71a5e6b4-5a29-4082-809f-4126c1f71589-kube-api-access-4s8p9\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd\" (UID: \"71a5e6b4-5a29-4082-809f-4126c1f71589\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd"
Sep 17 18:45:59.672294 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:59.672167    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a5e6b4-5a29-4082-809f-4126c1f71589-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd\" (UID: \"71a5e6b4-5a29-4082-809f-4126c1f71589\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd"
Sep 17 18:45:59.672294 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:59.672216    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a5e6b4-5a29-4082-809f-4126c1f71589-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd\" (UID: \"71a5e6b4-5a29-4082-809f-4126c1f71589\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd"
Sep 17 18:45:59.672294 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:59.672245    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a5e6b4-5a29-4082-809f-4126c1f71589-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd\" (UID: \"71a5e6b4-5a29-4082-809f-4126c1f71589\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd"
Sep 17 18:45:59.672294 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:59.672277    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a5e6b4-5a29-4082-809f-4126c1f71589-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd\" (UID: \"71a5e6b4-5a29-4082-809f-4126c1f71589\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd"
Sep 17 18:45:59.672538 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:59.672323    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a5e6b4-5a29-4082-809f-4126c1f71589-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd\" (UID: \"71a5e6b4-5a29-4082-809f-4126c1f71589\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd"
Sep 17 18:45:59.672613 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:59.672591    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a5e6b4-5a29-4082-809f-4126c1f71589-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd\" (UID: \"71a5e6b4-5a29-4082-809f-4126c1f71589\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd"
Sep 17 18:45:59.672662 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:59.672638    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a5e6b4-5a29-4082-809f-4126c1f71589-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd\" (UID: \"71a5e6b4-5a29-4082-809f-4126c1f71589\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd"
Sep 17 18:45:59.672697 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:59.672684    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a5e6b4-5a29-4082-809f-4126c1f71589-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd\" (UID: \"71a5e6b4-5a29-4082-809f-4126c1f71589\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd"
Sep 17 18:45:59.674503 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:59.674484    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a5e6b4-5a29-4082-809f-4126c1f71589-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd\" (UID: \"71a5e6b4-5a29-4082-809f-4126c1f71589\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd"
Sep 17 18:45:59.674630 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:59.674614    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a5e6b4-5a29-4082-809f-4126c1f71589-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd\" (UID: \"71a5e6b4-5a29-4082-809f-4126c1f71589\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd"
Sep 17 18:45:59.683168 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:59.683148    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s8p9\" (UniqueName: \"kubernetes.io/projected/71a5e6b4-5a29-4082-809f-4126c1f71589-kube-api-access-4s8p9\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd\" (UID: \"71a5e6b4-5a29-4082-809f-4126c1f71589\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd"
Sep 17 18:45:59.835548 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:45:59.835493    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd"
Sep 17 18:46:00.115537 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:00.115506    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd"]
Sep 17 18:46:00.117139 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:46:00.117106    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a5e6b4_5a29_4082_809f_4126c1f71589.slice/crio-f22bbaa253c249d913c2b0477762f264a489d3ba9452857528e2d5df00ad400b WatchSource:0}: Error finding container f22bbaa253c249d913c2b0477762f264a489d3ba9452857528e2d5df00ad400b: Status 404 returned error can't find the container with id f22bbaa253c249d913c2b0477762f264a489d3ba9452857528e2d5df00ad400b
Sep 17 18:46:00.771874 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:00.771835    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd" event={"ID":"71a5e6b4-5a29-4082-809f-4126c1f71589","Type":"ContainerStarted","Data":"b3d578a3e77352c5bbeb286a1f1bf0a4a98dc98a1ef3eb3497afa17445585414"}
Sep 17 18:46:00.772268 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:00.771895    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd" event={"ID":"71a5e6b4-5a29-4082-809f-4126c1f71589","Type":"ContainerStarted","Data":"f22bbaa253c249d913c2b0477762f264a489d3ba9452857528e2d5df00ad400b"}
Sep 17 18:46:00.798115 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:00.798062    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd" podStartSLOduration=1.798046436 podStartE2EDuration="1.798046436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:46:00.796588077 +0000 UTC m=+1018.728789595" watchObservedRunningTime="2026-09-17 18:46:00.798046436 +0000 UTC m=+1018.730247955"
Sep 17 18:46:00.836333 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:00.836306    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd"
Sep 17 18:46:02.187187 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:02.187149    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"
Sep 17 18:46:02.199462 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:02.199441    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"
Sep 17 18:46:02.795771 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:02.795742    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"
Sep 17 18:46:03.801185 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:03.801152    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"
Sep 17 18:46:07.499709 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:07.499671    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd"]
Sep 17 18:46:07.509186 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:46:07.509154    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:46:07.509186 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:46:07.509186 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:46:07.509186 ip-10-0-139-75 kubenswrapper[2565]:  > containerID="b3d578a3e77352c5bbeb286a1f1bf0a4a98dc98a1ef3eb3497afa17445585414" cmd=["/bin/sleep","15"]
Sep 17 18:46:07.509473 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:07.509223    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd" message="" err=<
Sep 17 18:46:07.509473 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:46:07.509473 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:46:07.509473 ip-10-0-139-75 kubenswrapper[2565]:  >
Sep 17 18:46:07.509473 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:46:07.509237    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:46:07.509473 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:46:07.509473 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:46:07.509473 ip-10-0-139-75 kubenswrapper[2565]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd" podUID="71a5e6b4-5a29-4082-809f-4126c1f71589" containerName="main" containerID="cri-o://b3d578a3e77352c5bbeb286a1f1bf0a4a98dc98a1ef3eb3497afa17445585414"
Sep 17 18:46:07.509473 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:07.509272    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd" podUID="71a5e6b4-5a29-4082-809f-4126c1f71589" containerName="main" containerID="cri-o://b3d578a3e77352c5bbeb286a1f1bf0a4a98dc98a1ef3eb3497afa17445585414" gracePeriod=60
Sep 17 18:46:07.764043 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:07.764012    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd"
Sep 17 18:46:07.811828 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:07.811800    2565 generic.go:356] "Generic (PLEG): container finished" podID="71a5e6b4-5a29-4082-809f-4126c1f71589" containerID="b3d578a3e77352c5bbeb286a1f1bf0a4a98dc98a1ef3eb3497afa17445585414" exitCode=0
Sep 17 18:46:07.811947 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:07.811843    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd"
Sep 17 18:46:07.811947 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:07.811891    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd" event={"ID":"71a5e6b4-5a29-4082-809f-4126c1f71589","Type":"ContainerDied","Data":"b3d578a3e77352c5bbeb286a1f1bf0a4a98dc98a1ef3eb3497afa17445585414"}
Sep 17 18:46:07.811947 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:07.811938    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd" event={"ID":"71a5e6b4-5a29-4082-809f-4126c1f71589","Type":"ContainerDied","Data":"f22bbaa253c249d913c2b0477762f264a489d3ba9452857528e2d5df00ad400b"}
Sep 17 18:46:07.812072 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:07.811960    2565 scope.go:117] "RemoveContainer" containerID="b3d578a3e77352c5bbeb286a1f1bf0a4a98dc98a1ef3eb3497afa17445585414"
Sep 17 18:46:07.820237 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:07.820193    2565 scope.go:117] "RemoveContainer" containerID="b3d578a3e77352c5bbeb286a1f1bf0a4a98dc98a1ef3eb3497afa17445585414"
Sep 17 18:46:07.820550 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:46:07.820530    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3d578a3e77352c5bbeb286a1f1bf0a4a98dc98a1ef3eb3497afa17445585414\": container with ID starting with b3d578a3e77352c5bbeb286a1f1bf0a4a98dc98a1ef3eb3497afa17445585414 not found: ID does not exist" containerID="b3d578a3e77352c5bbeb286a1f1bf0a4a98dc98a1ef3eb3497afa17445585414"
Sep 17 18:46:07.820625 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:07.820558    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3d578a3e77352c5bbeb286a1f1bf0a4a98dc98a1ef3eb3497afa17445585414"} err="failed to get container status \"b3d578a3e77352c5bbeb286a1f1bf0a4a98dc98a1ef3eb3497afa17445585414\": rpc error: code = NotFound desc = could not find container \"b3d578a3e77352c5bbeb286a1f1bf0a4a98dc98a1ef3eb3497afa17445585414\": container with ID starting with b3d578a3e77352c5bbeb286a1f1bf0a4a98dc98a1ef3eb3497afa17445585414 not found: ID does not exist"
Sep 17 18:46:07.835728 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:07.835709    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a5e6b4-5a29-4082-809f-4126c1f71589-model-cache\") pod \"71a5e6b4-5a29-4082-809f-4126c1f71589\" (UID: \"71a5e6b4-5a29-4082-809f-4126c1f71589\") "
Sep 17 18:46:07.835826 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:07.835743    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a5e6b4-5a29-4082-809f-4126c1f71589-tmp-dir\") pod \"71a5e6b4-5a29-4082-809f-4126c1f71589\" (UID: \"71a5e6b4-5a29-4082-809f-4126c1f71589\") "
Sep 17 18:46:07.835826 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:07.835770    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a5e6b4-5a29-4082-809f-4126c1f71589-tls-certs\") pod \"71a5e6b4-5a29-4082-809f-4126c1f71589\" (UID: \"71a5e6b4-5a29-4082-809f-4126c1f71589\") "
Sep 17 18:46:07.835826 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:07.835786    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a5e6b4-5a29-4082-809f-4126c1f71589-dshm\") pod \"71a5e6b4-5a29-4082-809f-4126c1f71589\" (UID: \"71a5e6b4-5a29-4082-809f-4126c1f71589\") "
Sep 17 18:46:07.835826 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:07.835816    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a5e6b4-5a29-4082-809f-4126c1f71589-home\") pod \"71a5e6b4-5a29-4082-809f-4126c1f71589\" (UID: \"71a5e6b4-5a29-4082-809f-4126c1f71589\") "
Sep 17 18:46:07.836014 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:07.835851    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s8p9\" (UniqueName: \"kubernetes.io/projected/71a5e6b4-5a29-4082-809f-4126c1f71589-kube-api-access-4s8p9\") pod \"71a5e6b4-5a29-4082-809f-4126c1f71589\" (UID: \"71a5e6b4-5a29-4082-809f-4126c1f71589\") "
Sep 17 18:46:07.836014 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:07.836001    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a5e6b4-5a29-4082-809f-4126c1f71589-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "71a5e6b4-5a29-4082-809f-4126c1f71589" (UID: "71a5e6b4-5a29-4082-809f-4126c1f71589"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:46:07.836101 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:07.836018    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a5e6b4-5a29-4082-809f-4126c1f71589-model-cache" (OuterVolumeSpecName: "model-cache") pod "71a5e6b4-5a29-4082-809f-4126c1f71589" (UID: "71a5e6b4-5a29-4082-809f-4126c1f71589"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:46:07.836245 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:07.836220    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a5e6b4-5a29-4082-809f-4126c1f71589-home" (OuterVolumeSpecName: "home") pod "71a5e6b4-5a29-4082-809f-4126c1f71589" (UID: "71a5e6b4-5a29-4082-809f-4126c1f71589"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:46:07.837985 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:07.837955    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71a5e6b4-5a29-4082-809f-4126c1f71589-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "71a5e6b4-5a29-4082-809f-4126c1f71589" (UID: "71a5e6b4-5a29-4082-809f-4126c1f71589"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:46:07.837985 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:07.837962    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a5e6b4-5a29-4082-809f-4126c1f71589-kube-api-access-4s8p9" (OuterVolumeSpecName: "kube-api-access-4s8p9") pod "71a5e6b4-5a29-4082-809f-4126c1f71589" (UID: "71a5e6b4-5a29-4082-809f-4126c1f71589"). InnerVolumeSpecName "kube-api-access-4s8p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:46:07.838131 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:07.838013    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a5e6b4-5a29-4082-809f-4126c1f71589-dshm" (OuterVolumeSpecName: "dshm") pod "71a5e6b4-5a29-4082-809f-4126c1f71589" (UID: "71a5e6b4-5a29-4082-809f-4126c1f71589"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:46:07.936808 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:07.936782    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a5e6b4-5a29-4082-809f-4126c1f71589-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:46:07.936808 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:07.936806    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a5e6b4-5a29-4082-809f-4126c1f71589-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:46:07.936917 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:07.936815    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a5e6b4-5a29-4082-809f-4126c1f71589-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:46:07.936917 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:07.936822    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a5e6b4-5a29-4082-809f-4126c1f71589-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:46:07.936917 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:07.936830    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a5e6b4-5a29-4082-809f-4126c1f71589-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:46:07.936917 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:07.936838    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s8p9\" (UniqueName: \"kubernetes.io/projected/71a5e6b4-5a29-4082-809f-4126c1f71589-kube-api-access-4s8p9\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:46:08.135587 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:08.135557    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd"]
Sep 17 18:46:08.139345 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:08.139320    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf296gd"]
Sep 17 18:46:08.607985 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:08.607954    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71a5e6b4-5a29-4082-809f-4126c1f71589" path="/var/lib/kubelet/pods/71a5e6b4-5a29-4082-809f-4126c1f71589/volumes"
Sep 17 18:46:32.340193 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.340158    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"]
Sep 17 18:46:32.340630 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.340560    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71a5e6b4-5a29-4082-809f-4126c1f71589" containerName="main"
Sep 17 18:46:32.340630 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.340572    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="71a5e6b4-5a29-4082-809f-4126c1f71589" containerName="main"
Sep 17 18:46:32.340630 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.340630    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="71a5e6b4-5a29-4082-809f-4126c1f71589" containerName="main"
Sep 17 18:46:32.343716 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.343695    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:46:32.346094 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.346071    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-8h246\""
Sep 17 18:46:32.346321 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.346284    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 17 18:46:32.346555 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.346538    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 17 18:46:32.346858 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.346833    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:46:32.347130 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.347112    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 17 18:46:32.348261 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.348240    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 17 18:46:32.357849 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.357828    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"]
Sep 17 18:46:32.410617 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.410594    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a85c292c-a2fd-4ff7-b74d-b2c25f980469-dshm\") pod \"auth-enabled-test-kserve-5bb9dd765c-q6nrw\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:46:32.410715 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.410634    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a85c292c-a2fd-4ff7-b74d-b2c25f980469-model-cache\") pod \"auth-enabled-test-kserve-5bb9dd765c-q6nrw\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:46:32.410715 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.410652    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a85c292c-a2fd-4ff7-b74d-b2c25f980469-tmp-dir\") pod \"auth-enabled-test-kserve-5bb9dd765c-q6nrw\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:46:32.410715 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.410670    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a85c292c-a2fd-4ff7-b74d-b2c25f980469-kserve-provision-location\") pod \"auth-enabled-test-kserve-5bb9dd765c-q6nrw\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:46:32.410715 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.410711    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a85c292c-a2fd-4ff7-b74d-b2c25f980469-cabundle-cert\") pod \"auth-enabled-test-kserve-5bb9dd765c-q6nrw\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:46:32.410856 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.410732    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a85c292c-a2fd-4ff7-b74d-b2c25f980469-home\") pod \"auth-enabled-test-kserve-5bb9dd765c-q6nrw\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:46:32.410856 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.410751    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phzgm\" (UniqueName: \"kubernetes.io/projected/a85c292c-a2fd-4ff7-b74d-b2c25f980469-kube-api-access-phzgm\") pod \"auth-enabled-test-kserve-5bb9dd765c-q6nrw\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:46:32.410856 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.410800    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a85c292c-a2fd-4ff7-b74d-b2c25f980469-tls-certs\") pod \"auth-enabled-test-kserve-5bb9dd765c-q6nrw\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:46:32.511669 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.511637    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a85c292c-a2fd-4ff7-b74d-b2c25f980469-model-cache\") pod \"auth-enabled-test-kserve-5bb9dd765c-q6nrw\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:46:32.511791 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.511674    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a85c292c-a2fd-4ff7-b74d-b2c25f980469-tmp-dir\") pod \"auth-enabled-test-kserve-5bb9dd765c-q6nrw\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:46:32.511791 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.511703    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a85c292c-a2fd-4ff7-b74d-b2c25f980469-kserve-provision-location\") pod \"auth-enabled-test-kserve-5bb9dd765c-q6nrw\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:46:32.511791 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.511751    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a85c292c-a2fd-4ff7-b74d-b2c25f980469-cabundle-cert\") pod \"auth-enabled-test-kserve-5bb9dd765c-q6nrw\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:46:32.511791 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.511785    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a85c292c-a2fd-4ff7-b74d-b2c25f980469-home\") pod \"auth-enabled-test-kserve-5bb9dd765c-q6nrw\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:46:32.512013 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.511812    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phzgm\" (UniqueName: \"kubernetes.io/projected/a85c292c-a2fd-4ff7-b74d-b2c25f980469-kube-api-access-phzgm\") pod \"auth-enabled-test-kserve-5bb9dd765c-q6nrw\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:46:32.512013 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.511848    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a85c292c-a2fd-4ff7-b74d-b2c25f980469-tls-certs\") pod \"auth-enabled-test-kserve-5bb9dd765c-q6nrw\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:46:32.512013 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.511909    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a85c292c-a2fd-4ff7-b74d-b2c25f980469-dshm\") pod \"auth-enabled-test-kserve-5bb9dd765c-q6nrw\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:46:32.512148 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.512032    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a85c292c-a2fd-4ff7-b74d-b2c25f980469-model-cache\") pod \"auth-enabled-test-kserve-5bb9dd765c-q6nrw\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:46:32.512148 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.512099    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a85c292c-a2fd-4ff7-b74d-b2c25f980469-tmp-dir\") pod \"auth-enabled-test-kserve-5bb9dd765c-q6nrw\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:46:32.512323 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.512304    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a85c292c-a2fd-4ff7-b74d-b2c25f980469-home\") pod \"auth-enabled-test-kserve-5bb9dd765c-q6nrw\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:46:32.512390 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.512332    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a85c292c-a2fd-4ff7-b74d-b2c25f980469-kserve-provision-location\") pod \"auth-enabled-test-kserve-5bb9dd765c-q6nrw\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:46:32.512576 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.512558    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a85c292c-a2fd-4ff7-b74d-b2c25f980469-cabundle-cert\") pod \"auth-enabled-test-kserve-5bb9dd765c-q6nrw\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:46:32.514213 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.514181    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a85c292c-a2fd-4ff7-b74d-b2c25f980469-dshm\") pod \"auth-enabled-test-kserve-5bb9dd765c-q6nrw\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:46:32.514400 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.514382    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a85c292c-a2fd-4ff7-b74d-b2c25f980469-tls-certs\") pod \"auth-enabled-test-kserve-5bb9dd765c-q6nrw\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:46:32.521985 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.521966    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phzgm\" (UniqueName: \"kubernetes.io/projected/a85c292c-a2fd-4ff7-b74d-b2c25f980469-kube-api-access-phzgm\") pod \"auth-enabled-test-kserve-5bb9dd765c-q6nrw\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:46:32.655215 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.655146    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:46:32.933825 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.933804    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"]
Sep 17 18:46:32.936001 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:46:32.935974    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85c292c_a2fd_4ff7_b74d_b2c25f980469.slice/crio-8e57cf994d26eb9ce82abd4e8fe8863259670ee6cf3c2d97e26911fd5394823d WatchSource:0}: Error finding container 8e57cf994d26eb9ce82abd4e8fe8863259670ee6cf3c2d97e26911fd5394823d: Status 404 returned error can't find the container with id 8e57cf994d26eb9ce82abd4e8fe8863259670ee6cf3c2d97e26911fd5394823d
Sep 17 18:46:32.947950 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:32.947920    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw" event={"ID":"a85c292c-a2fd-4ff7-b74d-b2c25f980469","Type":"ContainerStarted","Data":"8e57cf994d26eb9ce82abd4e8fe8863259670ee6cf3c2d97e26911fd5394823d"}
Sep 17 18:46:33.958978 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:33.958935    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw" event={"ID":"a85c292c-a2fd-4ff7-b74d-b2c25f980469","Type":"ContainerStarted","Data":"11f47204692c8b1ee801860a9662337a64dbb16612862f0c1efa3c0e2c4777a9"}
Sep 17 18:46:35.972662 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:35.972621    2565 generic.go:356] "Generic (PLEG): container finished" podID="a85c292c-a2fd-4ff7-b74d-b2c25f980469" containerID="11f47204692c8b1ee801860a9662337a64dbb16612862f0c1efa3c0e2c4777a9" exitCode=0
Sep 17 18:46:35.973030 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:46:35.972695    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw" event={"ID":"a85c292c-a2fd-4ff7-b74d-b2c25f980469","Type":"ContainerDied","Data":"11f47204692c8b1ee801860a9662337a64dbb16612862f0c1efa3c0e2c4777a9"}
Sep 17 18:47:05.182652 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:05.182613    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw" event={"ID":"a85c292c-a2fd-4ff7-b74d-b2c25f980469","Type":"ContainerStarted","Data":"2de1eaf9fbaca356f7d744753161a4c64bf74183afbb77af6c34651eb1387bbf"}
Sep 17 18:47:05.211337 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:05.211289    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw" podStartSLOduration=2.716380817 podStartE2EDuration="33.211272572s" totalImagesPullingTime="28.366511726s" totalInitContainerRuntime="2.128380029s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:46:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:47:05.207922355 +0000 UTC m=+1083.140123897" watchObservedRunningTime="2026-09-17 18:47:05.211272572 +0000 UTC m=+1083.143474090"
Sep 17 18:47:05.655409 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:05.655323    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:47:12.656065 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:12.656021    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:47:12.657782 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:12.657754    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw" podUID="a85c292c-a2fd-4ff7-b74d-b2c25f980469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 17 18:47:22.656653 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:22.656589    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw" podUID="a85c292c-a2fd-4ff7-b74d-b2c25f980469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 17 18:47:32.656220 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:32.656154    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw" podUID="a85c292c-a2fd-4ff7-b74d-b2c25f980469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 17 18:47:42.655711 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:42.655667    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw" podUID="a85c292c-a2fd-4ff7-b74d-b2c25f980469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 17 18:47:43.995262 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:43.995223    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"]
Sep 17 18:47:44.005415 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:47:44.005370    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:47:44.005415 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:47:44.005415 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:47:44.005415 ip-10-0-139-75 kubenswrapper[2565]:  > containerID="fbd7903a64d569b561f3d46721a5ae69314ab6bf7425659d9b9d9f530e6512f4" cmd=["/bin/sleep","15"]
Sep 17 18:47:44.005751 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:44.005440    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s" message="" err=<
Sep 17 18:47:44.005751 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:47:44.005751 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:47:44.005751 ip-10-0-139-75 kubenswrapper[2565]:  >
Sep 17 18:47:44.005751 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:47:44.005457    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:47:44.005751 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:47:44.005751 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:47:44.005751 ip-10-0-139-75 kubenswrapper[2565]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s" podUID="19181c95-7c8a-4419-89a2-55d370e1249c" containerName="main" containerID="cri-o://fbd7903a64d569b561f3d46721a5ae69314ab6bf7425659d9b9d9f530e6512f4"
Sep 17 18:47:44.005751 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:44.005534    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s" podUID="19181c95-7c8a-4419-89a2-55d370e1249c" containerName="main" containerID="cri-o://fbd7903a64d569b561f3d46721a5ae69314ab6bf7425659d9b9d9f530e6512f4" gracePeriod=60
Sep 17 18:47:44.222210 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:44.222136    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s" podUID="19181c95-7c8a-4419-89a2-55d370e1249c" containerName="llm-d-routing-sidecar" containerID="cri-o://42515b7a1bf8e60d697ffc32dba19fec8249e38af471515a26ab02082f673d5b" gracePeriod=60
Sep 17 18:47:44.406022 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:44.405958    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s" event={"ID":"19181c95-7c8a-4419-89a2-55d370e1249c","Type":"ContainerDied","Data":"fbd7903a64d569b561f3d46721a5ae69314ab6bf7425659d9b9d9f530e6512f4"}
Sep 17 18:47:44.406173 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:44.406051    2565 generic.go:356] "Generic (PLEG): container finished" podID="19181c95-7c8a-4419-89a2-55d370e1249c" containerID="fbd7903a64d569b561f3d46721a5ae69314ab6bf7425659d9b9d9f530e6512f4" exitCode=0
Sep 17 18:47:44.406173 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:44.406079    2565 generic.go:356] "Generic (PLEG): container finished" podID="19181c95-7c8a-4419-89a2-55d370e1249c" containerID="42515b7a1bf8e60d697ffc32dba19fec8249e38af471515a26ab02082f673d5b" exitCode=0
Sep 17 18:47:44.406173 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:44.406133    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s" event={"ID":"19181c95-7c8a-4419-89a2-55d370e1249c","Type":"ContainerDied","Data":"42515b7a1bf8e60d697ffc32dba19fec8249e38af471515a26ab02082f673d5b"}
Sep 17 18:47:44.482734 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:44.482711    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"
Sep 17 18:47:44.547617 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:44.547588    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpczk\" (UniqueName: \"kubernetes.io/projected/19181c95-7c8a-4419-89a2-55d370e1249c-kube-api-access-mpczk\") pod \"19181c95-7c8a-4419-89a2-55d370e1249c\" (UID: \"19181c95-7c8a-4419-89a2-55d370e1249c\") "
Sep 17 18:47:44.547792 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:44.547641    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19181c95-7c8a-4419-89a2-55d370e1249c-tls-certs\") pod \"19181c95-7c8a-4419-89a2-55d370e1249c\" (UID: \"19181c95-7c8a-4419-89a2-55d370e1249c\") "
Sep 17 18:47:44.547792 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:44.547675    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19181c95-7c8a-4419-89a2-55d370e1249c-model-cache\") pod \"19181c95-7c8a-4419-89a2-55d370e1249c\" (UID: \"19181c95-7c8a-4419-89a2-55d370e1249c\") "
Sep 17 18:47:44.547792 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:44.547695    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19181c95-7c8a-4419-89a2-55d370e1249c-dshm\") pod \"19181c95-7c8a-4419-89a2-55d370e1249c\" (UID: \"19181c95-7c8a-4419-89a2-55d370e1249c\") "
Sep 17 18:47:44.547792 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:44.547748    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19181c95-7c8a-4419-89a2-55d370e1249c-home\") pod \"19181c95-7c8a-4419-89a2-55d370e1249c\" (UID: \"19181c95-7c8a-4419-89a2-55d370e1249c\") "
Sep 17 18:47:44.547792 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:44.547783    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19181c95-7c8a-4419-89a2-55d370e1249c-tmp-dir\") pod \"19181c95-7c8a-4419-89a2-55d370e1249c\" (UID: \"19181c95-7c8a-4419-89a2-55d370e1249c\") "
Sep 17 18:47:44.548298 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:44.548266    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19181c95-7c8a-4419-89a2-55d370e1249c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "19181c95-7c8a-4419-89a2-55d370e1249c" (UID: "19181c95-7c8a-4419-89a2-55d370e1249c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:47:44.548462 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:44.548440    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19181c95-7c8a-4419-89a2-55d370e1249c-home" (OuterVolumeSpecName: "home") pod "19181c95-7c8a-4419-89a2-55d370e1249c" (UID: "19181c95-7c8a-4419-89a2-55d370e1249c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:47:44.548537 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:44.548509    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19181c95-7c8a-4419-89a2-55d370e1249c-model-cache" (OuterVolumeSpecName: "model-cache") pod "19181c95-7c8a-4419-89a2-55d370e1249c" (UID: "19181c95-7c8a-4419-89a2-55d370e1249c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:47:44.550505 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:44.550471    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19181c95-7c8a-4419-89a2-55d370e1249c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "19181c95-7c8a-4419-89a2-55d370e1249c" (UID: "19181c95-7c8a-4419-89a2-55d370e1249c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:47:44.550590 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:44.550494    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19181c95-7c8a-4419-89a2-55d370e1249c-kube-api-access-mpczk" (OuterVolumeSpecName: "kube-api-access-mpczk") pod "19181c95-7c8a-4419-89a2-55d370e1249c" (UID: "19181c95-7c8a-4419-89a2-55d370e1249c"). InnerVolumeSpecName "kube-api-access-mpczk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:47:44.550590 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:44.550481    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19181c95-7c8a-4419-89a2-55d370e1249c-dshm" (OuterVolumeSpecName: "dshm") pod "19181c95-7c8a-4419-89a2-55d370e1249c" (UID: "19181c95-7c8a-4419-89a2-55d370e1249c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:47:44.648631 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:44.648600    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19181c95-7c8a-4419-89a2-55d370e1249c-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:47:44.648631 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:44.648627    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19181c95-7c8a-4419-89a2-55d370e1249c-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:47:44.648631 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:44.648637    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19181c95-7c8a-4419-89a2-55d370e1249c-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:47:44.648830 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:44.648644    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19181c95-7c8a-4419-89a2-55d370e1249c-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:47:44.648830 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:44.648652    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19181c95-7c8a-4419-89a2-55d370e1249c-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:47:44.648830 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:44.648660    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpczk\" (UniqueName: \"kubernetes.io/projected/19181c95-7c8a-4419-89a2-55d370e1249c-kube-api-access-mpczk\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:47:45.413099 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:45.413065    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"
Sep 17 18:47:45.413099 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:45.413081    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s" event={"ID":"19181c95-7c8a-4419-89a2-55d370e1249c","Type":"ContainerDied","Data":"f549f5c88868a18b87e07ddb651792e37f72d3aa61399539528a77cdce202c38"}
Sep 17 18:47:45.413662 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:45.413137    2565 scope.go:117] "RemoveContainer" containerID="fbd7903a64d569b561f3d46721a5ae69314ab6bf7425659d9b9d9f530e6512f4"
Sep 17 18:47:45.424069 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:45.424045    2565 scope.go:117] "RemoveContainer" containerID="42515b7a1bf8e60d697ffc32dba19fec8249e38af471515a26ab02082f673d5b"
Sep 17 18:47:45.433058 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:45.433036    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"]
Sep 17 18:47:45.439511 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:45.439490    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-5xb9s"]
Sep 17 18:47:46.607897 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:46.607857    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19181c95-7c8a-4419-89a2-55d370e1249c" path="/var/lib/kubelet/pods/19181c95-7c8a-4419-89a2-55d370e1249c/volumes"
Sep 17 18:47:52.656537 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:47:52.656482    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw" podUID="a85c292c-a2fd-4ff7-b74d-b2c25f980469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 17 18:48:02.655741 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:02.655695    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw" podUID="a85c292c-a2fd-4ff7-b74d-b2c25f980469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 17 18:48:12.656446 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:12.656407    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw" podUID="a85c292c-a2fd-4ff7-b74d-b2c25f980469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 17 18:48:22.655940 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:22.655895    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw" podUID="a85c292c-a2fd-4ff7-b74d-b2c25f980469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 17 18:48:32.655961 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:32.655873    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw" podUID="a85c292c-a2fd-4ff7-b74d-b2c25f980469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 17 18:48:42.656548 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:42.656501    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw" podUID="a85c292c-a2fd-4ff7-b74d-b2c25f980469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 17 18:48:43.043469 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.043425    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"]
Sep 17 18:48:43.044065 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.044044    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19181c95-7c8a-4419-89a2-55d370e1249c" containerName="main"
Sep 17 18:48:43.044113 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.044071    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="19181c95-7c8a-4419-89a2-55d370e1249c" containerName="main"
Sep 17 18:48:43.044154 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.044110    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19181c95-7c8a-4419-89a2-55d370e1249c" containerName="llm-d-routing-sidecar"
Sep 17 18:48:43.044154 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.044121    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="19181c95-7c8a-4419-89a2-55d370e1249c" containerName="llm-d-routing-sidecar"
Sep 17 18:48:43.044288 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.044272    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="19181c95-7c8a-4419-89a2-55d370e1249c" containerName="main"
Sep 17 18:48:43.044325 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.044293    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="19181c95-7c8a-4419-89a2-55d370e1249c" containerName="llm-d-routing-sidecar"
Sep 17 18:48:43.049249 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.049227    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"
Sep 17 18:48:43.054257 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.054236    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 17 18:48:43.055057 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.055037    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 17 18:48:43.055171 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.055065    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-pbffz\""
Sep 17 18:48:43.055171 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.055072    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 17 18:48:43.055171 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.055119    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-9jsw2\""
Sep 17 18:48:43.075061 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.075036    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"]
Sep 17 18:48:43.127874 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.127837    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a960691-3587-4076-9f69-3793c41fac48-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq\" (UID: \"5a960691-3587-4076-9f69-3793c41fac48\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"
Sep 17 18:48:43.127874 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.127877    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a960691-3587-4076-9f69-3793c41fac48-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq\" (UID: \"5a960691-3587-4076-9f69-3793c41fac48\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"
Sep 17 18:48:43.128122 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.127964    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a960691-3587-4076-9f69-3793c41fac48-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq\" (UID: \"5a960691-3587-4076-9f69-3793c41fac48\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"
Sep 17 18:48:43.128122 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.127994    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a960691-3587-4076-9f69-3793c41fac48-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq\" (UID: \"5a960691-3587-4076-9f69-3793c41fac48\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"
Sep 17 18:48:43.128122 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.128031    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cbcr\" (UniqueName: \"kubernetes.io/projected/5a960691-3587-4076-9f69-3793c41fac48-kube-api-access-9cbcr\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq\" (UID: \"5a960691-3587-4076-9f69-3793c41fac48\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"
Sep 17 18:48:43.128265 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.128116    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a960691-3587-4076-9f69-3793c41fac48-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq\" (UID: \"5a960691-3587-4076-9f69-3793c41fac48\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"
Sep 17 18:48:43.228965 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.228925    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a960691-3587-4076-9f69-3793c41fac48-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq\" (UID: \"5a960691-3587-4076-9f69-3793c41fac48\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"
Sep 17 18:48:43.228965 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.228965    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a960691-3587-4076-9f69-3793c41fac48-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq\" (UID: \"5a960691-3587-4076-9f69-3793c41fac48\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"
Sep 17 18:48:43.229243 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.229014    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a960691-3587-4076-9f69-3793c41fac48-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq\" (UID: \"5a960691-3587-4076-9f69-3793c41fac48\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"
Sep 17 18:48:43.229243 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.229032    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a960691-3587-4076-9f69-3793c41fac48-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq\" (UID: \"5a960691-3587-4076-9f69-3793c41fac48\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"
Sep 17 18:48:43.229243 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.229056    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cbcr\" (UniqueName: \"kubernetes.io/projected/5a960691-3587-4076-9f69-3793c41fac48-kube-api-access-9cbcr\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq\" (UID: \"5a960691-3587-4076-9f69-3793c41fac48\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"
Sep 17 18:48:43.229423 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.229238    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a960691-3587-4076-9f69-3793c41fac48-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq\" (UID: \"5a960691-3587-4076-9f69-3793c41fac48\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"
Sep 17 18:48:43.229423 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.229402    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a960691-3587-4076-9f69-3793c41fac48-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq\" (UID: \"5a960691-3587-4076-9f69-3793c41fac48\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"
Sep 17 18:48:43.229550 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.229523    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a960691-3587-4076-9f69-3793c41fac48-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq\" (UID: \"5a960691-3587-4076-9f69-3793c41fac48\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"
Sep 17 18:48:43.229591 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.229558    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a960691-3587-4076-9f69-3793c41fac48-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq\" (UID: \"5a960691-3587-4076-9f69-3793c41fac48\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"
Sep 17 18:48:43.231285 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.231264    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a960691-3587-4076-9f69-3793c41fac48-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq\" (UID: \"5a960691-3587-4076-9f69-3793c41fac48\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"
Sep 17 18:48:43.231594 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.231578    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a960691-3587-4076-9f69-3793c41fac48-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq\" (UID: \"5a960691-3587-4076-9f69-3793c41fac48\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"
Sep 17 18:48:43.251385 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.251358    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cbcr\" (UniqueName: \"kubernetes.io/projected/5a960691-3587-4076-9f69-3793c41fac48-kube-api-access-9cbcr\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq\" (UID: \"5a960691-3587-4076-9f69-3793c41fac48\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"
Sep 17 18:48:43.362261 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.362162    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"
Sep 17 18:48:43.664928 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.664900    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"]
Sep 17 18:48:43.668066 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:48:43.668041    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a960691_3587_4076_9f69_3793c41fac48.slice/crio-ea993cf2f9c72abcd308ae8cb6fcbe354210a5e7cb1dc686c666816f9ecce342 WatchSource:0}: Error finding container ea993cf2f9c72abcd308ae8cb6fcbe354210a5e7cb1dc686c666816f9ecce342: Status 404 returned error can't find the container with id ea993cf2f9c72abcd308ae8cb6fcbe354210a5e7cb1dc686c666816f9ecce342
Sep 17 18:48:43.733170 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:43.733144    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq" event={"ID":"5a960691-3587-4076-9f69-3793c41fac48","Type":"ContainerStarted","Data":"ea993cf2f9c72abcd308ae8cb6fcbe354210a5e7cb1dc686c666816f9ecce342"}
Sep 17 18:48:44.741028 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:44.740989    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq" event={"ID":"5a960691-3587-4076-9f69-3793c41fac48","Type":"ContainerStarted","Data":"3d480e27d2560f6744ead2ef20e6e8eab537c25007ebf973313dbe692f9a98d0"}
Sep 17 18:48:44.741420 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:44.741064    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"
Sep 17 18:48:45.748885 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:45.748846    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq" event={"ID":"5a960691-3587-4076-9f69-3793c41fac48","Type":"ContainerStarted","Data":"3245bffcc86cbb7e01306adb492b657277c5708b686e0b1357dda5a36e803de0"}
Sep 17 18:48:45.792035 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:45.791975    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq" podStartSLOduration=3.7919594290000003 podStartE2EDuration="3.791959429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:48:45.791450899 +0000 UTC m=+1183.723652419" watchObservedRunningTime="2026-09-17 18:48:45.791959429 +0000 UTC m=+1183.724160948"
Sep 17 18:48:46.362752 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:46.362717    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"
Sep 17 18:48:52.665835 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:52.665800    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:48:52.673420 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:52.673397    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:48:53.362302 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:53.362268    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"
Sep 17 18:48:53.374880 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:53.374851    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"
Sep 17 18:48:53.387330 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:53.387305    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"
Sep 17 18:48:53.807169 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:48:53.807141    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"
Sep 17 18:49:25.234151 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.234113    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"]
Sep 17 18:49:25.243833 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:49:25.243792    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:49:25.243833 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:49:25.243833 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:49:25.243833 ip-10-0-139-75 kubenswrapper[2565]:  > containerID="3245bffcc86cbb7e01306adb492b657277c5708b686e0b1357dda5a36e803de0" cmd=["/bin/sleep","15"]
Sep 17 18:49:25.244124 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.243853    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq" message="" err=<
Sep 17 18:49:25.244124 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:49:25.244124 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:49:25.244124 ip-10-0-139-75 kubenswrapper[2565]:  >
Sep 17 18:49:25.244124 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:49:25.243872    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:49:25.244124 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:49:25.244124 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:49:25.244124 ip-10-0-139-75 kubenswrapper[2565]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq" podUID="5a960691-3587-4076-9f69-3793c41fac48" containerName="main" containerID="cri-o://3245bffcc86cbb7e01306adb492b657277c5708b686e0b1357dda5a36e803de0"
Sep 17 18:49:25.244124 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.243911    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq" podUID="5a960691-3587-4076-9f69-3793c41fac48" containerName="main" containerID="cri-o://3245bffcc86cbb7e01306adb492b657277c5708b686e0b1357dda5a36e803de0" gracePeriod=60
Sep 17 18:49:25.460739 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.460707    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq" podUID="5a960691-3587-4076-9f69-3793c41fac48" containerName="llm-d-routing-sidecar" containerID="cri-o://3d480e27d2560f6744ead2ef20e6e8eab537c25007ebf973313dbe692f9a98d0" gracePeriod=60
Sep 17 18:49:25.705818 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.705783    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"
Sep 17 18:49:25.789245 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.789174    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a960691-3587-4076-9f69-3793c41fac48-home\") pod \"5a960691-3587-4076-9f69-3793c41fac48\" (UID: \"5a960691-3587-4076-9f69-3793c41fac48\") "
Sep 17 18:49:25.789462 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.789296    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a960691-3587-4076-9f69-3793c41fac48-dshm\") pod \"5a960691-3587-4076-9f69-3793c41fac48\" (UID: \"5a960691-3587-4076-9f69-3793c41fac48\") "
Sep 17 18:49:25.789462 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.789344    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cbcr\" (UniqueName: \"kubernetes.io/projected/5a960691-3587-4076-9f69-3793c41fac48-kube-api-access-9cbcr\") pod \"5a960691-3587-4076-9f69-3793c41fac48\" (UID: \"5a960691-3587-4076-9f69-3793c41fac48\") "
Sep 17 18:49:25.789462 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.789391    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a960691-3587-4076-9f69-3793c41fac48-tls-certs\") pod \"5a960691-3587-4076-9f69-3793c41fac48\" (UID: \"5a960691-3587-4076-9f69-3793c41fac48\") "
Sep 17 18:49:25.789462 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.789434    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a960691-3587-4076-9f69-3793c41fac48-tmp-dir\") pod \"5a960691-3587-4076-9f69-3793c41fac48\" (UID: \"5a960691-3587-4076-9f69-3793c41fac48\") "
Sep 17 18:49:25.789769 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.789486    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a960691-3587-4076-9f69-3793c41fac48-model-cache\") pod \"5a960691-3587-4076-9f69-3793c41fac48\" (UID: \"5a960691-3587-4076-9f69-3793c41fac48\") "
Sep 17 18:49:25.789868 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.789832    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a960691-3587-4076-9f69-3793c41fac48-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5a960691-3587-4076-9f69-3793c41fac48" (UID: "5a960691-3587-4076-9f69-3793c41fac48"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:25.790038 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.790004    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a960691-3587-4076-9f69-3793c41fac48-home" (OuterVolumeSpecName: "home") pod "5a960691-3587-4076-9f69-3793c41fac48" (UID: "5a960691-3587-4076-9f69-3793c41fac48"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:25.790130 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.790058    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a960691-3587-4076-9f69-3793c41fac48-model-cache" (OuterVolumeSpecName: "model-cache") pod "5a960691-3587-4076-9f69-3793c41fac48" (UID: "5a960691-3587-4076-9f69-3793c41fac48"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:25.793069 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.793035    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a960691-3587-4076-9f69-3793c41fac48-dshm" (OuterVolumeSpecName: "dshm") pod "5a960691-3587-4076-9f69-3793c41fac48" (UID: "5a960691-3587-4076-9f69-3793c41fac48"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:25.793449 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.793420    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a960691-3587-4076-9f69-3793c41fac48-kube-api-access-9cbcr" (OuterVolumeSpecName: "kube-api-access-9cbcr") pod "5a960691-3587-4076-9f69-3793c41fac48" (UID: "5a960691-3587-4076-9f69-3793c41fac48"). InnerVolumeSpecName "kube-api-access-9cbcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:49:25.793739 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.793708    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a960691-3587-4076-9f69-3793c41fac48-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5a960691-3587-4076-9f69-3793c41fac48" (UID: "5a960691-3587-4076-9f69-3793c41fac48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:49:25.890934 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.890863    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a960691-3587-4076-9f69-3793c41fac48-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:49:25.890934 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.890891    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cbcr\" (UniqueName: \"kubernetes.io/projected/5a960691-3587-4076-9f69-3793c41fac48-kube-api-access-9cbcr\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:49:25.890934 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.890902    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a960691-3587-4076-9f69-3793c41fac48-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:49:25.890934 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.890911    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a960691-3587-4076-9f69-3793c41fac48-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:49:25.890934 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.890919    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a960691-3587-4076-9f69-3793c41fac48-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:49:25.890934 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.890926    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a960691-3587-4076-9f69-3793c41fac48-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:49:25.971338 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.971301    2565 generic.go:356] "Generic (PLEG): container finished" podID="5a960691-3587-4076-9f69-3793c41fac48" containerID="3245bffcc86cbb7e01306adb492b657277c5708b686e0b1357dda5a36e803de0" exitCode=0
Sep 17 18:49:25.971338 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.971333    2565 generic.go:356] "Generic (PLEG): container finished" podID="5a960691-3587-4076-9f69-3793c41fac48" containerID="3d480e27d2560f6744ead2ef20e6e8eab537c25007ebf973313dbe692f9a98d0" exitCode=0
Sep 17 18:49:25.971639 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.971387    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"
Sep 17 18:49:25.971639 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.971417    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq" event={"ID":"5a960691-3587-4076-9f69-3793c41fac48","Type":"ContainerDied","Data":"3245bffcc86cbb7e01306adb492b657277c5708b686e0b1357dda5a36e803de0"}
Sep 17 18:49:25.971639 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.971504    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq" event={"ID":"5a960691-3587-4076-9f69-3793c41fac48","Type":"ContainerDied","Data":"3d480e27d2560f6744ead2ef20e6e8eab537c25007ebf973313dbe692f9a98d0"}
Sep 17 18:49:25.971639 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.971549    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq" event={"ID":"5a960691-3587-4076-9f69-3793c41fac48","Type":"ContainerDied","Data":"ea993cf2f9c72abcd308ae8cb6fcbe354210a5e7cb1dc686c666816f9ecce342"}
Sep 17 18:49:25.971639 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.971591    2565 scope.go:117] "RemoveContainer" containerID="3245bffcc86cbb7e01306adb492b657277c5708b686e0b1357dda5a36e803de0"
Sep 17 18:49:25.986908 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.986671    2565 scope.go:117] "RemoveContainer" containerID="3d480e27d2560f6744ead2ef20e6e8eab537c25007ebf973313dbe692f9a98d0"
Sep 17 18:49:25.996739 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.996720    2565 scope.go:117] "RemoveContainer" containerID="3245bffcc86cbb7e01306adb492b657277c5708b686e0b1357dda5a36e803de0"
Sep 17 18:49:25.996991 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:49:25.996973    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3245bffcc86cbb7e01306adb492b657277c5708b686e0b1357dda5a36e803de0\": container with ID starting with 3245bffcc86cbb7e01306adb492b657277c5708b686e0b1357dda5a36e803de0 not found: ID does not exist" containerID="3245bffcc86cbb7e01306adb492b657277c5708b686e0b1357dda5a36e803de0"
Sep 17 18:49:25.997040 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.996999    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3245bffcc86cbb7e01306adb492b657277c5708b686e0b1357dda5a36e803de0"} err="failed to get container status \"3245bffcc86cbb7e01306adb492b657277c5708b686e0b1357dda5a36e803de0\": rpc error: code = NotFound desc = could not find container \"3245bffcc86cbb7e01306adb492b657277c5708b686e0b1357dda5a36e803de0\": container with ID starting with 3245bffcc86cbb7e01306adb492b657277c5708b686e0b1357dda5a36e803de0 not found: ID does not exist"
Sep 17 18:49:25.997040 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.997016    2565 scope.go:117] "RemoveContainer" containerID="3d480e27d2560f6744ead2ef20e6e8eab537c25007ebf973313dbe692f9a98d0"
Sep 17 18:49:25.997264 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:49:25.997246    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d480e27d2560f6744ead2ef20e6e8eab537c25007ebf973313dbe692f9a98d0\": container with ID starting with 3d480e27d2560f6744ead2ef20e6e8eab537c25007ebf973313dbe692f9a98d0 not found: ID does not exist" containerID="3d480e27d2560f6744ead2ef20e6e8eab537c25007ebf973313dbe692f9a98d0"
Sep 17 18:49:25.997314 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.997270    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d480e27d2560f6744ead2ef20e6e8eab537c25007ebf973313dbe692f9a98d0"} err="failed to get container status \"3d480e27d2560f6744ead2ef20e6e8eab537c25007ebf973313dbe692f9a98d0\": rpc error: code = NotFound desc = could not find container \"3d480e27d2560f6744ead2ef20e6e8eab537c25007ebf973313dbe692f9a98d0\": container with ID starting with 3d480e27d2560f6744ead2ef20e6e8eab537c25007ebf973313dbe692f9a98d0 not found: ID does not exist"
Sep 17 18:49:25.997314 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.997288    2565 scope.go:117] "RemoveContainer" containerID="3245bffcc86cbb7e01306adb492b657277c5708b686e0b1357dda5a36e803de0"
Sep 17 18:49:25.997566 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.997543    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3245bffcc86cbb7e01306adb492b657277c5708b686e0b1357dda5a36e803de0"} err="failed to get container status \"3245bffcc86cbb7e01306adb492b657277c5708b686e0b1357dda5a36e803de0\": rpc error: code = NotFound desc = could not find container \"3245bffcc86cbb7e01306adb492b657277c5708b686e0b1357dda5a36e803de0\": container with ID starting with 3245bffcc86cbb7e01306adb492b657277c5708b686e0b1357dda5a36e803de0 not found: ID does not exist"
Sep 17 18:49:25.997653 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.997568    2565 scope.go:117] "RemoveContainer" containerID="3d480e27d2560f6744ead2ef20e6e8eab537c25007ebf973313dbe692f9a98d0"
Sep 17 18:49:25.997815 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:25.997795    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d480e27d2560f6744ead2ef20e6e8eab537c25007ebf973313dbe692f9a98d0"} err="failed to get container status \"3d480e27d2560f6744ead2ef20e6e8eab537c25007ebf973313dbe692f9a98d0\": rpc error: code = NotFound desc = could not find container \"3d480e27d2560f6744ead2ef20e6e8eab537c25007ebf973313dbe692f9a98d0\": container with ID starting with 3d480e27d2560f6744ead2ef20e6e8eab537c25007ebf973313dbe692f9a98d0 not found: ID does not exist"
Sep 17 18:49:26.004796 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:26.004772    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"]
Sep 17 18:49:26.009959 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:26.009928    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-qwvjq"]
Sep 17 18:49:26.606691 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:26.606655    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a960691-3587-4076-9f69-3793c41fac48" path="/var/lib/kubelet/pods/5a960691-3587-4076-9f69-3793c41fac48/volumes"
Sep 17 18:49:27.125154 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:27.125121    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"]
Sep 17 18:49:42.148671 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:42.148604    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw" podUID="a85c292c-a2fd-4ff7-b74d-b2c25f980469" containerName="main" containerID="cri-o://2de1eaf9fbaca356f7d744753161a4c64bf74183afbb77af6c34651eb1387bbf" gracePeriod=45
Sep 17 18:49:42.667072 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:42.667028    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw" podUID="a85c292c-a2fd-4ff7-b74d-b2c25f980469" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 17 18:49:42.902194 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:42.902171    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:49:43.021280 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.021250    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a85c292c-a2fd-4ff7-b74d-b2c25f980469-model-cache\") pod \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") "
Sep 17 18:49:43.021432 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.021297    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a85c292c-a2fd-4ff7-b74d-b2c25f980469-home\") pod \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") "
Sep 17 18:49:43.021432 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.021316    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a85c292c-a2fd-4ff7-b74d-b2c25f980469-tmp-dir\") pod \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") "
Sep 17 18:49:43.021432 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.021336    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phzgm\" (UniqueName: \"kubernetes.io/projected/a85c292c-a2fd-4ff7-b74d-b2c25f980469-kube-api-access-phzgm\") pod \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") "
Sep 17 18:49:43.021432 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.021405    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a85c292c-a2fd-4ff7-b74d-b2c25f980469-tls-certs\") pod \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") "
Sep 17 18:49:43.021432 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.021425    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a85c292c-a2fd-4ff7-b74d-b2c25f980469-cabundle-cert\") pod \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") "
Sep 17 18:49:43.021691 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.021445    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a85c292c-a2fd-4ff7-b74d-b2c25f980469-kserve-provision-location\") pod \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") "
Sep 17 18:49:43.021691 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.021484    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a85c292c-a2fd-4ff7-b74d-b2c25f980469-dshm\") pod \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\" (UID: \"a85c292c-a2fd-4ff7-b74d-b2c25f980469\") "
Sep 17 18:49:43.021691 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.021562    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85c292c-a2fd-4ff7-b74d-b2c25f980469-model-cache" (OuterVolumeSpecName: "model-cache") pod "a85c292c-a2fd-4ff7-b74d-b2c25f980469" (UID: "a85c292c-a2fd-4ff7-b74d-b2c25f980469"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:43.021874 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.021854    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a85c292c-a2fd-4ff7-b74d-b2c25f980469-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:49:43.021925 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.021868    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a85c292c-a2fd-4ff7-b74d-b2c25f980469-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a85c292c-a2fd-4ff7-b74d-b2c25f980469" (UID: "a85c292c-a2fd-4ff7-b74d-b2c25f980469"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:49:43.022104 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.022059    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85c292c-a2fd-4ff7-b74d-b2c25f980469-home" (OuterVolumeSpecName: "home") pod "a85c292c-a2fd-4ff7-b74d-b2c25f980469" (UID: "a85c292c-a2fd-4ff7-b74d-b2c25f980469"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:43.023955 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.023924    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a85c292c-a2fd-4ff7-b74d-b2c25f980469-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a85c292c-a2fd-4ff7-b74d-b2c25f980469" (UID: "a85c292c-a2fd-4ff7-b74d-b2c25f980469"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:49:43.024060 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.023980    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a85c292c-a2fd-4ff7-b74d-b2c25f980469-kube-api-access-phzgm" (OuterVolumeSpecName: "kube-api-access-phzgm") pod "a85c292c-a2fd-4ff7-b74d-b2c25f980469" (UID: "a85c292c-a2fd-4ff7-b74d-b2c25f980469"). InnerVolumeSpecName "kube-api-access-phzgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:49:43.024125 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.024070    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85c292c-a2fd-4ff7-b74d-b2c25f980469-dshm" (OuterVolumeSpecName: "dshm") pod "a85c292c-a2fd-4ff7-b74d-b2c25f980469" (UID: "a85c292c-a2fd-4ff7-b74d-b2c25f980469"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:43.034403 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.034377    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85c292c-a2fd-4ff7-b74d-b2c25f980469-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a85c292c-a2fd-4ff7-b74d-b2c25f980469" (UID: "a85c292c-a2fd-4ff7-b74d-b2c25f980469"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:43.042269 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.042242    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a85c292c-a2fd-4ff7-b74d-b2c25f980469-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a85c292c-a2fd-4ff7-b74d-b2c25f980469" (UID: "a85c292c-a2fd-4ff7-b74d-b2c25f980469"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:43.067076 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.067048    2565 generic.go:356] "Generic (PLEG): container finished" podID="a85c292c-a2fd-4ff7-b74d-b2c25f980469" containerID="2de1eaf9fbaca356f7d744753161a4c64bf74183afbb77af6c34651eb1387bbf" exitCode=0
Sep 17 18:49:43.067157 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.067103    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"
Sep 17 18:49:43.067157 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.067125    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw" event={"ID":"a85c292c-a2fd-4ff7-b74d-b2c25f980469","Type":"ContainerDied","Data":"2de1eaf9fbaca356f7d744753161a4c64bf74183afbb77af6c34651eb1387bbf"}
Sep 17 18:49:43.067251 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.067181    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw" event={"ID":"a85c292c-a2fd-4ff7-b74d-b2c25f980469","Type":"ContainerDied","Data":"8e57cf994d26eb9ce82abd4e8fe8863259670ee6cf3c2d97e26911fd5394823d"}
Sep 17 18:49:43.067251 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.067226    2565 scope.go:117] "RemoveContainer" containerID="2de1eaf9fbaca356f7d744753161a4c64bf74183afbb77af6c34651eb1387bbf"
Sep 17 18:49:43.077293 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.077274    2565 scope.go:117] "RemoveContainer" containerID="11f47204692c8b1ee801860a9662337a64dbb16612862f0c1efa3c0e2c4777a9"
Sep 17 18:49:43.085012 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.084992    2565 scope.go:117] "RemoveContainer" containerID="2de1eaf9fbaca356f7d744753161a4c64bf74183afbb77af6c34651eb1387bbf"
Sep 17 18:49:43.085349 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:49:43.085330    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2de1eaf9fbaca356f7d744753161a4c64bf74183afbb77af6c34651eb1387bbf\": container with ID starting with 2de1eaf9fbaca356f7d744753161a4c64bf74183afbb77af6c34651eb1387bbf not found: ID does not exist" containerID="2de1eaf9fbaca356f7d744753161a4c64bf74183afbb77af6c34651eb1387bbf"
Sep 17 18:49:43.085445 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.085360    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2de1eaf9fbaca356f7d744753161a4c64bf74183afbb77af6c34651eb1387bbf"} err="failed to get container status \"2de1eaf9fbaca356f7d744753161a4c64bf74183afbb77af6c34651eb1387bbf\": rpc error: code = NotFound desc = could not find container \"2de1eaf9fbaca356f7d744753161a4c64bf74183afbb77af6c34651eb1387bbf\": container with ID starting with 2de1eaf9fbaca356f7d744753161a4c64bf74183afbb77af6c34651eb1387bbf not found: ID does not exist"
Sep 17 18:49:43.085445 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.085383    2565 scope.go:117] "RemoveContainer" containerID="11f47204692c8b1ee801860a9662337a64dbb16612862f0c1efa3c0e2c4777a9"
Sep 17 18:49:43.085626 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:49:43.085611    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11f47204692c8b1ee801860a9662337a64dbb16612862f0c1efa3c0e2c4777a9\": container with ID starting with 11f47204692c8b1ee801860a9662337a64dbb16612862f0c1efa3c0e2c4777a9 not found: ID does not exist" containerID="11f47204692c8b1ee801860a9662337a64dbb16612862f0c1efa3c0e2c4777a9"
Sep 17 18:49:43.085626 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.085633    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11f47204692c8b1ee801860a9662337a64dbb16612862f0c1efa3c0e2c4777a9"} err="failed to get container status \"11f47204692c8b1ee801860a9662337a64dbb16612862f0c1efa3c0e2c4777a9\": rpc error: code = NotFound desc = could not find container \"11f47204692c8b1ee801860a9662337a64dbb16612862f0c1efa3c0e2c4777a9\": container with ID starting with 11f47204692c8b1ee801860a9662337a64dbb16612862f0c1efa3c0e2c4777a9 not found: ID does not exist"
Sep 17 18:49:43.090783 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.090761    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"]
Sep 17 18:49:43.094445 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.094424    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-5bb9dd765c-q6nrw"]
Sep 17 18:49:43.122521 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.122500    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a85c292c-a2fd-4ff7-b74d-b2c25f980469-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:49:43.122521 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.122521    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a85c292c-a2fd-4ff7-b74d-b2c25f980469-cabundle-cert\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:49:43.122633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.122531    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a85c292c-a2fd-4ff7-b74d-b2c25f980469-kserve-provision-location\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:49:43.122633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.122540    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a85c292c-a2fd-4ff7-b74d-b2c25f980469-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:49:43.122633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.122550    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a85c292c-a2fd-4ff7-b74d-b2c25f980469-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:49:43.122633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.122557    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a85c292c-a2fd-4ff7-b74d-b2c25f980469-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:49:43.122633 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:43.122565    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phzgm\" (UniqueName: \"kubernetes.io/projected/a85c292c-a2fd-4ff7-b74d-b2c25f980469-kube-api-access-phzgm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:49:44.608220 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:49:44.608156    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a85c292c-a2fd-4ff7-b74d-b2c25f980469" path="/var/lib/kubelet/pods/a85c292c-a2fd-4ff7-b74d-b2c25f980469/volumes"
Sep 17 18:50:32.858613 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:32.858581    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"]
Sep 17 18:50:32.861005 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:32.858936    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a960691-3587-4076-9f69-3793c41fac48" containerName="llm-d-routing-sidecar"
Sep 17 18:50:32.861005 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:32.858948    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a960691-3587-4076-9f69-3793c41fac48" containerName="llm-d-routing-sidecar"
Sep 17 18:50:32.861005 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:32.858967    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a960691-3587-4076-9f69-3793c41fac48" containerName="main"
Sep 17 18:50:32.861005 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:32.858974    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a960691-3587-4076-9f69-3793c41fac48" containerName="main"
Sep 17 18:50:32.861005 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:32.858981    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a85c292c-a2fd-4ff7-b74d-b2c25f980469" containerName="main"
Sep 17 18:50:32.861005 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:32.858986    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a85c292c-a2fd-4ff7-b74d-b2c25f980469" containerName="main"
Sep 17 18:50:32.861005 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:32.858995    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a85c292c-a2fd-4ff7-b74d-b2c25f980469" containerName="storage-initializer"
Sep 17 18:50:32.861005 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:32.859000    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a85c292c-a2fd-4ff7-b74d-b2c25f980469" containerName="storage-initializer"
Sep 17 18:50:32.861005 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:32.859050    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a960691-3587-4076-9f69-3793c41fac48" containerName="main"
Sep 17 18:50:32.861005 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:32.859059    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a85c292c-a2fd-4ff7-b74d-b2c25f980469" containerName="main"
Sep 17 18:50:32.861005 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:32.859066    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a960691-3587-4076-9f69-3793c41fac48" containerName="llm-d-routing-sidecar"
Sep 17 18:50:32.862517 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:32.862500    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:50:32.865182 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:32.865153    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 17 18:50:32.865342 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:32.865168    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 17 18:50:32.865715 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:32.865695    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-4vhmb\""
Sep 17 18:50:32.865825 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:32.865731    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:50:32.865825 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:32.865731    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 17 18:50:32.865948 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:32.865848    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 17 18:50:32.877738 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:32.877715    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"]
Sep 17 18:50:32.978711 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:32.978679    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:50:32.978858 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:32.978712    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:50:32.978858 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:32.978738    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb2nb\" (UniqueName: \"kubernetes.io/projected/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-kube-api-access-sb2nb\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:50:32.978858 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:32.978829    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:50:32.979008 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:32.978864    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-dshm\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:50:32.979008 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:32.978893    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:50:32.979008 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:32.978955    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-home\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:50:32.979008 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:32.978987    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-model-cache\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:50:33.080021 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.079995    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-home\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:50:33.080136 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.080025    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-model-cache\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:50:33.080136 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.080051    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:50:33.080136 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.080067    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:50:33.080136 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.080086    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb2nb\" (UniqueName: \"kubernetes.io/projected/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-kube-api-access-sb2nb\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:50:33.080325 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.080138    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:50:33.080325 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.080171    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-dshm\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:50:33.080429 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.080336    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:50:33.080485 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.080472    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-model-cache\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:50:33.080542 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.080489    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-home\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:50:33.080605 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.080554    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:50:33.080697 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.080677    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:50:33.080956 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.080937    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:50:33.082412 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.082391    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-dshm\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:50:33.082578 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.082560    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:50:33.088888 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.088866    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb2nb\" (UniqueName: \"kubernetes.io/projected/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-kube-api-access-sb2nb\") pod \"tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:50:33.172969 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.172947    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:50:33.183532 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.183509    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj"]
Sep 17 18:50:33.187435 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.187411    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj"
Sep 17 18:50:33.207991 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.207923    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj"]
Sep 17 18:50:33.282476 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.282436    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9960d8af-172e-46ad-92cf-26a708705864-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj\" (UID: \"9960d8af-172e-46ad-92cf-26a708705864\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj"
Sep 17 18:50:33.282608 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.282517    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9960d8af-172e-46ad-92cf-26a708705864-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj\" (UID: \"9960d8af-172e-46ad-92cf-26a708705864\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj"
Sep 17 18:50:33.282608 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.282545    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9960d8af-172e-46ad-92cf-26a708705864-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj\" (UID: \"9960d8af-172e-46ad-92cf-26a708705864\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj"
Sep 17 18:50:33.282714 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.282637    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9960d8af-172e-46ad-92cf-26a708705864-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj\" (UID: \"9960d8af-172e-46ad-92cf-26a708705864\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj"
Sep 17 18:50:33.383495 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.383469    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9960d8af-172e-46ad-92cf-26a708705864-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj\" (UID: \"9960d8af-172e-46ad-92cf-26a708705864\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj"
Sep 17 18:50:33.383647 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.383503    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9960d8af-172e-46ad-92cf-26a708705864-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj\" (UID: \"9960d8af-172e-46ad-92cf-26a708705864\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj"
Sep 17 18:50:33.383647 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.383518    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9960d8af-172e-46ad-92cf-26a708705864-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj\" (UID: \"9960d8af-172e-46ad-92cf-26a708705864\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj"
Sep 17 18:50:33.383647 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.383567    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9960d8af-172e-46ad-92cf-26a708705864-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj\" (UID: \"9960d8af-172e-46ad-92cf-26a708705864\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj"
Sep 17 18:50:33.383921 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.383899    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9960d8af-172e-46ad-92cf-26a708705864-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj\" (UID: \"9960d8af-172e-46ad-92cf-26a708705864\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj"
Sep 17 18:50:33.383921 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.383915    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9960d8af-172e-46ad-92cf-26a708705864-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj\" (UID: \"9960d8af-172e-46ad-92cf-26a708705864\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj"
Sep 17 18:50:33.384185 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.384168    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9960d8af-172e-46ad-92cf-26a708705864-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj\" (UID: \"9960d8af-172e-46ad-92cf-26a708705864\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj"
Sep 17 18:50:33.385951 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.385928    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9960d8af-172e-46ad-92cf-26a708705864-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj\" (UID: \"9960d8af-172e-46ad-92cf-26a708705864\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj"
Sep 17 18:50:33.454534 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.454512    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"]
Sep 17 18:50:33.456079 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:50:33.456059    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c660442_a3d5_44e1_88a7_d502b2bdf2ac.slice/crio-9f9aa43f1bdd6b2bcabaaaa311da2a7d0d42df7538938d47920b63834db755eb WatchSource:0}: Error finding container 9f9aa43f1bdd6b2bcabaaaa311da2a7d0d42df7538938d47920b63834db755eb: Status 404 returned error can't find the container with id 9f9aa43f1bdd6b2bcabaaaa311da2a7d0d42df7538938d47920b63834db755eb
Sep 17 18:50:33.513976 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.513959    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj"
Sep 17 18:50:33.792662 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:33.792634    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj"]
Sep 17 18:50:33.794576 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:50:33.794549    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9960d8af_172e_46ad_92cf_26a708705864.slice/crio-e68375476602d751c787e640197ffcef51010ee78ce80a2758ec48eb23699355 WatchSource:0}: Error finding container e68375476602d751c787e640197ffcef51010ee78ce80a2758ec48eb23699355: Status 404 returned error can't find the container with id e68375476602d751c787e640197ffcef51010ee78ce80a2758ec48eb23699355
Sep 17 18:50:34.360815 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:34.360765    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8" event={"ID":"1c660442-a3d5-44e1-88a7-d502b2bdf2ac","Type":"ContainerStarted","Data":"b7ef2cfc2e323eac7306804428290335cd11701b9f918bc9ff490b67cd4dbe38"}
Sep 17 18:50:34.361355 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:34.360835    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8" event={"ID":"1c660442-a3d5-44e1-88a7-d502b2bdf2ac","Type":"ContainerStarted","Data":"9f9aa43f1bdd6b2bcabaaaa311da2a7d0d42df7538938d47920b63834db755eb"}
Sep 17 18:50:34.363578 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:34.363520    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj" event={"ID":"9960d8af-172e-46ad-92cf-26a708705864","Type":"ContainerStarted","Data":"e90f07429a1370b3ef93d9660d260ed325f26246dc3d5e2408dcb747f1ac258a"}
Sep 17 18:50:34.363728 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:34.363602    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj" event={"ID":"9960d8af-172e-46ad-92cf-26a708705864","Type":"ContainerStarted","Data":"e68375476602d751c787e640197ffcef51010ee78ce80a2758ec48eb23699355"}
Sep 17 18:50:36.378277 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:36.378246    2565 generic.go:356] "Generic (PLEG): container finished" podID="1c660442-a3d5-44e1-88a7-d502b2bdf2ac" containerID="b7ef2cfc2e323eac7306804428290335cd11701b9f918bc9ff490b67cd4dbe38" exitCode=0
Sep 17 18:50:36.378737 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:36.378332    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8" event={"ID":"1c660442-a3d5-44e1-88a7-d502b2bdf2ac","Type":"ContainerDied","Data":"b7ef2cfc2e323eac7306804428290335cd11701b9f918bc9ff490b67cd4dbe38"}
Sep 17 18:50:36.379995 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:36.379972    2565 generic.go:356] "Generic (PLEG): container finished" podID="9960d8af-172e-46ad-92cf-26a708705864" containerID="e90f07429a1370b3ef93d9660d260ed325f26246dc3d5e2408dcb747f1ac258a" exitCode=0
Sep 17 18:50:36.380075 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:36.380035    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj" event={"ID":"9960d8af-172e-46ad-92cf-26a708705864","Type":"ContainerDied","Data":"e90f07429a1370b3ef93d9660d260ed325f26246dc3d5e2408dcb747f1ac258a"}
Sep 17 18:50:37.386680 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:37.386644    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8" event={"ID":"1c660442-a3d5-44e1-88a7-d502b2bdf2ac","Type":"ContainerStarted","Data":"9a52ee7cc354957112a31ae0184abfa1a01d4b0641f8ead117268f5f8ba343bf"}
Sep 17 18:50:37.417237 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:37.417167    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8" podStartSLOduration=2.983591041 podStartE2EDuration="5.417152372s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.433561331s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:50:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:50:37.414934868 +0000 UTC m=+1295.347136391" watchObservedRunningTime="2026-09-17 18:50:37.417152372 +0000 UTC m=+1295.349353890"
Sep 17 18:50:38.173386 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:38.173350    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:50:43.173119 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:43.173080    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:50:43.186143 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:43.186121    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:50:43.440345 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:50:43.440230    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:51:52.843468 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:51:52.843420    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj" event={"ID":"9960d8af-172e-46ad-92cf-26a708705864","Type":"ContainerStarted","Data":"143ec6fdafbc74cd393f3c6d2c92e3d0c242d27e32bd0640f24f970b6bdcef70"}
Sep 17 18:51:52.844073 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:51:52.843500    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj"
Sep 17 18:51:52.862267 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:51:52.862215    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj" podStartSLOduration=1.971849966 podStartE2EDuration="1m19.862182836s" totalImagesPullingTime="1m15.458327009s" totalInitContainerRuntime="2.432005861s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:50:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:51:52.861192798 +0000 UTC m=+1370.793394318" watchObservedRunningTime="2026-09-17 18:51:52.862182836 +0000 UTC m=+1370.794384417"
Sep 17 18:52:23.860712 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:23.860672    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj"
Sep 17 18:52:24.936243 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:24.936213    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj_9960d8af-172e-46ad-92cf-26a708705864/main/0.log"
Sep 17 18:52:25.676342 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:25.676308    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"]
Sep 17 18:52:25.685819 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:52:25.685768    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:52:25.685819 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:52:25.685819 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:52:25.685819 ip-10-0-139-75 kubenswrapper[2565]:  > containerID="9a52ee7cc354957112a31ae0184abfa1a01d4b0641f8ead117268f5f8ba343bf" cmd=["/bin/sleep","15"]
Sep 17 18:52:25.686091 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:25.685844    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8" message="" err=<
Sep 17 18:52:25.686091 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:52:25.686091 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:52:25.686091 ip-10-0-139-75 kubenswrapper[2565]:  >
Sep 17 18:52:25.686091 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:52:25.685858    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:52:25.686091 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:52:25.686091 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:52:25.686091 ip-10-0-139-75 kubenswrapper[2565]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8" podUID="1c660442-a3d5-44e1-88a7-d502b2bdf2ac" containerName="main" containerID="cri-o://9a52ee7cc354957112a31ae0184abfa1a01d4b0641f8ead117268f5f8ba343bf"
Sep 17 18:52:25.686091 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:25.685905    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8" podUID="1c660442-a3d5-44e1-88a7-d502b2bdf2ac" containerName="main" containerID="cri-o://9a52ee7cc354957112a31ae0184abfa1a01d4b0641f8ead117268f5f8ba343bf" gracePeriod=60
Sep 17 18:52:25.691047 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:25.691019    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj"]
Sep 17 18:52:25.691252 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:25.691229    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj" podUID="9960d8af-172e-46ad-92cf-26a708705864" containerName="main" containerID="cri-o://143ec6fdafbc74cd393f3c6d2c92e3d0c242d27e32bd0640f24f970b6bdcef70" gracePeriod=30
Sep 17 18:52:26.041947 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.041914    2565 generic.go:356] "Generic (PLEG): container finished" podID="1c660442-a3d5-44e1-88a7-d502b2bdf2ac" containerID="9a52ee7cc354957112a31ae0184abfa1a01d4b0641f8ead117268f5f8ba343bf" exitCode=0
Sep 17 18:52:26.042339 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.041986    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8" event={"ID":"1c660442-a3d5-44e1-88a7-d502b2bdf2ac","Type":"ContainerDied","Data":"9a52ee7cc354957112a31ae0184abfa1a01d4b0641f8ead117268f5f8ba343bf"}
Sep 17 18:52:26.313987 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.313925    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj"
Sep 17 18:52:26.420489 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.420461    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9960d8af-172e-46ad-92cf-26a708705864-cabundle-cert\") pod \"9960d8af-172e-46ad-92cf-26a708705864\" (UID: \"9960d8af-172e-46ad-92cf-26a708705864\") "
Sep 17 18:52:26.420623 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.420520    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9960d8af-172e-46ad-92cf-26a708705864-tmp-dir\") pod \"9960d8af-172e-46ad-92cf-26a708705864\" (UID: \"9960d8af-172e-46ad-92cf-26a708705864\") "
Sep 17 18:52:26.420623 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.420574    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9960d8af-172e-46ad-92cf-26a708705864-kserve-provision-location\") pod \"9960d8af-172e-46ad-92cf-26a708705864\" (UID: \"9960d8af-172e-46ad-92cf-26a708705864\") "
Sep 17 18:52:26.420761 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.420628    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9960d8af-172e-46ad-92cf-26a708705864-tls-certs\") pod \"9960d8af-172e-46ad-92cf-26a708705864\" (UID: \"9960d8af-172e-46ad-92cf-26a708705864\") "
Sep 17 18:52:26.420845 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.420821    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9960d8af-172e-46ad-92cf-26a708705864-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9960d8af-172e-46ad-92cf-26a708705864" (UID: "9960d8af-172e-46ad-92cf-26a708705864"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:52:26.420912 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.420876    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9960d8af-172e-46ad-92cf-26a708705864-cabundle-cert\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:52:26.420995 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.420969    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9960d8af-172e-46ad-92cf-26a708705864-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9960d8af-172e-46ad-92cf-26a708705864" (UID: "9960d8af-172e-46ad-92cf-26a708705864"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:52:26.422842 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.422805    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9960d8af-172e-46ad-92cf-26a708705864-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9960d8af-172e-46ad-92cf-26a708705864" (UID: "9960d8af-172e-46ad-92cf-26a708705864"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:52:26.426212 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.426161    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8" podUID="1c660442-a3d5-44e1-88a7-d502b2bdf2ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Sep 17 18:52:26.447156 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.447122    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9960d8af-172e-46ad-92cf-26a708705864-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9960d8af-172e-46ad-92cf-26a708705864" (UID: "9960d8af-172e-46ad-92cf-26a708705864"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:52:26.521643 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.521616    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9960d8af-172e-46ad-92cf-26a708705864-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:52:26.521643 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.521643    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9960d8af-172e-46ad-92cf-26a708705864-kserve-provision-location\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:52:26.521806 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.521652    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9960d8af-172e-46ad-92cf-26a708705864-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:52:26.552390 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.552370    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:52:26.622150 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.622127    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-dshm\") pod \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") "
Sep 17 18:52:26.622283 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.622165    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-cabundle-cert\") pod \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") "
Sep 17 18:52:26.622283 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.622190    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-kserve-provision-location\") pod \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") "
Sep 17 18:52:26.622283 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.622231    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-model-cache\") pod \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") "
Sep 17 18:52:26.622283 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.622274    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sb2nb\" (UniqueName: \"kubernetes.io/projected/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-kube-api-access-sb2nb\") pod \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") "
Sep 17 18:52:26.622514 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.622307    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-home\") pod \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") "
Sep 17 18:52:26.622514 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.622391    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-tls-certs\") pod \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") "
Sep 17 18:52:26.622514 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.622419    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-tmp-dir\") pod \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\" (UID: \"1c660442-a3d5-44e1-88a7-d502b2bdf2ac\") "
Sep 17 18:52:26.622681 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.622535    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "1c660442-a3d5-44e1-88a7-d502b2bdf2ac" (UID: "1c660442-a3d5-44e1-88a7-d502b2bdf2ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:52:26.622681 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.622593    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-home" (OuterVolumeSpecName: "home") pod "1c660442-a3d5-44e1-88a7-d502b2bdf2ac" (UID: "1c660442-a3d5-44e1-88a7-d502b2bdf2ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:52:26.622681 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.622666    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1c660442-a3d5-44e1-88a7-d502b2bdf2ac" (UID: "1c660442-a3d5-44e1-88a7-d502b2bdf2ac"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:52:26.622839 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.622722    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:52:26.622839 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.622741    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:52:26.622950 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.622899    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1c660442-a3d5-44e1-88a7-d502b2bdf2ac" (UID: "1c660442-a3d5-44e1-88a7-d502b2bdf2ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:52:26.624350 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.624326    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-dshm" (OuterVolumeSpecName: "dshm") pod "1c660442-a3d5-44e1-88a7-d502b2bdf2ac" (UID: "1c660442-a3d5-44e1-88a7-d502b2bdf2ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:52:26.624467 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.624447    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-kube-api-access-sb2nb" (OuterVolumeSpecName: "kube-api-access-sb2nb") pod "1c660442-a3d5-44e1-88a7-d502b2bdf2ac" (UID: "1c660442-a3d5-44e1-88a7-d502b2bdf2ac"). InnerVolumeSpecName "kube-api-access-sb2nb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:52:26.624900 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.624881    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c660442-a3d5-44e1-88a7-d502b2bdf2ac" (UID: "1c660442-a3d5-44e1-88a7-d502b2bdf2ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:52:26.644262 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.644235    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1c660442-a3d5-44e1-88a7-d502b2bdf2ac" (UID: "1c660442-a3d5-44e1-88a7-d502b2bdf2ac"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:52:26.723058 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.723031    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-kserve-provision-location\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:52:26.723058 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.723053    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb2nb\" (UniqueName: \"kubernetes.io/projected/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-kube-api-access-sb2nb\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:52:26.723185 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.723064    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:52:26.723185 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.723074    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:52:26.723185 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.723081    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:52:26.723185 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:26.723088    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c660442-a3d5-44e1-88a7-d502b2bdf2ac-cabundle-cert\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:52:27.049326 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:27.049293    2565 generic.go:356] "Generic (PLEG): container finished" podID="9960d8af-172e-46ad-92cf-26a708705864" containerID="143ec6fdafbc74cd393f3c6d2c92e3d0c242d27e32bd0640f24f970b6bdcef70" exitCode=0
Sep 17 18:52:27.049711 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:27.049347    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj"
Sep 17 18:52:27.049711 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:27.049370    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj" event={"ID":"9960d8af-172e-46ad-92cf-26a708705864","Type":"ContainerDied","Data":"143ec6fdafbc74cd393f3c6d2c92e3d0c242d27e32bd0640f24f970b6bdcef70"}
Sep 17 18:52:27.049711 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:27.049422    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj" event={"ID":"9960d8af-172e-46ad-92cf-26a708705864","Type":"ContainerDied","Data":"e68375476602d751c787e640197ffcef51010ee78ce80a2758ec48eb23699355"}
Sep 17 18:52:27.049711 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:27.049447    2565 scope.go:117] "RemoveContainer" containerID="143ec6fdafbc74cd393f3c6d2c92e3d0c242d27e32bd0640f24f970b6bdcef70"
Sep 17 18:52:27.051250 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:27.051222    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8" event={"ID":"1c660442-a3d5-44e1-88a7-d502b2bdf2ac","Type":"ContainerDied","Data":"9f9aa43f1bdd6b2bcabaaaa311da2a7d0d42df7538938d47920b63834db755eb"}
Sep 17 18:52:27.051368 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:27.051322    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"
Sep 17 18:52:27.059406 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:27.059389    2565 scope.go:117] "RemoveContainer" containerID="e90f07429a1370b3ef93d9660d260ed325f26246dc3d5e2408dcb747f1ac258a"
Sep 17 18:52:27.067829 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:27.067788    2565 scope.go:117] "RemoveContainer" containerID="143ec6fdafbc74cd393f3c6d2c92e3d0c242d27e32bd0640f24f970b6bdcef70"
Sep 17 18:52:27.068369 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:52:27.068335    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"143ec6fdafbc74cd393f3c6d2c92e3d0c242d27e32bd0640f24f970b6bdcef70\": container with ID starting with 143ec6fdafbc74cd393f3c6d2c92e3d0c242d27e32bd0640f24f970b6bdcef70 not found: ID does not exist" containerID="143ec6fdafbc74cd393f3c6d2c92e3d0c242d27e32bd0640f24f970b6bdcef70"
Sep 17 18:52:27.068454 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:27.068380    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"143ec6fdafbc74cd393f3c6d2c92e3d0c242d27e32bd0640f24f970b6bdcef70"} err="failed to get container status \"143ec6fdafbc74cd393f3c6d2c92e3d0c242d27e32bd0640f24f970b6bdcef70\": rpc error: code = NotFound desc = could not find container \"143ec6fdafbc74cd393f3c6d2c92e3d0c242d27e32bd0640f24f970b6bdcef70\": container with ID starting with 143ec6fdafbc74cd393f3c6d2c92e3d0c242d27e32bd0640f24f970b6bdcef70 not found: ID does not exist"
Sep 17 18:52:27.068454 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:27.068404    2565 scope.go:117] "RemoveContainer" containerID="e90f07429a1370b3ef93d9660d260ed325f26246dc3d5e2408dcb747f1ac258a"
Sep 17 18:52:27.068699 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:52:27.068678    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e90f07429a1370b3ef93d9660d260ed325f26246dc3d5e2408dcb747f1ac258a\": container with ID starting with e90f07429a1370b3ef93d9660d260ed325f26246dc3d5e2408dcb747f1ac258a not found: ID does not exist" containerID="e90f07429a1370b3ef93d9660d260ed325f26246dc3d5e2408dcb747f1ac258a"
Sep 17 18:52:27.068763 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:27.068708    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e90f07429a1370b3ef93d9660d260ed325f26246dc3d5e2408dcb747f1ac258a"} err="failed to get container status \"e90f07429a1370b3ef93d9660d260ed325f26246dc3d5e2408dcb747f1ac258a\": rpc error: code = NotFound desc = could not find container \"e90f07429a1370b3ef93d9660d260ed325f26246dc3d5e2408dcb747f1ac258a\": container with ID starting with e90f07429a1370b3ef93d9660d260ed325f26246dc3d5e2408dcb747f1ac258a not found: ID does not exist"
Sep 17 18:52:27.068763 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:27.068727    2565 scope.go:117] "RemoveContainer" containerID="9a52ee7cc354957112a31ae0184abfa1a01d4b0641f8ead117268f5f8ba343bf"
Sep 17 18:52:27.070505 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:27.070481    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj"]
Sep 17 18:52:27.074511 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:27.074488    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-b676589fb-sgnhj"]
Sep 17 18:52:27.078315 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:27.078290    2565 scope.go:117] "RemoveContainer" containerID="b7ef2cfc2e323eac7306804428290335cd11701b9f918bc9ff490b67cd4dbe38"
Sep 17 18:52:27.092047 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:27.092025    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"]
Sep 17 18:52:27.100078 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:27.100058    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc76c4685-xdjt8"]
Sep 17 18:52:28.607925 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:28.607886    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c660442-a3d5-44e1-88a7-d502b2bdf2ac" path="/var/lib/kubelet/pods/1c660442-a3d5-44e1-88a7-d502b2bdf2ac/volumes"
Sep 17 18:52:28.608352 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:52:28.608336    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9960d8af-172e-46ad-92cf-26a708705864" path="/var/lib/kubelet/pods/9960d8af-172e-46ad-92cf-26a708705864/volumes"
Sep 17 18:53:36.836066 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:36.836028    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"]
Sep 17 18:53:36.836599 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:36.836378    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c660442-a3d5-44e1-88a7-d502b2bdf2ac" containerName="storage-initializer"
Sep 17 18:53:36.836599 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:36.836389    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c660442-a3d5-44e1-88a7-d502b2bdf2ac" containerName="storage-initializer"
Sep 17 18:53:36.836599 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:36.836403    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9960d8af-172e-46ad-92cf-26a708705864" containerName="storage-initializer"
Sep 17 18:53:36.836599 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:36.836408    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9960d8af-172e-46ad-92cf-26a708705864" containerName="storage-initializer"
Sep 17 18:53:36.836599 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:36.836414    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9960d8af-172e-46ad-92cf-26a708705864" containerName="main"
Sep 17 18:53:36.836599 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:36.836420    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9960d8af-172e-46ad-92cf-26a708705864" containerName="main"
Sep 17 18:53:36.836599 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:36.836431    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c660442-a3d5-44e1-88a7-d502b2bdf2ac" containerName="main"
Sep 17 18:53:36.836599 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:36.836436    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c660442-a3d5-44e1-88a7-d502b2bdf2ac" containerName="main"
Sep 17 18:53:36.836599 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:36.836497    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9960d8af-172e-46ad-92cf-26a708705864" containerName="main"
Sep 17 18:53:36.836599 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:36.836506    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c660442-a3d5-44e1-88a7-d502b2bdf2ac" containerName="main"
Sep 17 18:53:36.839525 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:36.839509    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:53:36.842345 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:36.842306    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 17 18:53:36.842583 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:36.842481    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:53:36.843394 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:36.843364    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 17 18:53:36.843394 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:36.843387    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 17 18:53:36.843564 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:36.843378    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 17 18:53:36.843564 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:36.843424    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-vjkxw\""
Sep 17 18:53:36.861510 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:36.861490    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"]
Sep 17 18:53:36.946001 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:36.945976    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/006769fb-d838-4a0d-a618-1daac8f489e7-home\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:53:36.946104 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:36.946003    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/006769fb-d838-4a0d-a618-1daac8f489e7-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:53:36.946104 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:36.946032    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/006769fb-d838-4a0d-a618-1daac8f489e7-tls-certs\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:53:36.946104 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:36.946052    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/006769fb-d838-4a0d-a618-1daac8f489e7-dshm\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:53:36.946252 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:36.946122    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/006769fb-d838-4a0d-a618-1daac8f489e7-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:53:36.946252 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:36.946157    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxk98\" (UniqueName: \"kubernetes.io/projected/006769fb-d838-4a0d-a618-1daac8f489e7-kube-api-access-vxk98\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:53:36.946252 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:36.946228    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/006769fb-d838-4a0d-a618-1daac8f489e7-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:53:36.946372 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:36.946263    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/006769fb-d838-4a0d-a618-1daac8f489e7-model-cache\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:53:37.047416 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.047390    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/006769fb-d838-4a0d-a618-1daac8f489e7-tls-certs\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:53:37.047514 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.047425    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/006769fb-d838-4a0d-a618-1daac8f489e7-dshm\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:53:37.047514 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.047444    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/006769fb-d838-4a0d-a618-1daac8f489e7-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:53:37.047514 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.047462    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxk98\" (UniqueName: \"kubernetes.io/projected/006769fb-d838-4a0d-a618-1daac8f489e7-kube-api-access-vxk98\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:53:37.047514 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.047496    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/006769fb-d838-4a0d-a618-1daac8f489e7-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:53:37.047678 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.047591    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/006769fb-d838-4a0d-a618-1daac8f489e7-model-cache\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:53:37.047678 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.047648    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/006769fb-d838-4a0d-a618-1daac8f489e7-home\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:53:37.047764 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.047675    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/006769fb-d838-4a0d-a618-1daac8f489e7-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:53:37.047995 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.047950    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/006769fb-d838-4a0d-a618-1daac8f489e7-model-cache\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:53:37.048113 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.048005    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/006769fb-d838-4a0d-a618-1daac8f489e7-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:53:37.048113 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.048025    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/006769fb-d838-4a0d-a618-1daac8f489e7-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:53:37.048113 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.048096    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/006769fb-d838-4a0d-a618-1daac8f489e7-home\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:53:37.048268 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.048154    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/006769fb-d838-4a0d-a618-1daac8f489e7-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:53:37.049756 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.049735    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/006769fb-d838-4a0d-a618-1daac8f489e7-dshm\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:53:37.050012 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.049994    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/006769fb-d838-4a0d-a618-1daac8f489e7-tls-certs\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:53:37.065362 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.059379    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxk98\" (UniqueName: \"kubernetes.io/projected/006769fb-d838-4a0d-a618-1daac8f489e7-kube-api-access-vxk98\") pod \"tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:53:37.102353 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.102282    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm"]
Sep 17 18:53:37.106818 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.106798    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm"
Sep 17 18:53:37.122665 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.122639    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm"]
Sep 17 18:53:37.148988 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.148952    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e32fc44e-89cd-4888-b189-a32ca6abb271-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm\" (UID: \"e32fc44e-89cd-4888-b189-a32ca6abb271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm"
Sep 17 18:53:37.148988 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.148983    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e32fc44e-89cd-4888-b189-a32ca6abb271-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm\" (UID: \"e32fc44e-89cd-4888-b189-a32ca6abb271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm"
Sep 17 18:53:37.149176 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.149020    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e32fc44e-89cd-4888-b189-a32ca6abb271-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm\" (UID: \"e32fc44e-89cd-4888-b189-a32ca6abb271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm"
Sep 17 18:53:37.149176 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.149044    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e32fc44e-89cd-4888-b189-a32ca6abb271-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm\" (UID: \"e32fc44e-89cd-4888-b189-a32ca6abb271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm"
Sep 17 18:53:37.149866 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.149836    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:53:37.249728 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.249693    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e32fc44e-89cd-4888-b189-a32ca6abb271-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm\" (UID: \"e32fc44e-89cd-4888-b189-a32ca6abb271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm"
Sep 17 18:53:37.249879 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.249734    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e32fc44e-89cd-4888-b189-a32ca6abb271-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm\" (UID: \"e32fc44e-89cd-4888-b189-a32ca6abb271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm"
Sep 17 18:53:37.249879 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.249822    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e32fc44e-89cd-4888-b189-a32ca6abb271-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm\" (UID: \"e32fc44e-89cd-4888-b189-a32ca6abb271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm"
Sep 17 18:53:37.249879 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.249849    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e32fc44e-89cd-4888-b189-a32ca6abb271-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm\" (UID: \"e32fc44e-89cd-4888-b189-a32ca6abb271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm"
Sep 17 18:53:37.250270 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.250246    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e32fc44e-89cd-4888-b189-a32ca6abb271-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm\" (UID: \"e32fc44e-89cd-4888-b189-a32ca6abb271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm"
Sep 17 18:53:37.250349 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.250259    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e32fc44e-89cd-4888-b189-a32ca6abb271-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm\" (UID: \"e32fc44e-89cd-4888-b189-a32ca6abb271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm"
Sep 17 18:53:37.250531 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.250513    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e32fc44e-89cd-4888-b189-a32ca6abb271-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm\" (UID: \"e32fc44e-89cd-4888-b189-a32ca6abb271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm"
Sep 17 18:53:37.252238 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.252222    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e32fc44e-89cd-4888-b189-a32ca6abb271-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm\" (UID: \"e32fc44e-89cd-4888-b189-a32ca6abb271\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm"
Sep 17 18:53:37.417289 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.417264    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm"
Sep 17 18:53:37.428340 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.428048    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"]
Sep 17 18:53:37.432493 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:53:37.432468    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod006769fb_d838_4a0d_a618_1daac8f489e7.slice/crio-2829f89d82b96c5d9cc35fb76a382eb5a4b88be03f4be92a2b31f37237560bcd WatchSource:0}: Error finding container 2829f89d82b96c5d9cc35fb76a382eb5a4b88be03f4be92a2b31f37237560bcd: Status 404 returned error can't find the container with id 2829f89d82b96c5d9cc35fb76a382eb5a4b88be03f4be92a2b31f37237560bcd
Sep 17 18:53:37.698445 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:37.698417    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm"]
Sep 17 18:53:37.700076 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:53:37.700051    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode32fc44e_89cd_4888_b189_a32ca6abb271.slice/crio-d66e4f9013913d7f6ecfbdb969938459f08c07f9999e24ccabb414d57e3ae282 WatchSource:0}: Error finding container d66e4f9013913d7f6ecfbdb969938459f08c07f9999e24ccabb414d57e3ae282: Status 404 returned error can't find the container with id d66e4f9013913d7f6ecfbdb969938459f08c07f9999e24ccabb414d57e3ae282
Sep 17 18:53:38.432819 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:38.432783    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm" event={"ID":"e32fc44e-89cd-4888-b189-a32ca6abb271","Type":"ContainerStarted","Data":"c579c58fa9754947cf6bf03fcf9dc8313ab5022b0cbc4b46ea1f810358a7c530"}
Sep 17 18:53:38.433331 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:38.432846    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm" event={"ID":"e32fc44e-89cd-4888-b189-a32ca6abb271","Type":"ContainerStarted","Data":"d66e4f9013913d7f6ecfbdb969938459f08c07f9999e24ccabb414d57e3ae282"}
Sep 17 18:53:38.434573 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:38.434543    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl" event={"ID":"006769fb-d838-4a0d-a618-1daac8f489e7","Type":"ContainerStarted","Data":"c2715127810ef65404181f079a2d4eeebdca83f6e9ac261929568ea95f182184"}
Sep 17 18:53:38.434691 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:38.434592    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl" event={"ID":"006769fb-d838-4a0d-a618-1daac8f489e7","Type":"ContainerStarted","Data":"2829f89d82b96c5d9cc35fb76a382eb5a4b88be03f4be92a2b31f37237560bcd"}
Sep 17 18:53:40.447942 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:40.447912    2565 generic.go:356] "Generic (PLEG): container finished" podID="006769fb-d838-4a0d-a618-1daac8f489e7" containerID="c2715127810ef65404181f079a2d4eeebdca83f6e9ac261929568ea95f182184" exitCode=0
Sep 17 18:53:40.448309 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:40.447987    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl" event={"ID":"006769fb-d838-4a0d-a618-1daac8f489e7","Type":"ContainerDied","Data":"c2715127810ef65404181f079a2d4eeebdca83f6e9ac261929568ea95f182184"}
Sep 17 18:53:40.449750 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:40.449730    2565 generic.go:356] "Generic (PLEG): container finished" podID="e32fc44e-89cd-4888-b189-a32ca6abb271" containerID="c579c58fa9754947cf6bf03fcf9dc8313ab5022b0cbc4b46ea1f810358a7c530" exitCode=0
Sep 17 18:53:40.449844 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:40.449799    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm" event={"ID":"e32fc44e-89cd-4888-b189-a32ca6abb271","Type":"ContainerDied","Data":"c579c58fa9754947cf6bf03fcf9dc8313ab5022b0cbc4b46ea1f810358a7c530"}
Sep 17 18:53:41.458173 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:41.458128    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm" event={"ID":"e32fc44e-89cd-4888-b189-a32ca6abb271","Type":"ContainerStarted","Data":"1b4c11d1ce05c6a7b16548424aed52b2e34afa341cc3ea450b2207fadc6901da"}
Sep 17 18:53:41.458654 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:41.458407    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm"
Sep 17 18:53:41.460347 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:41.460325    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl" event={"ID":"006769fb-d838-4a0d-a618-1daac8f489e7","Type":"ContainerStarted","Data":"29f3ab3ddbe74c97d9e6f8d578497a1d4781e58e649520950d22c99c8a9d3bb0"}
Sep 17 18:53:41.485259 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:41.485213    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm" podStartSLOduration=1.990866622 podStartE2EDuration="4.485182743s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.494316121s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:53:41.481395042 +0000 UTC m=+1479.413596552" watchObservedRunningTime="2026-09-17 18:53:41.485182743 +0000 UTC m=+1479.417384284"
Sep 17 18:53:41.506992 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:41.506952    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl" podStartSLOduration=3.045622237 podStartE2EDuration="5.506938226s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.461315989s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:53:41.503976315 +0000 UTC m=+1479.436177836" watchObservedRunningTime="2026-09-17 18:53:41.506938226 +0000 UTC m=+1479.439139744"
Sep 17 18:53:42.150465 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:42.150429    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:53:47.150667 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:47.150573    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:53:47.163626 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:47.163598    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:53:47.508377 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:53:47.508341    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:54:12.480333 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:12.480299    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm"
Sep 17 18:54:13.545481 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:13.545449    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm_e32fc44e-89cd-4888-b189-a32ca6abb271/main/0.log"
Sep 17 18:54:14.003063 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.003027    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm"]
Sep 17 18:54:14.003545 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.003480    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm" podUID="e32fc44e-89cd-4888-b189-a32ca6abb271" containerName="main" containerID="cri-o://1b4c11d1ce05c6a7b16548424aed52b2e34afa341cc3ea450b2207fadc6901da" gracePeriod=30
Sep 17 18:54:14.006648 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.006625    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"]
Sep 17 18:54:14.016056 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:54:14.016027    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:54:14.016056 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:54:14.016056 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:54:14.016056 ip-10-0-139-75 kubenswrapper[2565]:  > containerID="29f3ab3ddbe74c97d9e6f8d578497a1d4781e58e649520950d22c99c8a9d3bb0" cmd=["/bin/sleep","15"]
Sep 17 18:54:14.016292 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.016074    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl" message="" err=<
Sep 17 18:54:14.016292 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:54:14.016292 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:54:14.016292 ip-10-0-139-75 kubenswrapper[2565]:  >
Sep 17 18:54:14.016292 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:54:14.016092    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:54:14.016292 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:54:14.016292 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:54:14.016292 ip-10-0-139-75 kubenswrapper[2565]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl" podUID="006769fb-d838-4a0d-a618-1daac8f489e7" containerName="main" containerID="cri-o://29f3ab3ddbe74c97d9e6f8d578497a1d4781e58e649520950d22c99c8a9d3bb0"
Sep 17 18:54:14.016292 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.016128    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl" podUID="006769fb-d838-4a0d-a618-1daac8f489e7" containerName="main" containerID="cri-o://29f3ab3ddbe74c97d9e6f8d578497a1d4781e58e649520950d22c99c8a9d3bb0" gracePeriod=60
Sep 17 18:54:14.365892 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.365866    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:54:14.482460 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.482431    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/006769fb-d838-4a0d-a618-1daac8f489e7-home\") pod \"006769fb-d838-4a0d-a618-1daac8f489e7\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") "
Sep 17 18:54:14.482601 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.482478    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/006769fb-d838-4a0d-a618-1daac8f489e7-model-cache\") pod \"006769fb-d838-4a0d-a618-1daac8f489e7\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") "
Sep 17 18:54:14.482601 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.482518    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/006769fb-d838-4a0d-a618-1daac8f489e7-cabundle-cert\") pod \"006769fb-d838-4a0d-a618-1daac8f489e7\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") "
Sep 17 18:54:14.482695 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.482602    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/006769fb-d838-4a0d-a618-1daac8f489e7-tmp-dir\") pod \"006769fb-d838-4a0d-a618-1daac8f489e7\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") "
Sep 17 18:54:14.482695 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.482634    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxk98\" (UniqueName: \"kubernetes.io/projected/006769fb-d838-4a0d-a618-1daac8f489e7-kube-api-access-vxk98\") pod \"006769fb-d838-4a0d-a618-1daac8f489e7\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") "
Sep 17 18:54:14.482778 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.482735    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006769fb-d838-4a0d-a618-1daac8f489e7-home" (OuterVolumeSpecName: "home") pod "006769fb-d838-4a0d-a618-1daac8f489e7" (UID: "006769fb-d838-4a0d-a618-1daac8f489e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:14.482778 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.482754    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006769fb-d838-4a0d-a618-1daac8f489e7-model-cache" (OuterVolumeSpecName: "model-cache") pod "006769fb-d838-4a0d-a618-1daac8f489e7" (UID: "006769fb-d838-4a0d-a618-1daac8f489e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:14.482860 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.482774    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/006769fb-d838-4a0d-a618-1daac8f489e7-dshm\") pod \"006769fb-d838-4a0d-a618-1daac8f489e7\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") "
Sep 17 18:54:14.482860 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.482819    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006769fb-d838-4a0d-a618-1daac8f489e7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "006769fb-d838-4a0d-a618-1daac8f489e7" (UID: "006769fb-d838-4a0d-a618-1daac8f489e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:14.482860 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.482825    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/006769fb-d838-4a0d-a618-1daac8f489e7-kserve-provision-location\") pod \"006769fb-d838-4a0d-a618-1daac8f489e7\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") "
Sep 17 18:54:14.483004 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.482878    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/006769fb-d838-4a0d-a618-1daac8f489e7-tls-certs\") pod \"006769fb-d838-4a0d-a618-1daac8f489e7\" (UID: \"006769fb-d838-4a0d-a618-1daac8f489e7\") "
Sep 17 18:54:14.483004 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.482902    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/006769fb-d838-4a0d-a618-1daac8f489e7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "006769fb-d838-4a0d-a618-1daac8f489e7" (UID: "006769fb-d838-4a0d-a618-1daac8f489e7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:54:14.483193 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.483171    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/006769fb-d838-4a0d-a618-1daac8f489e7-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:54:14.483316 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.483222    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/006769fb-d838-4a0d-a618-1daac8f489e7-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:54:14.483316 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.483238    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/006769fb-d838-4a0d-a618-1daac8f489e7-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:54:14.483316 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.483251    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/006769fb-d838-4a0d-a618-1daac8f489e7-cabundle-cert\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:54:14.485026 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.485003    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/006769fb-d838-4a0d-a618-1daac8f489e7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "006769fb-d838-4a0d-a618-1daac8f489e7" (UID: "006769fb-d838-4a0d-a618-1daac8f489e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:54:14.485151 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.485098    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006769fb-d838-4a0d-a618-1daac8f489e7-dshm" (OuterVolumeSpecName: "dshm") pod "006769fb-d838-4a0d-a618-1daac8f489e7" (UID: "006769fb-d838-4a0d-a618-1daac8f489e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:14.485243 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.485187    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006769fb-d838-4a0d-a618-1daac8f489e7-kube-api-access-vxk98" (OuterVolumeSpecName: "kube-api-access-vxk98") pod "006769fb-d838-4a0d-a618-1daac8f489e7" (UID: "006769fb-d838-4a0d-a618-1daac8f489e7"). InnerVolumeSpecName "kube-api-access-vxk98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:54:14.505024 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.505000    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006769fb-d838-4a0d-a618-1daac8f489e7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "006769fb-d838-4a0d-a618-1daac8f489e7" (UID: "006769fb-d838-4a0d-a618-1daac8f489e7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:14.583706 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.583674    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxk98\" (UniqueName: \"kubernetes.io/projected/006769fb-d838-4a0d-a618-1daac8f489e7-kube-api-access-vxk98\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:54:14.583706 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.583707    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/006769fb-d838-4a0d-a618-1daac8f489e7-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:54:14.584061 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.583719    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/006769fb-d838-4a0d-a618-1daac8f489e7-kserve-provision-location\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:54:14.584061 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.583729    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/006769fb-d838-4a0d-a618-1daac8f489e7-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:54:14.647864 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.647838    2565 generic.go:356] "Generic (PLEG): container finished" podID="006769fb-d838-4a0d-a618-1daac8f489e7" containerID="29f3ab3ddbe74c97d9e6f8d578497a1d4781e58e649520950d22c99c8a9d3bb0" exitCode=0
Sep 17 18:54:14.647985 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.647887    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"
Sep 17 18:54:14.647985 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.647913    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl" event={"ID":"006769fb-d838-4a0d-a618-1daac8f489e7","Type":"ContainerDied","Data":"29f3ab3ddbe74c97d9e6f8d578497a1d4781e58e649520950d22c99c8a9d3bb0"}
Sep 17 18:54:14.647985 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.647975    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl" event={"ID":"006769fb-d838-4a0d-a618-1daac8f489e7","Type":"ContainerDied","Data":"2829f89d82b96c5d9cc35fb76a382eb5a4b88be03f4be92a2b31f37237560bcd"}
Sep 17 18:54:14.648135 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.648007    2565 scope.go:117] "RemoveContainer" containerID="29f3ab3ddbe74c97d9e6f8d578497a1d4781e58e649520950d22c99c8a9d3bb0"
Sep 17 18:54:14.650491 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.650459    2565 generic.go:356] "Generic (PLEG): container finished" podID="e32fc44e-89cd-4888-b189-a32ca6abb271" containerID="1b4c11d1ce05c6a7b16548424aed52b2e34afa341cc3ea450b2207fadc6901da" exitCode=0
Sep 17 18:54:14.650575 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.650508    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm" event={"ID":"e32fc44e-89cd-4888-b189-a32ca6abb271","Type":"ContainerDied","Data":"1b4c11d1ce05c6a7b16548424aed52b2e34afa341cc3ea450b2207fadc6901da"}
Sep 17 18:54:14.657599 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.657581    2565 scope.go:117] "RemoveContainer" containerID="c2715127810ef65404181f079a2d4eeebdca83f6e9ac261929568ea95f182184"
Sep 17 18:54:14.660050 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.660036    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm"
Sep 17 18:54:14.665814 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.665797    2565 scope.go:117] "RemoveContainer" containerID="29f3ab3ddbe74c97d9e6f8d578497a1d4781e58e649520950d22c99c8a9d3bb0"
Sep 17 18:54:14.666081 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:54:14.666052    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29f3ab3ddbe74c97d9e6f8d578497a1d4781e58e649520950d22c99c8a9d3bb0\": container with ID starting with 29f3ab3ddbe74c97d9e6f8d578497a1d4781e58e649520950d22c99c8a9d3bb0 not found: ID does not exist" containerID="29f3ab3ddbe74c97d9e6f8d578497a1d4781e58e649520950d22c99c8a9d3bb0"
Sep 17 18:54:14.666145 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.666088    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29f3ab3ddbe74c97d9e6f8d578497a1d4781e58e649520950d22c99c8a9d3bb0"} err="failed to get container status \"29f3ab3ddbe74c97d9e6f8d578497a1d4781e58e649520950d22c99c8a9d3bb0\": rpc error: code = NotFound desc = could not find container \"29f3ab3ddbe74c97d9e6f8d578497a1d4781e58e649520950d22c99c8a9d3bb0\": container with ID starting with 29f3ab3ddbe74c97d9e6f8d578497a1d4781e58e649520950d22c99c8a9d3bb0 not found: ID does not exist"
Sep 17 18:54:14.666145 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.666103    2565 scope.go:117] "RemoveContainer" containerID="c2715127810ef65404181f079a2d4eeebdca83f6e9ac261929568ea95f182184"
Sep 17 18:54:14.666398 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:54:14.666380    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2715127810ef65404181f079a2d4eeebdca83f6e9ac261929568ea95f182184\": container with ID starting with c2715127810ef65404181f079a2d4eeebdca83f6e9ac261929568ea95f182184 not found: ID does not exist" containerID="c2715127810ef65404181f079a2d4eeebdca83f6e9ac261929568ea95f182184"
Sep 17 18:54:14.666450 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.666403    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2715127810ef65404181f079a2d4eeebdca83f6e9ac261929568ea95f182184"} err="failed to get container status \"c2715127810ef65404181f079a2d4eeebdca83f6e9ac261929568ea95f182184\": rpc error: code = NotFound desc = could not find container \"c2715127810ef65404181f079a2d4eeebdca83f6e9ac261929568ea95f182184\": container with ID starting with c2715127810ef65404181f079a2d4eeebdca83f6e9ac261929568ea95f182184 not found: ID does not exist"
Sep 17 18:54:14.666496 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.666447    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"]
Sep 17 18:54:14.670629 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.670607    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-84f84dfc7b-nlfhl"]
Sep 17 18:54:14.785026 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.785007    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e32fc44e-89cd-4888-b189-a32ca6abb271-cabundle-cert\") pod \"e32fc44e-89cd-4888-b189-a32ca6abb271\" (UID: \"e32fc44e-89cd-4888-b189-a32ca6abb271\") "
Sep 17 18:54:14.785137 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.785087    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e32fc44e-89cd-4888-b189-a32ca6abb271-tmp-dir\") pod \"e32fc44e-89cd-4888-b189-a32ca6abb271\" (UID: \"e32fc44e-89cd-4888-b189-a32ca6abb271\") "
Sep 17 18:54:14.785189 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.785148    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e32fc44e-89cd-4888-b189-a32ca6abb271-kserve-provision-location\") pod \"e32fc44e-89cd-4888-b189-a32ca6abb271\" (UID: \"e32fc44e-89cd-4888-b189-a32ca6abb271\") "
Sep 17 18:54:14.785189 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.785181    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e32fc44e-89cd-4888-b189-a32ca6abb271-tls-certs\") pod \"e32fc44e-89cd-4888-b189-a32ca6abb271\" (UID: \"e32fc44e-89cd-4888-b189-a32ca6abb271\") "
Sep 17 18:54:14.785458 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.785436    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e32fc44e-89cd-4888-b189-a32ca6abb271-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e32fc44e-89cd-4888-b189-a32ca6abb271" (UID: "e32fc44e-89cd-4888-b189-a32ca6abb271"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:54:14.785533 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.785493    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32fc44e-89cd-4888-b189-a32ca6abb271-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e32fc44e-89cd-4888-b189-a32ca6abb271" (UID: "e32fc44e-89cd-4888-b189-a32ca6abb271"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:14.787177 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.787156    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e32fc44e-89cd-4888-b189-a32ca6abb271-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e32fc44e-89cd-4888-b189-a32ca6abb271" (UID: "e32fc44e-89cd-4888-b189-a32ca6abb271"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:54:14.808227 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.808188    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e32fc44e-89cd-4888-b189-a32ca6abb271-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e32fc44e-89cd-4888-b189-a32ca6abb271" (UID: "e32fc44e-89cd-4888-b189-a32ca6abb271"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:14.886093 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.886037    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e32fc44e-89cd-4888-b189-a32ca6abb271-kserve-provision-location\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:54:14.886093 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.886058    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e32fc44e-89cd-4888-b189-a32ca6abb271-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:54:14.886093 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.886069    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e32fc44e-89cd-4888-b189-a32ca6abb271-cabundle-cert\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:54:14.886093 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:14.886077    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e32fc44e-89cd-4888-b189-a32ca6abb271-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:54:15.657727 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:15.657696    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm"
Sep 17 18:54:15.658235 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:15.657696    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm" event={"ID":"e32fc44e-89cd-4888-b189-a32ca6abb271","Type":"ContainerDied","Data":"d66e4f9013913d7f6ecfbdb969938459f08c07f9999e24ccabb414d57e3ae282"}
Sep 17 18:54:15.658235 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:15.657854    2565 scope.go:117] "RemoveContainer" containerID="1b4c11d1ce05c6a7b16548424aed52b2e34afa341cc3ea450b2207fadc6901da"
Sep 17 18:54:15.668489 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:15.668472    2565 scope.go:117] "RemoveContainer" containerID="c579c58fa9754947cf6bf03fcf9dc8313ab5022b0cbc4b46ea1f810358a7c530"
Sep 17 18:54:15.691989 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:15.691958    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm"]
Sep 17 18:54:15.698960 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:15.698937    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-59958df9db-zlrrm"]
Sep 17 18:54:16.606438 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:16.606409    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="006769fb-d838-4a0d-a618-1daac8f489e7" path="/var/lib/kubelet/pods/006769fb-d838-4a0d-a618-1daac8f489e7/volumes"
Sep 17 18:54:16.606832 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:54:16.606818    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e32fc44e-89cd-4888-b189-a32ca6abb271" path="/var/lib/kubelet/pods/e32fc44e-89cd-4888-b189-a32ca6abb271/volumes"
Sep 17 18:55:02.233316 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.233281    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"]
Sep 17 18:55:02.233748 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.233655    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="006769fb-d838-4a0d-a618-1daac8f489e7" containerName="storage-initializer"
Sep 17 18:55:02.233748 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.233669    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="006769fb-d838-4a0d-a618-1daac8f489e7" containerName="storage-initializer"
Sep 17 18:55:02.233748 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.233681    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e32fc44e-89cd-4888-b189-a32ca6abb271" containerName="storage-initializer"
Sep 17 18:55:02.233748 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.233687    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e32fc44e-89cd-4888-b189-a32ca6abb271" containerName="storage-initializer"
Sep 17 18:55:02.233748 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.233704    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="006769fb-d838-4a0d-a618-1daac8f489e7" containerName="main"
Sep 17 18:55:02.233748 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.233709    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="006769fb-d838-4a0d-a618-1daac8f489e7" containerName="main"
Sep 17 18:55:02.233748 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.233719    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e32fc44e-89cd-4888-b189-a32ca6abb271" containerName="main"
Sep 17 18:55:02.233748 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.233727    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e32fc44e-89cd-4888-b189-a32ca6abb271" containerName="main"
Sep 17 18:55:02.234016 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.233785    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e32fc44e-89cd-4888-b189-a32ca6abb271" containerName="main"
Sep 17 18:55:02.234016 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.233794    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="006769fb-d838-4a0d-a618-1daac8f489e7" containerName="main"
Sep 17 18:55:02.237079 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.237059    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:55:02.239515 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.239491    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 17 18:55:02.239636 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.239542    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-ctd2t\""
Sep 17 18:55:02.239690 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.239630    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 17 18:55:02.239690 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.239656    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:55:02.239776 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.239716    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 17 18:55:02.240341 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.240325    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 17 18:55:02.249743 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.249721    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"]
Sep 17 18:55:02.315683 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.315651    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmv9x\" (UniqueName: \"kubernetes.io/projected/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-kube-api-access-pmv9x\") pod \"auth-disabled-test-kserve-5c9cfb7998-c86b4\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:55:02.315818 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.315710    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-dshm\") pod \"auth-disabled-test-kserve-5c9cfb7998-c86b4\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:55:02.315818 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.315742    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-kserve-provision-location\") pod \"auth-disabled-test-kserve-5c9cfb7998-c86b4\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:55:02.315818 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.315773    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-tmp-dir\") pod \"auth-disabled-test-kserve-5c9cfb7998-c86b4\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:55:02.315952 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.315855    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-model-cache\") pod \"auth-disabled-test-kserve-5c9cfb7998-c86b4\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:55:02.315952 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.315905    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-home\") pod \"auth-disabled-test-kserve-5c9cfb7998-c86b4\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:55:02.315952 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.315928    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-cabundle-cert\") pod \"auth-disabled-test-kserve-5c9cfb7998-c86b4\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:55:02.315952 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.315948    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-tls-certs\") pod \"auth-disabled-test-kserve-5c9cfb7998-c86b4\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:55:02.417264 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.417238    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmv9x\" (UniqueName: \"kubernetes.io/projected/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-kube-api-access-pmv9x\") pod \"auth-disabled-test-kserve-5c9cfb7998-c86b4\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:55:02.417371 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.417283    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-dshm\") pod \"auth-disabled-test-kserve-5c9cfb7998-c86b4\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:55:02.417371 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.417302    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-kserve-provision-location\") pod \"auth-disabled-test-kserve-5c9cfb7998-c86b4\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:55:02.417371 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.417330    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-tmp-dir\") pod \"auth-disabled-test-kserve-5c9cfb7998-c86b4\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:55:02.417371 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.417367    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-model-cache\") pod \"auth-disabled-test-kserve-5c9cfb7998-c86b4\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:55:02.417564 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.417396    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-home\") pod \"auth-disabled-test-kserve-5c9cfb7998-c86b4\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:55:02.417564 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.417426    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-cabundle-cert\") pod \"auth-disabled-test-kserve-5c9cfb7998-c86b4\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:55:02.417564 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.417452    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-tls-certs\") pod \"auth-disabled-test-kserve-5c9cfb7998-c86b4\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:55:02.417724 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.417710    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-kserve-provision-location\") pod \"auth-disabled-test-kserve-5c9cfb7998-c86b4\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:55:02.417785 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.417775    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-tmp-dir\") pod \"auth-disabled-test-kserve-5c9cfb7998-c86b4\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:55:02.417906 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.417886    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-model-cache\") pod \"auth-disabled-test-kserve-5c9cfb7998-c86b4\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:55:02.417976 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.417923    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-home\") pod \"auth-disabled-test-kserve-5c9cfb7998-c86b4\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:55:02.418143 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.418119    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-cabundle-cert\") pod \"auth-disabled-test-kserve-5c9cfb7998-c86b4\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:55:02.419784 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.419760    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-dshm\") pod \"auth-disabled-test-kserve-5c9cfb7998-c86b4\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:55:02.419978 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.419962    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-tls-certs\") pod \"auth-disabled-test-kserve-5c9cfb7998-c86b4\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:55:02.426179 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.426153    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmv9x\" (UniqueName: \"kubernetes.io/projected/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-kube-api-access-pmv9x\") pod \"auth-disabled-test-kserve-5c9cfb7998-c86b4\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:55:02.549909 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.549832    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-ctd2t\""
Sep 17 18:55:02.557782 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.557766    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:55:02.846090 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.846061    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"]
Sep 17 18:55:02.846853 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:55:02.846827    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a31b1a_35d4_433d_beb3_7d69b89eeb2c.slice/crio-a8cca146839b78b682cf870e8da19f5026d5d78cc0cb9ef2a7401b4cfac7b459 WatchSource:0}: Error finding container a8cca146839b78b682cf870e8da19f5026d5d78cc0cb9ef2a7401b4cfac7b459: Status 404 returned error can't find the container with id a8cca146839b78b682cf870e8da19f5026d5d78cc0cb9ef2a7401b4cfac7b459
Sep 17 18:55:02.851375 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.851356    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 17 18:55:02.913306 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:02.913281    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4" event={"ID":"32a31b1a-35d4-433d-beb3-7d69b89eeb2c","Type":"ContainerStarted","Data":"a8cca146839b78b682cf870e8da19f5026d5d78cc0cb9ef2a7401b4cfac7b459"}
Sep 17 18:55:03.920324 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:03.920287    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4" event={"ID":"32a31b1a-35d4-433d-beb3-7d69b89eeb2c","Type":"ContainerStarted","Data":"cd56c7f30df73b2cd19332b0621f0e48c07da76db4d5fbe30c3a38f39bfde9c6"}
Sep 17 18:55:05.931284 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:05.931251    2565 generic.go:356] "Generic (PLEG): container finished" podID="32a31b1a-35d4-433d-beb3-7d69b89eeb2c" containerID="cd56c7f30df73b2cd19332b0621f0e48c07da76db4d5fbe30c3a38f39bfde9c6" exitCode=0
Sep 17 18:55:05.931706 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:05.931284    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4" event={"ID":"32a31b1a-35d4-433d-beb3-7d69b89eeb2c","Type":"ContainerDied","Data":"cd56c7f30df73b2cd19332b0621f0e48c07da76db4d5fbe30c3a38f39bfde9c6"}
Sep 17 18:55:06.939333 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:06.939290    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4" event={"ID":"32a31b1a-35d4-433d-beb3-7d69b89eeb2c","Type":"ContainerStarted","Data":"3c2327433f2b493f7303ac49e6e92f12288973b7f82f9b33d16566f8eb8bc0d4"}
Sep 17 18:55:06.972583 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:06.972524    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4" podStartSLOduration=2.771961365 podStartE2EDuration="4.97250899s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.200547625s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:55:06.969329387 +0000 UTC m=+1564.901530906" watchObservedRunningTime="2026-09-17 18:55:06.97250899 +0000 UTC m=+1564.904710508"
Sep 17 18:55:07.558991 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:07.558957    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:55:12.558243 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:12.558192    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:55:12.559807 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:12.559777    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4" podUID="32a31b1a-35d4-433d-beb3-7d69b89eeb2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 18:55:21.602962 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.602928    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"]
Sep 17 18:55:21.607144 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.607120    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:55:21.609938 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.609913    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 17 18:55:21.610055 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.610015    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 17 18:55:21.610752 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.610722    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-fl45c\""
Sep 17 18:55:21.610752 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.610720    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 17 18:55:21.610925 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.610724    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 17 18:55:21.610925 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.610798    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:55:21.623530 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.623506    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"]
Sep 17 18:55:21.677458 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.677433    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z56n\" (UniqueName: \"kubernetes.io/projected/76fe11b0-8510-479e-848a-857159d5839b-kube-api-access-5z56n\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:55:21.677609 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.677469    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76fe11b0-8510-479e-848a-857159d5839b-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:55:21.677609 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.677547    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76fe11b0-8510-479e-848a-857159d5839b-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:55:21.677770 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.677615    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76fe11b0-8510-479e-848a-857159d5839b-model-cache\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:55:21.677770 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.677688    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76fe11b0-8510-479e-848a-857159d5839b-tls-certs\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:55:21.677770 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.677737    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76fe11b0-8510-479e-848a-857159d5839b-home\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:55:21.677770 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.677759    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76fe11b0-8510-479e-848a-857159d5839b-tmp-dir\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:55:21.677935 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.677795    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76fe11b0-8510-479e-848a-857159d5839b-dshm\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:55:21.779079 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.779055    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76fe11b0-8510-479e-848a-857159d5839b-model-cache\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:55:21.779193 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.779093    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76fe11b0-8510-479e-848a-857159d5839b-tls-certs\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:55:21.779193 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.779142    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76fe11b0-8510-479e-848a-857159d5839b-home\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:55:21.779193 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.779173    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76fe11b0-8510-479e-848a-857159d5839b-tmp-dir\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:55:21.779401 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.779243    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76fe11b0-8510-479e-848a-857159d5839b-dshm\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:55:21.779401 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.779276    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z56n\" (UniqueName: \"kubernetes.io/projected/76fe11b0-8510-479e-848a-857159d5839b-kube-api-access-5z56n\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:55:21.779401 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.779316    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76fe11b0-8510-479e-848a-857159d5839b-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:55:21.779401 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.779365    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76fe11b0-8510-479e-848a-857159d5839b-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:55:21.779607 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.779526    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76fe11b0-8510-479e-848a-857159d5839b-model-cache\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:55:21.779716 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.779697    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76fe11b0-8510-479e-848a-857159d5839b-tmp-dir\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:55:21.779802 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.779758    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76fe11b0-8510-479e-848a-857159d5839b-home\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:55:21.780002 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.779978    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76fe11b0-8510-479e-848a-857159d5839b-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:55:21.780391 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.780364    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76fe11b0-8510-479e-848a-857159d5839b-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:55:21.781636 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.781616    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76fe11b0-8510-479e-848a-857159d5839b-dshm\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:55:21.781862 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.781838    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76fe11b0-8510-479e-848a-857159d5839b-tls-certs\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:55:21.793123 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.793098    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z56n\" (UniqueName: \"kubernetes.io/projected/76fe11b0-8510-479e-848a-857159d5839b-kube-api-access-5z56n\") pod \"tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:55:21.920340 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.920313    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:55:21.933228 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.929022    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5"]
Sep 17 18:55:21.939963 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.939821    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5"
Sep 17 18:55:21.950868 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:21.950842    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5"]
Sep 17 18:55:22.081916 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:22.081879    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/451e2ee6-d170-4b0a-b912-171c9005360a-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-78966cd574-ks2g5\" (UID: \"451e2ee6-d170-4b0a-b912-171c9005360a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5"
Sep 17 18:55:22.082072 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:22.081987    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/451e2ee6-d170-4b0a-b912-171c9005360a-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-78966cd574-ks2g5\" (UID: \"451e2ee6-d170-4b0a-b912-171c9005360a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5"
Sep 17 18:55:22.082072 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:22.082040    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/451e2ee6-d170-4b0a-b912-171c9005360a-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-78966cd574-ks2g5\" (UID: \"451e2ee6-d170-4b0a-b912-171c9005360a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5"
Sep 17 18:55:22.082147 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:22.082083    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/451e2ee6-d170-4b0a-b912-171c9005360a-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-78966cd574-ks2g5\" (UID: \"451e2ee6-d170-4b0a-b912-171c9005360a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5"
Sep 17 18:55:22.183463 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:22.183386    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/451e2ee6-d170-4b0a-b912-171c9005360a-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-78966cd574-ks2g5\" (UID: \"451e2ee6-d170-4b0a-b912-171c9005360a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5"
Sep 17 18:55:22.183463 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:22.183437    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/451e2ee6-d170-4b0a-b912-171c9005360a-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-78966cd574-ks2g5\" (UID: \"451e2ee6-d170-4b0a-b912-171c9005360a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5"
Sep 17 18:55:22.183649 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:22.183490    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/451e2ee6-d170-4b0a-b912-171c9005360a-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-78966cd574-ks2g5\" (UID: \"451e2ee6-d170-4b0a-b912-171c9005360a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5"
Sep 17 18:55:22.183649 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:22.183531    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/451e2ee6-d170-4b0a-b912-171c9005360a-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-78966cd574-ks2g5\" (UID: \"451e2ee6-d170-4b0a-b912-171c9005360a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5"
Sep 17 18:55:22.183880 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:22.183858    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/451e2ee6-d170-4b0a-b912-171c9005360a-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-78966cd574-ks2g5\" (UID: \"451e2ee6-d170-4b0a-b912-171c9005360a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5"
Sep 17 18:55:22.183927 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:22.183907    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/451e2ee6-d170-4b0a-b912-171c9005360a-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-78966cd574-ks2g5\" (UID: \"451e2ee6-d170-4b0a-b912-171c9005360a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5"
Sep 17 18:55:22.184135 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:22.184111    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/451e2ee6-d170-4b0a-b912-171c9005360a-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-78966cd574-ks2g5\" (UID: \"451e2ee6-d170-4b0a-b912-171c9005360a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5"
Sep 17 18:55:22.185926 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:22.185901    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/451e2ee6-d170-4b0a-b912-171c9005360a-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-78966cd574-ks2g5\" (UID: \"451e2ee6-d170-4b0a-b912-171c9005360a\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5"
Sep 17 18:55:22.213552 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:22.213530    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"]
Sep 17 18:55:22.215010 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:55:22.214990    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76fe11b0_8510_479e_848a_857159d5839b.slice/crio-ea0177e3a19ef98dcbfd0e3bf1bb4c9e430d576e2dfb1e0fd10251859054bc83 WatchSource:0}: Error finding container ea0177e3a19ef98dcbfd0e3bf1bb4c9e430d576e2dfb1e0fd10251859054bc83: Status 404 returned error can't find the container with id ea0177e3a19ef98dcbfd0e3bf1bb4c9e430d576e2dfb1e0fd10251859054bc83
Sep 17 18:55:22.264406 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:22.264385    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5"
Sep 17 18:55:22.548258 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:22.548227    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5"]
Sep 17 18:55:22.549480 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:55:22.549449    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod451e2ee6_d170_4b0a_b912_171c9005360a.slice/crio-3b64b77b25aaa6c73ab7b2af239fb1e477a7d1cadabec2b3ffeb2c8de0f6044c WatchSource:0}: Error finding container 3b64b77b25aaa6c73ab7b2af239fb1e477a7d1cadabec2b3ffeb2c8de0f6044c: Status 404 returned error can't find the container with id 3b64b77b25aaa6c73ab7b2af239fb1e477a7d1cadabec2b3ffeb2c8de0f6044c
Sep 17 18:55:22.558573 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:22.558544    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4" podUID="32a31b1a-35d4-433d-beb3-7d69b89eeb2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 18:55:23.028478 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:23.028442    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn" event={"ID":"76fe11b0-8510-479e-848a-857159d5839b","Type":"ContainerStarted","Data":"2147547899a6a8a9a7c0a09a3778c9a4c21ef4eafec374778079083ec7de3d1b"}
Sep 17 18:55:23.028962 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:23.028505    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn" event={"ID":"76fe11b0-8510-479e-848a-857159d5839b","Type":"ContainerStarted","Data":"ea0177e3a19ef98dcbfd0e3bf1bb4c9e430d576e2dfb1e0fd10251859054bc83"}
Sep 17 18:55:23.030639 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:23.030611    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5" event={"ID":"451e2ee6-d170-4b0a-b912-171c9005360a","Type":"ContainerStarted","Data":"ee0694b0c5a417809389ebbedd4ed5b3db3da81801412a1b9ed4b72d31894079"}
Sep 17 18:55:23.030780 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:23.030673    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5" event={"ID":"451e2ee6-d170-4b0a-b912-171c9005360a","Type":"ContainerStarted","Data":"3b64b77b25aaa6c73ab7b2af239fb1e477a7d1cadabec2b3ffeb2c8de0f6044c"}
Sep 17 18:55:25.047897 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:25.047860    2565 generic.go:356] "Generic (PLEG): container finished" podID="76fe11b0-8510-479e-848a-857159d5839b" containerID="2147547899a6a8a9a7c0a09a3778c9a4c21ef4eafec374778079083ec7de3d1b" exitCode=0
Sep 17 18:55:25.048408 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:25.047925    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn" event={"ID":"76fe11b0-8510-479e-848a-857159d5839b","Type":"ContainerDied","Data":"2147547899a6a8a9a7c0a09a3778c9a4c21ef4eafec374778079083ec7de3d1b"}
Sep 17 18:55:26.055643 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:26.055610    2565 generic.go:356] "Generic (PLEG): container finished" podID="451e2ee6-d170-4b0a-b912-171c9005360a" containerID="ee0694b0c5a417809389ebbedd4ed5b3db3da81801412a1b9ed4b72d31894079" exitCode=0
Sep 17 18:55:26.056110 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:26.055685    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5" event={"ID":"451e2ee6-d170-4b0a-b912-171c9005360a","Type":"ContainerDied","Data":"ee0694b0c5a417809389ebbedd4ed5b3db3da81801412a1b9ed4b72d31894079"}
Sep 17 18:55:26.057938 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:26.057918    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn" event={"ID":"76fe11b0-8510-479e-848a-857159d5839b","Type":"ContainerStarted","Data":"d10a8627253184a5206fcf404b5a66d7baf44b0d6cf5384113cf22768499cd46"}
Sep 17 18:55:26.103669 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:26.103613    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn" podStartSLOduration=2.569459306 podStartE2EDuration="5.103594283s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.534134977s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:55:26.101013844 +0000 UTC m=+1584.033215363" watchObservedRunningTime="2026-09-17 18:55:26.103594283 +0000 UTC m=+1584.035795803"
Sep 17 18:55:26.921396 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:26.921353    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:55:27.069389 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:27.069335    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5" event={"ID":"451e2ee6-d170-4b0a-b912-171c9005360a","Type":"ContainerStarted","Data":"26ca8e24c2a4a0c3f5620d87a3446331371fb4ca9b0a89e19869bb0de536edaa"}
Sep 17 18:55:27.069870 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:27.069764    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5"
Sep 17 18:55:27.089477 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:27.089407    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5" podStartSLOduration=3.548517064 podStartE2EDuration="6.089387643s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.540870579s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:55:27.088754864 +0000 UTC m=+1585.020956383" watchObservedRunningTime="2026-09-17 18:55:27.089387643 +0000 UTC m=+1585.021589163"
Sep 17 18:55:31.920661 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:31.920622    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:55:31.943218 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:31.943162    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:55:32.111443 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:32.111414    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:55:32.558637 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:32.558582    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4" podUID="32a31b1a-35d4-433d-beb3-7d69b89eeb2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 18:55:42.558423 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:42.558374    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4" podUID="32a31b1a-35d4-433d-beb3-7d69b89eeb2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 18:55:52.558424 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:52.558373    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4" podUID="32a31b1a-35d4-433d-beb3-7d69b89eeb2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 18:55:58.088163 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:55:58.088082    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5"
Sep 17 18:56:00.012663 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:00.012623    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-78966cd574-ks2g5_451e2ee6-d170-4b0a-b912-171c9005360a/main/0.log"
Sep 17 18:56:00.880674 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:00.880642    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"]
Sep 17 18:56:00.890349 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:56:00.890309    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:56:00.890349 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:56:00.890349 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:56:00.890349 ip-10-0-139-75 kubenswrapper[2565]:  > containerID="d10a8627253184a5206fcf404b5a66d7baf44b0d6cf5384113cf22768499cd46" cmd=["/bin/sleep","15"]
Sep 17 18:56:00.890651 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:00.890386    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn" message="" err=<
Sep 17 18:56:00.890651 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:56:00.890651 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:56:00.890651 ip-10-0-139-75 kubenswrapper[2565]:  >
Sep 17 18:56:00.890651 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:56:00.890409    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:56:00.890651 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:56:00.890651 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:56:00.890651 ip-10-0-139-75 kubenswrapper[2565]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn" podUID="76fe11b0-8510-479e-848a-857159d5839b" containerName="main" containerID="cri-o://d10a8627253184a5206fcf404b5a66d7baf44b0d6cf5384113cf22768499cd46"
Sep 17 18:56:00.890651 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:00.890469    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn" podUID="76fe11b0-8510-479e-848a-857159d5839b" containerName="main" containerID="cri-o://d10a8627253184a5206fcf404b5a66d7baf44b0d6cf5384113cf22768499cd46" gracePeriod=60
Sep 17 18:56:00.901185 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:00.901095    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5"]
Sep 17 18:56:00.901365 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:00.901311    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5" podUID="451e2ee6-d170-4b0a-b912-171c9005360a" containerName="main" containerID="cri-o://26ca8e24c2a4a0c3f5620d87a3446331371fb4ca9b0a89e19869bb0de536edaa" gracePeriod=30
Sep 17 18:56:01.100094 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.100018    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn" podUID="76fe11b0-8510-479e-848a-857159d5839b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Sep 17 18:56:01.256681 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.256649    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:56:01.271582 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.271549    2565 generic.go:356] "Generic (PLEG): container finished" podID="76fe11b0-8510-479e-848a-857159d5839b" containerID="d10a8627253184a5206fcf404b5a66d7baf44b0d6cf5384113cf22768499cd46" exitCode=0
Sep 17 18:56:01.271709 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.271597    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"
Sep 17 18:56:01.271709 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.271601    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn" event={"ID":"76fe11b0-8510-479e-848a-857159d5839b","Type":"ContainerDied","Data":"d10a8627253184a5206fcf404b5a66d7baf44b0d6cf5384113cf22768499cd46"}
Sep 17 18:56:01.271709 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.271671    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn" event={"ID":"76fe11b0-8510-479e-848a-857159d5839b","Type":"ContainerDied","Data":"ea0177e3a19ef98dcbfd0e3bf1bb4c9e430d576e2dfb1e0fd10251859054bc83"}
Sep 17 18:56:01.271709 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.271704    2565 scope.go:117] "RemoveContainer" containerID="d10a8627253184a5206fcf404b5a66d7baf44b0d6cf5384113cf22768499cd46"
Sep 17 18:56:01.281969 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.281948    2565 scope.go:117] "RemoveContainer" containerID="2147547899a6a8a9a7c0a09a3778c9a4c21ef4eafec374778079083ec7de3d1b"
Sep 17 18:56:01.293974 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.293950    2565 scope.go:117] "RemoveContainer" containerID="d10a8627253184a5206fcf404b5a66d7baf44b0d6cf5384113cf22768499cd46"
Sep 17 18:56:01.294300 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:56:01.294273    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d10a8627253184a5206fcf404b5a66d7baf44b0d6cf5384113cf22768499cd46\": container with ID starting with d10a8627253184a5206fcf404b5a66d7baf44b0d6cf5384113cf22768499cd46 not found: ID does not exist" containerID="d10a8627253184a5206fcf404b5a66d7baf44b0d6cf5384113cf22768499cd46"
Sep 17 18:56:01.294401 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.294313    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d10a8627253184a5206fcf404b5a66d7baf44b0d6cf5384113cf22768499cd46"} err="failed to get container status \"d10a8627253184a5206fcf404b5a66d7baf44b0d6cf5384113cf22768499cd46\": rpc error: code = NotFound desc = could not find container \"d10a8627253184a5206fcf404b5a66d7baf44b0d6cf5384113cf22768499cd46\": container with ID starting with d10a8627253184a5206fcf404b5a66d7baf44b0d6cf5384113cf22768499cd46 not found: ID does not exist"
Sep 17 18:56:01.294401 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.294337    2565 scope.go:117] "RemoveContainer" containerID="2147547899a6a8a9a7c0a09a3778c9a4c21ef4eafec374778079083ec7de3d1b"
Sep 17 18:56:01.294652 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:56:01.294629    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2147547899a6a8a9a7c0a09a3778c9a4c21ef4eafec374778079083ec7de3d1b\": container with ID starting with 2147547899a6a8a9a7c0a09a3778c9a4c21ef4eafec374778079083ec7de3d1b not found: ID does not exist" containerID="2147547899a6a8a9a7c0a09a3778c9a4c21ef4eafec374778079083ec7de3d1b"
Sep 17 18:56:01.294713 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.294661    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2147547899a6a8a9a7c0a09a3778c9a4c21ef4eafec374778079083ec7de3d1b"} err="failed to get container status \"2147547899a6a8a9a7c0a09a3778c9a4c21ef4eafec374778079083ec7de3d1b\": rpc error: code = NotFound desc = could not find container \"2147547899a6a8a9a7c0a09a3778c9a4c21ef4eafec374778079083ec7de3d1b\": container with ID starting with 2147547899a6a8a9a7c0a09a3778c9a4c21ef4eafec374778079083ec7de3d1b not found: ID does not exist"
Sep 17 18:56:01.361469 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.361442    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76fe11b0-8510-479e-848a-857159d5839b-model-cache\") pod \"76fe11b0-8510-479e-848a-857159d5839b\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") "
Sep 17 18:56:01.361619 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.361534    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76fe11b0-8510-479e-848a-857159d5839b-home\") pod \"76fe11b0-8510-479e-848a-857159d5839b\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") "
Sep 17 18:56:01.361619 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.361575    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5z56n\" (UniqueName: \"kubernetes.io/projected/76fe11b0-8510-479e-848a-857159d5839b-kube-api-access-5z56n\") pod \"76fe11b0-8510-479e-848a-857159d5839b\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") "
Sep 17 18:56:01.361619 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.361612    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76fe11b0-8510-479e-848a-857159d5839b-kserve-provision-location\") pod \"76fe11b0-8510-479e-848a-857159d5839b\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") "
Sep 17 18:56:01.361762 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.361640    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76fe11b0-8510-479e-848a-857159d5839b-tmp-dir\") pod \"76fe11b0-8510-479e-848a-857159d5839b\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") "
Sep 17 18:56:01.361762 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.361664    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76fe11b0-8510-479e-848a-857159d5839b-dshm\") pod \"76fe11b0-8510-479e-848a-857159d5839b\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") "
Sep 17 18:56:01.361762 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.361703    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76fe11b0-8510-479e-848a-857159d5839b-cabundle-cert\") pod \"76fe11b0-8510-479e-848a-857159d5839b\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") "
Sep 17 18:56:01.361762 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.361724    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76fe11b0-8510-479e-848a-857159d5839b-model-cache" (OuterVolumeSpecName: "model-cache") pod "76fe11b0-8510-479e-848a-857159d5839b" (UID: "76fe11b0-8510-479e-848a-857159d5839b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:01.361975 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.361764    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76fe11b0-8510-479e-848a-857159d5839b-tls-certs\") pod \"76fe11b0-8510-479e-848a-857159d5839b\" (UID: \"76fe11b0-8510-479e-848a-857159d5839b\") "
Sep 17 18:56:01.361975 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.361813    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76fe11b0-8510-479e-848a-857159d5839b-home" (OuterVolumeSpecName: "home") pod "76fe11b0-8510-479e-848a-857159d5839b" (UID: "76fe11b0-8510-479e-848a-857159d5839b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:01.361975 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.361945    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76fe11b0-8510-479e-848a-857159d5839b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "76fe11b0-8510-479e-848a-857159d5839b" (UID: "76fe11b0-8510-479e-848a-857159d5839b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:01.362131 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.362066    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76fe11b0-8510-479e-848a-857159d5839b-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:56:01.362131 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.362084    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76fe11b0-8510-479e-848a-857159d5839b-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:56:01.362131 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.362099    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76fe11b0-8510-479e-848a-857159d5839b-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:56:01.362317 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.362138    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76fe11b0-8510-479e-848a-857159d5839b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "76fe11b0-8510-479e-848a-857159d5839b" (UID: "76fe11b0-8510-479e-848a-857159d5839b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:56:01.364469 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.364439    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76fe11b0-8510-479e-848a-857159d5839b-kube-api-access-5z56n" (OuterVolumeSpecName: "kube-api-access-5z56n") pod "76fe11b0-8510-479e-848a-857159d5839b" (UID: "76fe11b0-8510-479e-848a-857159d5839b"). InnerVolumeSpecName "kube-api-access-5z56n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:56:01.364585 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.364534    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76fe11b0-8510-479e-848a-857159d5839b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76fe11b0-8510-479e-848a-857159d5839b" (UID: "76fe11b0-8510-479e-848a-857159d5839b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:56:01.364585 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.364539    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76fe11b0-8510-479e-848a-857159d5839b-dshm" (OuterVolumeSpecName: "dshm") pod "76fe11b0-8510-479e-848a-857159d5839b" (UID: "76fe11b0-8510-479e-848a-857159d5839b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:01.386430 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.386375    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76fe11b0-8510-479e-848a-857159d5839b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "76fe11b0-8510-479e-848a-857159d5839b" (UID: "76fe11b0-8510-479e-848a-857159d5839b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:01.463155 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.463119    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76fe11b0-8510-479e-848a-857159d5839b-kserve-provision-location\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:56:01.463155 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.463149    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76fe11b0-8510-479e-848a-857159d5839b-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:56:01.463377 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.463164    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76fe11b0-8510-479e-848a-857159d5839b-cabundle-cert\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:56:01.463377 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.463178    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76fe11b0-8510-479e-848a-857159d5839b-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:56:01.463377 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.463192    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z56n\" (UniqueName: \"kubernetes.io/projected/76fe11b0-8510-479e-848a-857159d5839b-kube-api-access-5z56n\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:56:01.501144 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.501120    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5"
Sep 17 18:56:01.564456 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.564429    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/451e2ee6-d170-4b0a-b912-171c9005360a-cabundle-cert\") pod \"451e2ee6-d170-4b0a-b912-171c9005360a\" (UID: \"451e2ee6-d170-4b0a-b912-171c9005360a\") "
Sep 17 18:56:01.564618 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.564468    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/451e2ee6-d170-4b0a-b912-171c9005360a-tmp-dir\") pod \"451e2ee6-d170-4b0a-b912-171c9005360a\" (UID: \"451e2ee6-d170-4b0a-b912-171c9005360a\") "
Sep 17 18:56:01.564618 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.564496    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/451e2ee6-d170-4b0a-b912-171c9005360a-kserve-provision-location\") pod \"451e2ee6-d170-4b0a-b912-171c9005360a\" (UID: \"451e2ee6-d170-4b0a-b912-171c9005360a\") "
Sep 17 18:56:01.564618 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.564567    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/451e2ee6-d170-4b0a-b912-171c9005360a-tls-certs\") pod \"451e2ee6-d170-4b0a-b912-171c9005360a\" (UID: \"451e2ee6-d170-4b0a-b912-171c9005360a\") "
Sep 17 18:56:01.564823 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.564789    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451e2ee6-d170-4b0a-b912-171c9005360a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "451e2ee6-d170-4b0a-b912-171c9005360a" (UID: "451e2ee6-d170-4b0a-b912-171c9005360a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:01.564886 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.564830    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/451e2ee6-d170-4b0a-b912-171c9005360a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "451e2ee6-d170-4b0a-b912-171c9005360a" (UID: "451e2ee6-d170-4b0a-b912-171c9005360a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:56:01.566773 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.566743    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/451e2ee6-d170-4b0a-b912-171c9005360a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "451e2ee6-d170-4b0a-b912-171c9005360a" (UID: "451e2ee6-d170-4b0a-b912-171c9005360a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:56:01.586429 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.586399    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451e2ee6-d170-4b0a-b912-171c9005360a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "451e2ee6-d170-4b0a-b912-171c9005360a" (UID: "451e2ee6-d170-4b0a-b912-171c9005360a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:01.599233 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.599194    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"]
Sep 17 18:56:01.604079 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.604051    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-77c5c5bf8d-k2pdn"]
Sep 17 18:56:01.666283 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.666255    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/451e2ee6-d170-4b0a-b912-171c9005360a-cabundle-cert\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:56:01.666405 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.666288    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/451e2ee6-d170-4b0a-b912-171c9005360a-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:56:01.666405 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.666304    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/451e2ee6-d170-4b0a-b912-171c9005360a-kserve-provision-location\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:56:01.666405 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:01.666319    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/451e2ee6-d170-4b0a-b912-171c9005360a-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:56:02.282273 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:02.282174    2565 generic.go:356] "Generic (PLEG): container finished" podID="451e2ee6-d170-4b0a-b912-171c9005360a" containerID="26ca8e24c2a4a0c3f5620d87a3446331371fb4ca9b0a89e19869bb0de536edaa" exitCode=0
Sep 17 18:56:02.282273 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:02.282231    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5" event={"ID":"451e2ee6-d170-4b0a-b912-171c9005360a","Type":"ContainerDied","Data":"26ca8e24c2a4a0c3f5620d87a3446331371fb4ca9b0a89e19869bb0de536edaa"}
Sep 17 18:56:02.282273 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:02.282256    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5"
Sep 17 18:56:02.282851 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:02.282301    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5" event={"ID":"451e2ee6-d170-4b0a-b912-171c9005360a","Type":"ContainerDied","Data":"3b64b77b25aaa6c73ab7b2af239fb1e477a7d1cadabec2b3ffeb2c8de0f6044c"}
Sep 17 18:56:02.282851 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:02.282331    2565 scope.go:117] "RemoveContainer" containerID="26ca8e24c2a4a0c3f5620d87a3446331371fb4ca9b0a89e19869bb0de536edaa"
Sep 17 18:56:02.300219 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:02.300175    2565 scope.go:117] "RemoveContainer" containerID="ee0694b0c5a417809389ebbedd4ed5b3db3da81801412a1b9ed4b72d31894079"
Sep 17 18:56:02.308996 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:02.308974    2565 scope.go:117] "RemoveContainer" containerID="26ca8e24c2a4a0c3f5620d87a3446331371fb4ca9b0a89e19869bb0de536edaa"
Sep 17 18:56:02.309280 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:56:02.309258    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26ca8e24c2a4a0c3f5620d87a3446331371fb4ca9b0a89e19869bb0de536edaa\": container with ID starting with 26ca8e24c2a4a0c3f5620d87a3446331371fb4ca9b0a89e19869bb0de536edaa not found: ID does not exist" containerID="26ca8e24c2a4a0c3f5620d87a3446331371fb4ca9b0a89e19869bb0de536edaa"
Sep 17 18:56:02.309359 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:02.309287    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26ca8e24c2a4a0c3f5620d87a3446331371fb4ca9b0a89e19869bb0de536edaa"} err="failed to get container status \"26ca8e24c2a4a0c3f5620d87a3446331371fb4ca9b0a89e19869bb0de536edaa\": rpc error: code = NotFound desc = could not find container \"26ca8e24c2a4a0c3f5620d87a3446331371fb4ca9b0a89e19869bb0de536edaa\": container with ID starting with 26ca8e24c2a4a0c3f5620d87a3446331371fb4ca9b0a89e19869bb0de536edaa not found: ID does not exist"
Sep 17 18:56:02.309359 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:02.309303    2565 scope.go:117] "RemoveContainer" containerID="ee0694b0c5a417809389ebbedd4ed5b3db3da81801412a1b9ed4b72d31894079"
Sep 17 18:56:02.309546 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:56:02.309527    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee0694b0c5a417809389ebbedd4ed5b3db3da81801412a1b9ed4b72d31894079\": container with ID starting with ee0694b0c5a417809389ebbedd4ed5b3db3da81801412a1b9ed4b72d31894079 not found: ID does not exist" containerID="ee0694b0c5a417809389ebbedd4ed5b3db3da81801412a1b9ed4b72d31894079"
Sep 17 18:56:02.309589 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:02.309552    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee0694b0c5a417809389ebbedd4ed5b3db3da81801412a1b9ed4b72d31894079"} err="failed to get container status \"ee0694b0c5a417809389ebbedd4ed5b3db3da81801412a1b9ed4b72d31894079\": rpc error: code = NotFound desc = could not find container \"ee0694b0c5a417809389ebbedd4ed5b3db3da81801412a1b9ed4b72d31894079\": container with ID starting with ee0694b0c5a417809389ebbedd4ed5b3db3da81801412a1b9ed4b72d31894079 not found: ID does not exist"
Sep 17 18:56:02.314006 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:02.313979    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5"]
Sep 17 18:56:02.320730 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:02.320706    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-78966cd574-ks2g5"]
Sep 17 18:56:02.558594 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:02.558502    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4" podUID="32a31b1a-35d4-433d-beb3-7d69b89eeb2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 18:56:02.608393 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:02.608361    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="451e2ee6-d170-4b0a-b912-171c9005360a" path="/var/lib/kubelet/pods/451e2ee6-d170-4b0a-b912-171c9005360a/volumes"
Sep 17 18:56:02.608886 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:02.608870    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76fe11b0-8510-479e-848a-857159d5839b" path="/var/lib/kubelet/pods/76fe11b0-8510-479e-848a-857159d5839b/volumes"
Sep 17 18:56:12.558988 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:12.558944    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4" podUID="32a31b1a-35d4-433d-beb3-7d69b89eeb2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 18:56:22.558549 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:22.558508    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4" podUID="32a31b1a-35d4-433d-beb3-7d69b89eeb2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 18:56:32.559117 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:32.559067    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4" podUID="32a31b1a-35d4-433d-beb3-7d69b89eeb2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 18:56:42.558481 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:42.558439    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4" podUID="32a31b1a-35d4-433d-beb3-7d69b89eeb2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 18:56:52.558689 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:56:52.558647    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4" podUID="32a31b1a-35d4-433d-beb3-7d69b89eeb2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 18:57:01.034740 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.034703    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"]
Sep 17 18:57:01.035142 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.035060    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="451e2ee6-d170-4b0a-b912-171c9005360a" containerName="storage-initializer"
Sep 17 18:57:01.035142 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.035071    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="451e2ee6-d170-4b0a-b912-171c9005360a" containerName="storage-initializer"
Sep 17 18:57:01.035142 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.035080    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="451e2ee6-d170-4b0a-b912-171c9005360a" containerName="main"
Sep 17 18:57:01.035142 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.035086    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="451e2ee6-d170-4b0a-b912-171c9005360a" containerName="main"
Sep 17 18:57:01.035142 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.035097    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76fe11b0-8510-479e-848a-857159d5839b" containerName="storage-initializer"
Sep 17 18:57:01.035142 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.035102    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="76fe11b0-8510-479e-848a-857159d5839b" containerName="storage-initializer"
Sep 17 18:57:01.035142 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.035112    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76fe11b0-8510-479e-848a-857159d5839b" containerName="main"
Sep 17 18:57:01.035142 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.035117    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="76fe11b0-8510-479e-848a-857159d5839b" containerName="main"
Sep 17 18:57:01.035446 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.035172    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="451e2ee6-d170-4b0a-b912-171c9005360a" containerName="main"
Sep 17 18:57:01.035446 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.035181    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="76fe11b0-8510-479e-848a-857159d5839b" containerName="main"
Sep 17 18:57:01.041526 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.041508    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"
Sep 17 18:57:01.051029 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.050981    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 17 18:57:01.051029 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.051006    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 17 18:57:01.051239 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.051006    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-zww98\""
Sep 17 18:57:01.051239 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.050981    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 17 18:57:01.062393 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.062371    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"]
Sep 17 18:57:01.173614 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.173586    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf\" (UID: \"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"
Sep 17 18:57:01.173730 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.173618    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf\" (UID: \"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"
Sep 17 18:57:01.173730 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.173677    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wdcp\" (UniqueName: \"kubernetes.io/projected/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-kube-api-access-5wdcp\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf\" (UID: \"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"
Sep 17 18:57:01.173823 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.173783    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf\" (UID: \"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"
Sep 17 18:57:01.173877 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.173836    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf\" (UID: \"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"
Sep 17 18:57:01.173877 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.173855    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf\" (UID: \"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"
Sep 17 18:57:01.274881 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.274856    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf\" (UID: \"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"
Sep 17 18:57:01.275115 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.274886    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf\" (UID: \"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"
Sep 17 18:57:01.275115 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.274956    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wdcp\" (UniqueName: \"kubernetes.io/projected/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-kube-api-access-5wdcp\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf\" (UID: \"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"
Sep 17 18:57:01.275115 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.274999    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf\" (UID: \"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"
Sep 17 18:57:01.275115 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.275052    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf\" (UID: \"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"
Sep 17 18:57:01.275115 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.275081    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf\" (UID: \"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"
Sep 17 18:57:01.275592 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.275521    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf\" (UID: \"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"
Sep 17 18:57:01.276078 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.276058    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf\" (UID: \"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"
Sep 17 18:57:01.276233 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.276188    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf\" (UID: \"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"
Sep 17 18:57:01.278358 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.278334    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf\" (UID: \"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"
Sep 17 18:57:01.278857 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.278837    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf\" (UID: \"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"
Sep 17 18:57:01.293193 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.293133    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wdcp\" (UniqueName: \"kubernetes.io/projected/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-kube-api-access-5wdcp\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf\" (UID: \"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"
Sep 17 18:57:01.351985 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.351961    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"
Sep 17 18:57:01.626143 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:01.626119    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"]
Sep 17 18:57:01.627368 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:57:01.627337    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77dbf9e6_be2a_4eff_bd8d_8e5a31922ee6.slice/crio-e0db9b070927c19aa1dc5b2ff73fc256ec9f655132d719ae41ffd919a0edd641 WatchSource:0}: Error finding container e0db9b070927c19aa1dc5b2ff73fc256ec9f655132d719ae41ffd919a0edd641: Status 404 returned error can't find the container with id e0db9b070927c19aa1dc5b2ff73fc256ec9f655132d719ae41ffd919a0edd641
Sep 17 18:57:02.569132 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:02.569097    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:57:02.577062 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:02.577033    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:57:02.617228 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:02.617176    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf" event={"ID":"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6","Type":"ContainerStarted","Data":"f6b532572d5cbbbf35b313f3d1545088f640d47c49673cfbddf4baa75ee068e1"}
Sep 17 18:57:02.617390 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:02.617242    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf" event={"ID":"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6","Type":"ContainerStarted","Data":"e0db9b070927c19aa1dc5b2ff73fc256ec9f655132d719ae41ffd919a0edd641"}
Sep 17 18:57:02.660830 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:02.660774    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf" podStartSLOduration=1.660759905 podStartE2EDuration="1.660759905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:57:02.65685057 +0000 UTC m=+1680.589052089" watchObservedRunningTime="2026-09-17 18:57:02.660759905 +0000 UTC m=+1680.592961423"
Sep 17 18:57:03.352771 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:03.352740    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"
Sep 17 18:57:11.352967 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:11.352933    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"
Sep 17 18:57:11.365012 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:11.364989    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"
Sep 17 18:57:11.674061 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:11.674033    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"
Sep 17 18:57:22.025882 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:22.025847    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"]
Sep 17 18:57:35.076014 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.075922    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"]
Sep 17 18:57:35.085278 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:57:35.085244    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:57:35.085278 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:57:35.085278 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:57:35.085278 ip-10-0-139-75 kubenswrapper[2565]:  > containerID="f6b532572d5cbbbf35b313f3d1545088f640d47c49673cfbddf4baa75ee068e1" cmd=["/bin/sleep","15"]
Sep 17 18:57:35.085555 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.085300    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf" message="" err=<
Sep 17 18:57:35.085555 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:57:35.085555 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:57:35.085555 ip-10-0-139-75 kubenswrapper[2565]:  >
Sep 17 18:57:35.085555 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:57:35.085317    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:57:35.085555 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:57:35.085555 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 18:57:35.085555 ip-10-0-139-75 kubenswrapper[2565]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf" podUID="77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6" containerName="main" containerID="cri-o://f6b532572d5cbbbf35b313f3d1545088f640d47c49673cfbddf4baa75ee068e1"
Sep 17 18:57:35.085555 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.085372    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf" podUID="77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6" containerName="main" containerID="cri-o://f6b532572d5cbbbf35b313f3d1545088f640d47c49673cfbddf4baa75ee068e1" gracePeriod=60
Sep 17 18:57:35.438659 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.438628    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"
Sep 17 18:57:35.534501 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.534470    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wdcp\" (UniqueName: \"kubernetes.io/projected/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-kube-api-access-5wdcp\") pod \"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6\" (UID: \"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6\") "
Sep 17 18:57:35.534635 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.534545    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-model-cache\") pod \"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6\" (UID: \"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6\") "
Sep 17 18:57:35.534635 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.534570    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-tls-certs\") pod \"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6\" (UID: \"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6\") "
Sep 17 18:57:35.534635 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.534594    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-home\") pod \"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6\" (UID: \"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6\") "
Sep 17 18:57:35.534635 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.534627    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-dshm\") pod \"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6\" (UID: \"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6\") "
Sep 17 18:57:35.534863 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.534658    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-tmp-dir\") pod \"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6\" (UID: \"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6\") "
Sep 17 18:57:35.534863 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.534824    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-model-cache" (OuterVolumeSpecName: "model-cache") pod "77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6" (UID: "77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:57:35.534978 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.534937    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:57:35.534978 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.534940    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-home" (OuterVolumeSpecName: "home") pod "77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6" (UID: "77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:57:35.534978 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.534972    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6" (UID: "77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:57:35.536751 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.536716    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-dshm" (OuterVolumeSpecName: "dshm") pod "77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6" (UID: "77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:57:35.536861 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.536752    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6" (UID: "77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:57:35.536861 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.536804    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-kube-api-access-5wdcp" (OuterVolumeSpecName: "kube-api-access-5wdcp") pod "77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6" (UID: "77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6"). InnerVolumeSpecName "kube-api-access-5wdcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:57:35.636128 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.636057    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:57:35.636128 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.636083    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wdcp\" (UniqueName: \"kubernetes.io/projected/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-kube-api-access-5wdcp\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:57:35.636128 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.636098    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:57:35.636128 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.636107    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:57:35.636128 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.636116    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:57:35.797096 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.797067    2565 generic.go:356] "Generic (PLEG): container finished" podID="77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6" containerID="f6b532572d5cbbbf35b313f3d1545088f640d47c49673cfbddf4baa75ee068e1" exitCode=0
Sep 17 18:57:35.797244 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.797102    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf" event={"ID":"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6","Type":"ContainerDied","Data":"f6b532572d5cbbbf35b313f3d1545088f640d47c49673cfbddf4baa75ee068e1"}
Sep 17 18:57:35.797244 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.797120    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"
Sep 17 18:57:35.797244 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.797165    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf" event={"ID":"77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6","Type":"ContainerDied","Data":"e0db9b070927c19aa1dc5b2ff73fc256ec9f655132d719ae41ffd919a0edd641"}
Sep 17 18:57:35.797244 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.797187    2565 scope.go:117] "RemoveContainer" containerID="f6b532572d5cbbbf35b313f3d1545088f640d47c49673cfbddf4baa75ee068e1"
Sep 17 18:57:35.806986 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.806969    2565 scope.go:117] "RemoveContainer" containerID="f6b532572d5cbbbf35b313f3d1545088f640d47c49673cfbddf4baa75ee068e1"
Sep 17 18:57:35.807250 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:57:35.807230    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6b532572d5cbbbf35b313f3d1545088f640d47c49673cfbddf4baa75ee068e1\": container with ID starting with f6b532572d5cbbbf35b313f3d1545088f640d47c49673cfbddf4baa75ee068e1 not found: ID does not exist" containerID="f6b532572d5cbbbf35b313f3d1545088f640d47c49673cfbddf4baa75ee068e1"
Sep 17 18:57:35.807299 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.807257    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6b532572d5cbbbf35b313f3d1545088f640d47c49673cfbddf4baa75ee068e1"} err="failed to get container status \"f6b532572d5cbbbf35b313f3d1545088f640d47c49673cfbddf4baa75ee068e1\": rpc error: code = NotFound desc = could not find container \"f6b532572d5cbbbf35b313f3d1545088f640d47c49673cfbddf4baa75ee068e1\": container with ID starting with f6b532572d5cbbbf35b313f3d1545088f640d47c49673cfbddf4baa75ee068e1 not found: ID does not exist"
Sep 17 18:57:35.819274 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.819232    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"]
Sep 17 18:57:35.821389 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:35.821368    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-pzwwf"]
Sep 17 18:57:36.606811 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:36.606782    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6" path="/var/lib/kubelet/pods/77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6/volumes"
Sep 17 18:57:37.048265 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.048227    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4" podUID="32a31b1a-35d4-433d-beb3-7d69b89eeb2c" containerName="main" containerID="cri-o://3c2327433f2b493f7303ac49e6e92f12288973b7f82f9b33d16566f8eb8bc0d4" gracePeriod=45
Sep 17 18:57:37.570637 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.570590    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4" podUID="32a31b1a-35d4-433d-beb3-7d69b89eeb2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Sep 17 18:57:37.803253 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.803229    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:57:37.809812 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.809790    2565 generic.go:356] "Generic (PLEG): container finished" podID="32a31b1a-35d4-433d-beb3-7d69b89eeb2c" containerID="3c2327433f2b493f7303ac49e6e92f12288973b7f82f9b33d16566f8eb8bc0d4" exitCode=0
Sep 17 18:57:37.809922 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.809834    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"
Sep 17 18:57:37.809922 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.809870    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4" event={"ID":"32a31b1a-35d4-433d-beb3-7d69b89eeb2c","Type":"ContainerDied","Data":"3c2327433f2b493f7303ac49e6e92f12288973b7f82f9b33d16566f8eb8bc0d4"}
Sep 17 18:57:37.810049 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.809930    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4" event={"ID":"32a31b1a-35d4-433d-beb3-7d69b89eeb2c","Type":"ContainerDied","Data":"a8cca146839b78b682cf870e8da19f5026d5d78cc0cb9ef2a7401b4cfac7b459"}
Sep 17 18:57:37.810049 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.809965    2565 scope.go:117] "RemoveContainer" containerID="3c2327433f2b493f7303ac49e6e92f12288973b7f82f9b33d16566f8eb8bc0d4"
Sep 17 18:57:37.818481 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.818466    2565 scope.go:117] "RemoveContainer" containerID="cd56c7f30df73b2cd19332b0621f0e48c07da76db4d5fbe30c3a38f39bfde9c6"
Sep 17 18:57:37.830848 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.830829    2565 scope.go:117] "RemoveContainer" containerID="3c2327433f2b493f7303ac49e6e92f12288973b7f82f9b33d16566f8eb8bc0d4"
Sep 17 18:57:37.831121 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:57:37.831102    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c2327433f2b493f7303ac49e6e92f12288973b7f82f9b33d16566f8eb8bc0d4\": container with ID starting with 3c2327433f2b493f7303ac49e6e92f12288973b7f82f9b33d16566f8eb8bc0d4 not found: ID does not exist" containerID="3c2327433f2b493f7303ac49e6e92f12288973b7f82f9b33d16566f8eb8bc0d4"
Sep 17 18:57:37.831223 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.831125    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c2327433f2b493f7303ac49e6e92f12288973b7f82f9b33d16566f8eb8bc0d4"} err="failed to get container status \"3c2327433f2b493f7303ac49e6e92f12288973b7f82f9b33d16566f8eb8bc0d4\": rpc error: code = NotFound desc = could not find container \"3c2327433f2b493f7303ac49e6e92f12288973b7f82f9b33d16566f8eb8bc0d4\": container with ID starting with 3c2327433f2b493f7303ac49e6e92f12288973b7f82f9b33d16566f8eb8bc0d4 not found: ID does not exist"
Sep 17 18:57:37.831223 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.831142    2565 scope.go:117] "RemoveContainer" containerID="cd56c7f30df73b2cd19332b0621f0e48c07da76db4d5fbe30c3a38f39bfde9c6"
Sep 17 18:57:37.831431 ip-10-0-139-75 kubenswrapper[2565]: E0917 18:57:37.831405    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd56c7f30df73b2cd19332b0621f0e48c07da76db4d5fbe30c3a38f39bfde9c6\": container with ID starting with cd56c7f30df73b2cd19332b0621f0e48c07da76db4d5fbe30c3a38f39bfde9c6 not found: ID does not exist" containerID="cd56c7f30df73b2cd19332b0621f0e48c07da76db4d5fbe30c3a38f39bfde9c6"
Sep 17 18:57:37.831487 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.831440    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd56c7f30df73b2cd19332b0621f0e48c07da76db4d5fbe30c3a38f39bfde9c6"} err="failed to get container status \"cd56c7f30df73b2cd19332b0621f0e48c07da76db4d5fbe30c3a38f39bfde9c6\": rpc error: code = NotFound desc = could not find container \"cd56c7f30df73b2cd19332b0621f0e48c07da76db4d5fbe30c3a38f39bfde9c6\": container with ID starting with cd56c7f30df73b2cd19332b0621f0e48c07da76db4d5fbe30c3a38f39bfde9c6 not found: ID does not exist"
Sep 17 18:57:37.854009 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.853982    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-tls-certs\") pod \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") "
Sep 17 18:57:37.854118 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.854026    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-model-cache\") pod \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") "
Sep 17 18:57:37.854118 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.854055    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-dshm\") pod \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") "
Sep 17 18:57:37.854118 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.854087    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-kserve-provision-location\") pod \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") "
Sep 17 18:57:37.854370 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.854149    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-tmp-dir\") pod \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") "
Sep 17 18:57:37.854370 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.854177    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-home\") pod \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") "
Sep 17 18:57:37.854370 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.854282    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-cabundle-cert\") pod \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") "
Sep 17 18:57:37.854370 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.854342    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmv9x\" (UniqueName: \"kubernetes.io/projected/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-kube-api-access-pmv9x\") pod \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\" (UID: \"32a31b1a-35d4-433d-beb3-7d69b89eeb2c\") "
Sep 17 18:57:37.854933 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.854856    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-home" (OuterVolumeSpecName: "home") pod "32a31b1a-35d4-433d-beb3-7d69b89eeb2c" (UID: "32a31b1a-35d4-433d-beb3-7d69b89eeb2c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:57:37.855317 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.854294    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-model-cache" (OuterVolumeSpecName: "model-cache") pod "32a31b1a-35d4-433d-beb3-7d69b89eeb2c" (UID: "32a31b1a-35d4-433d-beb3-7d69b89eeb2c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:57:37.855620 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.855590    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "32a31b1a-35d4-433d-beb3-7d69b89eeb2c" (UID: "32a31b1a-35d4-433d-beb3-7d69b89eeb2c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:57:37.856845 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.856805    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32a31b1a-35d4-433d-beb3-7d69b89eeb2c" (UID: "32a31b1a-35d4-433d-beb3-7d69b89eeb2c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:57:37.857176 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.857151    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-dshm" (OuterVolumeSpecName: "dshm") pod "32a31b1a-35d4-433d-beb3-7d69b89eeb2c" (UID: "32a31b1a-35d4-433d-beb3-7d69b89eeb2c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:57:37.857372 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.857155    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-kube-api-access-pmv9x" (OuterVolumeSpecName: "kube-api-access-pmv9x") pod "32a31b1a-35d4-433d-beb3-7d69b89eeb2c" (UID: "32a31b1a-35d4-433d-beb3-7d69b89eeb2c"). InnerVolumeSpecName "kube-api-access-pmv9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:57:37.866643 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.866615    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "32a31b1a-35d4-433d-beb3-7d69b89eeb2c" (UID: "32a31b1a-35d4-433d-beb3-7d69b89eeb2c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:57:37.872574 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.872553    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "32a31b1a-35d4-433d-beb3-7d69b89eeb2c" (UID: "32a31b1a-35d4-433d-beb3-7d69b89eeb2c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:57:37.955825 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.955801    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-cabundle-cert\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:57:37.955825 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.955824    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmv9x\" (UniqueName: \"kubernetes.io/projected/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-kube-api-access-pmv9x\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:57:37.955948 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.955835    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:57:37.955948 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.955849    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:57:37.955948 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.955859    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:57:37.955948 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.955867    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-kserve-provision-location\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:57:37.955948 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.955875    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:57:37.955948 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:37.955882    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32a31b1a-35d4-433d-beb3-7d69b89eeb2c-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 18:57:38.139080 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:38.139055    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"]
Sep 17 18:57:38.142376 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:38.142352    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5c9cfb7998-c86b4"]
Sep 17 18:57:38.607385 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:57:38.607354    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32a31b1a-35d4-433d-beb3-7d69b89eeb2c" path="/var/lib/kubelet/pods/32a31b1a-35d4-433d-beb3-7d69b89eeb2c/volumes"
Sep 17 18:58:15.812022 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:15.811986    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"]
Sep 17 18:58:15.812590 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:15.812569    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6" containerName="main"
Sep 17 18:58:15.812677 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:15.812594    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6" containerName="main"
Sep 17 18:58:15.812677 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:15.812608    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32a31b1a-35d4-433d-beb3-7d69b89eeb2c" containerName="main"
Sep 17 18:58:15.812677 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:15.812615    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="32a31b1a-35d4-433d-beb3-7d69b89eeb2c" containerName="main"
Sep 17 18:58:15.812677 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:15.812633    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32a31b1a-35d4-433d-beb3-7d69b89eeb2c" containerName="storage-initializer"
Sep 17 18:58:15.812677 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:15.812642    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="32a31b1a-35d4-433d-beb3-7d69b89eeb2c" containerName="storage-initializer"
Sep 17 18:58:15.812921 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:15.812741    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="32a31b1a-35d4-433d-beb3-7d69b89eeb2c" containerName="main"
Sep 17 18:58:15.812921 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:15.812754    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="77dbf9e6-be2a-4eff-bd8d-8e5a31922ee6" containerName="main"
Sep 17 18:58:15.815822 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:15.815801    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"
Sep 17 18:58:15.819236 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:15.819193    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 17 18:58:15.819339 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:15.819273    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-sqd7s\""
Sep 17 18:58:15.819339 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:15.819290    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 17 18:58:15.819540 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:15.819517    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Sep 17 18:58:15.826910 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:15.826884    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"]
Sep 17 18:58:15.929483 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:15.929452    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22e2f024-e1d4-4e59-b631-a94518b5cac6-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t\" (UID: \"22e2f024-e1d4-4e59-b631-a94518b5cac6\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"
Sep 17 18:58:15.929614 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:15.929494    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22e2f024-e1d4-4e59-b631-a94518b5cac6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t\" (UID: \"22e2f024-e1d4-4e59-b631-a94518b5cac6\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"
Sep 17 18:58:15.929614 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:15.929560    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjvzv\" (UniqueName: \"kubernetes.io/projected/22e2f024-e1d4-4e59-b631-a94518b5cac6-kube-api-access-pjvzv\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t\" (UID: \"22e2f024-e1d4-4e59-b631-a94518b5cac6\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"
Sep 17 18:58:15.929705 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:15.929616    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22e2f024-e1d4-4e59-b631-a94518b5cac6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t\" (UID: \"22e2f024-e1d4-4e59-b631-a94518b5cac6\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"
Sep 17 18:58:15.929705 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:15.929639    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22e2f024-e1d4-4e59-b631-a94518b5cac6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t\" (UID: \"22e2f024-e1d4-4e59-b631-a94518b5cac6\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"
Sep 17 18:58:15.929705 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:15.929667    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22e2f024-e1d4-4e59-b631-a94518b5cac6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t\" (UID: \"22e2f024-e1d4-4e59-b631-a94518b5cac6\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"
Sep 17 18:58:16.030253 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:16.030222    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22e2f024-e1d4-4e59-b631-a94518b5cac6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t\" (UID: \"22e2f024-e1d4-4e59-b631-a94518b5cac6\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"
Sep 17 18:58:16.030253 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:16.030254    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22e2f024-e1d4-4e59-b631-a94518b5cac6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t\" (UID: \"22e2f024-e1d4-4e59-b631-a94518b5cac6\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"
Sep 17 18:58:16.030453 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:16.030273    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22e2f024-e1d4-4e59-b631-a94518b5cac6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t\" (UID: \"22e2f024-e1d4-4e59-b631-a94518b5cac6\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"
Sep 17 18:58:16.030453 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:16.030325    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22e2f024-e1d4-4e59-b631-a94518b5cac6-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t\" (UID: \"22e2f024-e1d4-4e59-b631-a94518b5cac6\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"
Sep 17 18:58:16.030453 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:16.030355    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22e2f024-e1d4-4e59-b631-a94518b5cac6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t\" (UID: \"22e2f024-e1d4-4e59-b631-a94518b5cac6\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"
Sep 17 18:58:16.030608 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:16.030505    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjvzv\" (UniqueName: \"kubernetes.io/projected/22e2f024-e1d4-4e59-b631-a94518b5cac6-kube-api-access-pjvzv\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t\" (UID: \"22e2f024-e1d4-4e59-b631-a94518b5cac6\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"
Sep 17 18:58:16.030687 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:16.030662    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22e2f024-e1d4-4e59-b631-a94518b5cac6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t\" (UID: \"22e2f024-e1d4-4e59-b631-a94518b5cac6\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"
Sep 17 18:58:16.030752 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:16.030735    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22e2f024-e1d4-4e59-b631-a94518b5cac6-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t\" (UID: \"22e2f024-e1d4-4e59-b631-a94518b5cac6\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"
Sep 17 18:58:16.030827 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:16.030805    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22e2f024-e1d4-4e59-b631-a94518b5cac6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t\" (UID: \"22e2f024-e1d4-4e59-b631-a94518b5cac6\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"
Sep 17 18:58:16.032620 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:16.032601    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22e2f024-e1d4-4e59-b631-a94518b5cac6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t\" (UID: \"22e2f024-e1d4-4e59-b631-a94518b5cac6\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"
Sep 17 18:58:16.032779 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:16.032763    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22e2f024-e1d4-4e59-b631-a94518b5cac6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t\" (UID: \"22e2f024-e1d4-4e59-b631-a94518b5cac6\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"
Sep 17 18:58:16.038168 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:16.038138    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjvzv\" (UniqueName: \"kubernetes.io/projected/22e2f024-e1d4-4e59-b631-a94518b5cac6-kube-api-access-pjvzv\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t\" (UID: \"22e2f024-e1d4-4e59-b631-a94518b5cac6\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"
Sep 17 18:58:16.129278 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:16.129222    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"
Sep 17 18:58:16.417472 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:16.417441    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"]
Sep 17 18:58:16.418088 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:58:16.418061    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e2f024_e1d4_4e59_b631_a94518b5cac6.slice/crio-7c56b6c6cff59c353dc0190758487e3dea04531a8a53518659a009fe2fd915e9 WatchSource:0}: Error finding container 7c56b6c6cff59c353dc0190758487e3dea04531a8a53518659a009fe2fd915e9: Status 404 returned error can't find the container with id 7c56b6c6cff59c353dc0190758487e3dea04531a8a53518659a009fe2fd915e9
Sep 17 18:58:17.020771 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:17.020741    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t" event={"ID":"22e2f024-e1d4-4e59-b631-a94518b5cac6","Type":"ContainerStarted","Data":"35fcf134fc69b4fb4d4cf4d57660351809748978b1ccb2e710bd29a0c964e469"}
Sep 17 18:58:17.021126 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:17.020794    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t" event={"ID":"22e2f024-e1d4-4e59-b631-a94518b5cac6","Type":"ContainerStarted","Data":"7c56b6c6cff59c353dc0190758487e3dea04531a8a53518659a009fe2fd915e9"}
Sep 17 18:58:17.049014 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:17.048957    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t" podStartSLOduration=2.048941358 podStartE2EDuration="2.048941358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:58:17.04730666 +0000 UTC m=+1754.979508193" watchObservedRunningTime="2026-09-17 18:58:17.048941358 +0000 UTC m=+1754.981142876"
Sep 17 18:58:17.130190 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:17.130166    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"
Sep 17 18:58:26.129942 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:26.129909    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"
Sep 17 18:58:26.142351 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:26.142327    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"
Sep 17 18:58:27.081700 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:58:27.081670    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"
Sep 17 18:59:43.531447 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:43.531413    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"]
Sep 17 18:59:43.534790 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:43.534769    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"
Sep 17 18:59:43.537603 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:43.537562    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 17 18:59:43.538408 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:43.538388    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-wlg4n\""
Sep 17 18:59:43.538485 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:43.538388    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 17 18:59:43.539637 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:43.539622    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 17 18:59:43.554429 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:43.554405    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"]
Sep 17 18:59:43.649848 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:43.649819    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6\" (UID: \"10c73b9e-b713-4a4b-bac9-0ef3290efd9e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"
Sep 17 18:59:43.649966 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:43.649860    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-927b5\" (UniqueName: \"kubernetes.io/projected/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-kube-api-access-927b5\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6\" (UID: \"10c73b9e-b713-4a4b-bac9-0ef3290efd9e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"
Sep 17 18:59:43.649966 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:43.649917    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6\" (UID: \"10c73b9e-b713-4a4b-bac9-0ef3290efd9e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"
Sep 17 18:59:43.650064 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:43.649977    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6\" (UID: \"10c73b9e-b713-4a4b-bac9-0ef3290efd9e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"
Sep 17 18:59:43.650064 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:43.649999    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6\" (UID: \"10c73b9e-b713-4a4b-bac9-0ef3290efd9e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"
Sep 17 18:59:43.650064 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:43.650013    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6\" (UID: \"10c73b9e-b713-4a4b-bac9-0ef3290efd9e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"
Sep 17 18:59:43.750567 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:43.750541    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6\" (UID: \"10c73b9e-b713-4a4b-bac9-0ef3290efd9e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"
Sep 17 18:59:43.750697 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:43.750587    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-927b5\" (UniqueName: \"kubernetes.io/projected/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-kube-api-access-927b5\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6\" (UID: \"10c73b9e-b713-4a4b-bac9-0ef3290efd9e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"
Sep 17 18:59:43.750697 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:43.750618    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6\" (UID: \"10c73b9e-b713-4a4b-bac9-0ef3290efd9e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"
Sep 17 18:59:43.750697 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:43.750671    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6\" (UID: \"10c73b9e-b713-4a4b-bac9-0ef3290efd9e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"
Sep 17 18:59:43.750860 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:43.750697    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6\" (UID: \"10c73b9e-b713-4a4b-bac9-0ef3290efd9e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"
Sep 17 18:59:43.750860 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:43.750723    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6\" (UID: \"10c73b9e-b713-4a4b-bac9-0ef3290efd9e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"
Sep 17 18:59:43.751037 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:43.751014    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6\" (UID: \"10c73b9e-b713-4a4b-bac9-0ef3290efd9e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"
Sep 17 18:59:43.751085 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:43.751067    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6\" (UID: \"10c73b9e-b713-4a4b-bac9-0ef3290efd9e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"
Sep 17 18:59:43.751130 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:43.751111    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6\" (UID: \"10c73b9e-b713-4a4b-bac9-0ef3290efd9e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"
Sep 17 18:59:43.753018 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:43.752998    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6\" (UID: \"10c73b9e-b713-4a4b-bac9-0ef3290efd9e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"
Sep 17 18:59:43.753219 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:43.753187    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6\" (UID: \"10c73b9e-b713-4a4b-bac9-0ef3290efd9e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"
Sep 17 18:59:43.765091 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:43.765071    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-927b5\" (UniqueName: \"kubernetes.io/projected/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-kube-api-access-927b5\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6\" (UID: \"10c73b9e-b713-4a4b-bac9-0ef3290efd9e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"
Sep 17 18:59:43.845230 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:43.845153    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"
Sep 17 18:59:44.137375 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:44.137349    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"]
Sep 17 18:59:44.138647 ip-10-0-139-75 kubenswrapper[2565]: W0917 18:59:44.138617    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c73b9e_b713_4a4b_bac9_0ef3290efd9e.slice/crio-b54b629906a5677c201d2d4b2b6f8d18d4c25a14f1baa41030843b7868da9753 WatchSource:0}: Error finding container b54b629906a5677c201d2d4b2b6f8d18d4c25a14f1baa41030843b7868da9753: Status 404 returned error can't find the container with id b54b629906a5677c201d2d4b2b6f8d18d4c25a14f1baa41030843b7868da9753
Sep 17 18:59:44.474056 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:44.474013    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6" event={"ID":"10c73b9e-b713-4a4b-bac9-0ef3290efd9e","Type":"ContainerStarted","Data":"105f494b49a66dc90b5ffc9edd4af2c42500b331451c3c29b725ada73e31b909"}
Sep 17 18:59:44.474256 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:44.474079    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6" event={"ID":"10c73b9e-b713-4a4b-bac9-0ef3290efd9e","Type":"ContainerStarted","Data":"b54b629906a5677c201d2d4b2b6f8d18d4c25a14f1baa41030843b7868da9753"}
Sep 17 18:59:44.496690 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:44.496644    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6" podStartSLOduration=1.496627174 podStartE2EDuration="1.496627174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:59:44.494002997 +0000 UTC m=+1842.426204515" watchObservedRunningTime="2026-09-17 18:59:44.496627174 +0000 UTC m=+1842.428828710"
Sep 17 18:59:44.845872 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:44.845792    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"
Sep 17 18:59:53.845904 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:53.845872    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"
Sep 17 18:59:53.858597 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:53.858573    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"
Sep 17 18:59:54.539674 ip-10-0-139-75 kubenswrapper[2565]: I0917 18:59:54.539648    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"
Sep 17 19:00:18.051012 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.050975    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"]
Sep 17 19:00:18.060334 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:00:18.060292    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:00:18.060334 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:00:18.060334 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 19:00:18.060334 ip-10-0-139-75 kubenswrapper[2565]:  > containerID="105f494b49a66dc90b5ffc9edd4af2c42500b331451c3c29b725ada73e31b909" cmd=["/bin/sleep","15"]
Sep 17 19:00:18.060630 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.060356    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6" message="" err=<
Sep 17 19:00:18.060630 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:00:18.060630 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 19:00:18.060630 ip-10-0-139-75 kubenswrapper[2565]:  >
Sep 17 19:00:18.060630 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:00:18.060374    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:00:18.060630 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:00:18.060630 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 19:00:18.060630 ip-10-0-139-75 kubenswrapper[2565]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6" podUID="10c73b9e-b713-4a4b-bac9-0ef3290efd9e" containerName="main" containerID="cri-o://105f494b49a66dc90b5ffc9edd4af2c42500b331451c3c29b725ada73e31b909"
Sep 17 19:00:18.060630 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.060419    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6" podUID="10c73b9e-b713-4a4b-bac9-0ef3290efd9e" containerName="main" containerID="cri-o://105f494b49a66dc90b5ffc9edd4af2c42500b331451c3c29b725ada73e31b909" gracePeriod=60
Sep 17 19:00:18.413000 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.412969    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"
Sep 17 19:00:18.513965 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.513939    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-927b5\" (UniqueName: \"kubernetes.io/projected/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-kube-api-access-927b5\") pod \"10c73b9e-b713-4a4b-bac9-0ef3290efd9e\" (UID: \"10c73b9e-b713-4a4b-bac9-0ef3290efd9e\") "
Sep 17 19:00:18.514098 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.513988    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-tmp-dir\") pod \"10c73b9e-b713-4a4b-bac9-0ef3290efd9e\" (UID: \"10c73b9e-b713-4a4b-bac9-0ef3290efd9e\") "
Sep 17 19:00:18.514098 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.514022    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-tls-certs\") pod \"10c73b9e-b713-4a4b-bac9-0ef3290efd9e\" (UID: \"10c73b9e-b713-4a4b-bac9-0ef3290efd9e\") "
Sep 17 19:00:18.514098 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.514044    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-home\") pod \"10c73b9e-b713-4a4b-bac9-0ef3290efd9e\" (UID: \"10c73b9e-b713-4a4b-bac9-0ef3290efd9e\") "
Sep 17 19:00:18.514098 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.514061    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-dshm\") pod \"10c73b9e-b713-4a4b-bac9-0ef3290efd9e\" (UID: \"10c73b9e-b713-4a4b-bac9-0ef3290efd9e\") "
Sep 17 19:00:18.514351 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.514145    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-model-cache\") pod \"10c73b9e-b713-4a4b-bac9-0ef3290efd9e\" (UID: \"10c73b9e-b713-4a4b-bac9-0ef3290efd9e\") "
Sep 17 19:00:18.514408 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.514352    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "10c73b9e-b713-4a4b-bac9-0ef3290efd9e" (UID: "10c73b9e-b713-4a4b-bac9-0ef3290efd9e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:00:18.514408 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.514365    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-home" (OuterVolumeSpecName: "home") pod "10c73b9e-b713-4a4b-bac9-0ef3290efd9e" (UID: "10c73b9e-b713-4a4b-bac9-0ef3290efd9e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:00:18.514489 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.514446    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-model-cache" (OuterVolumeSpecName: "model-cache") pod "10c73b9e-b713-4a4b-bac9-0ef3290efd9e" (UID: "10c73b9e-b713-4a4b-bac9-0ef3290efd9e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:00:18.516242 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.516216    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "10c73b9e-b713-4a4b-bac9-0ef3290efd9e" (UID: "10c73b9e-b713-4a4b-bac9-0ef3290efd9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:00:18.516346 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.516216    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-kube-api-access-927b5" (OuterVolumeSpecName: "kube-api-access-927b5") pod "10c73b9e-b713-4a4b-bac9-0ef3290efd9e" (UID: "10c73b9e-b713-4a4b-bac9-0ef3290efd9e"). InnerVolumeSpecName "kube-api-access-927b5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:00:18.516346 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.516219    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-dshm" (OuterVolumeSpecName: "dshm") pod "10c73b9e-b713-4a4b-bac9-0ef3290efd9e" (UID: "10c73b9e-b713-4a4b-bac9-0ef3290efd9e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:00:18.615181 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.615153    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:00:18.615289 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.615219    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-927b5\" (UniqueName: \"kubernetes.io/projected/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-kube-api-access-927b5\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:00:18.615289 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.615232    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:00:18.615289 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.615244    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:00:18.615289 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.615252    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:00:18.615289 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.615259    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c73b9e-b713-4a4b-bac9-0ef3290efd9e-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:00:18.656464 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.656432    2565 generic.go:356] "Generic (PLEG): container finished" podID="10c73b9e-b713-4a4b-bac9-0ef3290efd9e" containerID="105f494b49a66dc90b5ffc9edd4af2c42500b331451c3c29b725ada73e31b909" exitCode=0
Sep 17 19:00:18.656585 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.656479    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"
Sep 17 19:00:18.656585 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.656483    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6" event={"ID":"10c73b9e-b713-4a4b-bac9-0ef3290efd9e","Type":"ContainerDied","Data":"105f494b49a66dc90b5ffc9edd4af2c42500b331451c3c29b725ada73e31b909"}
Sep 17 19:00:18.656585 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.656554    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6" event={"ID":"10c73b9e-b713-4a4b-bac9-0ef3290efd9e","Type":"ContainerDied","Data":"b54b629906a5677c201d2d4b2b6f8d18d4c25a14f1baa41030843b7868da9753"}
Sep 17 19:00:18.656769 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.656589    2565 scope.go:117] "RemoveContainer" containerID="105f494b49a66dc90b5ffc9edd4af2c42500b331451c3c29b725ada73e31b909"
Sep 17 19:00:18.665694 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.665677    2565 scope.go:117] "RemoveContainer" containerID="105f494b49a66dc90b5ffc9edd4af2c42500b331451c3c29b725ada73e31b909"
Sep 17 19:00:18.665950 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:00:18.665934    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"105f494b49a66dc90b5ffc9edd4af2c42500b331451c3c29b725ada73e31b909\": container with ID starting with 105f494b49a66dc90b5ffc9edd4af2c42500b331451c3c29b725ada73e31b909 not found: ID does not exist" containerID="105f494b49a66dc90b5ffc9edd4af2c42500b331451c3c29b725ada73e31b909"
Sep 17 19:00:18.665996 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.665957    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"105f494b49a66dc90b5ffc9edd4af2c42500b331451c3c29b725ada73e31b909"} err="failed to get container status \"105f494b49a66dc90b5ffc9edd4af2c42500b331451c3c29b725ada73e31b909\": rpc error: code = NotFound desc = could not find container \"105f494b49a66dc90b5ffc9edd4af2c42500b331451c3c29b725ada73e31b909\": container with ID starting with 105f494b49a66dc90b5ffc9edd4af2c42500b331451c3c29b725ada73e31b909 not found: ID does not exist"
Sep 17 19:00:18.678470 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.678447    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"]
Sep 17 19:00:18.688553 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:18.688533    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954svk6"]
Sep 17 19:00:20.607680 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:20.607634    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10c73b9e-b713-4a4b-bac9-0ef3290efd9e" path="/var/lib/kubelet/pods/10c73b9e-b713-4a4b-bac9-0ef3290efd9e/volumes"
Sep 17 19:00:35.635548 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:35.635429    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"]
Sep 17 19:00:35.644600 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:00:35.644560    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:00:35.644600 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:00:35.644600 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 19:00:35.644600 ip-10-0-139-75 kubenswrapper[2565]:  > containerID="35fcf134fc69b4fb4d4cf4d57660351809748978b1ccb2e710bd29a0c964e469" cmd=["/bin/sleep","15"]
Sep 17 19:00:35.644841 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:35.644620    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t" message="" err=<
Sep 17 19:00:35.644841 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:00:35.644841 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 19:00:35.644841 ip-10-0-139-75 kubenswrapper[2565]:  >
Sep 17 19:00:35.644841 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:00:35.644637    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:00:35.644841 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:00:35.644841 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 19:00:35.644841 ip-10-0-139-75 kubenswrapper[2565]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t" podUID="22e2f024-e1d4-4e59-b631-a94518b5cac6" containerName="main" containerID="cri-o://35fcf134fc69b4fb4d4cf4d57660351809748978b1ccb2e710bd29a0c964e469"
Sep 17 19:00:35.644841 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:35.644693    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t" podUID="22e2f024-e1d4-4e59-b631-a94518b5cac6" containerName="main" containerID="cri-o://35fcf134fc69b4fb4d4cf4d57660351809748978b1ccb2e710bd29a0c964e469" gracePeriod=60
Sep 17 19:00:35.996592 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:35.996570    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"
Sep 17 19:00:36.140851 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:36.140820    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22e2f024-e1d4-4e59-b631-a94518b5cac6-tls-certs\") pod \"22e2f024-e1d4-4e59-b631-a94518b5cac6\" (UID: \"22e2f024-e1d4-4e59-b631-a94518b5cac6\") "
Sep 17 19:00:36.141007 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:36.140859    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22e2f024-e1d4-4e59-b631-a94518b5cac6-dshm\") pod \"22e2f024-e1d4-4e59-b631-a94518b5cac6\" (UID: \"22e2f024-e1d4-4e59-b631-a94518b5cac6\") "
Sep 17 19:00:36.141007 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:36.140890    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjvzv\" (UniqueName: \"kubernetes.io/projected/22e2f024-e1d4-4e59-b631-a94518b5cac6-kube-api-access-pjvzv\") pod \"22e2f024-e1d4-4e59-b631-a94518b5cac6\" (UID: \"22e2f024-e1d4-4e59-b631-a94518b5cac6\") "
Sep 17 19:00:36.141007 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:36.141003    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22e2f024-e1d4-4e59-b631-a94518b5cac6-tmp-dir\") pod \"22e2f024-e1d4-4e59-b631-a94518b5cac6\" (UID: \"22e2f024-e1d4-4e59-b631-a94518b5cac6\") "
Sep 17 19:00:36.141177 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:36.141031    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22e2f024-e1d4-4e59-b631-a94518b5cac6-model-cache\") pod \"22e2f024-e1d4-4e59-b631-a94518b5cac6\" (UID: \"22e2f024-e1d4-4e59-b631-a94518b5cac6\") "
Sep 17 19:00:36.141177 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:36.141060    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22e2f024-e1d4-4e59-b631-a94518b5cac6-home\") pod \"22e2f024-e1d4-4e59-b631-a94518b5cac6\" (UID: \"22e2f024-e1d4-4e59-b631-a94518b5cac6\") "
Sep 17 19:00:36.141358 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:36.141319    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e2f024-e1d4-4e59-b631-a94518b5cac6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "22e2f024-e1d4-4e59-b631-a94518b5cac6" (UID: "22e2f024-e1d4-4e59-b631-a94518b5cac6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:00:36.141358 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:36.141340    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e2f024-e1d4-4e59-b631-a94518b5cac6-model-cache" (OuterVolumeSpecName: "model-cache") pod "22e2f024-e1d4-4e59-b631-a94518b5cac6" (UID: "22e2f024-e1d4-4e59-b631-a94518b5cac6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:00:36.141493 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:36.141465    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e2f024-e1d4-4e59-b631-a94518b5cac6-home" (OuterVolumeSpecName: "home") pod "22e2f024-e1d4-4e59-b631-a94518b5cac6" (UID: "22e2f024-e1d4-4e59-b631-a94518b5cac6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:00:36.143138 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:36.143109    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e2f024-e1d4-4e59-b631-a94518b5cac6-dshm" (OuterVolumeSpecName: "dshm") pod "22e2f024-e1d4-4e59-b631-a94518b5cac6" (UID: "22e2f024-e1d4-4e59-b631-a94518b5cac6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:00:36.143290 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:36.143270    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e2f024-e1d4-4e59-b631-a94518b5cac6-kube-api-access-pjvzv" (OuterVolumeSpecName: "kube-api-access-pjvzv") pod "22e2f024-e1d4-4e59-b631-a94518b5cac6" (UID: "22e2f024-e1d4-4e59-b631-a94518b5cac6"). InnerVolumeSpecName "kube-api-access-pjvzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:00:36.143375 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:36.143316    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22e2f024-e1d4-4e59-b631-a94518b5cac6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "22e2f024-e1d4-4e59-b631-a94518b5cac6" (UID: "22e2f024-e1d4-4e59-b631-a94518b5cac6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:00:36.242040 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:36.241982    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22e2f024-e1d4-4e59-b631-a94518b5cac6-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:00:36.242040 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:36.242003    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22e2f024-e1d4-4e59-b631-a94518b5cac6-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:00:36.242040 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:36.242013    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22e2f024-e1d4-4e59-b631-a94518b5cac6-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:00:36.242040 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:36.242020    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22e2f024-e1d4-4e59-b631-a94518b5cac6-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:00:36.242040 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:36.242028    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22e2f024-e1d4-4e59-b631-a94518b5cac6-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:00:36.242040 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:36.242035    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjvzv\" (UniqueName: \"kubernetes.io/projected/22e2f024-e1d4-4e59-b631-a94518b5cac6-kube-api-access-pjvzv\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:00:36.754906 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:36.754876    2565 generic.go:356] "Generic (PLEG): container finished" podID="22e2f024-e1d4-4e59-b631-a94518b5cac6" containerID="35fcf134fc69b4fb4d4cf4d57660351809748978b1ccb2e710bd29a0c964e469" exitCode=0
Sep 17 19:00:36.755376 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:36.754931    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"
Sep 17 19:00:36.755376 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:36.754958    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t" event={"ID":"22e2f024-e1d4-4e59-b631-a94518b5cac6","Type":"ContainerDied","Data":"35fcf134fc69b4fb4d4cf4d57660351809748978b1ccb2e710bd29a0c964e469"}
Sep 17 19:00:36.755376 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:36.755009    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t" event={"ID":"22e2f024-e1d4-4e59-b631-a94518b5cac6","Type":"ContainerDied","Data":"7c56b6c6cff59c353dc0190758487e3dea04531a8a53518659a009fe2fd915e9"}
Sep 17 19:00:36.755376 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:36.755034    2565 scope.go:117] "RemoveContainer" containerID="35fcf134fc69b4fb4d4cf4d57660351809748978b1ccb2e710bd29a0c964e469"
Sep 17 19:00:36.764826 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:36.764809    2565 scope.go:117] "RemoveContainer" containerID="35fcf134fc69b4fb4d4cf4d57660351809748978b1ccb2e710bd29a0c964e469"
Sep 17 19:00:36.765174 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:00:36.765142    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35fcf134fc69b4fb4d4cf4d57660351809748978b1ccb2e710bd29a0c964e469\": container with ID starting with 35fcf134fc69b4fb4d4cf4d57660351809748978b1ccb2e710bd29a0c964e469 not found: ID does not exist" containerID="35fcf134fc69b4fb4d4cf4d57660351809748978b1ccb2e710bd29a0c964e469"
Sep 17 19:00:36.765262 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:36.765185    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35fcf134fc69b4fb4d4cf4d57660351809748978b1ccb2e710bd29a0c964e469"} err="failed to get container status \"35fcf134fc69b4fb4d4cf4d57660351809748978b1ccb2e710bd29a0c964e469\": rpc error: code = NotFound desc = could not find container \"35fcf134fc69b4fb4d4cf4d57660351809748978b1ccb2e710bd29a0c964e469\": container with ID starting with 35fcf134fc69b4fb4d4cf4d57660351809748978b1ccb2e710bd29a0c964e469 not found: ID does not exist"
Sep 17 19:00:36.791182 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:36.791160    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"]
Sep 17 19:00:36.806236 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:36.806215    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997dw95t"]
Sep 17 19:00:38.607069 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:38.607025    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22e2f024-e1d4-4e59-b631-a94518b5cac6" path="/var/lib/kubelet/pods/22e2f024-e1d4-4e59-b631-a94518b5cac6/volumes"
Sep 17 19:00:46.228576 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.228540    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"]
Sep 17 19:00:46.231000 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.228879    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10c73b9e-b713-4a4b-bac9-0ef3290efd9e" containerName="main"
Sep 17 19:00:46.231000 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.228892    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="10c73b9e-b713-4a4b-bac9-0ef3290efd9e" containerName="main"
Sep 17 19:00:46.231000 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.228910    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22e2f024-e1d4-4e59-b631-a94518b5cac6" containerName="main"
Sep 17 19:00:46.231000 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.228915    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="22e2f024-e1d4-4e59-b631-a94518b5cac6" containerName="main"
Sep 17 19:00:46.231000 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.228984    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="10c73b9e-b713-4a4b-bac9-0ef3290efd9e" containerName="main"
Sep 17 19:00:46.231000 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.228993    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="22e2f024-e1d4-4e59-b631-a94518b5cac6" containerName="main"
Sep 17 19:00:46.232020 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.232003    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:00:46.234426 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.234403    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 17 19:00:46.234628 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.234614    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Sep 17 19:00:46.234727 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.234707    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:00:46.234787 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.234747    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 17 19:00:46.234787 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.234707    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-9rmwz\""
Sep 17 19:00:46.234912 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.234809    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 17 19:00:46.248187 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.248159    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"]
Sep 17 19:00:46.423078 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.423046    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:00:46.423078 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.423081    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:00:46.423324 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.423100    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:00:46.423324 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.423145    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:00:46.423324 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.423180    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:00:46.423324 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.423229    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzsjf\" (UniqueName: \"kubernetes.io/projected/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-kube-api-access-lzsjf\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:00:46.423324 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.423250    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:00:46.423324 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.423320    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:00:46.523884 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.523807    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:00:46.523884 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.523839    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:00:46.523884 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.523857    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:00:46.523884 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.523883    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:00:46.524189 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.523906    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:00:46.524189 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.523942    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzsjf\" (UniqueName: \"kubernetes.io/projected/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-kube-api-access-lzsjf\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:00:46.524189 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.523969    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:00:46.524189 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.524025    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:00:46.524439 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.524327    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:00:46.524501 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.524449    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:00:46.524598 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.524575    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:00:46.524598 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.524586    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:00:46.524716 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.524652    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:00:46.526274 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.526250    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:00:46.526523 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.526505    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:00:46.533584 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.533555    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzsjf\" (UniqueName: \"kubernetes.io/projected/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-kube-api-access-lzsjf\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:00:46.542258 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.542235    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:00:46.822036 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:46.822012    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"]
Sep 17 19:00:46.824539 ip-10-0-139-75 kubenswrapper[2565]: W0917 19:00:46.824509    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5bc8e2e_06d1_48c2_bcf3_3e51b7ddd374.slice/crio-7b8bcac01040eca838a08a52cb89fc1170dbc93e29b9abb1fe8f99c3da79ab3f WatchSource:0}: Error finding container 7b8bcac01040eca838a08a52cb89fc1170dbc93e29b9abb1fe8f99c3da79ab3f: Status 404 returned error can't find the container with id 7b8bcac01040eca838a08a52cb89fc1170dbc93e29b9abb1fe8f99c3da79ab3f
Sep 17 19:00:47.815443 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:47.815403    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g" event={"ID":"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374","Type":"ContainerStarted","Data":"6932e83878d881fd9bb2a3d06d86a2806fa02c15cc39464a8f447de53cf53a82"}
Sep 17 19:00:47.815835 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:47.815463    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g" event={"ID":"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374","Type":"ContainerStarted","Data":"7b8bcac01040eca838a08a52cb89fc1170dbc93e29b9abb1fe8f99c3da79ab3f"}
Sep 17 19:00:49.828254 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:49.828219    2565 generic.go:356] "Generic (PLEG): container finished" podID="b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374" containerID="6932e83878d881fd9bb2a3d06d86a2806fa02c15cc39464a8f447de53cf53a82" exitCode=0
Sep 17 19:00:49.828713 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:49.828289    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g" event={"ID":"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374","Type":"ContainerDied","Data":"6932e83878d881fd9bb2a3d06d86a2806fa02c15cc39464a8f447de53cf53a82"}
Sep 17 19:00:50.836738 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:50.836701    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g" event={"ID":"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374","Type":"ContainerStarted","Data":"6707c7553a9d24179629ff79b69d808778d62cf4eb869bd56a5f11adf8285a90"}
Sep 17 19:00:50.866424 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:50.866366    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g" podStartSLOduration=2.667513202 podStartE2EDuration="4.866338874s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.198825672s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:00:50.864150238 +0000 UTC m=+1908.796351758" watchObservedRunningTime="2026-09-17 19:00:50.866338874 +0000 UTC m=+1908.798540394"
Sep 17 19:00:51.542425 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:51.542391    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:00:56.543081 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:56.543037    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:00:56.544544 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:00:56.544516    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g" podUID="b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 17 19:01:06.543652 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:01:06.543595    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g" podUID="b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 17 19:01:16.542794 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:01:16.542752    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g" podUID="b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 17 19:01:26.542721 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:01:26.542677    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g" podUID="b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 17 19:01:36.543588 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:01:36.543544    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g" podUID="b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 17 19:01:46.543367 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:01:46.543324    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g" podUID="b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 17 19:01:56.543582 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:01:56.543487    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g" podUID="b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 17 19:02:06.542753 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:06.542711    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g" podUID="b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 17 19:02:16.543219 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:16.543156    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g" podUID="b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 17 19:02:26.542908 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:26.542856    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g" podUID="b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 17 19:02:36.542805 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:36.542754    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g" podUID="b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Sep 17 19:02:46.556693 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:46.556655    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:02:46.564452 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:46.564421    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:02:52.116889 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:52.116852    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"]
Sep 17 19:02:52.120713 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:52.120691    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"
Sep 17 19:02:52.127168 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:52.127112    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 17 19:02:52.128059 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:52.128035    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 17 19:02:52.128132 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:52.128061    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-8m6kk\""
Sep 17 19:02:52.128219 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:52.128149    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 17 19:02:52.134511 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:52.134488    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x7zj\" (UniqueName: \"kubernetes.io/projected/824f127c-e1d7-4ff9-8a8d-16390cb84909-kube-api-access-8x7zj\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt\" (UID: \"824f127c-e1d7-4ff9-8a8d-16390cb84909\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"
Sep 17 19:02:52.134617 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:52.134525    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/824f127c-e1d7-4ff9-8a8d-16390cb84909-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt\" (UID: \"824f127c-e1d7-4ff9-8a8d-16390cb84909\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"
Sep 17 19:02:52.134617 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:52.134565    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/824f127c-e1d7-4ff9-8a8d-16390cb84909-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt\" (UID: \"824f127c-e1d7-4ff9-8a8d-16390cb84909\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"
Sep 17 19:02:52.134617 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:52.134588    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/824f127c-e1d7-4ff9-8a8d-16390cb84909-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt\" (UID: \"824f127c-e1d7-4ff9-8a8d-16390cb84909\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"
Sep 17 19:02:52.134617 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:52.134607    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/824f127c-e1d7-4ff9-8a8d-16390cb84909-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt\" (UID: \"824f127c-e1d7-4ff9-8a8d-16390cb84909\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"
Sep 17 19:02:52.134801 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:52.134680    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/824f127c-e1d7-4ff9-8a8d-16390cb84909-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt\" (UID: \"824f127c-e1d7-4ff9-8a8d-16390cb84909\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"
Sep 17 19:02:52.134951 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:52.134932    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"]
Sep 17 19:02:52.236007 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:52.235961    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/824f127c-e1d7-4ff9-8a8d-16390cb84909-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt\" (UID: \"824f127c-e1d7-4ff9-8a8d-16390cb84909\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"
Sep 17 19:02:52.236007 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:52.236006    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/824f127c-e1d7-4ff9-8a8d-16390cb84909-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt\" (UID: \"824f127c-e1d7-4ff9-8a8d-16390cb84909\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"
Sep 17 19:02:52.236007 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:52.236023    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/824f127c-e1d7-4ff9-8a8d-16390cb84909-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt\" (UID: \"824f127c-e1d7-4ff9-8a8d-16390cb84909\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"
Sep 17 19:02:52.236374 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:52.236071    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/824f127c-e1d7-4ff9-8a8d-16390cb84909-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt\" (UID: \"824f127c-e1d7-4ff9-8a8d-16390cb84909\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"
Sep 17 19:02:52.236374 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:52.236103    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x7zj\" (UniqueName: \"kubernetes.io/projected/824f127c-e1d7-4ff9-8a8d-16390cb84909-kube-api-access-8x7zj\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt\" (UID: \"824f127c-e1d7-4ff9-8a8d-16390cb84909\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"
Sep 17 19:02:52.236374 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:52.236143    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/824f127c-e1d7-4ff9-8a8d-16390cb84909-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt\" (UID: \"824f127c-e1d7-4ff9-8a8d-16390cb84909\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"
Sep 17 19:02:52.236585 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:52.236454    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/824f127c-e1d7-4ff9-8a8d-16390cb84909-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt\" (UID: \"824f127c-e1d7-4ff9-8a8d-16390cb84909\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"
Sep 17 19:02:52.236585 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:52.236473    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/824f127c-e1d7-4ff9-8a8d-16390cb84909-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt\" (UID: \"824f127c-e1d7-4ff9-8a8d-16390cb84909\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"
Sep 17 19:02:52.236585 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:52.236550    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/824f127c-e1d7-4ff9-8a8d-16390cb84909-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt\" (UID: \"824f127c-e1d7-4ff9-8a8d-16390cb84909\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"
Sep 17 19:02:52.238447 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:52.238420    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/824f127c-e1d7-4ff9-8a8d-16390cb84909-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt\" (UID: \"824f127c-e1d7-4ff9-8a8d-16390cb84909\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"
Sep 17 19:02:52.238723 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:52.238705    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/824f127c-e1d7-4ff9-8a8d-16390cb84909-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt\" (UID: \"824f127c-e1d7-4ff9-8a8d-16390cb84909\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"
Sep 17 19:02:52.246596 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:52.246578    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x7zj\" (UniqueName: \"kubernetes.io/projected/824f127c-e1d7-4ff9-8a8d-16390cb84909-kube-api-access-8x7zj\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt\" (UID: \"824f127c-e1d7-4ff9-8a8d-16390cb84909\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"
Sep 17 19:02:52.430978 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:52.430945    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"
Sep 17 19:02:52.715008 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:52.714974    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"]
Sep 17 19:02:52.717645 ip-10-0-139-75 kubenswrapper[2565]: W0917 19:02:52.717621    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod824f127c_e1d7_4ff9_8a8d_16390cb84909.slice/crio-bd1023909642e1b624273b895232a0a4e3e7ef9632b033989bdb761abec981a9 WatchSource:0}: Error finding container bd1023909642e1b624273b895232a0a4e3e7ef9632b033989bdb761abec981a9: Status 404 returned error can't find the container with id bd1023909642e1b624273b895232a0a4e3e7ef9632b033989bdb761abec981a9
Sep 17 19:02:53.507153 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:53.507119    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt" event={"ID":"824f127c-e1d7-4ff9-8a8d-16390cb84909","Type":"ContainerStarted","Data":"9325b21c9c9e2ac8a0ba24380f8e5f63e4d64ce77d133f3c83e1e71bff1a20a2"}
Sep 17 19:02:53.507559 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:53.507182    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt" event={"ID":"824f127c-e1d7-4ff9-8a8d-16390cb84909","Type":"ContainerStarted","Data":"bd1023909642e1b624273b895232a0a4e3e7ef9632b033989bdb761abec981a9"}
Sep 17 19:02:53.541070 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:53.541022    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt" podStartSLOduration=1.541004072 podStartE2EDuration="1.541004072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:02:53.540148746 +0000 UTC m=+2031.472350266" watchObservedRunningTime="2026-09-17 19:02:53.541004072 +0000 UTC m=+2031.473205593"
Sep 17 19:02:54.432076 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:02:54.432044    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"
Sep 17 19:03:02.431326 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:02.431278    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"
Sep 17 19:03:02.444079 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:02.444047    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"
Sep 17 19:03:02.566918 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:02.566888    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"
Sep 17 19:03:12.514094 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:12.514053    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"]
Sep 17 19:03:26.628026 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:26.627942    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"]
Sep 17 19:03:26.636902 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:03:26.636866    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:03:26.636902 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:03:26.636902 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 19:03:26.636902 ip-10-0-139-75 kubenswrapper[2565]:  > containerID="9325b21c9c9e2ac8a0ba24380f8e5f63e4d64ce77d133f3c83e1e71bff1a20a2" cmd=["/bin/sleep","15"]
Sep 17 19:03:26.637178 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:26.636931    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt" message="" err=<
Sep 17 19:03:26.637178 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:03:26.637178 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 19:03:26.637178 ip-10-0-139-75 kubenswrapper[2565]:  >
Sep 17 19:03:26.637178 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:03:26.636948    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:03:26.637178 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:03:26.637178 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 19:03:26.637178 ip-10-0-139-75 kubenswrapper[2565]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt" podUID="824f127c-e1d7-4ff9-8a8d-16390cb84909" containerName="main" containerID="cri-o://9325b21c9c9e2ac8a0ba24380f8e5f63e4d64ce77d133f3c83e1e71bff1a20a2"
Sep 17 19:03:26.637178 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:26.636990    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt" podUID="824f127c-e1d7-4ff9-8a8d-16390cb84909" containerName="main" containerID="cri-o://9325b21c9c9e2ac8a0ba24380f8e5f63e4d64ce77d133f3c83e1e71bff1a20a2" gracePeriod=60
Sep 17 19:03:27.390268 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:27.390244    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"
Sep 17 19:03:27.508463 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:27.508398    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/824f127c-e1d7-4ff9-8a8d-16390cb84909-tmp-dir\") pod \"824f127c-e1d7-4ff9-8a8d-16390cb84909\" (UID: \"824f127c-e1d7-4ff9-8a8d-16390cb84909\") "
Sep 17 19:03:27.508463 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:27.508437    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8x7zj\" (UniqueName: \"kubernetes.io/projected/824f127c-e1d7-4ff9-8a8d-16390cb84909-kube-api-access-8x7zj\") pod \"824f127c-e1d7-4ff9-8a8d-16390cb84909\" (UID: \"824f127c-e1d7-4ff9-8a8d-16390cb84909\") "
Sep 17 19:03:27.508463 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:27.508457    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/824f127c-e1d7-4ff9-8a8d-16390cb84909-model-cache\") pod \"824f127c-e1d7-4ff9-8a8d-16390cb84909\" (UID: \"824f127c-e1d7-4ff9-8a8d-16390cb84909\") "
Sep 17 19:03:27.508692 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:27.508477    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/824f127c-e1d7-4ff9-8a8d-16390cb84909-home\") pod \"824f127c-e1d7-4ff9-8a8d-16390cb84909\" (UID: \"824f127c-e1d7-4ff9-8a8d-16390cb84909\") "
Sep 17 19:03:27.508692 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:27.508525    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/824f127c-e1d7-4ff9-8a8d-16390cb84909-dshm\") pod \"824f127c-e1d7-4ff9-8a8d-16390cb84909\" (UID: \"824f127c-e1d7-4ff9-8a8d-16390cb84909\") "
Sep 17 19:03:27.508692 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:27.508674    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/824f127c-e1d7-4ff9-8a8d-16390cb84909-tls-certs\") pod \"824f127c-e1d7-4ff9-8a8d-16390cb84909\" (UID: \"824f127c-e1d7-4ff9-8a8d-16390cb84909\") "
Sep 17 19:03:27.508841 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:27.508699    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824f127c-e1d7-4ff9-8a8d-16390cb84909-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "824f127c-e1d7-4ff9-8a8d-16390cb84909" (UID: "824f127c-e1d7-4ff9-8a8d-16390cb84909"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:03:27.508841 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:27.508703    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824f127c-e1d7-4ff9-8a8d-16390cb84909-model-cache" (OuterVolumeSpecName: "model-cache") pod "824f127c-e1d7-4ff9-8a8d-16390cb84909" (UID: "824f127c-e1d7-4ff9-8a8d-16390cb84909"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:03:27.508841 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:27.508771    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824f127c-e1d7-4ff9-8a8d-16390cb84909-home" (OuterVolumeSpecName: "home") pod "824f127c-e1d7-4ff9-8a8d-16390cb84909" (UID: "824f127c-e1d7-4ff9-8a8d-16390cb84909"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:03:27.509011 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:27.508995    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/824f127c-e1d7-4ff9-8a8d-16390cb84909-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:03:27.509069 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:27.509017    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/824f127c-e1d7-4ff9-8a8d-16390cb84909-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:03:27.509069 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:27.509031    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/824f127c-e1d7-4ff9-8a8d-16390cb84909-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:03:27.510748 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:27.510715    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/824f127c-e1d7-4ff9-8a8d-16390cb84909-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "824f127c-e1d7-4ff9-8a8d-16390cb84909" (UID: "824f127c-e1d7-4ff9-8a8d-16390cb84909"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:03:27.510862 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:27.510778    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824f127c-e1d7-4ff9-8a8d-16390cb84909-kube-api-access-8x7zj" (OuterVolumeSpecName: "kube-api-access-8x7zj") pod "824f127c-e1d7-4ff9-8a8d-16390cb84909" (UID: "824f127c-e1d7-4ff9-8a8d-16390cb84909"). InnerVolumeSpecName "kube-api-access-8x7zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:03:27.510862 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:27.510803    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824f127c-e1d7-4ff9-8a8d-16390cb84909-dshm" (OuterVolumeSpecName: "dshm") pod "824f127c-e1d7-4ff9-8a8d-16390cb84909" (UID: "824f127c-e1d7-4ff9-8a8d-16390cb84909"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:03:27.536482 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:27.536454    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g" podUID="b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374" containerName="main" containerID="cri-o://6707c7553a9d24179629ff79b69d808778d62cf4eb869bd56a5f11adf8285a90" gracePeriod=45
Sep 17 19:03:27.610391 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:27.610354    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/824f127c-e1d7-4ff9-8a8d-16390cb84909-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:03:27.610391 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:27.610385    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/824f127c-e1d7-4ff9-8a8d-16390cb84909-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:03:27.610391 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:27.610396    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x7zj\" (UniqueName: \"kubernetes.io/projected/824f127c-e1d7-4ff9-8a8d-16390cb84909-kube-api-access-8x7zj\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:03:27.686394 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:27.686353    2565 generic.go:356] "Generic (PLEG): container finished" podID="824f127c-e1d7-4ff9-8a8d-16390cb84909" containerID="9325b21c9c9e2ac8a0ba24380f8e5f63e4d64ce77d133f3c83e1e71bff1a20a2" exitCode=0
Sep 17 19:03:27.686766 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:27.686408    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"
Sep 17 19:03:27.686766 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:27.686440    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt" event={"ID":"824f127c-e1d7-4ff9-8a8d-16390cb84909","Type":"ContainerDied","Data":"9325b21c9c9e2ac8a0ba24380f8e5f63e4d64ce77d133f3c83e1e71bff1a20a2"}
Sep 17 19:03:27.686766 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:27.686496    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt" event={"ID":"824f127c-e1d7-4ff9-8a8d-16390cb84909","Type":"ContainerDied","Data":"bd1023909642e1b624273b895232a0a4e3e7ef9632b033989bdb761abec981a9"}
Sep 17 19:03:27.686766 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:27.686517    2565 scope.go:117] "RemoveContainer" containerID="9325b21c9c9e2ac8a0ba24380f8e5f63e4d64ce77d133f3c83e1e71bff1a20a2"
Sep 17 19:03:27.696057 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:27.696039    2565 scope.go:117] "RemoveContainer" containerID="9325b21c9c9e2ac8a0ba24380f8e5f63e4d64ce77d133f3c83e1e71bff1a20a2"
Sep 17 19:03:27.696366 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:03:27.696340    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9325b21c9c9e2ac8a0ba24380f8e5f63e4d64ce77d133f3c83e1e71bff1a20a2\": container with ID starting with 9325b21c9c9e2ac8a0ba24380f8e5f63e4d64ce77d133f3c83e1e71bff1a20a2 not found: ID does not exist" containerID="9325b21c9c9e2ac8a0ba24380f8e5f63e4d64ce77d133f3c83e1e71bff1a20a2"
Sep 17 19:03:27.696433 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:27.696376    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9325b21c9c9e2ac8a0ba24380f8e5f63e4d64ce77d133f3c83e1e71bff1a20a2"} err="failed to get container status \"9325b21c9c9e2ac8a0ba24380f8e5f63e4d64ce77d133f3c83e1e71bff1a20a2\": rpc error: code = NotFound desc = could not find container \"9325b21c9c9e2ac8a0ba24380f8e5f63e4d64ce77d133f3c83e1e71bff1a20a2\": container with ID starting with 9325b21c9c9e2ac8a0ba24380f8e5f63e4d64ce77d133f3c83e1e71bff1a20a2 not found: ID does not exist"
Sep 17 19:03:27.708852 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:27.708831    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"]
Sep 17 19:03:27.717686 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:27.717657    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-944mt"]
Sep 17 19:03:28.292480 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.292456    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:03:28.416510 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.416488    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-tls-certs\") pod \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") "
Sep 17 19:03:28.416657 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.416533    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-cabundle-cert\") pod \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") "
Sep 17 19:03:28.416657 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.416574    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-tmp-dir\") pod \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") "
Sep 17 19:03:28.416784 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.416730    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzsjf\" (UniqueName: \"kubernetes.io/projected/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-kube-api-access-lzsjf\") pod \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") "
Sep 17 19:03:28.416784 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.416771    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-dshm\") pod \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") "
Sep 17 19:03:28.416904 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.416802    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-model-cache\") pod \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") "
Sep 17 19:03:28.416904 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.416833    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-kserve-provision-location\") pod \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") "
Sep 17 19:03:28.416904 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.416898    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-home\") pod \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\" (UID: \"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374\") "
Sep 17 19:03:28.417057 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.416933    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374" (UID: "b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:03:28.417116 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.417070    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-model-cache" (OuterVolumeSpecName: "model-cache") pod "b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374" (UID: "b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:03:28.417252 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.417232    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-cabundle-cert\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:03:28.417322 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.417260    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:03:28.417675 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.417649    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-home" (OuterVolumeSpecName: "home") pod "b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374" (UID: "b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:03:28.418884 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.418860    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374" (UID: "b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:03:28.419187 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.419170    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-dshm" (OuterVolumeSpecName: "dshm") pod "b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374" (UID: "b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:03:28.419344 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.419326    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-kube-api-access-lzsjf" (OuterVolumeSpecName: "kube-api-access-lzsjf") pod "b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374" (UID: "b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374"). InnerVolumeSpecName "kube-api-access-lzsjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:03:28.430581 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.430560    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374" (UID: "b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:03:28.441669 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.441646    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374" (UID: "b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:03:28.518115 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.518087    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:03:28.518115 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.518113    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzsjf\" (UniqueName: \"kubernetes.io/projected/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-kube-api-access-lzsjf\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:03:28.518286 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.518126    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:03:28.518286 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.518140    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-kserve-provision-location\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:03:28.518286 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.518151    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:03:28.518286 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.518165    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:03:28.607038 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.607009    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="824f127c-e1d7-4ff9-8a8d-16390cb84909" path="/var/lib/kubelet/pods/824f127c-e1d7-4ff9-8a8d-16390cb84909/volumes"
Sep 17 19:03:28.700047 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.699990    2565 generic.go:356] "Generic (PLEG): container finished" podID="b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374" containerID="6707c7553a9d24179629ff79b69d808778d62cf4eb869bd56a5f11adf8285a90" exitCode=0
Sep 17 19:03:28.700423 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.700051    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"
Sep 17 19:03:28.700423 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.700059    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g" event={"ID":"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374","Type":"ContainerDied","Data":"6707c7553a9d24179629ff79b69d808778d62cf4eb869bd56a5f11adf8285a90"}
Sep 17 19:03:28.700423 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.700128    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g" event={"ID":"b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374","Type":"ContainerDied","Data":"7b8bcac01040eca838a08a52cb89fc1170dbc93e29b9abb1fe8f99c3da79ab3f"}
Sep 17 19:03:28.700423 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.700155    2565 scope.go:117] "RemoveContainer" containerID="6707c7553a9d24179629ff79b69d808778d62cf4eb869bd56a5f11adf8285a90"
Sep 17 19:03:28.709403 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.709384    2565 scope.go:117] "RemoveContainer" containerID="6932e83878d881fd9bb2a3d06d86a2806fa02c15cc39464a8f447de53cf53a82"
Sep 17 19:03:28.716958 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.716940    2565 scope.go:117] "RemoveContainer" containerID="6707c7553a9d24179629ff79b69d808778d62cf4eb869bd56a5f11adf8285a90"
Sep 17 19:03:28.717216 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:03:28.717181    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6707c7553a9d24179629ff79b69d808778d62cf4eb869bd56a5f11adf8285a90\": container with ID starting with 6707c7553a9d24179629ff79b69d808778d62cf4eb869bd56a5f11adf8285a90 not found: ID does not exist" containerID="6707c7553a9d24179629ff79b69d808778d62cf4eb869bd56a5f11adf8285a90"
Sep 17 19:03:28.717285 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.717223    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6707c7553a9d24179629ff79b69d808778d62cf4eb869bd56a5f11adf8285a90"} err="failed to get container status \"6707c7553a9d24179629ff79b69d808778d62cf4eb869bd56a5f11adf8285a90\": rpc error: code = NotFound desc = could not find container \"6707c7553a9d24179629ff79b69d808778d62cf4eb869bd56a5f11adf8285a90\": container with ID starting with 6707c7553a9d24179629ff79b69d808778d62cf4eb869bd56a5f11adf8285a90 not found: ID does not exist"
Sep 17 19:03:28.717285 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.717240    2565 scope.go:117] "RemoveContainer" containerID="6932e83878d881fd9bb2a3d06d86a2806fa02c15cc39464a8f447de53cf53a82"
Sep 17 19:03:28.717489 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:03:28.717474    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6932e83878d881fd9bb2a3d06d86a2806fa02c15cc39464a8f447de53cf53a82\": container with ID starting with 6932e83878d881fd9bb2a3d06d86a2806fa02c15cc39464a8f447de53cf53a82 not found: ID does not exist" containerID="6932e83878d881fd9bb2a3d06d86a2806fa02c15cc39464a8f447de53cf53a82"
Sep 17 19:03:28.717531 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.717493    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6932e83878d881fd9bb2a3d06d86a2806fa02c15cc39464a8f447de53cf53a82"} err="failed to get container status \"6932e83878d881fd9bb2a3d06d86a2806fa02c15cc39464a8f447de53cf53a82\": rpc error: code = NotFound desc = could not find container \"6932e83878d881fd9bb2a3d06d86a2806fa02c15cc39464a8f447de53cf53a82\": container with ID starting with 6932e83878d881fd9bb2a3d06d86a2806fa02c15cc39464a8f447de53cf53a82 not found: ID does not exist"
Sep 17 19:03:28.721727 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.721706    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"]
Sep 17 19:03:28.737295 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:28.737262    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-d94ccffb7-fvb4g"]
Sep 17 19:03:30.606638 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:03:30.606599    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374" path="/var/lib/kubelet/pods/b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374/volumes"
Sep 17 19:04:09.028063 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.028024    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"]
Sep 17 19:04:09.028486 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.028387    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374" containerName="main"
Sep 17 19:04:09.028486 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.028399    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374" containerName="main"
Sep 17 19:04:09.028486 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.028410    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="824f127c-e1d7-4ff9-8a8d-16390cb84909" containerName="main"
Sep 17 19:04:09.028486 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.028416    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="824f127c-e1d7-4ff9-8a8d-16390cb84909" containerName="main"
Sep 17 19:04:09.028486 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.028422    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374" containerName="storage-initializer"
Sep 17 19:04:09.028486 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.028427    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374" containerName="storage-initializer"
Sep 17 19:04:09.028683 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.028495    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5bc8e2e-06d1-48c2-bcf3-3e51b7ddd374" containerName="main"
Sep 17 19:04:09.028683 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.028503    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="824f127c-e1d7-4ff9-8a8d-16390cb84909" containerName="main"
Sep 17 19:04:09.031777 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.031759    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"
Sep 17 19:04:09.034658 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.034633    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 17 19:04:09.035041 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.035023    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-52l5t\""
Sep 17 19:04:09.035521 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.035506    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 17 19:04:09.037455 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.037411    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 17 19:04:09.051930 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.051907    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"]
Sep 17 19:04:09.111113 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.111085    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/393eb93a-bc25-486d-b53d-c7f0b37ab894-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9\" (UID: \"393eb93a-bc25-486d-b53d-c7f0b37ab894\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"
Sep 17 19:04:09.111113 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.111115    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkpxq\" (UniqueName: \"kubernetes.io/projected/393eb93a-bc25-486d-b53d-c7f0b37ab894-kube-api-access-qkpxq\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9\" (UID: \"393eb93a-bc25-486d-b53d-c7f0b37ab894\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"
Sep 17 19:04:09.111364 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.111140    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/393eb93a-bc25-486d-b53d-c7f0b37ab894-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9\" (UID: \"393eb93a-bc25-486d-b53d-c7f0b37ab894\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"
Sep 17 19:04:09.111364 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.111222    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/393eb93a-bc25-486d-b53d-c7f0b37ab894-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9\" (UID: \"393eb93a-bc25-486d-b53d-c7f0b37ab894\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"
Sep 17 19:04:09.111364 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.111310    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/393eb93a-bc25-486d-b53d-c7f0b37ab894-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9\" (UID: \"393eb93a-bc25-486d-b53d-c7f0b37ab894\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"
Sep 17 19:04:09.111364 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.111343    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/393eb93a-bc25-486d-b53d-c7f0b37ab894-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9\" (UID: \"393eb93a-bc25-486d-b53d-c7f0b37ab894\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"
Sep 17 19:04:09.111539 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.111399    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393eb93a-bc25-486d-b53d-c7f0b37ab894-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9\" (UID: \"393eb93a-bc25-486d-b53d-c7f0b37ab894\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"
Sep 17 19:04:09.212344 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.212309    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/393eb93a-bc25-486d-b53d-c7f0b37ab894-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9\" (UID: \"393eb93a-bc25-486d-b53d-c7f0b37ab894\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"
Sep 17 19:04:09.212472 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.212353    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/393eb93a-bc25-486d-b53d-c7f0b37ab894-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9\" (UID: \"393eb93a-bc25-486d-b53d-c7f0b37ab894\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"
Sep 17 19:04:09.212472 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.212414    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/393eb93a-bc25-486d-b53d-c7f0b37ab894-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9\" (UID: \"393eb93a-bc25-486d-b53d-c7f0b37ab894\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"
Sep 17 19:04:09.212472 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.212443    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/393eb93a-bc25-486d-b53d-c7f0b37ab894-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9\" (UID: \"393eb93a-bc25-486d-b53d-c7f0b37ab894\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"
Sep 17 19:04:09.212636 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.212487    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393eb93a-bc25-486d-b53d-c7f0b37ab894-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9\" (UID: \"393eb93a-bc25-486d-b53d-c7f0b37ab894\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"
Sep 17 19:04:09.212636 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.212529    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/393eb93a-bc25-486d-b53d-c7f0b37ab894-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9\" (UID: \"393eb93a-bc25-486d-b53d-c7f0b37ab894\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"
Sep 17 19:04:09.212636 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.212557    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkpxq\" (UniqueName: \"kubernetes.io/projected/393eb93a-bc25-486d-b53d-c7f0b37ab894-kube-api-access-qkpxq\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9\" (UID: \"393eb93a-bc25-486d-b53d-c7f0b37ab894\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"
Sep 17 19:04:09.212852 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.212829    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/393eb93a-bc25-486d-b53d-c7f0b37ab894-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9\" (UID: \"393eb93a-bc25-486d-b53d-c7f0b37ab894\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"
Sep 17 19:04:09.212928 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.212847    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/393eb93a-bc25-486d-b53d-c7f0b37ab894-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9\" (UID: \"393eb93a-bc25-486d-b53d-c7f0b37ab894\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"
Sep 17 19:04:09.212928 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.212891    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/393eb93a-bc25-486d-b53d-c7f0b37ab894-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9\" (UID: \"393eb93a-bc25-486d-b53d-c7f0b37ab894\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"
Sep 17 19:04:09.213005 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.212955    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/393eb93a-bc25-486d-b53d-c7f0b37ab894-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9\" (UID: \"393eb93a-bc25-486d-b53d-c7f0b37ab894\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"
Sep 17 19:04:09.214559 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.214538    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/393eb93a-bc25-486d-b53d-c7f0b37ab894-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9\" (UID: \"393eb93a-bc25-486d-b53d-c7f0b37ab894\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"
Sep 17 19:04:09.214939 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.214923    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393eb93a-bc25-486d-b53d-c7f0b37ab894-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9\" (UID: \"393eb93a-bc25-486d-b53d-c7f0b37ab894\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"
Sep 17 19:04:09.222063 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.222039    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkpxq\" (UniqueName: \"kubernetes.io/projected/393eb93a-bc25-486d-b53d-c7f0b37ab894-kube-api-access-qkpxq\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9\" (UID: \"393eb93a-bc25-486d-b53d-c7f0b37ab894\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"
Sep 17 19:04:09.342459 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.342399    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"
Sep 17 19:04:09.635720 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.635693    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"]
Sep 17 19:04:09.685903 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.685881    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:04:09.930819 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:09.930791    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9" event={"ID":"393eb93a-bc25-486d-b53d-c7f0b37ab894","Type":"ContainerStarted","Data":"d2774f70b2b41b7216b42f22b8a416199554e24e92f189c9bcc4fbb9297a5461"}
Sep 17 19:04:17.978433 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:17.978398    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9" event={"ID":"393eb93a-bc25-486d-b53d-c7f0b37ab894","Type":"ContainerStarted","Data":"e1e1d8ac0c0f92899dc9fa5fcdef4f75fad924475b204d52a95846c82ef697f8"}
Sep 17 19:04:18.985069 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:18.985033    2565 generic.go:356] "Generic (PLEG): container finished" podID="393eb93a-bc25-486d-b53d-c7f0b37ab894" containerID="e1e1d8ac0c0f92899dc9fa5fcdef4f75fad924475b204d52a95846c82ef697f8" exitCode=0
Sep 17 19:04:18.985538 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:18.985087    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9" event={"ID":"393eb93a-bc25-486d-b53d-c7f0b37ab894","Type":"ContainerDied","Data":"e1e1d8ac0c0f92899dc9fa5fcdef4f75fad924475b204d52a95846c82ef697f8"}
Sep 17 19:04:19.994630 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:19.994595    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9" event={"ID":"393eb93a-bc25-486d-b53d-c7f0b37ab894","Type":"ContainerStarted","Data":"f1a5aaed4e394dc6102c4711fe4820ea9809fcf3bdba7ef9bcd01048d8d1200b"}
Sep 17 19:04:19.995020 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:19.994665    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9" event={"ID":"393eb93a-bc25-486d-b53d-c7f0b37ab894","Type":"ContainerStarted","Data":"cfbe9d95d060c7afa1cd48df3059b1414b216608a95dd5f69c3f73748f6848ec"}
Sep 17 19:04:20.018837 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:20.018787    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9" podStartSLOduration=2.202901042 podStartE2EDuration="11.018772662s" totalImagesPullingTime="7.330875319s" totalInitContainerRuntime="1.484996301s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:04:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:04:20.016530957 +0000 UTC m=+2117.948732477" watchObservedRunningTime="2026-09-17 19:04:20.018772662 +0000 UTC m=+2117.950974181"
Sep 17 19:04:20.343568 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:20.343474    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"
Sep 17 19:04:29.343444 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:29.343401    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"
Sep 17 19:04:29.345140 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:29.345107    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9" podUID="393eb93a-bc25-486d-b53d-c7f0b37ab894" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 17 19:04:39.343113 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:39.343063    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9" podUID="393eb93a-bc25-486d-b53d-c7f0b37ab894" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 17 19:04:49.343742 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:49.343693    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9" podUID="393eb93a-bc25-486d-b53d-c7f0b37ab894" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 17 19:04:59.343280 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:04:59.343166    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9" podUID="393eb93a-bc25-486d-b53d-c7f0b37ab894" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 17 19:05:09.343218 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:05:09.343146    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9" podUID="393eb93a-bc25-486d-b53d-c7f0b37ab894" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 17 19:05:19.343259 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:05:19.343216    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9" podUID="393eb93a-bc25-486d-b53d-c7f0b37ab894" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 17 19:05:29.343497 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:05:29.343446    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9" podUID="393eb93a-bc25-486d-b53d-c7f0b37ab894" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 17 19:05:39.343804 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:05:39.343752    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9" podUID="393eb93a-bc25-486d-b53d-c7f0b37ab894" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 17 19:05:49.343098 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:05:49.343054    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9" podUID="393eb93a-bc25-486d-b53d-c7f0b37ab894" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 17 19:05:59.343387 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:05:59.343344    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9" podUID="393eb93a-bc25-486d-b53d-c7f0b37ab894" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 17 19:06:09.353646 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:09.353616    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"
Sep 17 19:06:09.362789 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:09.362763    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"
Sep 17 19:06:14.155747 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:14.155713    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"]
Sep 17 19:06:14.156274 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:14.156029    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9" podUID="393eb93a-bc25-486d-b53d-c7f0b37ab894" containerName="modelcar" containerID="cri-o://f1a5aaed4e394dc6102c4711fe4820ea9809fcf3bdba7ef9bcd01048d8d1200b" gracePeriod=60
Sep 17 19:06:14.637959 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:14.637931    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9_393eb93a-bc25-486d-b53d-c7f0b37ab894/modelcar/0.log"
Sep 17 19:06:14.638681 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:14.638659    2565 generic.go:356] "Generic (PLEG): container finished" podID="393eb93a-bc25-486d-b53d-c7f0b37ab894" containerID="f1a5aaed4e394dc6102c4711fe4820ea9809fcf3bdba7ef9bcd01048d8d1200b" exitCode=143
Sep 17 19:06:14.638775 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:14.638726    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9" event={"ID":"393eb93a-bc25-486d-b53d-c7f0b37ab894","Type":"ContainerDied","Data":"f1a5aaed4e394dc6102c4711fe4820ea9809fcf3bdba7ef9bcd01048d8d1200b"}
Sep 17 19:06:29.178145 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:29.178039    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9" podUID="393eb93a-bc25-486d-b53d-c7f0b37ab894" containerName="main" containerID="cri-o://cfbe9d95d060c7afa1cd48df3059b1414b216608a95dd5f69c3f73748f6848ec" gracePeriod=45
Sep 17 19:06:29.354434 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:29.354388    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9" podUID="393eb93a-bc25-486d-b53d-c7f0b37ab894" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Sep 17 19:06:30.430397 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.430373    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9_393eb93a-bc25-486d-b53d-c7f0b37ab894/modelcar/0.log"
Sep 17 19:06:30.433193 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.433175    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"
Sep 17 19:06:30.476379 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.476322    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/393eb93a-bc25-486d-b53d-c7f0b37ab894-model-cache\") pod \"393eb93a-bc25-486d-b53d-c7f0b37ab894\" (UID: \"393eb93a-bc25-486d-b53d-c7f0b37ab894\") "
Sep 17 19:06:30.476379 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.476357    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/393eb93a-bc25-486d-b53d-c7f0b37ab894-dshm\") pod \"393eb93a-bc25-486d-b53d-c7f0b37ab894\" (UID: \"393eb93a-bc25-486d-b53d-c7f0b37ab894\") "
Sep 17 19:06:30.476379 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.476380    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/393eb93a-bc25-486d-b53d-c7f0b37ab894-tmp-dir\") pod \"393eb93a-bc25-486d-b53d-c7f0b37ab894\" (UID: \"393eb93a-bc25-486d-b53d-c7f0b37ab894\") "
Sep 17 19:06:30.476613 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.476423    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkpxq\" (UniqueName: \"kubernetes.io/projected/393eb93a-bc25-486d-b53d-c7f0b37ab894-kube-api-access-qkpxq\") pod \"393eb93a-bc25-486d-b53d-c7f0b37ab894\" (UID: \"393eb93a-bc25-486d-b53d-c7f0b37ab894\") "
Sep 17 19:06:30.476613 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.476470    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/393eb93a-bc25-486d-b53d-c7f0b37ab894-home\") pod \"393eb93a-bc25-486d-b53d-c7f0b37ab894\" (UID: \"393eb93a-bc25-486d-b53d-c7f0b37ab894\") "
Sep 17 19:06:30.476613 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.476485    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/393eb93a-bc25-486d-b53d-c7f0b37ab894-kserve-provision-location\") pod \"393eb93a-bc25-486d-b53d-c7f0b37ab894\" (UID: \"393eb93a-bc25-486d-b53d-c7f0b37ab894\") "
Sep 17 19:06:30.476613 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.476504    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393eb93a-bc25-486d-b53d-c7f0b37ab894-tls-certs\") pod \"393eb93a-bc25-486d-b53d-c7f0b37ab894\" (UID: \"393eb93a-bc25-486d-b53d-c7f0b37ab894\") "
Sep 17 19:06:30.476613 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.476557    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393eb93a-bc25-486d-b53d-c7f0b37ab894-model-cache" (OuterVolumeSpecName: "model-cache") pod "393eb93a-bc25-486d-b53d-c7f0b37ab894" (UID: "393eb93a-bc25-486d-b53d-c7f0b37ab894"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:06:30.476882 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.476782    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/393eb93a-bc25-486d-b53d-c7f0b37ab894-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:06:30.477086 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.477054    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393eb93a-bc25-486d-b53d-c7f0b37ab894-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "393eb93a-bc25-486d-b53d-c7f0b37ab894" (UID: "393eb93a-bc25-486d-b53d-c7f0b37ab894"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:06:30.477312 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.477261    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393eb93a-bc25-486d-b53d-c7f0b37ab894-home" (OuterVolumeSpecName: "home") pod "393eb93a-bc25-486d-b53d-c7f0b37ab894" (UID: "393eb93a-bc25-486d-b53d-c7f0b37ab894"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:06:30.478770 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.478729    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393eb93a-bc25-486d-b53d-c7f0b37ab894-dshm" (OuterVolumeSpecName: "dshm") pod "393eb93a-bc25-486d-b53d-c7f0b37ab894" (UID: "393eb93a-bc25-486d-b53d-c7f0b37ab894"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:06:30.478876 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.478811    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393eb93a-bc25-486d-b53d-c7f0b37ab894-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "393eb93a-bc25-486d-b53d-c7f0b37ab894" (UID: "393eb93a-bc25-486d-b53d-c7f0b37ab894"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:06:30.478876 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.478826    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393eb93a-bc25-486d-b53d-c7f0b37ab894-kube-api-access-qkpxq" (OuterVolumeSpecName: "kube-api-access-qkpxq") pod "393eb93a-bc25-486d-b53d-c7f0b37ab894" (UID: "393eb93a-bc25-486d-b53d-c7f0b37ab894"). InnerVolumeSpecName "kube-api-access-qkpxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:06:30.495679 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.495646    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393eb93a-bc25-486d-b53d-c7f0b37ab894-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "393eb93a-bc25-486d-b53d-c7f0b37ab894" (UID: "393eb93a-bc25-486d-b53d-c7f0b37ab894"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:06:30.577911 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.577891    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/393eb93a-bc25-486d-b53d-c7f0b37ab894-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:06:30.577911 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.577913    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/393eb93a-bc25-486d-b53d-c7f0b37ab894-kserve-provision-location\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:06:30.578052 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.577923    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393eb93a-bc25-486d-b53d-c7f0b37ab894-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:06:30.578052 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.577931    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/393eb93a-bc25-486d-b53d-c7f0b37ab894-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:06:30.578052 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.577939    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/393eb93a-bc25-486d-b53d-c7f0b37ab894-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:06:30.578052 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.577946    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkpxq\" (UniqueName: \"kubernetes.io/projected/393eb93a-bc25-486d-b53d-c7f0b37ab894-kube-api-access-qkpxq\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:06:30.729663 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.729613    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9_393eb93a-bc25-486d-b53d-c7f0b37ab894/modelcar/0.log"
Sep 17 19:06:30.730302 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.730280    2565 generic.go:356] "Generic (PLEG): container finished" podID="393eb93a-bc25-486d-b53d-c7f0b37ab894" containerID="cfbe9d95d060c7afa1cd48df3059b1414b216608a95dd5f69c3f73748f6848ec" exitCode=0
Sep 17 19:06:30.730388 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.730312    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9" event={"ID":"393eb93a-bc25-486d-b53d-c7f0b37ab894","Type":"ContainerDied","Data":"cfbe9d95d060c7afa1cd48df3059b1414b216608a95dd5f69c3f73748f6848ec"}
Sep 17 19:06:30.730388 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.730343    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"
Sep 17 19:06:30.730388 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.730357    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9" event={"ID":"393eb93a-bc25-486d-b53d-c7f0b37ab894","Type":"ContainerDied","Data":"d2774f70b2b41b7216b42f22b8a416199554e24e92f189c9bcc4fbb9297a5461"}
Sep 17 19:06:30.730388 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.730385    2565 scope.go:117] "RemoveContainer" containerID="f1a5aaed4e394dc6102c4711fe4820ea9809fcf3bdba7ef9bcd01048d8d1200b"
Sep 17 19:06:30.739573 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.739551    2565 scope.go:117] "RemoveContainer" containerID="cfbe9d95d060c7afa1cd48df3059b1414b216608a95dd5f69c3f73748f6848ec"
Sep 17 19:06:30.747189 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.747173    2565 scope.go:117] "RemoveContainer" containerID="e1e1d8ac0c0f92899dc9fa5fcdef4f75fad924475b204d52a95846c82ef697f8"
Sep 17 19:06:30.754003 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.753946    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"]
Sep 17 19:06:30.755770 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.755753    2565 scope.go:117] "RemoveContainer" containerID="f1a5aaed4e394dc6102c4711fe4820ea9809fcf3bdba7ef9bcd01048d8d1200b"
Sep 17 19:06:30.756025 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:06:30.756007    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1a5aaed4e394dc6102c4711fe4820ea9809fcf3bdba7ef9bcd01048d8d1200b\": container with ID starting with f1a5aaed4e394dc6102c4711fe4820ea9809fcf3bdba7ef9bcd01048d8d1200b not found: ID does not exist" containerID="f1a5aaed4e394dc6102c4711fe4820ea9809fcf3bdba7ef9bcd01048d8d1200b"
Sep 17 19:06:30.756077 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.756035    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1a5aaed4e394dc6102c4711fe4820ea9809fcf3bdba7ef9bcd01048d8d1200b"} err="failed to get container status \"f1a5aaed4e394dc6102c4711fe4820ea9809fcf3bdba7ef9bcd01048d8d1200b\": rpc error: code = NotFound desc = could not find container \"f1a5aaed4e394dc6102c4711fe4820ea9809fcf3bdba7ef9bcd01048d8d1200b\": container with ID starting with f1a5aaed4e394dc6102c4711fe4820ea9809fcf3bdba7ef9bcd01048d8d1200b not found: ID does not exist"
Sep 17 19:06:30.756077 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.756053    2565 scope.go:117] "RemoveContainer" containerID="cfbe9d95d060c7afa1cd48df3059b1414b216608a95dd5f69c3f73748f6848ec"
Sep 17 19:06:30.756401 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:06:30.756383    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfbe9d95d060c7afa1cd48df3059b1414b216608a95dd5f69c3f73748f6848ec\": container with ID starting with cfbe9d95d060c7afa1cd48df3059b1414b216608a95dd5f69c3f73748f6848ec not found: ID does not exist" containerID="cfbe9d95d060c7afa1cd48df3059b1414b216608a95dd5f69c3f73748f6848ec"
Sep 17 19:06:30.756462 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.756406    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfbe9d95d060c7afa1cd48df3059b1414b216608a95dd5f69c3f73748f6848ec"} err="failed to get container status \"cfbe9d95d060c7afa1cd48df3059b1414b216608a95dd5f69c3f73748f6848ec\": rpc error: code = NotFound desc = could not find container \"cfbe9d95d060c7afa1cd48df3059b1414b216608a95dd5f69c3f73748f6848ec\": container with ID starting with cfbe9d95d060c7afa1cd48df3059b1414b216608a95dd5f69c3f73748f6848ec not found: ID does not exist"
Sep 17 19:06:30.756462 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.756420    2565 scope.go:117] "RemoveContainer" containerID="e1e1d8ac0c0f92899dc9fa5fcdef4f75fad924475b204d52a95846c82ef697f8"
Sep 17 19:06:30.756626 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.756606    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9"]
Sep 17 19:06:30.756672 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:06:30.756648    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1e1d8ac0c0f92899dc9fa5fcdef4f75fad924475b204d52a95846c82ef697f8\": container with ID starting with e1e1d8ac0c0f92899dc9fa5fcdef4f75fad924475b204d52a95846c82ef697f8 not found: ID does not exist" containerID="e1e1d8ac0c0f92899dc9fa5fcdef4f75fad924475b204d52a95846c82ef697f8"
Sep 17 19:06:30.756708 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:30.756667    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1e1d8ac0c0f92899dc9fa5fcdef4f75fad924475b204d52a95846c82ef697f8"} err="failed to get container status \"e1e1d8ac0c0f92899dc9fa5fcdef4f75fad924475b204d52a95846c82ef697f8\": rpc error: code = NotFound desc = could not find container \"e1e1d8ac0c0f92899dc9fa5fcdef4f75fad924475b204d52a95846c82ef697f8\": container with ID starting with e1e1d8ac0c0f92899dc9fa5fcdef4f75fad924475b204d52a95846c82ef697f8 not found: ID does not exist"
Sep 17 19:06:31.355759 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:31.355718    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dctwv9" podUID="393eb93a-bc25-486d-b53d-c7f0b37ab894" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 19:06:32.607064 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:06:32.607029    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="393eb93a-bc25-486d-b53d-c7f0b37ab894" path="/var/lib/kubelet/pods/393eb93a-bc25-486d-b53d-c7f0b37ab894/volumes"
Sep 17 19:07:10.459821 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.459788    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"]
Sep 17 19:07:10.460271 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.460137    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="393eb93a-bc25-486d-b53d-c7f0b37ab894" containerName="main"
Sep 17 19:07:10.460271 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.460148    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="393eb93a-bc25-486d-b53d-c7f0b37ab894" containerName="main"
Sep 17 19:07:10.460271 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.460160    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="393eb93a-bc25-486d-b53d-c7f0b37ab894" containerName="modelcar"
Sep 17 19:07:10.460271 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.460167    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="393eb93a-bc25-486d-b53d-c7f0b37ab894" containerName="modelcar"
Sep 17 19:07:10.460271 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.460185    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="393eb93a-bc25-486d-b53d-c7f0b37ab894" containerName="modelcar-init"
Sep 17 19:07:10.460271 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.460191    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="393eb93a-bc25-486d-b53d-c7f0b37ab894" containerName="modelcar-init"
Sep 17 19:07:10.460271 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.460254    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="393eb93a-bc25-486d-b53d-c7f0b37ab894" containerName="main"
Sep 17 19:07:10.460271 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.460262    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="393eb93a-bc25-486d-b53d-c7f0b37ab894" containerName="modelcar"
Sep 17 19:07:10.463368 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.463341    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:07:10.470165 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.470141    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Sep 17 19:07:10.470389 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.470361    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:07:10.470773 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.470755    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-9tppj\""
Sep 17 19:07:10.470939 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.470922    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 17 19:07:10.471096 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.471077    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 17 19:07:10.477055 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.477036    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 17 19:07:10.500718 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.500689    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"]
Sep 17 19:07:10.569918 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.569892    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61cad67c-e2ce-4de1-855f-ba213ea50071-cabundle-cert\") pod \"custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:07:10.570024 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.569923    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61cad67c-e2ce-4de1-855f-ba213ea50071-dshm\") pod \"custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:07:10.570024 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.569955    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61cad67c-e2ce-4de1-855f-ba213ea50071-tmp-dir\") pod \"custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:07:10.570024 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.569993    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61cad67c-e2ce-4de1-855f-ba213ea50071-tls-certs\") pod \"custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:07:10.570161 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.570028    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61cad67c-e2ce-4de1-855f-ba213ea50071-home\") pod \"custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:07:10.570161 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.570080    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ll4v\" (UniqueName: \"kubernetes.io/projected/61cad67c-e2ce-4de1-855f-ba213ea50071-kube-api-access-7ll4v\") pod \"custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:07:10.570161 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.570103    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61cad67c-e2ce-4de1-855f-ba213ea50071-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:07:10.570161 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.570149    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61cad67c-e2ce-4de1-855f-ba213ea50071-model-cache\") pod \"custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:07:10.671109 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.671079    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ll4v\" (UniqueName: \"kubernetes.io/projected/61cad67c-e2ce-4de1-855f-ba213ea50071-kube-api-access-7ll4v\") pod \"custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:07:10.671275 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.671115    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61cad67c-e2ce-4de1-855f-ba213ea50071-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:07:10.671275 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.671246    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61cad67c-e2ce-4de1-855f-ba213ea50071-model-cache\") pod \"custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:07:10.671403 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.671290    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61cad67c-e2ce-4de1-855f-ba213ea50071-cabundle-cert\") pod \"custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:07:10.671403 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.671321    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61cad67c-e2ce-4de1-855f-ba213ea50071-dshm\") pod \"custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:07:10.671403 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.671357    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61cad67c-e2ce-4de1-855f-ba213ea50071-tmp-dir\") pod \"custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:07:10.671403 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.671385    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61cad67c-e2ce-4de1-855f-ba213ea50071-tls-certs\") pod \"custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:07:10.671619 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.671436    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61cad67c-e2ce-4de1-855f-ba213ea50071-home\") pod \"custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:07:10.671619 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.671583    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61cad67c-e2ce-4de1-855f-ba213ea50071-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:07:10.671698 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.671631    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61cad67c-e2ce-4de1-855f-ba213ea50071-model-cache\") pod \"custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:07:10.671743 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.671714    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61cad67c-e2ce-4de1-855f-ba213ea50071-tmp-dir\") pod \"custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:07:10.671790 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.671773    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61cad67c-e2ce-4de1-855f-ba213ea50071-home\") pod \"custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:07:10.671990 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.671967    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61cad67c-e2ce-4de1-855f-ba213ea50071-cabundle-cert\") pod \"custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:07:10.673682 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.673661    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61cad67c-e2ce-4de1-855f-ba213ea50071-dshm\") pod \"custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:07:10.673877 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.673858    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61cad67c-e2ce-4de1-855f-ba213ea50071-tls-certs\") pod \"custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:07:10.684953 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.684928    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ll4v\" (UniqueName: \"kubernetes.io/projected/61cad67c-e2ce-4de1-855f-ba213ea50071-kube-api-access-7ll4v\") pod \"custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:07:10.773704 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:10.773642    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:07:11.088602 ip-10-0-139-75 kubenswrapper[2565]: W0917 19:07:11.088568    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61cad67c_e2ce_4de1_855f_ba213ea50071.slice/crio-8ea00d538241f87386ba55519e2a1d7f51c22bd3d2c2f4c201fc45905f25bd01 WatchSource:0}: Error finding container 8ea00d538241f87386ba55519e2a1d7f51c22bd3d2c2f4c201fc45905f25bd01: Status 404 returned error can't find the container with id 8ea00d538241f87386ba55519e2a1d7f51c22bd3d2c2f4c201fc45905f25bd01
Sep 17 19:07:11.089044 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:11.088963    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"]
Sep 17 19:07:11.951392 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:11.951352    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf" event={"ID":"61cad67c-e2ce-4de1-855f-ba213ea50071","Type":"ContainerStarted","Data":"e3f3ac1067fc448e93250e2a3e065a60dda831122914f52008f12c0173eb63f0"}
Sep 17 19:07:11.951853 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:11.951419    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf" event={"ID":"61cad67c-e2ce-4de1-855f-ba213ea50071","Type":"ContainerStarted","Data":"8ea00d538241f87386ba55519e2a1d7f51c22bd3d2c2f4c201fc45905f25bd01"}
Sep 17 19:07:13.965278 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:13.965242    2565 generic.go:356] "Generic (PLEG): container finished" podID="61cad67c-e2ce-4de1-855f-ba213ea50071" containerID="e3f3ac1067fc448e93250e2a3e065a60dda831122914f52008f12c0173eb63f0" exitCode=0
Sep 17 19:07:13.965686 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:13.965319    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf" event={"ID":"61cad67c-e2ce-4de1-855f-ba213ea50071","Type":"ContainerDied","Data":"e3f3ac1067fc448e93250e2a3e065a60dda831122914f52008f12c0173eb63f0"}
Sep 17 19:07:14.973105 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:14.973065    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf" event={"ID":"61cad67c-e2ce-4de1-855f-ba213ea50071","Type":"ContainerStarted","Data":"100318897aeb9c8154e68efdd98a726638cf29cb9486ea5b5940f290b0f11972"}
Sep 17 19:07:15.774765 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:15.774726    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:07:20.774729 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:20.774685    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:07:20.776324 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:20.776294    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf" podUID="61cad67c-e2ce-4de1-855f-ba213ea50071" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 17 19:07:21.537632 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:21.537577    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf" podStartSLOduration=9.33403771 podStartE2EDuration="11.537561156s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.203523446s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:21.519098121 +0000 UTC m=+2299.451299662" watchObservedRunningTime="2026-09-17 19:07:21.537561156 +0000 UTC m=+2299.469762675"
Sep 17 19:07:30.774362 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:30.774313    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf" podUID="61cad67c-e2ce-4de1-855f-ba213ea50071" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 17 19:07:40.774909 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:40.774866    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf" podUID="61cad67c-e2ce-4de1-855f-ba213ea50071" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 17 19:07:50.774619 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:07:50.774563    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf" podUID="61cad67c-e2ce-4de1-855f-ba213ea50071" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 17 19:08:00.775009 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:00.774927    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf" podUID="61cad67c-e2ce-4de1-855f-ba213ea50071" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 17 19:08:10.774081 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:10.774041    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf" podUID="61cad67c-e2ce-4de1-855f-ba213ea50071" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 17 19:08:20.773981 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:20.773937    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf" podUID="61cad67c-e2ce-4de1-855f-ba213ea50071" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 17 19:08:30.774338 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:30.774291    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf" podUID="61cad67c-e2ce-4de1-855f-ba213ea50071" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 17 19:08:32.806847 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:32.806811    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"]
Sep 17 19:08:32.810917 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:32.810888    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:08:32.814454 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:32.814429    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 17 19:08:32.814807 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:32.814790    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:08:32.815016 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:32.815000    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 17 19:08:32.815237 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:32.815217    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-dtdtq\""
Sep 17 19:08:32.815298 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:32.815241    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 17 19:08:32.815298 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:32.815256    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 17 19:08:32.826313 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:32.826278    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"]
Sep 17 19:08:32.943983 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:32.943924    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce9bb840-5666-4fc2-925d-1784411fc2e2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:08:32.943983 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:32.943990    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce9bb840-5666-4fc2-925d-1784411fc2e2-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:08:32.944322 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:32.944031    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce9bb840-5666-4fc2-925d-1784411fc2e2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:08:32.944322 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:32.944060    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce9bb840-5666-4fc2-925d-1784411fc2e2-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:08:32.944322 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:32.944170    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4b9h\" (UniqueName: \"kubernetes.io/projected/ce9bb840-5666-4fc2-925d-1784411fc2e2-kube-api-access-n4b9h\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:08:32.944322 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:32.944244    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce9bb840-5666-4fc2-925d-1784411fc2e2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:08:32.944322 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:32.944271    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce9bb840-5666-4fc2-925d-1784411fc2e2-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:08:32.944322 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:32.944292    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce9bb840-5666-4fc2-925d-1784411fc2e2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:08:33.045131 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:33.045097    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce9bb840-5666-4fc2-925d-1784411fc2e2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:08:33.045320 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:33.045140    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce9bb840-5666-4fc2-925d-1784411fc2e2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:08:33.045320 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:33.045166    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce9bb840-5666-4fc2-925d-1784411fc2e2-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:08:33.045320 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:33.045191    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce9bb840-5666-4fc2-925d-1784411fc2e2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:08:33.045320 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:33.045245    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce9bb840-5666-4fc2-925d-1784411fc2e2-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:08:33.045320 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:33.045321    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4b9h\" (UniqueName: \"kubernetes.io/projected/ce9bb840-5666-4fc2-925d-1784411fc2e2-kube-api-access-n4b9h\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:08:33.045635 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:33.045372    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce9bb840-5666-4fc2-925d-1784411fc2e2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:08:33.045635 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:33.045397    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce9bb840-5666-4fc2-925d-1784411fc2e2-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:08:33.045635 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:33.045572    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce9bb840-5666-4fc2-925d-1784411fc2e2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:08:33.045835 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:33.045811    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce9bb840-5666-4fc2-925d-1784411fc2e2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:08:33.045905 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:33.045836    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce9bb840-5666-4fc2-925d-1784411fc2e2-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:08:33.045970 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:33.045917    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce9bb840-5666-4fc2-925d-1784411fc2e2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:08:33.045970 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:33.045948    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce9bb840-5666-4fc2-925d-1784411fc2e2-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:08:33.047665 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:33.047639    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce9bb840-5666-4fc2-925d-1784411fc2e2-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:08:33.047989 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:33.047967    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce9bb840-5666-4fc2-925d-1784411fc2e2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:08:33.053718 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:33.053695    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4b9h\" (UniqueName: \"kubernetes.io/projected/ce9bb840-5666-4fc2-925d-1784411fc2e2-kube-api-access-n4b9h\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:08:33.122264 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:33.122160    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:08:33.407121 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:33.407087    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"]
Sep 17 19:08:33.407633 ip-10-0-139-75 kubenswrapper[2565]: W0917 19:08:33.407606    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9bb840_5666_4fc2_925d_1784411fc2e2.slice/crio-89e445a915a3046f24c0512a59ca61ea48b59a9f9d0f81ef7f64486a86343827 WatchSource:0}: Error finding container 89e445a915a3046f24c0512a59ca61ea48b59a9f9d0f81ef7f64486a86343827: Status 404 returned error can't find the container with id 89e445a915a3046f24c0512a59ca61ea48b59a9f9d0f81ef7f64486a86343827
Sep 17 19:08:34.407498 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:34.407463    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd" event={"ID":"ce9bb840-5666-4fc2-925d-1784411fc2e2","Type":"ContainerStarted","Data":"2810b3f168e7b4e327909cd04402866f73e01222af6a4d480c95443759110331"}
Sep 17 19:08:34.407878 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:34.407516    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd" event={"ID":"ce9bb840-5666-4fc2-925d-1784411fc2e2","Type":"ContainerStarted","Data":"89e445a915a3046f24c0512a59ca61ea48b59a9f9d0f81ef7f64486a86343827"}
Sep 17 19:08:37.435649 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:37.435613    2565 generic.go:356] "Generic (PLEG): container finished" podID="ce9bb840-5666-4fc2-925d-1784411fc2e2" containerID="2810b3f168e7b4e327909cd04402866f73e01222af6a4d480c95443759110331" exitCode=0
Sep 17 19:08:37.436057 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:37.435687    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd" event={"ID":"ce9bb840-5666-4fc2-925d-1784411fc2e2","Type":"ContainerDied","Data":"2810b3f168e7b4e327909cd04402866f73e01222af6a4d480c95443759110331"}
Sep 17 19:08:38.445568 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:38.445530    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd" event={"ID":"ce9bb840-5666-4fc2-925d-1784411fc2e2","Type":"ContainerStarted","Data":"2eacd1e1cde71d8f58ad23d89086f026e9adf12a5eebb3129ccd3dfa6e323e81"}
Sep 17 19:08:38.473055 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:38.473002    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd" podStartSLOduration=3.51753102 podStartE2EDuration="6.47298675s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.95545573s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:08:38.469862298 +0000 UTC m=+2376.402063817" watchObservedRunningTime="2026-09-17 19:08:38.47298675 +0000 UTC m=+2376.405188269"
Sep 17 19:08:39.122613 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:39.122569    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:08:40.774286 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:40.774237    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf" podUID="61cad67c-e2ce-4de1-855f-ba213ea50071" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 17 19:08:43.123378 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:43.123336    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:08:43.124994 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:43.124959    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd" podUID="ce9bb840-5666-4fc2-925d-1784411fc2e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 19:08:50.775046 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:50.775000    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf" podUID="61cad67c-e2ce-4de1-855f-ba213ea50071" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 17 19:08:53.123499 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:08:53.123457    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd" podUID="ce9bb840-5666-4fc2-925d-1784411fc2e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 19:09:00.774337 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:00.774283    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf" podUID="61cad67c-e2ce-4de1-855f-ba213ea50071" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 17 19:09:03.123343 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:03.123294    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd" podUID="ce9bb840-5666-4fc2-925d-1784411fc2e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 19:09:10.784309 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:10.784274    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:09:10.792597 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:10.792575    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:09:13.123742 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:13.123686    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd" podUID="ce9bb840-5666-4fc2-925d-1784411fc2e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 19:09:15.535377 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:15.535339    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"]
Sep 17 19:09:23.122789 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:23.122745    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd" podUID="ce9bb840-5666-4fc2-925d-1784411fc2e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 19:09:30.562768 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:30.562624    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf" podUID="61cad67c-e2ce-4de1-855f-ba213ea50071" containerName="main" containerID="cri-o://100318897aeb9c8154e68efdd98a726638cf29cb9486ea5b5940f290b0f11972" gracePeriod=45
Sep 17 19:09:30.785621 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:30.785575    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf" podUID="61cad67c-e2ce-4de1-855f-ba213ea50071" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 17 19:09:31.310566 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.310544    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:09:31.466721 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.466685    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61cad67c-e2ce-4de1-855f-ba213ea50071-tls-certs\") pod \"61cad67c-e2ce-4de1-855f-ba213ea50071\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") "
Sep 17 19:09:31.466887 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.466741    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61cad67c-e2ce-4de1-855f-ba213ea50071-cabundle-cert\") pod \"61cad67c-e2ce-4de1-855f-ba213ea50071\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") "
Sep 17 19:09:31.466887 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.466779    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61cad67c-e2ce-4de1-855f-ba213ea50071-tmp-dir\") pod \"61cad67c-e2ce-4de1-855f-ba213ea50071\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") "
Sep 17 19:09:31.466887 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.466813    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61cad67c-e2ce-4de1-855f-ba213ea50071-dshm\") pod \"61cad67c-e2ce-4de1-855f-ba213ea50071\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") "
Sep 17 19:09:31.466887 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.466840    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61cad67c-e2ce-4de1-855f-ba213ea50071-home\") pod \"61cad67c-e2ce-4de1-855f-ba213ea50071\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") "
Sep 17 19:09:31.466887 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.466868    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ll4v\" (UniqueName: \"kubernetes.io/projected/61cad67c-e2ce-4de1-855f-ba213ea50071-kube-api-access-7ll4v\") pod \"61cad67c-e2ce-4de1-855f-ba213ea50071\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") "
Sep 17 19:09:31.467187 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.466942    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61cad67c-e2ce-4de1-855f-ba213ea50071-kserve-provision-location\") pod \"61cad67c-e2ce-4de1-855f-ba213ea50071\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") "
Sep 17 19:09:31.467187 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.466984    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61cad67c-e2ce-4de1-855f-ba213ea50071-model-cache\") pod \"61cad67c-e2ce-4de1-855f-ba213ea50071\" (UID: \"61cad67c-e2ce-4de1-855f-ba213ea50071\") "
Sep 17 19:09:31.467564 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.467527    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61cad67c-e2ce-4de1-855f-ba213ea50071-model-cache" (OuterVolumeSpecName: "model-cache") pod "61cad67c-e2ce-4de1-855f-ba213ea50071" (UID: "61cad67c-e2ce-4de1-855f-ba213ea50071"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:09:31.468927 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.468894    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61cad67c-e2ce-4de1-855f-ba213ea50071-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "61cad67c-e2ce-4de1-855f-ba213ea50071" (UID: "61cad67c-e2ce-4de1-855f-ba213ea50071"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:09:31.469125 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.469096    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61cad67c-e2ce-4de1-855f-ba213ea50071-dshm" (OuterVolumeSpecName: "dshm") pod "61cad67c-e2ce-4de1-855f-ba213ea50071" (UID: "61cad67c-e2ce-4de1-855f-ba213ea50071"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:09:31.469445 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.469404    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61cad67c-e2ce-4de1-855f-ba213ea50071-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "61cad67c-e2ce-4de1-855f-ba213ea50071" (UID: "61cad67c-e2ce-4de1-855f-ba213ea50071"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:09:31.469557 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.469526    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61cad67c-e2ce-4de1-855f-ba213ea50071-kube-api-access-7ll4v" (OuterVolumeSpecName: "kube-api-access-7ll4v") pod "61cad67c-e2ce-4de1-855f-ba213ea50071" (UID: "61cad67c-e2ce-4de1-855f-ba213ea50071"). InnerVolumeSpecName "kube-api-access-7ll4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:09:31.469909 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.469882    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61cad67c-e2ce-4de1-855f-ba213ea50071-home" (OuterVolumeSpecName: "home") pod "61cad67c-e2ce-4de1-855f-ba213ea50071" (UID: "61cad67c-e2ce-4de1-855f-ba213ea50071"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:09:31.474882 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.474853    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61cad67c-e2ce-4de1-855f-ba213ea50071-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "61cad67c-e2ce-4de1-855f-ba213ea50071" (UID: "61cad67c-e2ce-4de1-855f-ba213ea50071"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:09:31.479905 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.479885    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61cad67c-e2ce-4de1-855f-ba213ea50071-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "61cad67c-e2ce-4de1-855f-ba213ea50071" (UID: "61cad67c-e2ce-4de1-855f-ba213ea50071"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:09:31.567751 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.567709    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61cad67c-e2ce-4de1-855f-ba213ea50071-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:09:31.567751 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.567734    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61cad67c-e2ce-4de1-855f-ba213ea50071-cabundle-cert\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:09:31.567751 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.567743    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61cad67c-e2ce-4de1-855f-ba213ea50071-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:09:31.567751 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.567751    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61cad67c-e2ce-4de1-855f-ba213ea50071-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:09:31.567751 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.567760    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61cad67c-e2ce-4de1-855f-ba213ea50071-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:09:31.568432 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.567768    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ll4v\" (UniqueName: \"kubernetes.io/projected/61cad67c-e2ce-4de1-855f-ba213ea50071-kube-api-access-7ll4v\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:09:31.568432 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.567777    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61cad67c-e2ce-4de1-855f-ba213ea50071-kserve-provision-location\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:09:31.568432 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.567785    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61cad67c-e2ce-4de1-855f-ba213ea50071-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:09:31.768294 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.768187    2565 generic.go:356] "Generic (PLEG): container finished" podID="61cad67c-e2ce-4de1-855f-ba213ea50071" containerID="100318897aeb9c8154e68efdd98a726638cf29cb9486ea5b5940f290b0f11972" exitCode=0
Sep 17 19:09:31.768294 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.768239    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf" event={"ID":"61cad67c-e2ce-4de1-855f-ba213ea50071","Type":"ContainerDied","Data":"100318897aeb9c8154e68efdd98a726638cf29cb9486ea5b5940f290b0f11972"}
Sep 17 19:09:31.768294 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.768271    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"
Sep 17 19:09:31.768294 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.768290    2565 scope.go:117] "RemoveContainer" containerID="100318897aeb9c8154e68efdd98a726638cf29cb9486ea5b5940f290b0f11972"
Sep 17 19:09:31.768653 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.768281    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf" event={"ID":"61cad67c-e2ce-4de1-855f-ba213ea50071","Type":"ContainerDied","Data":"8ea00d538241f87386ba55519e2a1d7f51c22bd3d2c2f4c201fc45905f25bd01"}
Sep 17 19:09:31.779851 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.779835    2565 scope.go:117] "RemoveContainer" containerID="e3f3ac1067fc448e93250e2a3e065a60dda831122914f52008f12c0173eb63f0"
Sep 17 19:09:31.788121 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.788103    2565 scope.go:117] "RemoveContainer" containerID="100318897aeb9c8154e68efdd98a726638cf29cb9486ea5b5940f290b0f11972"
Sep 17 19:09:31.788405 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:09:31.788383    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"100318897aeb9c8154e68efdd98a726638cf29cb9486ea5b5940f290b0f11972\": container with ID starting with 100318897aeb9c8154e68efdd98a726638cf29cb9486ea5b5940f290b0f11972 not found: ID does not exist" containerID="100318897aeb9c8154e68efdd98a726638cf29cb9486ea5b5940f290b0f11972"
Sep 17 19:09:31.788497 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.788413    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"100318897aeb9c8154e68efdd98a726638cf29cb9486ea5b5940f290b0f11972"} err="failed to get container status \"100318897aeb9c8154e68efdd98a726638cf29cb9486ea5b5940f290b0f11972\": rpc error: code = NotFound desc = could not find container \"100318897aeb9c8154e68efdd98a726638cf29cb9486ea5b5940f290b0f11972\": container with ID starting with 100318897aeb9c8154e68efdd98a726638cf29cb9486ea5b5940f290b0f11972 not found: ID does not exist"
Sep 17 19:09:31.788497 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.788436    2565 scope.go:117] "RemoveContainer" containerID="e3f3ac1067fc448e93250e2a3e065a60dda831122914f52008f12c0173eb63f0"
Sep 17 19:09:31.788735 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:09:31.788709    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3f3ac1067fc448e93250e2a3e065a60dda831122914f52008f12c0173eb63f0\": container with ID starting with e3f3ac1067fc448e93250e2a3e065a60dda831122914f52008f12c0173eb63f0 not found: ID does not exist" containerID="e3f3ac1067fc448e93250e2a3e065a60dda831122914f52008f12c0173eb63f0"
Sep 17 19:09:31.788803 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.788742    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3f3ac1067fc448e93250e2a3e065a60dda831122914f52008f12c0173eb63f0"} err="failed to get container status \"e3f3ac1067fc448e93250e2a3e065a60dda831122914f52008f12c0173eb63f0\": rpc error: code = NotFound desc = could not find container \"e3f3ac1067fc448e93250e2a3e065a60dda831122914f52008f12c0173eb63f0\": container with ID starting with e3f3ac1067fc448e93250e2a3e065a60dda831122914f52008f12c0173eb63f0 not found: ID does not exist"
Sep 17 19:09:31.791754 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.791728    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"]
Sep 17 19:09:31.796741 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:31.796720    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-76c9d5c7c4-sw8qf"]
Sep 17 19:09:32.607311 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:32.607282    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61cad67c-e2ce-4de1-855f-ba213ea50071" path="/var/lib/kubelet/pods/61cad67c-e2ce-4de1-855f-ba213ea50071/volumes"
Sep 17 19:09:33.122761 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:33.122718    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd" podUID="ce9bb840-5666-4fc2-925d-1784411fc2e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 19:09:43.123182 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:43.123137    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd" podUID="ce9bb840-5666-4fc2-925d-1784411fc2e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 19:09:53.132999 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:53.132967    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:09:53.140619 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:53.140593    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:09:54.357579 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:09:54.357542    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"]
Sep 17 19:10:05.802878 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:05.802846    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"]
Sep 17 19:10:05.803308 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:05.803232    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61cad67c-e2ce-4de1-855f-ba213ea50071" containerName="main"
Sep 17 19:10:05.803308 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:05.803245    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="61cad67c-e2ce-4de1-855f-ba213ea50071" containerName="main"
Sep 17 19:10:05.803308 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:05.803256    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61cad67c-e2ce-4de1-855f-ba213ea50071" containerName="storage-initializer"
Sep 17 19:10:05.803308 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:05.803262    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="61cad67c-e2ce-4de1-855f-ba213ea50071" containerName="storage-initializer"
Sep 17 19:10:05.803453 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:05.803316    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="61cad67c-e2ce-4de1-855f-ba213ea50071" containerName="main"
Sep 17 19:10:05.807845 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:05.807825    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:05.810503 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:05.810471    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Sep 17 19:10:05.810757 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:05.810733    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 17 19:10:05.811021 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:05.810999    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 17 19:10:05.811242 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:05.811217    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-m2gjp\""
Sep 17 19:10:05.820858 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:05.820834    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"]
Sep 17 19:10:05.939158 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:05.939133    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-istio-envoy\") pod \"router-gateway-1-openshift-default-cc7cb45dd-jk656\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:05.939302 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:05.939165    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-cc7cb45dd-jk656\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:05.939302 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:05.939186    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhqzr\" (UniqueName: \"kubernetes.io/projected/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-kube-api-access-bhqzr\") pod \"router-gateway-1-openshift-default-cc7cb45dd-jk656\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:05.939302 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:05.939241    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-istio-data\") pod \"router-gateway-1-openshift-default-cc7cb45dd-jk656\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:05.939302 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:05.939274    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-istio-token\") pod \"router-gateway-1-openshift-default-cc7cb45dd-jk656\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:05.939446 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:05.939341    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-credential-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-jk656\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:05.939446 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:05.939377    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-workload-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-jk656\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:05.939446 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:05.939398    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-istio-podinfo\") pod \"router-gateway-1-openshift-default-cc7cb45dd-jk656\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:05.939446 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:05.939433    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-workload-certs\") pod \"router-gateway-1-openshift-default-cc7cb45dd-jk656\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:06.040547 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:06.040520    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-istio-envoy\") pod \"router-gateway-1-openshift-default-cc7cb45dd-jk656\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:06.040667 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:06.040553    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-cc7cb45dd-jk656\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:06.040667 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:06.040574    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhqzr\" (UniqueName: \"kubernetes.io/projected/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-kube-api-access-bhqzr\") pod \"router-gateway-1-openshift-default-cc7cb45dd-jk656\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:06.040667 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:06.040603    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-istio-data\") pod \"router-gateway-1-openshift-default-cc7cb45dd-jk656\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:06.040667 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:06.040620    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-istio-token\") pod \"router-gateway-1-openshift-default-cc7cb45dd-jk656\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:06.040667 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:06.040646    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-credential-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-jk656\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:06.040667 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:06.040664    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-workload-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-jk656\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:06.040982 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:06.040694    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-istio-podinfo\") pod \"router-gateway-1-openshift-default-cc7cb45dd-jk656\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:06.040982 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:06.040727    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-workload-certs\") pod \"router-gateway-1-openshift-default-cc7cb45dd-jk656\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:06.041091 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:06.041019    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-istio-data\") pod \"router-gateway-1-openshift-default-cc7cb45dd-jk656\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:06.041091 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:06.041066    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-credential-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-jk656\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:06.041232 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:06.041113    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-workload-certs\") pod \"router-gateway-1-openshift-default-cc7cb45dd-jk656\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:06.041351 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:06.041327    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-workload-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-jk656\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:06.041430 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:06.041352    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-cc7cb45dd-jk656\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:06.042918 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:06.042892    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-istio-envoy\") pod \"router-gateway-1-openshift-default-cc7cb45dd-jk656\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:06.043446 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:06.043427    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-istio-podinfo\") pod \"router-gateway-1-openshift-default-cc7cb45dd-jk656\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:06.049338 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:06.049317    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-istio-token\") pod \"router-gateway-1-openshift-default-cc7cb45dd-jk656\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:06.049465 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:06.049445    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhqzr\" (UniqueName: \"kubernetes.io/projected/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-kube-api-access-bhqzr\") pod \"router-gateway-1-openshift-default-cc7cb45dd-jk656\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:06.120833 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:06.120778    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:06.404049 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:06.404020    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"]
Sep 17 19:10:06.405600 ip-10-0-139-75 kubenswrapper[2565]: W0917 19:10:06.405559    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0e9e6fe_80d9_4ca7_9e15_1cfb3d7fc546.slice/crio-82ec8dd17ffe008b708ab30858cdb8ab3ed0f446837f2a8976d198a74e6593ff WatchSource:0}: Error finding container 82ec8dd17ffe008b708ab30858cdb8ab3ed0f446837f2a8976d198a74e6593ff: Status 404 returned error can't find the container with id 82ec8dd17ffe008b708ab30858cdb8ab3ed0f446837f2a8976d198a74e6593ff
Sep 17 19:10:06.407875 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:06.407826    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 17 19:10:06.407961 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:06.407930    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 17 19:10:06.408005 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:06.407980    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 17 19:10:06.963495 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:06.963443    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656" event={"ID":"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546","Type":"ContainerStarted","Data":"d9d3aa23d19048b16477042f82239a23d230df26b048f5998a99942ca846bc06"}
Sep 17 19:10:06.964053 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:06.963524    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656" event={"ID":"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546","Type":"ContainerStarted","Data":"82ec8dd17ffe008b708ab30858cdb8ab3ed0f446837f2a8976d198a74e6593ff"}
Sep 17 19:10:06.989616 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:06.989564    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656" podStartSLOduration=1.9895492799999999 podStartE2EDuration="1.98954928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:10:06.986777052 +0000 UTC m=+2464.918978569" watchObservedRunningTime="2026-09-17 19:10:06.98954928 +0000 UTC m=+2464.921750800"
Sep 17 19:10:07.121345 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:07.121309    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:07.122941 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:07.122908    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656" podUID="c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 17 19:10:08.121671 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:08.121629    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656" podUID="c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 17 19:10:09.121646 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:09.121605    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656" podUID="c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 17 19:10:09.916163 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:09.916115    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd" podUID="ce9bb840-5666-4fc2-925d-1784411fc2e2" containerName="main" containerID="cri-o://2eacd1e1cde71d8f58ad23d89086f026e9adf12a5eebb3129ccd3dfa6e323e81" gracePeriod=45
Sep 17 19:10:10.121447 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.121406    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656" podUID="c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 17 19:10:10.133784 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.133750    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd" podUID="ce9bb840-5666-4fc2-925d-1784411fc2e2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 17 19:10:10.613367 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.613307    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:10:10.685376 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.685336    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce9bb840-5666-4fc2-925d-1784411fc2e2-home\") pod \"ce9bb840-5666-4fc2-925d-1784411fc2e2\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") "
Sep 17 19:10:10.685376 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.685374    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce9bb840-5666-4fc2-925d-1784411fc2e2-tmp-dir\") pod \"ce9bb840-5666-4fc2-925d-1784411fc2e2\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") "
Sep 17 19:10:10.685633 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.685401    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4b9h\" (UniqueName: \"kubernetes.io/projected/ce9bb840-5666-4fc2-925d-1784411fc2e2-kube-api-access-n4b9h\") pod \"ce9bb840-5666-4fc2-925d-1784411fc2e2\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") "
Sep 17 19:10:10.685633 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.685489    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce9bb840-5666-4fc2-925d-1784411fc2e2-model-cache\") pod \"ce9bb840-5666-4fc2-925d-1784411fc2e2\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") "
Sep 17 19:10:10.685633 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.685523    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce9bb840-5666-4fc2-925d-1784411fc2e2-tls-certs\") pod \"ce9bb840-5666-4fc2-925d-1784411fc2e2\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") "
Sep 17 19:10:10.685633 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.685549    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce9bb840-5666-4fc2-925d-1784411fc2e2-cabundle-cert\") pod \"ce9bb840-5666-4fc2-925d-1784411fc2e2\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") "
Sep 17 19:10:10.685866 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.685640    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce9bb840-5666-4fc2-925d-1784411fc2e2-dshm\") pod \"ce9bb840-5666-4fc2-925d-1784411fc2e2\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") "
Sep 17 19:10:10.685866 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.685676    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce9bb840-5666-4fc2-925d-1784411fc2e2-kserve-provision-location\") pod \"ce9bb840-5666-4fc2-925d-1784411fc2e2\" (UID: \"ce9bb840-5666-4fc2-925d-1784411fc2e2\") "
Sep 17 19:10:10.685866 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.685734    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9bb840-5666-4fc2-925d-1784411fc2e2-model-cache" (OuterVolumeSpecName: "model-cache") pod "ce9bb840-5666-4fc2-925d-1784411fc2e2" (UID: "ce9bb840-5666-4fc2-925d-1784411fc2e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:10:10.685866 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.685748    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9bb840-5666-4fc2-925d-1784411fc2e2-home" (OuterVolumeSpecName: "home") pod "ce9bb840-5666-4fc2-925d-1784411fc2e2" (UID: "ce9bb840-5666-4fc2-925d-1784411fc2e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:10:10.685866 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.685844    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9bb840-5666-4fc2-925d-1784411fc2e2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ce9bb840-5666-4fc2-925d-1784411fc2e2" (UID: "ce9bb840-5666-4fc2-925d-1784411fc2e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:10:10.686147 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.685976    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce9bb840-5666-4fc2-925d-1784411fc2e2-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:10:10.686147 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.685996    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce9bb840-5666-4fc2-925d-1784411fc2e2-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:10:10.686147 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.686008    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce9bb840-5666-4fc2-925d-1784411fc2e2-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:10:10.686147 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.686010    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce9bb840-5666-4fc2-925d-1784411fc2e2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ce9bb840-5666-4fc2-925d-1784411fc2e2" (UID: "ce9bb840-5666-4fc2-925d-1784411fc2e2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:10:10.687909 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.687881    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce9bb840-5666-4fc2-925d-1784411fc2e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ce9bb840-5666-4fc2-925d-1784411fc2e2" (UID: "ce9bb840-5666-4fc2-925d-1784411fc2e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:10:10.688127 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.688111    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9bb840-5666-4fc2-925d-1784411fc2e2-dshm" (OuterVolumeSpecName: "dshm") pod "ce9bb840-5666-4fc2-925d-1784411fc2e2" (UID: "ce9bb840-5666-4fc2-925d-1784411fc2e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:10:10.688236 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.688221    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce9bb840-5666-4fc2-925d-1784411fc2e2-kube-api-access-n4b9h" (OuterVolumeSpecName: "kube-api-access-n4b9h") pod "ce9bb840-5666-4fc2-925d-1784411fc2e2" (UID: "ce9bb840-5666-4fc2-925d-1784411fc2e2"). InnerVolumeSpecName "kube-api-access-n4b9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:10:10.708844 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.708816    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9bb840-5666-4fc2-925d-1784411fc2e2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ce9bb840-5666-4fc2-925d-1784411fc2e2" (UID: "ce9bb840-5666-4fc2-925d-1784411fc2e2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:10:10.786431 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.786397    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce9bb840-5666-4fc2-925d-1784411fc2e2-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:10:10.786431 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.786425    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce9bb840-5666-4fc2-925d-1784411fc2e2-cabundle-cert\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:10:10.786431 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.786435    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce9bb840-5666-4fc2-925d-1784411fc2e2-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:10:10.786696 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.786444    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce9bb840-5666-4fc2-925d-1784411fc2e2-kserve-provision-location\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:10:10.786696 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.786453    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4b9h\" (UniqueName: \"kubernetes.io/projected/ce9bb840-5666-4fc2-925d-1784411fc2e2-kube-api-access-n4b9h\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:10:10.990927 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.990894    2565 generic.go:356] "Generic (PLEG): container finished" podID="ce9bb840-5666-4fc2-925d-1784411fc2e2" containerID="2eacd1e1cde71d8f58ad23d89086f026e9adf12a5eebb3129ccd3dfa6e323e81" exitCode=0
Sep 17 19:10:10.991438 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.990952    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"
Sep 17 19:10:10.991438 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.990974    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd" event={"ID":"ce9bb840-5666-4fc2-925d-1784411fc2e2","Type":"ContainerDied","Data":"2eacd1e1cde71d8f58ad23d89086f026e9adf12a5eebb3129ccd3dfa6e323e81"}
Sep 17 19:10:10.991438 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.991049    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd" event={"ID":"ce9bb840-5666-4fc2-925d-1784411fc2e2","Type":"ContainerDied","Data":"89e445a915a3046f24c0512a59ca61ea48b59a9f9d0f81ef7f64486a86343827"}
Sep 17 19:10:10.991438 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:10.991085    2565 scope.go:117] "RemoveContainer" containerID="2eacd1e1cde71d8f58ad23d89086f026e9adf12a5eebb3129ccd3dfa6e323e81"
Sep 17 19:10:11.000729 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:11.000714    2565 scope.go:117] "RemoveContainer" containerID="2810b3f168e7b4e327909cd04402866f73e01222af6a4d480c95443759110331"
Sep 17 19:10:11.008957 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:11.008930    2565 scope.go:117] "RemoveContainer" containerID="2eacd1e1cde71d8f58ad23d89086f026e9adf12a5eebb3129ccd3dfa6e323e81"
Sep 17 19:10:11.009232 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:10:11.009189    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eacd1e1cde71d8f58ad23d89086f026e9adf12a5eebb3129ccd3dfa6e323e81\": container with ID starting with 2eacd1e1cde71d8f58ad23d89086f026e9adf12a5eebb3129ccd3dfa6e323e81 not found: ID does not exist" containerID="2eacd1e1cde71d8f58ad23d89086f026e9adf12a5eebb3129ccd3dfa6e323e81"
Sep 17 19:10:11.009339 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:11.009240    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eacd1e1cde71d8f58ad23d89086f026e9adf12a5eebb3129ccd3dfa6e323e81"} err="failed to get container status \"2eacd1e1cde71d8f58ad23d89086f026e9adf12a5eebb3129ccd3dfa6e323e81\": rpc error: code = NotFound desc = could not find container \"2eacd1e1cde71d8f58ad23d89086f026e9adf12a5eebb3129ccd3dfa6e323e81\": container with ID starting with 2eacd1e1cde71d8f58ad23d89086f026e9adf12a5eebb3129ccd3dfa6e323e81 not found: ID does not exist"
Sep 17 19:10:11.009339 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:11.009256    2565 scope.go:117] "RemoveContainer" containerID="2810b3f168e7b4e327909cd04402866f73e01222af6a4d480c95443759110331"
Sep 17 19:10:11.009540 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:10:11.009518    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2810b3f168e7b4e327909cd04402866f73e01222af6a4d480c95443759110331\": container with ID starting with 2810b3f168e7b4e327909cd04402866f73e01222af6a4d480c95443759110331 not found: ID does not exist" containerID="2810b3f168e7b4e327909cd04402866f73e01222af6a4d480c95443759110331"
Sep 17 19:10:11.009585 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:11.009549    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2810b3f168e7b4e327909cd04402866f73e01222af6a4d480c95443759110331"} err="failed to get container status \"2810b3f168e7b4e327909cd04402866f73e01222af6a4d480c95443759110331\": rpc error: code = NotFound desc = could not find container \"2810b3f168e7b4e327909cd04402866f73e01222af6a4d480c95443759110331\": container with ID starting with 2810b3f168e7b4e327909cd04402866f73e01222af6a4d480c95443759110331 not found: ID does not exist"
Sep 17 19:10:11.024606 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:11.024581    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"]
Sep 17 19:10:11.037599 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:11.037579    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7b9856b4cd-nkdgd"]
Sep 17 19:10:11.121299 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:11.121262    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656" podUID="c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 17 19:10:12.121497 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:12.121459    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656" podUID="c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 17 19:10:12.606515 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:12.606473    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce9bb840-5666-4fc2-925d-1784411fc2e2" path="/var/lib/kubelet/pods/ce9bb840-5666-4fc2-925d-1784411fc2e2/volumes"
Sep 17 19:10:13.125361 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:13.125331    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:13.125777 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:13.125610    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:13.126314 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:13.126284    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:10:24.531388 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.531343    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"]
Sep 17 19:10:24.531916 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.531822    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce9bb840-5666-4fc2-925d-1784411fc2e2" containerName="main"
Sep 17 19:10:24.531916 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.531841    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce9bb840-5666-4fc2-925d-1784411fc2e2" containerName="main"
Sep 17 19:10:24.531916 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.531850    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce9bb840-5666-4fc2-925d-1784411fc2e2" containerName="storage-initializer"
Sep 17 19:10:24.531916 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.531858    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce9bb840-5666-4fc2-925d-1784411fc2e2" containerName="storage-initializer"
Sep 17 19:10:24.532142 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.531925    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce9bb840-5666-4fc2-925d-1784411fc2e2" containerName="main"
Sep 17 19:10:24.536491 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.536469    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:10:24.538822 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.538793    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 17 19:10:24.538963 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.538844    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-4cc6b\""
Sep 17 19:10:24.538963 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.538925    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 17 19:10:24.539063 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.539014    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:10:24.553493 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.553471    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"]
Sep 17 19:10:24.688911 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.688880    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5679fd4-b807-4f87-b61f-618e07f2dfdd-dshm\") pod \"router-with-refs-test-kserve-79568f98fd-d8k4q\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:10:24.688911 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.688910    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5679fd4-b807-4f87-b61f-618e07f2dfdd-cabundle-cert\") pod \"router-with-refs-test-kserve-79568f98fd-d8k4q\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:10:24.689092 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.688930    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsmnh\" (UniqueName: \"kubernetes.io/projected/a5679fd4-b807-4f87-b61f-618e07f2dfdd-kube-api-access-nsmnh\") pod \"router-with-refs-test-kserve-79568f98fd-d8k4q\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:10:24.689092 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.689016    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5679fd4-b807-4f87-b61f-618e07f2dfdd-home\") pod \"router-with-refs-test-kserve-79568f98fd-d8k4q\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:10:24.689092 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.689048    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5679fd4-b807-4f87-b61f-618e07f2dfdd-model-cache\") pod \"router-with-refs-test-kserve-79568f98fd-d8k4q\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:10:24.689210 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.689102    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5679fd4-b807-4f87-b61f-618e07f2dfdd-tmp-dir\") pod \"router-with-refs-test-kserve-79568f98fd-d8k4q\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:10:24.689210 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.689146    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5679fd4-b807-4f87-b61f-618e07f2dfdd-kserve-provision-location\") pod \"router-with-refs-test-kserve-79568f98fd-d8k4q\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:10:24.689210 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.689166    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5679fd4-b807-4f87-b61f-618e07f2dfdd-tls-certs\") pod \"router-with-refs-test-kserve-79568f98fd-d8k4q\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:10:24.789738 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.789675    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5679fd4-b807-4f87-b61f-618e07f2dfdd-dshm\") pod \"router-with-refs-test-kserve-79568f98fd-d8k4q\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:10:24.789738 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.789704    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5679fd4-b807-4f87-b61f-618e07f2dfdd-cabundle-cert\") pod \"router-with-refs-test-kserve-79568f98fd-d8k4q\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:10:24.789738 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.789720    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsmnh\" (UniqueName: \"kubernetes.io/projected/a5679fd4-b807-4f87-b61f-618e07f2dfdd-kube-api-access-nsmnh\") pod \"router-with-refs-test-kserve-79568f98fd-d8k4q\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:10:24.789988 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.789863    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5679fd4-b807-4f87-b61f-618e07f2dfdd-home\") pod \"router-with-refs-test-kserve-79568f98fd-d8k4q\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:10:24.789988 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.789891    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5679fd4-b807-4f87-b61f-618e07f2dfdd-model-cache\") pod \"router-with-refs-test-kserve-79568f98fd-d8k4q\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:10:24.789988 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.789928    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5679fd4-b807-4f87-b61f-618e07f2dfdd-tmp-dir\") pod \"router-with-refs-test-kserve-79568f98fd-d8k4q\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:10:24.789988 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.789966    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5679fd4-b807-4f87-b61f-618e07f2dfdd-kserve-provision-location\") pod \"router-with-refs-test-kserve-79568f98fd-d8k4q\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:10:24.790220 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.789998    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5679fd4-b807-4f87-b61f-618e07f2dfdd-tls-certs\") pod \"router-with-refs-test-kserve-79568f98fd-d8k4q\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:10:24.790317 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.790288    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5679fd4-b807-4f87-b61f-618e07f2dfdd-home\") pod \"router-with-refs-test-kserve-79568f98fd-d8k4q\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:10:24.790383 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.790333    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5679fd4-b807-4f87-b61f-618e07f2dfdd-model-cache\") pod \"router-with-refs-test-kserve-79568f98fd-d8k4q\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:10:24.790448 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.790377    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5679fd4-b807-4f87-b61f-618e07f2dfdd-kserve-provision-location\") pod \"router-with-refs-test-kserve-79568f98fd-d8k4q\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:10:24.790506 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.790473    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5679fd4-b807-4f87-b61f-618e07f2dfdd-tmp-dir\") pod \"router-with-refs-test-kserve-79568f98fd-d8k4q\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:10:24.790610 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.790588    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5679fd4-b807-4f87-b61f-618e07f2dfdd-cabundle-cert\") pod \"router-with-refs-test-kserve-79568f98fd-d8k4q\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:10:24.792069 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.792048    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5679fd4-b807-4f87-b61f-618e07f2dfdd-dshm\") pod \"router-with-refs-test-kserve-79568f98fd-d8k4q\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:10:24.792381 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.792365    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5679fd4-b807-4f87-b61f-618e07f2dfdd-tls-certs\") pod \"router-with-refs-test-kserve-79568f98fd-d8k4q\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:10:24.797590 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.797568    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsmnh\" (UniqueName: \"kubernetes.io/projected/a5679fd4-b807-4f87-b61f-618e07f2dfdd-kube-api-access-nsmnh\") pod \"router-with-refs-test-kserve-79568f98fd-d8k4q\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:10:24.847559 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:24.847531    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:10:25.143057 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:25.143021    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"]
Sep 17 19:10:25.144364 ip-10-0-139-75 kubenswrapper[2565]: W0917 19:10:25.144335    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5679fd4_b807_4f87_b61f_618e07f2dfdd.slice/crio-5cd10e1b8f14eeef7e20836758915f7fed025827f5d71eddf5d78b670ee85a03 WatchSource:0}: Error finding container 5cd10e1b8f14eeef7e20836758915f7fed025827f5d71eddf5d78b670ee85a03: Status 404 returned error can't find the container with id 5cd10e1b8f14eeef7e20836758915f7fed025827f5d71eddf5d78b670ee85a03
Sep 17 19:10:26.074596 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:26.074553    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q" event={"ID":"a5679fd4-b807-4f87-b61f-618e07f2dfdd","Type":"ContainerStarted","Data":"d4d00dcf23ece0b5663764cea6b9f57c3a041df0d43182862210166e5c78ad0f"}
Sep 17 19:10:26.074994 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:26.074631    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q" event={"ID":"a5679fd4-b807-4f87-b61f-618e07f2dfdd","Type":"ContainerStarted","Data":"5cd10e1b8f14eeef7e20836758915f7fed025827f5d71eddf5d78b670ee85a03"}
Sep 17 19:10:28.087753 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:28.087712    2565 generic.go:356] "Generic (PLEG): container finished" podID="a5679fd4-b807-4f87-b61f-618e07f2dfdd" containerID="d4d00dcf23ece0b5663764cea6b9f57c3a041df0d43182862210166e5c78ad0f" exitCode=0
Sep 17 19:10:28.088124 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:28.087785    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q" event={"ID":"a5679fd4-b807-4f87-b61f-618e07f2dfdd","Type":"ContainerDied","Data":"d4d00dcf23ece0b5663764cea6b9f57c3a041df0d43182862210166e5c78ad0f"}
Sep 17 19:10:29.095398 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:29.095358    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q" event={"ID":"a5679fd4-b807-4f87-b61f-618e07f2dfdd","Type":"ContainerStarted","Data":"528157fc60b4d8de6fe50ae888598ded79d192cecfe2307c3e0e4c2aae60b835"}
Sep 17 19:10:29.127494 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:29.127423    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q" podStartSLOduration=2.978166341 podStartE2EDuration="5.127403977s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.149237636s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:10:29.125812274 +0000 UTC m=+2487.058013792" watchObservedRunningTime="2026-09-17 19:10:29.127403977 +0000 UTC m=+2487.059605498"
Sep 17 19:10:29.848552 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:29.848514    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:10:34.847884 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:34.847832    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:10:34.849235 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:34.849190    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q" podUID="a5679fd4-b807-4f87-b61f-618e07f2dfdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 17 19:10:44.848382 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:44.848333    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q" podUID="a5679fd4-b807-4f87-b61f-618e07f2dfdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 17 19:10:54.848316 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:10:54.848270    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q" podUID="a5679fd4-b807-4f87-b61f-618e07f2dfdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 17 19:11:04.848658 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:11:04.848604    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q" podUID="a5679fd4-b807-4f87-b61f-618e07f2dfdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 17 19:11:14.847976 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:11:14.847924    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q" podUID="a5679fd4-b807-4f87-b61f-618e07f2dfdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 17 19:11:24.848154 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:11:24.848110    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q" podUID="a5679fd4-b807-4f87-b61f-618e07f2dfdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 17 19:11:34.848265 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:11:34.848186    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q" podUID="a5679fd4-b807-4f87-b61f-618e07f2dfdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 17 19:11:44.848575 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:11:44.848530    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q" podUID="a5679fd4-b807-4f87-b61f-618e07f2dfdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 17 19:11:54.848893 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:11:54.848847    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q" podUID="a5679fd4-b807-4f87-b61f-618e07f2dfdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 17 19:12:04.848655 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:04.848614    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q" podUID="a5679fd4-b807-4f87-b61f-618e07f2dfdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 17 19:12:14.848640 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:14.848593    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q" podUID="a5679fd4-b807-4f87-b61f-618e07f2dfdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 17 19:12:24.858014 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:24.857983    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:12:24.865941 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:24.865918    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:12:29.624132 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:29.624098    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"]
Sep 17 19:12:44.647012 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:44.646940    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q" podUID="a5679fd4-b807-4f87-b61f-618e07f2dfdd" containerName="main" containerID="cri-o://528157fc60b4d8de6fe50ae888598ded79d192cecfe2307c3e0e4c2aae60b835" gracePeriod=45
Sep 17 19:12:44.858577 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:44.858533    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q" podUID="a5679fd4-b807-4f87-b61f-618e07f2dfdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 17 19:12:45.392021 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.391995    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:12:45.456732 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.456668    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5679fd4-b807-4f87-b61f-618e07f2dfdd-dshm\") pod \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") "
Sep 17 19:12:45.456732 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.456703    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5679fd4-b807-4f87-b61f-618e07f2dfdd-cabundle-cert\") pod \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") "
Sep 17 19:12:45.456929 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.456751    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5679fd4-b807-4f87-b61f-618e07f2dfdd-home\") pod \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") "
Sep 17 19:12:45.456929 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.456803    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5679fd4-b807-4f87-b61f-618e07f2dfdd-model-cache\") pod \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") "
Sep 17 19:12:45.456929 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.456820    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5679fd4-b807-4f87-b61f-618e07f2dfdd-kserve-provision-location\") pod \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") "
Sep 17 19:12:45.456929 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.456841    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5679fd4-b807-4f87-b61f-618e07f2dfdd-tmp-dir\") pod \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") "
Sep 17 19:12:45.456929 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.456879    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5679fd4-b807-4f87-b61f-618e07f2dfdd-tls-certs\") pod \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") "
Sep 17 19:12:45.456929 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.456928    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsmnh\" (UniqueName: \"kubernetes.io/projected/a5679fd4-b807-4f87-b61f-618e07f2dfdd-kube-api-access-nsmnh\") pod \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\" (UID: \"a5679fd4-b807-4f87-b61f-618e07f2dfdd\") "
Sep 17 19:12:45.457254 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.457104    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5679fd4-b807-4f87-b61f-618e07f2dfdd-model-cache" (OuterVolumeSpecName: "model-cache") pod "a5679fd4-b807-4f87-b61f-618e07f2dfdd" (UID: "a5679fd4-b807-4f87-b61f-618e07f2dfdd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:12:45.457254 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.457247    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5679fd4-b807-4f87-b61f-618e07f2dfdd-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:12:45.457528 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.457476    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5679fd4-b807-4f87-b61f-618e07f2dfdd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a5679fd4-b807-4f87-b61f-618e07f2dfdd" (UID: "a5679fd4-b807-4f87-b61f-618e07f2dfdd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:12:45.457620 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.457574    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5679fd4-b807-4f87-b61f-618e07f2dfdd-home" (OuterVolumeSpecName: "home") pod "a5679fd4-b807-4f87-b61f-618e07f2dfdd" (UID: "a5679fd4-b807-4f87-b61f-618e07f2dfdd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:12:45.459424 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.459397    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5679fd4-b807-4f87-b61f-618e07f2dfdd-dshm" (OuterVolumeSpecName: "dshm") pod "a5679fd4-b807-4f87-b61f-618e07f2dfdd" (UID: "a5679fd4-b807-4f87-b61f-618e07f2dfdd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:12:45.459534 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.459425    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5679fd4-b807-4f87-b61f-618e07f2dfdd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5679fd4-b807-4f87-b61f-618e07f2dfdd" (UID: "a5679fd4-b807-4f87-b61f-618e07f2dfdd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:12:45.459534 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.459475    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5679fd4-b807-4f87-b61f-618e07f2dfdd-kube-api-access-nsmnh" (OuterVolumeSpecName: "kube-api-access-nsmnh") pod "a5679fd4-b807-4f87-b61f-618e07f2dfdd" (UID: "a5679fd4-b807-4f87-b61f-618e07f2dfdd"). InnerVolumeSpecName "kube-api-access-nsmnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:12:45.475028 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.474997    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5679fd4-b807-4f87-b61f-618e07f2dfdd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a5679fd4-b807-4f87-b61f-618e07f2dfdd" (UID: "a5679fd4-b807-4f87-b61f-618e07f2dfdd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:12:45.486712 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.486686    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5679fd4-b807-4f87-b61f-618e07f2dfdd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a5679fd4-b807-4f87-b61f-618e07f2dfdd" (UID: "a5679fd4-b807-4f87-b61f-618e07f2dfdd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:12:45.557949 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.557925    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5679fd4-b807-4f87-b61f-618e07f2dfdd-cabundle-cert\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:12:45.557949 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.557950    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5679fd4-b807-4f87-b61f-618e07f2dfdd-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:12:45.558071 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.557975    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5679fd4-b807-4f87-b61f-618e07f2dfdd-kserve-provision-location\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:12:45.558071 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.557985    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5679fd4-b807-4f87-b61f-618e07f2dfdd-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:12:45.558071 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.557995    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5679fd4-b807-4f87-b61f-618e07f2dfdd-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:12:45.558071 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.558004    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsmnh\" (UniqueName: \"kubernetes.io/projected/a5679fd4-b807-4f87-b61f-618e07f2dfdd-kube-api-access-nsmnh\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:12:45.558071 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.558012    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5679fd4-b807-4f87-b61f-618e07f2dfdd-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:12:45.859100 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.859023    2565 generic.go:356] "Generic (PLEG): container finished" podID="a5679fd4-b807-4f87-b61f-618e07f2dfdd" containerID="528157fc60b4d8de6fe50ae888598ded79d192cecfe2307c3e0e4c2aae60b835" exitCode=0
Sep 17 19:12:45.859100 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.859088    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"
Sep 17 19:12:45.859546 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.859106    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q" event={"ID":"a5679fd4-b807-4f87-b61f-618e07f2dfdd","Type":"ContainerDied","Data":"528157fc60b4d8de6fe50ae888598ded79d192cecfe2307c3e0e4c2aae60b835"}
Sep 17 19:12:45.859546 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.859179    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q" event={"ID":"a5679fd4-b807-4f87-b61f-618e07f2dfdd","Type":"ContainerDied","Data":"5cd10e1b8f14eeef7e20836758915f7fed025827f5d71eddf5d78b670ee85a03"}
Sep 17 19:12:45.859546 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.859222    2565 scope.go:117] "RemoveContainer" containerID="528157fc60b4d8de6fe50ae888598ded79d192cecfe2307c3e0e4c2aae60b835"
Sep 17 19:12:45.869448 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.869429    2565 scope.go:117] "RemoveContainer" containerID="d4d00dcf23ece0b5663764cea6b9f57c3a041df0d43182862210166e5c78ad0f"
Sep 17 19:12:45.879874 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.879854    2565 scope.go:117] "RemoveContainer" containerID="528157fc60b4d8de6fe50ae888598ded79d192cecfe2307c3e0e4c2aae60b835"
Sep 17 19:12:45.880141 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:12:45.880122    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"528157fc60b4d8de6fe50ae888598ded79d192cecfe2307c3e0e4c2aae60b835\": container with ID starting with 528157fc60b4d8de6fe50ae888598ded79d192cecfe2307c3e0e4c2aae60b835 not found: ID does not exist" containerID="528157fc60b4d8de6fe50ae888598ded79d192cecfe2307c3e0e4c2aae60b835"
Sep 17 19:12:45.880189 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.880148    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"528157fc60b4d8de6fe50ae888598ded79d192cecfe2307c3e0e4c2aae60b835"} err="failed to get container status \"528157fc60b4d8de6fe50ae888598ded79d192cecfe2307c3e0e4c2aae60b835\": rpc error: code = NotFound desc = could not find container \"528157fc60b4d8de6fe50ae888598ded79d192cecfe2307c3e0e4c2aae60b835\": container with ID starting with 528157fc60b4d8de6fe50ae888598ded79d192cecfe2307c3e0e4c2aae60b835 not found: ID does not exist"
Sep 17 19:12:45.880189 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.880172    2565 scope.go:117] "RemoveContainer" containerID="d4d00dcf23ece0b5663764cea6b9f57c3a041df0d43182862210166e5c78ad0f"
Sep 17 19:12:45.880535 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:12:45.880510    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4d00dcf23ece0b5663764cea6b9f57c3a041df0d43182862210166e5c78ad0f\": container with ID starting with d4d00dcf23ece0b5663764cea6b9f57c3a041df0d43182862210166e5c78ad0f not found: ID does not exist" containerID="d4d00dcf23ece0b5663764cea6b9f57c3a041df0d43182862210166e5c78ad0f"
Sep 17 19:12:45.880635 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.880546    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4d00dcf23ece0b5663764cea6b9f57c3a041df0d43182862210166e5c78ad0f"} err="failed to get container status \"d4d00dcf23ece0b5663764cea6b9f57c3a041df0d43182862210166e5c78ad0f\": rpc error: code = NotFound desc = could not find container \"d4d00dcf23ece0b5663764cea6b9f57c3a041df0d43182862210166e5c78ad0f\": container with ID starting with d4d00dcf23ece0b5663764cea6b9f57c3a041df0d43182862210166e5c78ad0f not found: ID does not exist"
Sep 17 19:12:45.883399 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.883378    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"]
Sep 17 19:12:45.887525 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:45.887495    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-79568f98fd-d8k4q"]
Sep 17 19:12:46.607235 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:12:46.607185    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5679fd4-b807-4f87-b61f-618e07f2dfdd" path="/var/lib/kubelet/pods/a5679fd4-b807-4f87-b61f-618e07f2dfdd/volumes"
Sep 17 19:13:06.407034 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.407000    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"]
Sep 17 19:13:06.407438 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.407396    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5679fd4-b807-4f87-b61f-618e07f2dfdd" containerName="main"
Sep 17 19:13:06.407438 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.407409    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5679fd4-b807-4f87-b61f-618e07f2dfdd" containerName="main"
Sep 17 19:13:06.407438 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.407424    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5679fd4-b807-4f87-b61f-618e07f2dfdd" containerName="storage-initializer"
Sep 17 19:13:06.407438 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.407430    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5679fd4-b807-4f87-b61f-618e07f2dfdd" containerName="storage-initializer"
Sep 17 19:13:06.407621 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.407492    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5679fd4-b807-4f87-b61f-618e07f2dfdd" containerName="main"
Sep 17 19:13:06.412040 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.412016    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:13:06.414544 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.414520    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-cqxf2\""
Sep 17 19:13:06.414662 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.414520    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 17 19:13:06.414662 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.414527    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:13:06.414662 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.414527    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 17 19:13:06.414904 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.414886    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 17 19:13:06.415130 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.415115    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 17 19:13:06.424243 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.424220    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"]
Sep 17 19:13:06.518216 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.518182    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a355547-32a3-46e7-bdbe-2bdad136df25-tls-certs\") pod \"stop-feature-test-kserve-7d8f57947b-xlcn7\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:13:06.518327 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.518259    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zndc\" (UniqueName: \"kubernetes.io/projected/0a355547-32a3-46e7-bdbe-2bdad136df25-kube-api-access-6zndc\") pod \"stop-feature-test-kserve-7d8f57947b-xlcn7\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:13:06.518327 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.518308    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a355547-32a3-46e7-bdbe-2bdad136df25-kserve-provision-location\") pod \"stop-feature-test-kserve-7d8f57947b-xlcn7\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:13:06.518443 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.518338    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a355547-32a3-46e7-bdbe-2bdad136df25-model-cache\") pod \"stop-feature-test-kserve-7d8f57947b-xlcn7\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:13:06.518512 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.518486    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a355547-32a3-46e7-bdbe-2bdad136df25-dshm\") pod \"stop-feature-test-kserve-7d8f57947b-xlcn7\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:13:06.518570 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.518531    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a355547-32a3-46e7-bdbe-2bdad136df25-home\") pod \"stop-feature-test-kserve-7d8f57947b-xlcn7\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:13:06.518626 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.518578    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a355547-32a3-46e7-bdbe-2bdad136df25-cabundle-cert\") pod \"stop-feature-test-kserve-7d8f57947b-xlcn7\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:13:06.518685 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.518652    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a355547-32a3-46e7-bdbe-2bdad136df25-tmp-dir\") pod \"stop-feature-test-kserve-7d8f57947b-xlcn7\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:13:06.619722 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.619692    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a355547-32a3-46e7-bdbe-2bdad136df25-dshm\") pod \"stop-feature-test-kserve-7d8f57947b-xlcn7\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:13:06.619722 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.619722    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a355547-32a3-46e7-bdbe-2bdad136df25-home\") pod \"stop-feature-test-kserve-7d8f57947b-xlcn7\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:13:06.619918 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.619744    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a355547-32a3-46e7-bdbe-2bdad136df25-cabundle-cert\") pod \"stop-feature-test-kserve-7d8f57947b-xlcn7\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:13:06.619918 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.619774    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a355547-32a3-46e7-bdbe-2bdad136df25-tmp-dir\") pod \"stop-feature-test-kserve-7d8f57947b-xlcn7\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:13:06.619918 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.619813    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a355547-32a3-46e7-bdbe-2bdad136df25-tls-certs\") pod \"stop-feature-test-kserve-7d8f57947b-xlcn7\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:13:06.619918 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.619830    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zndc\" (UniqueName: \"kubernetes.io/projected/0a355547-32a3-46e7-bdbe-2bdad136df25-kube-api-access-6zndc\") pod \"stop-feature-test-kserve-7d8f57947b-xlcn7\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:13:06.619918 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.619856    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a355547-32a3-46e7-bdbe-2bdad136df25-kserve-provision-location\") pod \"stop-feature-test-kserve-7d8f57947b-xlcn7\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:13:06.619918 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.619882    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a355547-32a3-46e7-bdbe-2bdad136df25-model-cache\") pod \"stop-feature-test-kserve-7d8f57947b-xlcn7\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:13:06.620276 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.620176    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a355547-32a3-46e7-bdbe-2bdad136df25-home\") pod \"stop-feature-test-kserve-7d8f57947b-xlcn7\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:13:06.620367 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.620342    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a355547-32a3-46e7-bdbe-2bdad136df25-tmp-dir\") pod \"stop-feature-test-kserve-7d8f57947b-xlcn7\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:13:06.620422 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.620374    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a355547-32a3-46e7-bdbe-2bdad136df25-model-cache\") pod \"stop-feature-test-kserve-7d8f57947b-xlcn7\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:13:06.620484 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.620461    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a355547-32a3-46e7-bdbe-2bdad136df25-kserve-provision-location\") pod \"stop-feature-test-kserve-7d8f57947b-xlcn7\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:13:06.620619 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.620602    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a355547-32a3-46e7-bdbe-2bdad136df25-cabundle-cert\") pod \"stop-feature-test-kserve-7d8f57947b-xlcn7\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:13:06.621952 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.621930    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a355547-32a3-46e7-bdbe-2bdad136df25-dshm\") pod \"stop-feature-test-kserve-7d8f57947b-xlcn7\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:13:06.622301 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.622282    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a355547-32a3-46e7-bdbe-2bdad136df25-tls-certs\") pod \"stop-feature-test-kserve-7d8f57947b-xlcn7\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:13:06.629498 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.629475    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zndc\" (UniqueName: \"kubernetes.io/projected/0a355547-32a3-46e7-bdbe-2bdad136df25-kube-api-access-6zndc\") pod \"stop-feature-test-kserve-7d8f57947b-xlcn7\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:13:06.723508 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:06.723428    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:13:07.007294 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:07.007264    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"]
Sep 17 19:13:07.008512 ip-10-0-139-75 kubenswrapper[2565]: W0917 19:13:07.008484    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a355547_32a3_46e7_bdbe_2bdad136df25.slice/crio-f98af1b93a6632850f6d63cf614b586aeef344cf5c7af110fde6545bfc31e678 WatchSource:0}: Error finding container f98af1b93a6632850f6d63cf614b586aeef344cf5c7af110fde6545bfc31e678: Status 404 returned error can't find the container with id f98af1b93a6632850f6d63cf614b586aeef344cf5c7af110fde6545bfc31e678
Sep 17 19:13:07.978220 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:07.978162    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7" event={"ID":"0a355547-32a3-46e7-bdbe-2bdad136df25","Type":"ContainerStarted","Data":"d20650476e49f560be898d61b3141f7215dc584d92ff9d9073aab6b5865e0ddf"}
Sep 17 19:13:07.978649 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:07.978245    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7" event={"ID":"0a355547-32a3-46e7-bdbe-2bdad136df25","Type":"ContainerStarted","Data":"f98af1b93a6632850f6d63cf614b586aeef344cf5c7af110fde6545bfc31e678"}
Sep 17 19:13:09.991222 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:09.991163    2565 generic.go:356] "Generic (PLEG): container finished" podID="0a355547-32a3-46e7-bdbe-2bdad136df25" containerID="d20650476e49f560be898d61b3141f7215dc584d92ff9d9073aab6b5865e0ddf" exitCode=0
Sep 17 19:13:09.991647 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:09.991238    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7" event={"ID":"0a355547-32a3-46e7-bdbe-2bdad136df25","Type":"ContainerDied","Data":"d20650476e49f560be898d61b3141f7215dc584d92ff9d9073aab6b5865e0ddf"}
Sep 17 19:13:10.998529 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:10.998489    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7" event={"ID":"0a355547-32a3-46e7-bdbe-2bdad136df25","Type":"ContainerStarted","Data":"e364467ff83ba26e3237ae6f2a925ccbbdf4028aa55ebaef0c07aaf9eb6346eb"}
Sep 17 19:13:11.025214 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:11.025154    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7" podStartSLOduration=2.914193609 podStartE2EDuration="5.025135065s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.110941456s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:13:11.022044831 +0000 UTC m=+2648.954246351" watchObservedRunningTime="2026-09-17 19:13:11.025135065 +0000 UTC m=+2648.957336586"
Sep 17 19:13:11.724254 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:11.724223    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:13:16.724447 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:16.724412    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:13:16.726042 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:16.726010    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7" podUID="0a355547-32a3-46e7-bdbe-2bdad136df25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 17 19:13:24.643953 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:24.643916    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"]
Sep 17 19:13:24.644457 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:24.644136    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656" podUID="c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546" containerName="istio-proxy" containerID="cri-o://d9d3aa23d19048b16477042f82239a23d230df26b048f5998a99942ca846bc06" gracePeriod=30
Sep 17 19:13:26.724506 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:26.724453    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7" podUID="0a355547-32a3-46e7-bdbe-2bdad136df25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 17 19:13:28.128286 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:28.128242    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656" podUID="c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:13:29.220859 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.220818    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"]
Sep 17 19:13:29.224337 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.224308    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:13:29.229112 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.229086    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:13:29.229483 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.229453    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 17 19:13:29.229622 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.229595    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 17 19:13:29.230162 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.230117    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 17 19:13:29.230303 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.230126    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 17 19:13:29.230303 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.230247    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-qsj2m\""
Sep 17 19:13:29.243886 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.243859    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"]
Sep 17 19:13:29.409619 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.409577    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef265e97-434b-4fa2-808b-1d559395a15f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:13:29.409801 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.409667    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef265e97-434b-4fa2-808b-1d559395a15f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:13:29.409801 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.409759    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef265e97-434b-4fa2-808b-1d559395a15f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:13:29.409886 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.409813    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef265e97-434b-4fa2-808b-1d559395a15f-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:13:29.409925 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.409885    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef265e97-434b-4fa2-808b-1d559395a15f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:13:29.409961 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.409919    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef265e97-434b-4fa2-808b-1d559395a15f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:13:29.409961 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.409945    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2w58\" (UniqueName: \"kubernetes.io/projected/ef265e97-434b-4fa2-808b-1d559395a15f-kube-api-access-j2w58\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:13:29.410029 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.409974    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef265e97-434b-4fa2-808b-1d559395a15f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:13:29.511286 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.511181    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef265e97-434b-4fa2-808b-1d559395a15f-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:13:29.511286 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.511269    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef265e97-434b-4fa2-808b-1d559395a15f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:13:29.511523 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.511291    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef265e97-434b-4fa2-808b-1d559395a15f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:13:29.511523 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.511313    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2w58\" (UniqueName: \"kubernetes.io/projected/ef265e97-434b-4fa2-808b-1d559395a15f-kube-api-access-j2w58\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:13:29.511523 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.511352    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef265e97-434b-4fa2-808b-1d559395a15f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:13:29.511696 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.511514    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef265e97-434b-4fa2-808b-1d559395a15f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:13:29.511696 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.511617    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef265e97-434b-4fa2-808b-1d559395a15f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:13:29.511696 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.511683    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef265e97-434b-4fa2-808b-1d559395a15f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:13:29.511863 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.511700    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef265e97-434b-4fa2-808b-1d559395a15f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:13:29.511863 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.511717    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef265e97-434b-4fa2-808b-1d559395a15f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:13:29.512060 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.512035    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef265e97-434b-4fa2-808b-1d559395a15f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:13:29.512386 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.512366    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef265e97-434b-4fa2-808b-1d559395a15f-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:13:29.512520 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.512384    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef265e97-434b-4fa2-808b-1d559395a15f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:13:29.514810 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.514787    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef265e97-434b-4fa2-808b-1d559395a15f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:13:29.514970 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.514896    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef265e97-434b-4fa2-808b-1d559395a15f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:13:29.528685 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.528655    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2w58\" (UniqueName: \"kubernetes.io/projected/ef265e97-434b-4fa2-808b-1d559395a15f-kube-api-access-j2w58\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:13:29.535580 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.535552    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:13:29.824850 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.824816    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"]
Sep 17 19:13:29.826182 ip-10-0-139-75 kubenswrapper[2565]: W0917 19:13:29.826156    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef265e97_434b_4fa2_808b_1d559395a15f.slice/crio-f2638434ca1532116d71917b5dbe4ba50100eebc99e70c6f621e549fda065601 WatchSource:0}: Error finding container f2638434ca1532116d71917b5dbe4ba50100eebc99e70c6f621e549fda065601: Status 404 returned error can't find the container with id f2638434ca1532116d71917b5dbe4ba50100eebc99e70c6f621e549fda065601
Sep 17 19:13:29.891328 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:29.891303    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:13:30.016874 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.016796    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-workload-socket\") pod \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") "
Sep 17 19:13:30.016874 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.016856    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-istio-token\") pod \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") "
Sep 17 19:13:30.017089 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.016895    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhqzr\" (UniqueName: \"kubernetes.io/projected/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-kube-api-access-bhqzr\") pod \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") "
Sep 17 19:13:30.017089 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.016931    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-workload-certs\") pod \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") "
Sep 17 19:13:30.017089 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.016959    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-istiod-ca-cert\") pod \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") "
Sep 17 19:13:30.017089 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.017035    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-istio-data\") pod \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") "
Sep 17 19:13:30.017089 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.017057    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546" (UID: "c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:13:30.017089 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.017065    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-istio-envoy\") pod \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") "
Sep 17 19:13:30.017445 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.017122    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-istio-podinfo\") pod \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") "
Sep 17 19:13:30.017445 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.017190    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-credential-socket\") pod \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\" (UID: \"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546\") "
Sep 17 19:13:30.017445 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.017226    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546" (UID: "c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:13:30.017622 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.017472    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-workload-socket\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:13:30.017622 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.017490    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-workload-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:13:30.017622 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.017509    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546" (UID: "c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:13:30.017622 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.017549    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-istio-data" (OuterVolumeSpecName: "istio-data") pod "c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546" (UID: "c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:13:30.017841 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.017681    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546" (UID: "c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:13:30.019593 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.019556    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-kube-api-access-bhqzr" (OuterVolumeSpecName: "kube-api-access-bhqzr") pod "c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546" (UID: "c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546"). InnerVolumeSpecName "kube-api-access-bhqzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:13:30.019706 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.019585    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546" (UID: "c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:13:30.019751 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.019728    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-istio-token" (OuterVolumeSpecName: "istio-token") pod "c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546" (UID: "c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:13:30.019791 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.019738    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546" (UID: "c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 19:13:30.112513 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.112466    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb" event={"ID":"ef265e97-434b-4fa2-808b-1d559395a15f","Type":"ContainerStarted","Data":"cfc046b2d6d91631db21abc43361286b2ced683cc32e9629f6503e705ab83866"}
Sep 17 19:13:30.112680 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.112549    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb" event={"ID":"ef265e97-434b-4fa2-808b-1d559395a15f","Type":"ContainerStarted","Data":"f2638434ca1532116d71917b5dbe4ba50100eebc99e70c6f621e549fda065601"}
Sep 17 19:13:30.113891 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.113862    2565 generic.go:356] "Generic (PLEG): container finished" podID="c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546" containerID="d9d3aa23d19048b16477042f82239a23d230df26b048f5998a99942ca846bc06" exitCode=0
Sep 17 19:13:30.114020 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.113901    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656" event={"ID":"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546","Type":"ContainerDied","Data":"d9d3aa23d19048b16477042f82239a23d230df26b048f5998a99942ca846bc06"}
Sep 17 19:13:30.114020 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.113912    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"
Sep 17 19:13:30.114020 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.113948    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656" event={"ID":"c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546","Type":"ContainerDied","Data":"82ec8dd17ffe008b708ab30858cdb8ab3ed0f446837f2a8976d198a74e6593ff"}
Sep 17 19:13:30.114020 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.113985    2565 scope.go:117] "RemoveContainer" containerID="d9d3aa23d19048b16477042f82239a23d230df26b048f5998a99942ca846bc06"
Sep 17 19:13:30.118016 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.117981    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-istio-token\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:13:30.118016 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.118007    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhqzr\" (UniqueName: \"kubernetes.io/projected/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-kube-api-access-bhqzr\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:13:30.118224 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.118025    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-istiod-ca-cert\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:13:30.118224 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.118040    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-istio-data\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:13:30.118224 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.118053    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-istio-envoy\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:13:30.118224 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.118066    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-istio-podinfo\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:13:30.118224 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.118079    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546-credential-socket\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:13:30.123701 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.123676    2565 scope.go:117] "RemoveContainer" containerID="d9d3aa23d19048b16477042f82239a23d230df26b048f5998a99942ca846bc06"
Sep 17 19:13:30.124043 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:13:30.124015    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9d3aa23d19048b16477042f82239a23d230df26b048f5998a99942ca846bc06\": container with ID starting with d9d3aa23d19048b16477042f82239a23d230df26b048f5998a99942ca846bc06 not found: ID does not exist" containerID="d9d3aa23d19048b16477042f82239a23d230df26b048f5998a99942ca846bc06"
Sep 17 19:13:30.124118 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.124056    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9d3aa23d19048b16477042f82239a23d230df26b048f5998a99942ca846bc06"} err="failed to get container status \"d9d3aa23d19048b16477042f82239a23d230df26b048f5998a99942ca846bc06\": rpc error: code = NotFound desc = could not find container \"d9d3aa23d19048b16477042f82239a23d230df26b048f5998a99942ca846bc06\": container with ID starting with d9d3aa23d19048b16477042f82239a23d230df26b048f5998a99942ca846bc06 not found: ID does not exist"
Sep 17 19:13:30.153120 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.153080    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"]
Sep 17 19:13:30.164084 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.164046    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-jk656"]
Sep 17 19:13:30.609029 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:30.608990    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546" path="/var/lib/kubelet/pods/c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546/volumes"
Sep 17 19:13:33.133423 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:33.133389    2565 generic.go:356] "Generic (PLEG): container finished" podID="ef265e97-434b-4fa2-808b-1d559395a15f" containerID="cfc046b2d6d91631db21abc43361286b2ced683cc32e9629f6503e705ab83866" exitCode=0
Sep 17 19:13:33.133844 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:33.133472    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb" event={"ID":"ef265e97-434b-4fa2-808b-1d559395a15f","Type":"ContainerDied","Data":"cfc046b2d6d91631db21abc43361286b2ced683cc32e9629f6503e705ab83866"}
Sep 17 19:13:34.142071 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:34.142033    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb" event={"ID":"ef265e97-434b-4fa2-808b-1d559395a15f","Type":"ContainerStarted","Data":"6b9db24fc72c796f3b5d8a1d2f5cff5faf8d0113b88995b23ca67ddbefedcda6"}
Sep 17 19:13:34.168782 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:34.168728    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb" podStartSLOduration=2.87268325 podStartE2EDuration="5.168712638s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.296029388s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:13:34.166768635 +0000 UTC m=+2672.098970155" watchObservedRunningTime="2026-09-17 19:13:34.168712638 +0000 UTC m=+2672.100914156"
Sep 17 19:13:36.724561 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:36.724513    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7" podUID="0a355547-32a3-46e7-bdbe-2bdad136df25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 17 19:13:39.535978 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:39.535920    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:13:39.537155 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:39.537122    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:13:39.539382 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:39.539349    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb" podUID="ef265e97-434b-4fa2-808b-1d559395a15f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 17 19:13:46.724886 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:46.724833    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7" podUID="0a355547-32a3-46e7-bdbe-2bdad136df25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 17 19:13:49.536446 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:49.536401    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb" podUID="ef265e97-434b-4fa2-808b-1d559395a15f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 17 19:13:56.723892 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:56.723785    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7" podUID="0a355547-32a3-46e7-bdbe-2bdad136df25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 17 19:13:59.536151 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:13:59.536101    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb" podUID="ef265e97-434b-4fa2-808b-1d559395a15f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 17 19:14:06.724562 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:14:06.724514    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7" podUID="0a355547-32a3-46e7-bdbe-2bdad136df25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 17 19:14:09.536400 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:14:09.536353    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb" podUID="ef265e97-434b-4fa2-808b-1d559395a15f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 17 19:14:16.724026 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:14:16.723983    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7" podUID="0a355547-32a3-46e7-bdbe-2bdad136df25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 17 19:14:19.535968 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:14:19.535924    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb" podUID="ef265e97-434b-4fa2-808b-1d559395a15f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 17 19:14:26.724836 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:14:26.724785    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7" podUID="0a355547-32a3-46e7-bdbe-2bdad136df25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 17 19:14:29.536618 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:14:29.536554    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb" podUID="ef265e97-434b-4fa2-808b-1d559395a15f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 17 19:14:36.724768 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:14:36.724709    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7" podUID="0a355547-32a3-46e7-bdbe-2bdad136df25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 17 19:14:39.536043 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:14:39.535989    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb" podUID="ef265e97-434b-4fa2-808b-1d559395a15f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 17 19:14:46.724719 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:14:46.724677    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7" podUID="0a355547-32a3-46e7-bdbe-2bdad136df25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 17 19:14:49.536751 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:14:49.536706    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb" podUID="ef265e97-434b-4fa2-808b-1d559395a15f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 17 19:14:56.724128 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:14:56.724082    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7" podUID="0a355547-32a3-46e7-bdbe-2bdad136df25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 17 19:14:59.536776 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:14:59.536733    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb" podUID="ef265e97-434b-4fa2-808b-1d559395a15f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 17 19:15:06.724497 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:06.724447    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7" podUID="0a355547-32a3-46e7-bdbe-2bdad136df25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 17 19:15:09.536675 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:09.536634    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb" podUID="ef265e97-434b-4fa2-808b-1d559395a15f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 17 19:15:16.733763 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:16.733726    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:15:16.742345 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:16.742306    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:15:18.445632 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:18.445581    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"]
Sep 17 19:15:19.536256 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:19.536195    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb" podUID="ef265e97-434b-4fa2-808b-1d559395a15f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 17 19:15:29.536972 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:29.536878    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb" podUID="ef265e97-434b-4fa2-808b-1d559395a15f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 17 19:15:33.468529 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:33.468489    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7" podUID="0a355547-32a3-46e7-bdbe-2bdad136df25" containerName="main" containerID="cri-o://e364467ff83ba26e3237ae6f2a925ccbbdf4028aa55ebaef0c07aaf9eb6346eb" gracePeriod=45
Sep 17 19:15:33.735332 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:33.735246    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7" podUID="0a355547-32a3-46e7-bdbe-2bdad136df25" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 17 19:15:34.431893 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.431866    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:15:34.505458 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.505381    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a355547-32a3-46e7-bdbe-2bdad136df25-cabundle-cert\") pod \"0a355547-32a3-46e7-bdbe-2bdad136df25\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") "
Sep 17 19:15:34.505458 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.505416    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a355547-32a3-46e7-bdbe-2bdad136df25-dshm\") pod \"0a355547-32a3-46e7-bdbe-2bdad136df25\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") "
Sep 17 19:15:34.505937 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.505506    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zndc\" (UniqueName: \"kubernetes.io/projected/0a355547-32a3-46e7-bdbe-2bdad136df25-kube-api-access-6zndc\") pod \"0a355547-32a3-46e7-bdbe-2bdad136df25\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") "
Sep 17 19:15:34.505937 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.505528    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a355547-32a3-46e7-bdbe-2bdad136df25-tmp-dir\") pod \"0a355547-32a3-46e7-bdbe-2bdad136df25\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") "
Sep 17 19:15:34.505937 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.505557    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a355547-32a3-46e7-bdbe-2bdad136df25-model-cache\") pod \"0a355547-32a3-46e7-bdbe-2bdad136df25\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") "
Sep 17 19:15:34.505937 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.505590    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a355547-32a3-46e7-bdbe-2bdad136df25-home\") pod \"0a355547-32a3-46e7-bdbe-2bdad136df25\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") "
Sep 17 19:15:34.505937 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.505617    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a355547-32a3-46e7-bdbe-2bdad136df25-tls-certs\") pod \"0a355547-32a3-46e7-bdbe-2bdad136df25\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") "
Sep 17 19:15:34.505937 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.505893    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a355547-32a3-46e7-bdbe-2bdad136df25-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0a355547-32a3-46e7-bdbe-2bdad136df25" (UID: "0a355547-32a3-46e7-bdbe-2bdad136df25"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:15:34.506243 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.505943    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a355547-32a3-46e7-bdbe-2bdad136df25-model-cache" (OuterVolumeSpecName: "model-cache") pod "0a355547-32a3-46e7-bdbe-2bdad136df25" (UID: "0a355547-32a3-46e7-bdbe-2bdad136df25"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:15:34.506633 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.506604    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a355547-32a3-46e7-bdbe-2bdad136df25-home" (OuterVolumeSpecName: "home") pod "0a355547-32a3-46e7-bdbe-2bdad136df25" (UID: "0a355547-32a3-46e7-bdbe-2bdad136df25"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:15:34.506759 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.506738    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a355547-32a3-46e7-bdbe-2bdad136df25-kserve-provision-location\") pod \"0a355547-32a3-46e7-bdbe-2bdad136df25\" (UID: \"0a355547-32a3-46e7-bdbe-2bdad136df25\") "
Sep 17 19:15:34.507031 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.507011    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a355547-32a3-46e7-bdbe-2bdad136df25-cabundle-cert\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:15:34.507127 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.507035    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a355547-32a3-46e7-bdbe-2bdad136df25-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:15:34.507127 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.507051    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a355547-32a3-46e7-bdbe-2bdad136df25-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:15:34.507835 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.507807    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a355547-32a3-46e7-bdbe-2bdad136df25-kube-api-access-6zndc" (OuterVolumeSpecName: "kube-api-access-6zndc") pod "0a355547-32a3-46e7-bdbe-2bdad136df25" (UID: "0a355547-32a3-46e7-bdbe-2bdad136df25"). InnerVolumeSpecName "kube-api-access-6zndc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:15:34.508020 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.507997    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a355547-32a3-46e7-bdbe-2bdad136df25-dshm" (OuterVolumeSpecName: "dshm") pod "0a355547-32a3-46e7-bdbe-2bdad136df25" (UID: "0a355547-32a3-46e7-bdbe-2bdad136df25"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:15:34.508566 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.508544    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a355547-32a3-46e7-bdbe-2bdad136df25-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0a355547-32a3-46e7-bdbe-2bdad136df25" (UID: "0a355547-32a3-46e7-bdbe-2bdad136df25"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:15:34.514621 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.514597    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a355547-32a3-46e7-bdbe-2bdad136df25-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0a355547-32a3-46e7-bdbe-2bdad136df25" (UID: "0a355547-32a3-46e7-bdbe-2bdad136df25"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:15:34.524582 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.524543    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a355547-32a3-46e7-bdbe-2bdad136df25-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0a355547-32a3-46e7-bdbe-2bdad136df25" (UID: "0a355547-32a3-46e7-bdbe-2bdad136df25"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:15:34.607816 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.607790    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zndc\" (UniqueName: \"kubernetes.io/projected/0a355547-32a3-46e7-bdbe-2bdad136df25-kube-api-access-6zndc\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:15:34.607816 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.607815    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a355547-32a3-46e7-bdbe-2bdad136df25-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:15:34.607968 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.607826    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a355547-32a3-46e7-bdbe-2bdad136df25-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:15:34.607968 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.607836    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a355547-32a3-46e7-bdbe-2bdad136df25-kserve-provision-location\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:15:34.607968 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.607844    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a355547-32a3-46e7-bdbe-2bdad136df25-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:15:34.842499 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.842406    2565 generic.go:356] "Generic (PLEG): container finished" podID="0a355547-32a3-46e7-bdbe-2bdad136df25" containerID="e364467ff83ba26e3237ae6f2a925ccbbdf4028aa55ebaef0c07aaf9eb6346eb" exitCode=0
Sep 17 19:15:34.842499 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.842464    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"
Sep 17 19:15:34.842499 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.842486    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7" event={"ID":"0a355547-32a3-46e7-bdbe-2bdad136df25","Type":"ContainerDied","Data":"e364467ff83ba26e3237ae6f2a925ccbbdf4028aa55ebaef0c07aaf9eb6346eb"}
Sep 17 19:15:34.842749 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.842546    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7" event={"ID":"0a355547-32a3-46e7-bdbe-2bdad136df25","Type":"ContainerDied","Data":"f98af1b93a6632850f6d63cf614b586aeef344cf5c7af110fde6545bfc31e678"}
Sep 17 19:15:34.842749 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.842571    2565 scope.go:117] "RemoveContainer" containerID="e364467ff83ba26e3237ae6f2a925ccbbdf4028aa55ebaef0c07aaf9eb6346eb"
Sep 17 19:15:34.851849 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.851825    2565 scope.go:117] "RemoveContainer" containerID="d20650476e49f560be898d61b3141f7215dc584d92ff9d9073aab6b5865e0ddf"
Sep 17 19:15:34.860037 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.860013    2565 scope.go:117] "RemoveContainer" containerID="e364467ff83ba26e3237ae6f2a925ccbbdf4028aa55ebaef0c07aaf9eb6346eb"
Sep 17 19:15:34.860387 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:15:34.860364    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e364467ff83ba26e3237ae6f2a925ccbbdf4028aa55ebaef0c07aaf9eb6346eb\": container with ID starting with e364467ff83ba26e3237ae6f2a925ccbbdf4028aa55ebaef0c07aaf9eb6346eb not found: ID does not exist" containerID="e364467ff83ba26e3237ae6f2a925ccbbdf4028aa55ebaef0c07aaf9eb6346eb"
Sep 17 19:15:34.860485 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.860395    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e364467ff83ba26e3237ae6f2a925ccbbdf4028aa55ebaef0c07aaf9eb6346eb"} err="failed to get container status \"e364467ff83ba26e3237ae6f2a925ccbbdf4028aa55ebaef0c07aaf9eb6346eb\": rpc error: code = NotFound desc = could not find container \"e364467ff83ba26e3237ae6f2a925ccbbdf4028aa55ebaef0c07aaf9eb6346eb\": container with ID starting with e364467ff83ba26e3237ae6f2a925ccbbdf4028aa55ebaef0c07aaf9eb6346eb not found: ID does not exist"
Sep 17 19:15:34.860485 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.860412    2565 scope.go:117] "RemoveContainer" containerID="d20650476e49f560be898d61b3141f7215dc584d92ff9d9073aab6b5865e0ddf"
Sep 17 19:15:34.860708 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:15:34.860679    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d20650476e49f560be898d61b3141f7215dc584d92ff9d9073aab6b5865e0ddf\": container with ID starting with d20650476e49f560be898d61b3141f7215dc584d92ff9d9073aab6b5865e0ddf not found: ID does not exist" containerID="d20650476e49f560be898d61b3141f7215dc584d92ff9d9073aab6b5865e0ddf"
Sep 17 19:15:34.860759 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.860712    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d20650476e49f560be898d61b3141f7215dc584d92ff9d9073aab6b5865e0ddf"} err="failed to get container status \"d20650476e49f560be898d61b3141f7215dc584d92ff9d9073aab6b5865e0ddf\": rpc error: code = NotFound desc = could not find container \"d20650476e49f560be898d61b3141f7215dc584d92ff9d9073aab6b5865e0ddf\": container with ID starting with d20650476e49f560be898d61b3141f7215dc584d92ff9d9073aab6b5865e0ddf not found: ID does not exist"
Sep 17 19:15:34.863050 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.863028    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"]
Sep 17 19:15:34.868417 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:34.868392    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7d8f57947b-xlcn7"]
Sep 17 19:15:36.609575 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:36.609541    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a355547-32a3-46e7-bdbe-2bdad136df25" path="/var/lib/kubelet/pods/0a355547-32a3-46e7-bdbe-2bdad136df25/volumes"
Sep 17 19:15:39.536562 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:39.536513    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb" podUID="ef265e97-434b-4fa2-808b-1d559395a15f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 17 19:15:49.545830 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:49.545796    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:15:49.553473 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:49.553444    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:15:58.901457 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:15:58.901416    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"]
Sep 17 19:16:13.923348 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:13.923299    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb" podUID="ef265e97-434b-4fa2-808b-1d559395a15f" containerName="main" containerID="cri-o://6b9db24fc72c796f3b5d8a1d2f5cff5faf8d0113b88995b23ca67ddbefedcda6" gracePeriod=45
Sep 17 19:16:14.684976 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:14.684950    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:16:14.823045 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:14.822976    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef265e97-434b-4fa2-808b-1d559395a15f-tls-certs\") pod \"ef265e97-434b-4fa2-808b-1d559395a15f\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") "
Sep 17 19:16:14.823045 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:14.823011    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef265e97-434b-4fa2-808b-1d559395a15f-home\") pod \"ef265e97-434b-4fa2-808b-1d559395a15f\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") "
Sep 17 19:16:14.823261 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:14.823070    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef265e97-434b-4fa2-808b-1d559395a15f-dshm\") pod \"ef265e97-434b-4fa2-808b-1d559395a15f\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") "
Sep 17 19:16:14.823261 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:14.823101    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef265e97-434b-4fa2-808b-1d559395a15f-tmp-dir\") pod \"ef265e97-434b-4fa2-808b-1d559395a15f\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") "
Sep 17 19:16:14.823261 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:14.823144    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef265e97-434b-4fa2-808b-1d559395a15f-cabundle-cert\") pod \"ef265e97-434b-4fa2-808b-1d559395a15f\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") "
Sep 17 19:16:14.823261 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:14.823223    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef265e97-434b-4fa2-808b-1d559395a15f-kserve-provision-location\") pod \"ef265e97-434b-4fa2-808b-1d559395a15f\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") "
Sep 17 19:16:14.823261 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:14.823250    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2w58\" (UniqueName: \"kubernetes.io/projected/ef265e97-434b-4fa2-808b-1d559395a15f-kube-api-access-j2w58\") pod \"ef265e97-434b-4fa2-808b-1d559395a15f\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") "
Sep 17 19:16:14.823556 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:14.823298    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef265e97-434b-4fa2-808b-1d559395a15f-model-cache\") pod \"ef265e97-434b-4fa2-808b-1d559395a15f\" (UID: \"ef265e97-434b-4fa2-808b-1d559395a15f\") "
Sep 17 19:16:14.823733 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:14.823692    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef265e97-434b-4fa2-808b-1d559395a15f-home" (OuterVolumeSpecName: "home") pod "ef265e97-434b-4fa2-808b-1d559395a15f" (UID: "ef265e97-434b-4fa2-808b-1d559395a15f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:16:14.823833 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:14.823727    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef265e97-434b-4fa2-808b-1d559395a15f-model-cache" (OuterVolumeSpecName: "model-cache") pod "ef265e97-434b-4fa2-808b-1d559395a15f" (UID: "ef265e97-434b-4fa2-808b-1d559395a15f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:16:14.825259 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:14.823637    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef265e97-434b-4fa2-808b-1d559395a15f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ef265e97-434b-4fa2-808b-1d559395a15f" (UID: "ef265e97-434b-4fa2-808b-1d559395a15f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:16:14.825524 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:14.825501    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef265e97-434b-4fa2-808b-1d559395a15f-kube-api-access-j2w58" (OuterVolumeSpecName: "kube-api-access-j2w58") pod "ef265e97-434b-4fa2-808b-1d559395a15f" (UID: "ef265e97-434b-4fa2-808b-1d559395a15f"). InnerVolumeSpecName "kube-api-access-j2w58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:16:14.825679 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:14.825658    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef265e97-434b-4fa2-808b-1d559395a15f-dshm" (OuterVolumeSpecName: "dshm") pod "ef265e97-434b-4fa2-808b-1d559395a15f" (UID: "ef265e97-434b-4fa2-808b-1d559395a15f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:16:14.825741 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:14.825674    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef265e97-434b-4fa2-808b-1d559395a15f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ef265e97-434b-4fa2-808b-1d559395a15f" (UID: "ef265e97-434b-4fa2-808b-1d559395a15f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:16:14.838327 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:14.838293    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef265e97-434b-4fa2-808b-1d559395a15f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ef265e97-434b-4fa2-808b-1d559395a15f" (UID: "ef265e97-434b-4fa2-808b-1d559395a15f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:16:14.844924 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:14.844893    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef265e97-434b-4fa2-808b-1d559395a15f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ef265e97-434b-4fa2-808b-1d559395a15f" (UID: "ef265e97-434b-4fa2-808b-1d559395a15f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:16:14.924312 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:14.924286    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef265e97-434b-4fa2-808b-1d559395a15f-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:16:14.924312 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:14.924309    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef265e97-434b-4fa2-808b-1d559395a15f-cabundle-cert\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:16:14.924629 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:14.924320    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef265e97-434b-4fa2-808b-1d559395a15f-kserve-provision-location\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:16:14.924629 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:14.924329    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2w58\" (UniqueName: \"kubernetes.io/projected/ef265e97-434b-4fa2-808b-1d559395a15f-kube-api-access-j2w58\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:16:14.924629 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:14.924337    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef265e97-434b-4fa2-808b-1d559395a15f-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:16:14.924629 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:14.924346    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef265e97-434b-4fa2-808b-1d559395a15f-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:16:14.924629 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:14.924353    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef265e97-434b-4fa2-808b-1d559395a15f-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:16:14.924629 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:14.924360    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef265e97-434b-4fa2-808b-1d559395a15f-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:16:15.070086 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:15.070056    2565 generic.go:356] "Generic (PLEG): container finished" podID="ef265e97-434b-4fa2-808b-1d559395a15f" containerID="6b9db24fc72c796f3b5d8a1d2f5cff5faf8d0113b88995b23ca67ddbefedcda6" exitCode=0
Sep 17 19:16:15.070227 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:15.070102    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"
Sep 17 19:16:15.070227 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:15.070139    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb" event={"ID":"ef265e97-434b-4fa2-808b-1d559395a15f","Type":"ContainerDied","Data":"6b9db24fc72c796f3b5d8a1d2f5cff5faf8d0113b88995b23ca67ddbefedcda6"}
Sep 17 19:16:15.070227 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:15.070216    2565 scope.go:117] "RemoveContainer" containerID="6b9db24fc72c796f3b5d8a1d2f5cff5faf8d0113b88995b23ca67ddbefedcda6"
Sep 17 19:16:15.070227 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:15.070222    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb" event={"ID":"ef265e97-434b-4fa2-808b-1d559395a15f","Type":"ContainerDied","Data":"f2638434ca1532116d71917b5dbe4ba50100eebc99e70c6f621e549fda065601"}
Sep 17 19:16:15.079494 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:15.079475    2565 scope.go:117] "RemoveContainer" containerID="cfc046b2d6d91631db21abc43361286b2ced683cc32e9629f6503e705ab83866"
Sep 17 19:16:15.087113 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:15.087095    2565 scope.go:117] "RemoveContainer" containerID="6b9db24fc72c796f3b5d8a1d2f5cff5faf8d0113b88995b23ca67ddbefedcda6"
Sep 17 19:16:15.087377 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:16:15.087359    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b9db24fc72c796f3b5d8a1d2f5cff5faf8d0113b88995b23ca67ddbefedcda6\": container with ID starting with 6b9db24fc72c796f3b5d8a1d2f5cff5faf8d0113b88995b23ca67ddbefedcda6 not found: ID does not exist" containerID="6b9db24fc72c796f3b5d8a1d2f5cff5faf8d0113b88995b23ca67ddbefedcda6"
Sep 17 19:16:15.087423 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:15.087385    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9db24fc72c796f3b5d8a1d2f5cff5faf8d0113b88995b23ca67ddbefedcda6"} err="failed to get container status \"6b9db24fc72c796f3b5d8a1d2f5cff5faf8d0113b88995b23ca67ddbefedcda6\": rpc error: code = NotFound desc = could not find container \"6b9db24fc72c796f3b5d8a1d2f5cff5faf8d0113b88995b23ca67ddbefedcda6\": container with ID starting with 6b9db24fc72c796f3b5d8a1d2f5cff5faf8d0113b88995b23ca67ddbefedcda6 not found: ID does not exist"
Sep 17 19:16:15.087423 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:15.087401    2565 scope.go:117] "RemoveContainer" containerID="cfc046b2d6d91631db21abc43361286b2ced683cc32e9629f6503e705ab83866"
Sep 17 19:16:15.087667 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:16:15.087648    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfc046b2d6d91631db21abc43361286b2ced683cc32e9629f6503e705ab83866\": container with ID starting with cfc046b2d6d91631db21abc43361286b2ced683cc32e9629f6503e705ab83866 not found: ID does not exist" containerID="cfc046b2d6d91631db21abc43361286b2ced683cc32e9629f6503e705ab83866"
Sep 17 19:16:15.087714 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:15.087674    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfc046b2d6d91631db21abc43361286b2ced683cc32e9629f6503e705ab83866"} err="failed to get container status \"cfc046b2d6d91631db21abc43361286b2ced683cc32e9629f6503e705ab83866\": rpc error: code = NotFound desc = could not find container \"cfc046b2d6d91631db21abc43361286b2ced683cc32e9629f6503e705ab83866\": container with ID starting with cfc046b2d6d91631db21abc43361286b2ced683cc32e9629f6503e705ab83866 not found: ID does not exist"
Sep 17 19:16:15.092924 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:15.092902    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"]
Sep 17 19:16:15.099157 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:15.099136    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f5cp4rb"]
Sep 17 19:16:16.606888 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:16.606854    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef265e97-434b-4fa2-808b-1d559395a15f" path="/var/lib/kubelet/pods/ef265e97-434b-4fa2-808b-1d559395a15f/volumes"
Sep 17 19:16:26.434397 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.434364    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"]
Sep 17 19:16:26.434783 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.434697    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a355547-32a3-46e7-bdbe-2bdad136df25" containerName="storage-initializer"
Sep 17 19:16:26.434783 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.434709    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a355547-32a3-46e7-bdbe-2bdad136df25" containerName="storage-initializer"
Sep 17 19:16:26.434783 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.434717    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef265e97-434b-4fa2-808b-1d559395a15f" containerName="storage-initializer"
Sep 17 19:16:26.434783 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.434723    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef265e97-434b-4fa2-808b-1d559395a15f" containerName="storage-initializer"
Sep 17 19:16:26.434783 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.434732    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a355547-32a3-46e7-bdbe-2bdad136df25" containerName="main"
Sep 17 19:16:26.434783 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.434738    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a355547-32a3-46e7-bdbe-2bdad136df25" containerName="main"
Sep 17 19:16:26.434783 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.434745    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef265e97-434b-4fa2-808b-1d559395a15f" containerName="main"
Sep 17 19:16:26.434783 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.434750    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef265e97-434b-4fa2-808b-1d559395a15f" containerName="main"
Sep 17 19:16:26.434783 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.434766    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546" containerName="istio-proxy"
Sep 17 19:16:26.434783 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.434773    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546" containerName="istio-proxy"
Sep 17 19:16:26.435115 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.434837    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef265e97-434b-4fa2-808b-1d559395a15f" containerName="main"
Sep 17 19:16:26.435115 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.434847    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0e9e6fe-80d9-4ca7-9e15-1cfb3d7fc546" containerName="istio-proxy"
Sep 17 19:16:26.435115 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.434853    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a355547-32a3-46e7-bdbe-2bdad136df25" containerName="main"
Sep 17 19:16:26.439827 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.439805    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:16:26.442422 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.442400    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 17 19:16:26.442541 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.442432    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-t6jp9\""
Sep 17 19:16:26.442807 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.442785    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 17 19:16:26.443436 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.443419    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:16:26.443908 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.443872    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 17 19:16:26.444151 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.444131    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 17 19:16:26.457394 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.457369    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"]
Sep 17 19:16:26.501154 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.501128    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-tls-certs\") pod \"lora-single-adapter-test-kserve-54c49b4774-fp728\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:16:26.501288 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.501175    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-tmp-dir\") pod \"lora-single-adapter-test-kserve-54c49b4774-fp728\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:16:26.501288 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.501217    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-dshm\") pod \"lora-single-adapter-test-kserve-54c49b4774-fp728\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:16:26.501384 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.501278    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-54c49b4774-fp728\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:16:26.501384 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.501315    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-cabundle-cert\") pod \"lora-single-adapter-test-kserve-54c49b4774-fp728\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:16:26.501384 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.501354    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h4xm\" (UniqueName: \"kubernetes.io/projected/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-kube-api-access-2h4xm\") pod \"lora-single-adapter-test-kserve-54c49b4774-fp728\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:16:26.501384 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.501380    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-home\") pod \"lora-single-adapter-test-kserve-54c49b4774-fp728\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:16:26.501534 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.501400    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-model-cache\") pod \"lora-single-adapter-test-kserve-54c49b4774-fp728\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:16:26.601803 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.601775    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-tls-certs\") pod \"lora-single-adapter-test-kserve-54c49b4774-fp728\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:16:26.601924 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.601817    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-tmp-dir\") pod \"lora-single-adapter-test-kserve-54c49b4774-fp728\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:16:26.601924 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.601845    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-dshm\") pod \"lora-single-adapter-test-kserve-54c49b4774-fp728\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:16:26.601924 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.601876    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-54c49b4774-fp728\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:16:26.601924 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.601904    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-cabundle-cert\") pod \"lora-single-adapter-test-kserve-54c49b4774-fp728\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:16:26.602152 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.601960    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h4xm\" (UniqueName: \"kubernetes.io/projected/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-kube-api-access-2h4xm\") pod \"lora-single-adapter-test-kserve-54c49b4774-fp728\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:16:26.602152 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.601990    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-home\") pod \"lora-single-adapter-test-kserve-54c49b4774-fp728\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:16:26.602152 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.602019    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-model-cache\") pod \"lora-single-adapter-test-kserve-54c49b4774-fp728\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:16:26.602367 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.602340    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-54c49b4774-fp728\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:16:26.602467 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.602447    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-model-cache\") pod \"lora-single-adapter-test-kserve-54c49b4774-fp728\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:16:26.602618 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.602589    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-home\") pod \"lora-single-adapter-test-kserve-54c49b4774-fp728\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:16:26.602780 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.602759    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-cabundle-cert\") pod \"lora-single-adapter-test-kserve-54c49b4774-fp728\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:16:26.602821 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.602766    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-tmp-dir\") pod \"lora-single-adapter-test-kserve-54c49b4774-fp728\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:16:26.604327 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.604289    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-dshm\") pod \"lora-single-adapter-test-kserve-54c49b4774-fp728\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:16:26.604617 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.604596    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-tls-certs\") pod \"lora-single-adapter-test-kserve-54c49b4774-fp728\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:16:26.609950 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.609929    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h4xm\" (UniqueName: \"kubernetes.io/projected/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-kube-api-access-2h4xm\") pod \"lora-single-adapter-test-kserve-54c49b4774-fp728\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:16:26.751186 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:26.751134    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:16:27.035331 ip-10-0-139-75 kubenswrapper[2565]: W0917 19:16:27.035304    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ec6c9f_9ec3_4ded_9722_16a45d0db1c8.slice/crio-f33cc7dc345f57b9e6ee1082ea6a26a47627099467130f5e70f24034b00100cc WatchSource:0}: Error finding container f33cc7dc345f57b9e6ee1082ea6a26a47627099467130f5e70f24034b00100cc: Status 404 returned error can't find the container with id f33cc7dc345f57b9e6ee1082ea6a26a47627099467130f5e70f24034b00100cc
Sep 17 19:16:27.037097 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:27.037069    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"]
Sep 17 19:16:27.132124 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:27.132096    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728" event={"ID":"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8","Type":"ContainerStarted","Data":"b4ab81a3f67cfd49c12c12024d71e38daf0fbfe0baaa4103c186cb5f84c50921"}
Sep 17 19:16:27.132231 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:27.132147    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728" event={"ID":"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8","Type":"ContainerStarted","Data":"f33cc7dc345f57b9e6ee1082ea6a26a47627099467130f5e70f24034b00100cc"}
Sep 17 19:16:30.149977 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:30.149894    2565 generic.go:356] "Generic (PLEG): container finished" podID="62ec6c9f-9ec3-4ded-9722-16a45d0db1c8" containerID="b4ab81a3f67cfd49c12c12024d71e38daf0fbfe0baaa4103c186cb5f84c50921" exitCode=0
Sep 17 19:16:30.150332 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:30.149971    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728" event={"ID":"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8","Type":"ContainerDied","Data":"b4ab81a3f67cfd49c12c12024d71e38daf0fbfe0baaa4103c186cb5f84c50921"}
Sep 17 19:16:31.159174 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:31.159133    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728" event={"ID":"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8","Type":"ContainerStarted","Data":"9b6b2dd985d27b3992ff79208b44114b4c29bd49de9d6188bf30815ee06f8c44"}
Sep 17 19:16:31.183251 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:31.183175    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728" podStartSLOduration=2.453868674 podStartE2EDuration="5.183160007s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.729291333s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:16:31.181388318 +0000 UTC m=+2849.113589842" watchObservedRunningTime="2026-09-17 19:16:31.183160007 +0000 UTC m=+2849.115361525"
Sep 17 19:16:31.751950 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:31.751913    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:16:36.751541 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:36.751506    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:16:36.753188 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:36.753156    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728" podUID="62ec6c9f-9ec3-4ded-9722-16a45d0db1c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 17 19:16:46.751687 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:46.751645    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728" podUID="62ec6c9f-9ec3-4ded-9722-16a45d0db1c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 17 19:16:54.887070 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:54.887033    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"]
Sep 17 19:16:54.890766 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:54.890739    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:16:54.893265 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:54.893219    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 17 19:16:54.893957 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:54.893524    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:16:54.893957 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:54.893633    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 17 19:16:54.894141 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:54.894068    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 17 19:16:54.894237 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:54.894170    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-5kpg5\""
Sep 17 19:16:54.894390 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:54.894365    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 17 19:16:54.911683 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:54.911659    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"]
Sep 17 19:16:54.932242 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:54.932189    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:16:54.932374 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:54.932301    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:16:54.932448 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:54.932373    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:16:54.932448 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:54.932405    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:16:54.932448 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:54.932443    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pqhw\" (UniqueName: \"kubernetes.io/projected/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-kube-api-access-7pqhw\") pod \"custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:16:54.932640 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:54.932473    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:16:54.932640 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:54.932511    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:16:54.932640 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:54.932584    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:16:55.034008 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:55.033931    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:16:55.034008 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:55.033976    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:16:55.034008 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:55.034006    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:16:55.034293 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:55.034038    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:16:55.034293 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:55.034054    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:16:55.034293 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:55.034072    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pqhw\" (UniqueName: \"kubernetes.io/projected/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-kube-api-access-7pqhw\") pod \"custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:16:55.034293 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:55.034090    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:16:55.034527 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:55.034326    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:16:55.034776 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:55.034688    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:16:55.034776 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:55.034723    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:16:55.034776 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:55.034758    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:16:55.034954 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:55.034783    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:16:55.035056 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:55.035034    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:16:55.036630 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:55.036599    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:16:55.036776 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:55.036760    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:16:55.043812 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:55.043789    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pqhw\" (UniqueName: \"kubernetes.io/projected/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-kube-api-access-7pqhw\") pod \"custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:16:55.203923 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:55.203877    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:16:55.491014 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:55.490982    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"]
Sep 17 19:16:55.493097 ip-10-0-139-75 kubenswrapper[2565]: W0917 19:16:55.493071    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae4de399_cc36_49cd_9708_0ac6bcc04ed7.slice/crio-683d8cd281d985911b1d73e4a134a33e7f65d797ac4f96cdaa12f36ca51fe1b1 WatchSource:0}: Error finding container 683d8cd281d985911b1d73e4a134a33e7f65d797ac4f96cdaa12f36ca51fe1b1: Status 404 returned error can't find the container with id 683d8cd281d985911b1d73e4a134a33e7f65d797ac4f96cdaa12f36ca51fe1b1
Sep 17 19:16:56.304945 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:56.304898    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn" event={"ID":"ae4de399-cc36-49cd-9708-0ac6bcc04ed7","Type":"ContainerStarted","Data":"f30479a3d769fb4ba653f82bba370eacc39716d813617c8c9c31c5c9690caa80"}
Sep 17 19:16:56.305399 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:56.304977    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn" event={"ID":"ae4de399-cc36-49cd-9708-0ac6bcc04ed7","Type":"ContainerStarted","Data":"683d8cd281d985911b1d73e4a134a33e7f65d797ac4f96cdaa12f36ca51fe1b1"}
Sep 17 19:16:56.751851 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:56.751799    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728" podUID="62ec6c9f-9ec3-4ded-9722-16a45d0db1c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 17 19:16:58.320296 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:58.320190    2565 generic.go:356] "Generic (PLEG): container finished" podID="ae4de399-cc36-49cd-9708-0ac6bcc04ed7" containerID="f30479a3d769fb4ba653f82bba370eacc39716d813617c8c9c31c5c9690caa80" exitCode=0
Sep 17 19:16:58.320296 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:58.320239    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn" event={"ID":"ae4de399-cc36-49cd-9708-0ac6bcc04ed7","Type":"ContainerDied","Data":"f30479a3d769fb4ba653f82bba370eacc39716d813617c8c9c31c5c9690caa80"}
Sep 17 19:16:59.327764 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:59.327717    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn" event={"ID":"ae4de399-cc36-49cd-9708-0ac6bcc04ed7","Type":"ContainerStarted","Data":"bfc1d691002e6e69fdb03b3271def5ec7958cfcdebd7fbc050c7930b618d646f"}
Sep 17 19:16:59.366415 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:16:59.366349    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn" podStartSLOduration=2.932978442 podStartE2EDuration="5.366323966s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.433345524s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:16:59.365539272 +0000 UTC m=+2877.297740792" watchObservedRunningTime="2026-09-17 19:16:59.366323966 +0000 UTC m=+2877.298525487"
Sep 17 19:17:05.204265 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:17:05.204225    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:17:05.204699 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:17:05.204293    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:17:05.206385 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:17:05.206355    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn" podUID="ae4de399-cc36-49cd-9708-0ac6bcc04ed7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 17 19:17:06.752619 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:17:06.752573    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728" podUID="62ec6c9f-9ec3-4ded-9722-16a45d0db1c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 17 19:17:15.204793 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:17:15.204739    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn" podUID="ae4de399-cc36-49cd-9708-0ac6bcc04ed7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 17 19:17:16.752516 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:17:16.752463    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728" podUID="62ec6c9f-9ec3-4ded-9722-16a45d0db1c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 17 19:17:25.204622 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:17:25.204573    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn" podUID="ae4de399-cc36-49cd-9708-0ac6bcc04ed7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 17 19:17:26.752097 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:17:26.752043    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728" podUID="62ec6c9f-9ec3-4ded-9722-16a45d0db1c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 17 19:17:35.204953 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:17:35.204907    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn" podUID="ae4de399-cc36-49cd-9708-0ac6bcc04ed7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 17 19:17:36.751850 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:17:36.751805    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728" podUID="62ec6c9f-9ec3-4ded-9722-16a45d0db1c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 17 19:17:45.204620 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:17:45.204568    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn" podUID="ae4de399-cc36-49cd-9708-0ac6bcc04ed7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 17 19:17:46.762356 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:17:46.762312    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:17:46.770117 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:17:46.770083    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:17:55.205386 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:17:55.205337    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn" podUID="ae4de399-cc36-49cd-9708-0ac6bcc04ed7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 17 19:18:05.205308 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:05.205262    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn" podUID="ae4de399-cc36-49cd-9708-0ac6bcc04ed7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 17 19:18:15.205152 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:15.205088    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn" podUID="ae4de399-cc36-49cd-9708-0ac6bcc04ed7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 17 19:18:15.694804 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:15.694766    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-54c49b4774-fp728_62ec6c9f-9ec3-4ded-9722-16a45d0db1c8/storage-initializer/0.log"
Sep 17 19:18:15.705274 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:15.705249    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-54c49b4774-fp728_62ec6c9f-9ec3-4ded-9722-16a45d0db1c8/main/0.log"
Sep 17 19:18:17.121923 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:17.121892    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"]
Sep 17 19:18:25.205067 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:25.204981    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn" podUID="ae4de399-cc36-49cd-9708-0ac6bcc04ed7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 17 19:18:32.143618 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:32.143554    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728" podUID="62ec6c9f-9ec3-4ded-9722-16a45d0db1c8" containerName="main" containerID="cri-o://9b6b2dd985d27b3992ff79208b44114b4c29bd49de9d6188bf30815ee06f8c44" gracePeriod=45
Sep 17 19:18:32.763343 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:32.763295    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728" podUID="62ec6c9f-9ec3-4ded-9722-16a45d0db1c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 17 19:18:32.906037 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:32.906007    2565 generic.go:356] "Generic (PLEG): container finished" podID="62ec6c9f-9ec3-4ded-9722-16a45d0db1c8" containerID="9b6b2dd985d27b3992ff79208b44114b4c29bd49de9d6188bf30815ee06f8c44" exitCode=0
Sep 17 19:18:32.906189 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:32.906074    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728" event={"ID":"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8","Type":"ContainerDied","Data":"9b6b2dd985d27b3992ff79208b44114b4c29bd49de9d6188bf30815ee06f8c44"}
Sep 17 19:18:33.004906 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.004883    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:18:33.102299 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.102228    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-kserve-provision-location\") pod \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") "
Sep 17 19:18:33.102299 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.102273    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-model-cache\") pod \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") "
Sep 17 19:18:33.102511 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.102301    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-tls-certs\") pod \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") "
Sep 17 19:18:33.102511 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.102337    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h4xm\" (UniqueName: \"kubernetes.io/projected/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-kube-api-access-2h4xm\") pod \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") "
Sep 17 19:18:33.102511 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.102361    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-tmp-dir\") pod \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") "
Sep 17 19:18:33.102511 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.102386    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-dshm\") pod \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") "
Sep 17 19:18:33.102511 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.102429    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-home\") pod \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") "
Sep 17 19:18:33.102511 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.102497    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-cabundle-cert\") pod \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\" (UID: \"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8\") "
Sep 17 19:18:33.102819 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.102538    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-model-cache" (OuterVolumeSpecName: "model-cache") pod "62ec6c9f-9ec3-4ded-9722-16a45d0db1c8" (UID: "62ec6c9f-9ec3-4ded-9722-16a45d0db1c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:18:33.102819 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.102770    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:18:33.102915 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.102824    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "62ec6c9f-9ec3-4ded-9722-16a45d0db1c8" (UID: "62ec6c9f-9ec3-4ded-9722-16a45d0db1c8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:18:33.102915 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.102879    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-home" (OuterVolumeSpecName: "home") pod "62ec6c9f-9ec3-4ded-9722-16a45d0db1c8" (UID: "62ec6c9f-9ec3-4ded-9722-16a45d0db1c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:18:33.105158 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.105134    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-dshm" (OuterVolumeSpecName: "dshm") pod "62ec6c9f-9ec3-4ded-9722-16a45d0db1c8" (UID: "62ec6c9f-9ec3-4ded-9722-16a45d0db1c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:18:33.105158 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.105147    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62ec6c9f-9ec3-4ded-9722-16a45d0db1c8" (UID: "62ec6c9f-9ec3-4ded-9722-16a45d0db1c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:18:33.105336 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.105158    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-kube-api-access-2h4xm" (OuterVolumeSpecName: "kube-api-access-2h4xm") pod "62ec6c9f-9ec3-4ded-9722-16a45d0db1c8" (UID: "62ec6c9f-9ec3-4ded-9722-16a45d0db1c8"). InnerVolumeSpecName "kube-api-access-2h4xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:18:33.119376 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.119327    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "62ec6c9f-9ec3-4ded-9722-16a45d0db1c8" (UID: "62ec6c9f-9ec3-4ded-9722-16a45d0db1c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:18:33.133194 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.133166    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "62ec6c9f-9ec3-4ded-9722-16a45d0db1c8" (UID: "62ec6c9f-9ec3-4ded-9722-16a45d0db1c8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:18:33.204116 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.204091    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-kserve-provision-location\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:18:33.204116 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.204115    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:18:33.204457 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.204125    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h4xm\" (UniqueName: \"kubernetes.io/projected/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-kube-api-access-2h4xm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:18:33.204457 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.204135    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:18:33.204457 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.204144    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:18:33.204457 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.204153    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:18:33.204457 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.204161    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8-cabundle-cert\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:18:33.912684 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.912643    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728" event={"ID":"62ec6c9f-9ec3-4ded-9722-16a45d0db1c8","Type":"ContainerDied","Data":"f33cc7dc345f57b9e6ee1082ea6a26a47627099467130f5e70f24034b00100cc"}
Sep 17 19:18:33.912875 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.912740    2565 scope.go:117] "RemoveContainer" containerID="9b6b2dd985d27b3992ff79208b44114b4c29bd49de9d6188bf30815ee06f8c44"
Sep 17 19:18:33.912875 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.912664    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"
Sep 17 19:18:33.921961 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.921939    2565 scope.go:117] "RemoveContainer" containerID="b4ab81a3f67cfd49c12c12024d71e38daf0fbfe0baaa4103c186cb5f84c50921"
Sep 17 19:18:33.941364 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.941343    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"]
Sep 17 19:18:33.947437 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:33.947416    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-54c49b4774-fp728"]
Sep 17 19:18:34.606833 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:34.606801    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62ec6c9f-9ec3-4ded-9722-16a45d0db1c8" path="/var/lib/kubelet/pods/62ec6c9f-9ec3-4ded-9722-16a45d0db1c8/volumes"
Sep 17 19:18:35.204588 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:35.204542    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn" podUID="ae4de399-cc36-49cd-9708-0ac6bcc04ed7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 17 19:18:45.205152 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.205100    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn" podUID="ae4de399-cc36-49cd-9708-0ac6bcc04ed7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 17 19:18:45.672892 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.672859    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"]
Sep 17 19:18:45.673277 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.673263    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62ec6c9f-9ec3-4ded-9722-16a45d0db1c8" containerName="main"
Sep 17 19:18:45.673327 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.673279    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62ec6c9f-9ec3-4ded-9722-16a45d0db1c8" containerName="main"
Sep 17 19:18:45.673327 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.673294    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62ec6c9f-9ec3-4ded-9722-16a45d0db1c8" containerName="storage-initializer"
Sep 17 19:18:45.673327 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.673301    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62ec6c9f-9ec3-4ded-9722-16a45d0db1c8" containerName="storage-initializer"
Sep 17 19:18:45.673425 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.673357    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="62ec6c9f-9ec3-4ded-9722-16a45d0db1c8" containerName="main"
Sep 17 19:18:45.676626 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.676607    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:18:45.680783 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.680763    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:18:45.682977 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.682956    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 17 19:18:45.683267 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.683245    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 17 19:18:45.683811 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.683788    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-npsp6\""
Sep 17 19:18:45.684100 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.684081    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 17 19:18:45.686963 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.686945    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 17 19:18:45.717362 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.717328    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"]
Sep 17 19:18:45.817693 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.817648    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-model-cache\") pod \"lora-multiple-adapters-test-kserve-5f848df9d5-76bkz\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:18:45.817693 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.817695    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-5f848df9d5-76bkz\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:18:45.817947 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.817780    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjllp\" (UniqueName: \"kubernetes.io/projected/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-kube-api-access-vjllp\") pod \"lora-multiple-adapters-test-kserve-5f848df9d5-76bkz\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:18:45.817947 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.817836    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-home\") pod \"lora-multiple-adapters-test-kserve-5f848df9d5-76bkz\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:18:45.817947 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.817890    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-tls-certs\") pod \"lora-multiple-adapters-test-kserve-5f848df9d5-76bkz\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:18:45.817947 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.817922    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-5f848df9d5-76bkz\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:18:45.818133 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.817958    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-dshm\") pod \"lora-multiple-adapters-test-kserve-5f848df9d5-76bkz\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:18:45.818133 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.817999    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-5f848df9d5-76bkz\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:18:45.919285 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.919250    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-tls-certs\") pod \"lora-multiple-adapters-test-kserve-5f848df9d5-76bkz\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:18:45.919285 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.919289    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-5f848df9d5-76bkz\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:18:45.919518 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.919315    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-dshm\") pod \"lora-multiple-adapters-test-kserve-5f848df9d5-76bkz\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:18:45.919518 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.919335    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-5f848df9d5-76bkz\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:18:45.919518 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.919363    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-model-cache\") pod \"lora-multiple-adapters-test-kserve-5f848df9d5-76bkz\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:18:45.919518 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.919396    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-5f848df9d5-76bkz\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:18:45.919518 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.919443    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjllp\" (UniqueName: \"kubernetes.io/projected/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-kube-api-access-vjllp\") pod \"lora-multiple-adapters-test-kserve-5f848df9d5-76bkz\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:18:45.919518 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.919477    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-home\") pod \"lora-multiple-adapters-test-kserve-5f848df9d5-76bkz\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:18:45.919847 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.919730    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-5f848df9d5-76bkz\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:18:45.919847 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.919815    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-model-cache\") pod \"lora-multiple-adapters-test-kserve-5f848df9d5-76bkz\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:18:45.920126 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.920101    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-home\") pod \"lora-multiple-adapters-test-kserve-5f848df9d5-76bkz\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:18:45.920269 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.920134    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-5f848df9d5-76bkz\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:18:45.920269 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.920137    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-5f848df9d5-76bkz\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:18:45.921854 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.921816    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-dshm\") pod \"lora-multiple-adapters-test-kserve-5f848df9d5-76bkz\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:18:45.922011 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.921990    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-tls-certs\") pod \"lora-multiple-adapters-test-kserve-5f848df9d5-76bkz\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:18:45.931245 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.931171    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjllp\" (UniqueName: \"kubernetes.io/projected/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-kube-api-access-vjllp\") pod \"lora-multiple-adapters-test-kserve-5f848df9d5-76bkz\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:18:45.986880 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:45.986843    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:18:46.271822 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:46.271795    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"]
Sep 17 19:18:46.274223 ip-10-0-139-75 kubenswrapper[2565]: W0917 19:18:46.274180    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd240afea_ff4c_4dbb_9f1e_6ccc9717db22.slice/crio-1a84b5fe95522a74c39cfa0b08ad95ae901a7409249bacc3fb58805f6d145239 WatchSource:0}: Error finding container 1a84b5fe95522a74c39cfa0b08ad95ae901a7409249bacc3fb58805f6d145239: Status 404 returned error can't find the container with id 1a84b5fe95522a74c39cfa0b08ad95ae901a7409249bacc3fb58805f6d145239
Sep 17 19:18:46.989583 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:46.989539    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz" event={"ID":"d240afea-ff4c-4dbb-9f1e-6ccc9717db22","Type":"ContainerStarted","Data":"4084ca9613228eb7096e8cd53db2468f772109409f360b46e86b51265942f432"}
Sep 17 19:18:46.989816 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:46.989607    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz" event={"ID":"d240afea-ff4c-4dbb-9f1e-6ccc9717db22","Type":"ContainerStarted","Data":"1a84b5fe95522a74c39cfa0b08ad95ae901a7409249bacc3fb58805f6d145239"}
Sep 17 19:18:50.010096 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:50.010062    2565 generic.go:356] "Generic (PLEG): container finished" podID="d240afea-ff4c-4dbb-9f1e-6ccc9717db22" containerID="4084ca9613228eb7096e8cd53db2468f772109409f360b46e86b51265942f432" exitCode=0
Sep 17 19:18:50.010562 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:50.010137    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz" event={"ID":"d240afea-ff4c-4dbb-9f1e-6ccc9717db22","Type":"ContainerDied","Data":"4084ca9613228eb7096e8cd53db2468f772109409f360b46e86b51265942f432"}
Sep 17 19:18:51.018314 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:51.018283    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz" event={"ID":"d240afea-ff4c-4dbb-9f1e-6ccc9717db22","Type":"ContainerStarted","Data":"32883d8c0723ec1c582d2fc0efd5d60658147dccb6ca97b9a611b78dd3679580"}
Sep 17 19:18:51.080801 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:51.080740    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz" podStartSLOduration=3.102095633 podStartE2EDuration="6.080723104s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.978627471s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:18:51.075578281 +0000 UTC m=+2989.007779800" watchObservedRunningTime="2026-09-17 19:18:51.080723104 +0000 UTC m=+2989.012924622"
Sep 17 19:18:51.987692 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:51.987646    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:18:55.214777 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:55.214733    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:18:55.222746 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:55.222717    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:18:55.987063 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:55.987024    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:18:55.988676 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:18:55.988640    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz" podUID="d240afea-ff4c-4dbb-9f1e-6ccc9717db22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 17 19:19:05.988299 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:05.988246    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz" podUID="d240afea-ff4c-4dbb-9f1e-6ccc9717db22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 17 19:19:11.295901 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:11.295861    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"]
Sep 17 19:19:15.987422 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:15.987377    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz" podUID="d240afea-ff4c-4dbb-9f1e-6ccc9717db22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 17 19:19:25.987541 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:25.987496    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz" podUID="d240afea-ff4c-4dbb-9f1e-6ccc9717db22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 17 19:19:26.319742 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:26.319648    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn" podUID="ae4de399-cc36-49cd-9708-0ac6bcc04ed7" containerName="main" containerID="cri-o://bfc1d691002e6e69fdb03b3271def5ec7958cfcdebd7fbc050c7930b618d646f" gracePeriod=45
Sep 17 19:19:27.071057 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.071031    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:19:27.177416 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.177388    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-kserve-provision-location\") pod \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") "
Sep 17 19:19:27.177584 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.177435    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pqhw\" (UniqueName: \"kubernetes.io/projected/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-kube-api-access-7pqhw\") pod \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") "
Sep 17 19:19:27.177584 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.177454    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-cabundle-cert\") pod \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") "
Sep 17 19:19:27.177584 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.177493    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-tls-certs\") pod \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") "
Sep 17 19:19:27.177584 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.177511    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-dshm\") pod \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") "
Sep 17 19:19:27.177821 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.177602    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-model-cache\") pod \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") "
Sep 17 19:19:27.177821 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.177624    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-tmp-dir\") pod \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") "
Sep 17 19:19:27.177821 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.177651    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-home\") pod \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\" (UID: \"ae4de399-cc36-49cd-9708-0ac6bcc04ed7\") "
Sep 17 19:19:27.178235 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.178151    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ae4de399-cc36-49cd-9708-0ac6bcc04ed7" (UID: "ae4de399-cc36-49cd-9708-0ac6bcc04ed7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:19:27.178476 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.178437    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae4de399-cc36-49cd-9708-0ac6bcc04ed7" (UID: "ae4de399-cc36-49cd-9708-0ac6bcc04ed7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:19:27.178562 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.178475    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-home" (OuterVolumeSpecName: "home") pod "ae4de399-cc36-49cd-9708-0ac6bcc04ed7" (UID: "ae4de399-cc36-49cd-9708-0ac6bcc04ed7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:19:27.179949 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.179917    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae4de399-cc36-49cd-9708-0ac6bcc04ed7" (UID: "ae4de399-cc36-49cd-9708-0ac6bcc04ed7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:19:27.180127 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.180100    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-dshm" (OuterVolumeSpecName: "dshm") pod "ae4de399-cc36-49cd-9708-0ac6bcc04ed7" (UID: "ae4de399-cc36-49cd-9708-0ac6bcc04ed7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:19:27.180127 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.180115    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-kube-api-access-7pqhw" (OuterVolumeSpecName: "kube-api-access-7pqhw") pod "ae4de399-cc36-49cd-9708-0ac6bcc04ed7" (UID: "ae4de399-cc36-49cd-9708-0ac6bcc04ed7"). InnerVolumeSpecName "kube-api-access-7pqhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:19:27.192733 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.192703    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae4de399-cc36-49cd-9708-0ac6bcc04ed7" (UID: "ae4de399-cc36-49cd-9708-0ac6bcc04ed7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:19:27.211386 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.211351    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ae4de399-cc36-49cd-9708-0ac6bcc04ed7" (UID: "ae4de399-cc36-49cd-9708-0ac6bcc04ed7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:19:27.220517 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.220490    2565 generic.go:356] "Generic (PLEG): container finished" podID="ae4de399-cc36-49cd-9708-0ac6bcc04ed7" containerID="bfc1d691002e6e69fdb03b3271def5ec7958cfcdebd7fbc050c7930b618d646f" exitCode=0
Sep 17 19:19:27.220621 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.220546    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"
Sep 17 19:19:27.220621 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.220573    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn" event={"ID":"ae4de399-cc36-49cd-9708-0ac6bcc04ed7","Type":"ContainerDied","Data":"bfc1d691002e6e69fdb03b3271def5ec7958cfcdebd7fbc050c7930b618d646f"}
Sep 17 19:19:27.220745 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.220633    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn" event={"ID":"ae4de399-cc36-49cd-9708-0ac6bcc04ed7","Type":"ContainerDied","Data":"683d8cd281d985911b1d73e4a134a33e7f65d797ac4f96cdaa12f36ca51fe1b1"}
Sep 17 19:19:27.220745 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.220657    2565 scope.go:117] "RemoveContainer" containerID="bfc1d691002e6e69fdb03b3271def5ec7958cfcdebd7fbc050c7930b618d646f"
Sep 17 19:19:27.230105 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.230084    2565 scope.go:117] "RemoveContainer" containerID="f30479a3d769fb4ba653f82bba370eacc39716d813617c8c9c31c5c9690caa80"
Sep 17 19:19:27.239232 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.239210    2565 scope.go:117] "RemoveContainer" containerID="bfc1d691002e6e69fdb03b3271def5ec7958cfcdebd7fbc050c7930b618d646f"
Sep 17 19:19:27.239492 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:19:27.239474    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfc1d691002e6e69fdb03b3271def5ec7958cfcdebd7fbc050c7930b618d646f\": container with ID starting with bfc1d691002e6e69fdb03b3271def5ec7958cfcdebd7fbc050c7930b618d646f not found: ID does not exist" containerID="bfc1d691002e6e69fdb03b3271def5ec7958cfcdebd7fbc050c7930b618d646f"
Sep 17 19:19:27.239561 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.239501    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfc1d691002e6e69fdb03b3271def5ec7958cfcdebd7fbc050c7930b618d646f"} err="failed to get container status \"bfc1d691002e6e69fdb03b3271def5ec7958cfcdebd7fbc050c7930b618d646f\": rpc error: code = NotFound desc = could not find container \"bfc1d691002e6e69fdb03b3271def5ec7958cfcdebd7fbc050c7930b618d646f\": container with ID starting with bfc1d691002e6e69fdb03b3271def5ec7958cfcdebd7fbc050c7930b618d646f not found: ID does not exist"
Sep 17 19:19:27.239561 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.239518    2565 scope.go:117] "RemoveContainer" containerID="f30479a3d769fb4ba653f82bba370eacc39716d813617c8c9c31c5c9690caa80"
Sep 17 19:19:27.239778 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:19:27.239762    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f30479a3d769fb4ba653f82bba370eacc39716d813617c8c9c31c5c9690caa80\": container with ID starting with f30479a3d769fb4ba653f82bba370eacc39716d813617c8c9c31c5c9690caa80 not found: ID does not exist" containerID="f30479a3d769fb4ba653f82bba370eacc39716d813617c8c9c31c5c9690caa80"
Sep 17 19:19:27.239823 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.239783    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f30479a3d769fb4ba653f82bba370eacc39716d813617c8c9c31c5c9690caa80"} err="failed to get container status \"f30479a3d769fb4ba653f82bba370eacc39716d813617c8c9c31c5c9690caa80\": rpc error: code = NotFound desc = could not find container \"f30479a3d769fb4ba653f82bba370eacc39716d813617c8c9c31c5c9690caa80\": container with ID starting with f30479a3d769fb4ba653f82bba370eacc39716d813617c8c9c31c5c9690caa80 not found: ID does not exist"
Sep 17 19:19:27.246936 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.246906    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"]
Sep 17 19:19:27.251362 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.251341    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-599dfcb45-n2pfn"]
Sep 17 19:19:27.278483 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.278459    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:19:27.278483 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.278485    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:19:27.278610 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.278494    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:19:27.278610 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.278503    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-kserve-provision-location\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:19:27.278610 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.278517    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pqhw\" (UniqueName: \"kubernetes.io/projected/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-kube-api-access-7pqhw\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:19:27.278610 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.278528    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-cabundle-cert\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:19:27.278610 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.278536    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:19:27.278610 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:27.278543    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae4de399-cc36-49cd-9708-0ac6bcc04ed7-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:19:28.608558 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:28.608522    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae4de399-cc36-49cd-9708-0ac6bcc04ed7" path="/var/lib/kubelet/pods/ae4de399-cc36-49cd-9708-0ac6bcc04ed7/volumes"
Sep 17 19:19:35.987799 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:35.987745    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz" podUID="d240afea-ff4c-4dbb-9f1e-6ccc9717db22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 17 19:19:45.988109 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:45.988064    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz" podUID="d240afea-ff4c-4dbb-9f1e-6ccc9717db22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 17 19:19:55.988121 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:19:55.988029    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz" podUID="d240afea-ff4c-4dbb-9f1e-6ccc9717db22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 17 19:20:05.997296 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:05.997258    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:20:06.005105 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:06.005077    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:20:08.673770 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.673737    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"]
Sep 17 19:20:08.674148 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.674086    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae4de399-cc36-49cd-9708-0ac6bcc04ed7" containerName="storage-initializer"
Sep 17 19:20:08.674148 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.674097    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae4de399-cc36-49cd-9708-0ac6bcc04ed7" containerName="storage-initializer"
Sep 17 19:20:08.674148 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.674117    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae4de399-cc36-49cd-9708-0ac6bcc04ed7" containerName="main"
Sep 17 19:20:08.674148 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.674123    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae4de399-cc36-49cd-9708-0ac6bcc04ed7" containerName="main"
Sep 17 19:20:08.674306 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.674180    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae4de399-cc36-49cd-9708-0ac6bcc04ed7" containerName="main"
Sep 17 19:20:08.678328 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.678304    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:20:08.680843 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.680818    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 17 19:20:08.682305 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.682283    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 17 19:20:08.682404 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.682327    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-2c4zw\""
Sep 17 19:20:08.682465 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.682445    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:20:08.682543 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.682525    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 17 19:20:08.689424 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.689401    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 17 19:20:08.699964 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.699938    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"]
Sep 17 19:20:08.717448 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.717421    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d47ba3fa-9af3-4ba4-9155-abc592d68000-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:20:08.717545 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.717465    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d47ba3fa-9af3-4ba4-9155-abc592d68000-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:20:08.717545 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.717489    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d47ba3fa-9af3-4ba4-9155-abc592d68000-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:20:08.717545 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.717513    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-284jv\" (UniqueName: \"kubernetes.io/projected/d47ba3fa-9af3-4ba4-9155-abc592d68000-kube-api-access-284jv\") pod \"router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:20:08.717685 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.717550    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d47ba3fa-9af3-4ba4-9155-abc592d68000-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:20:08.717685 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.717623    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d47ba3fa-9af3-4ba4-9155-abc592d68000-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:20:08.717685 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.717665    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d47ba3fa-9af3-4ba4-9155-abc592d68000-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:20:08.717783 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.717693    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d47ba3fa-9af3-4ba4-9155-abc592d68000-home\") pod \"router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:20:08.818876 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.818850    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d47ba3fa-9af3-4ba4-9155-abc592d68000-home\") pod \"router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:20:08.818971 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.818880    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d47ba3fa-9af3-4ba4-9155-abc592d68000-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:20:08.818971 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.818905    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d47ba3fa-9af3-4ba4-9155-abc592d68000-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:20:08.819082 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.819008    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d47ba3fa-9af3-4ba4-9155-abc592d68000-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:20:08.819082 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.819041    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-284jv\" (UniqueName: \"kubernetes.io/projected/d47ba3fa-9af3-4ba4-9155-abc592d68000-kube-api-access-284jv\") pod \"router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:20:08.819225 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.819096    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d47ba3fa-9af3-4ba4-9155-abc592d68000-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:20:08.819225 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.819159    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d47ba3fa-9af3-4ba4-9155-abc592d68000-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:20:08.819225 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.819188    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d47ba3fa-9af3-4ba4-9155-abc592d68000-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:20:08.819406 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.819258    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d47ba3fa-9af3-4ba4-9155-abc592d68000-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:20:08.819406 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.819317    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d47ba3fa-9af3-4ba4-9155-abc592d68000-home\") pod \"router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:20:08.819523 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.819493    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d47ba3fa-9af3-4ba4-9155-abc592d68000-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:20:08.819582 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.819553    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d47ba3fa-9af3-4ba4-9155-abc592d68000-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:20:08.819952 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.819931    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d47ba3fa-9af3-4ba4-9155-abc592d68000-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:20:08.821421 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.821399    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d47ba3fa-9af3-4ba4-9155-abc592d68000-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:20:08.821701 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.821680    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d47ba3fa-9af3-4ba4-9155-abc592d68000-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:20:08.829240 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.829192    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-284jv\" (UniqueName: \"kubernetes.io/projected/d47ba3fa-9af3-4ba4-9155-abc592d68000-kube-api-access-284jv\") pod \"router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:20:08.989414 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:08.989336    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:20:09.277052 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:09.276971    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"]
Sep 17 19:20:09.280700 ip-10-0-139-75 kubenswrapper[2565]: W0917 19:20:09.280662    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd47ba3fa_9af3_4ba4_9155_abc592d68000.slice/crio-8ac9053486261987059a875a1a88564de68034cd5a4bd513ead295688be311bc WatchSource:0}: Error finding container 8ac9053486261987059a875a1a88564de68034cd5a4bd513ead295688be311bc: Status 404 returned error can't find the container with id 8ac9053486261987059a875a1a88564de68034cd5a4bd513ead295688be311bc
Sep 17 19:20:09.449763 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:09.449730    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng" event={"ID":"d47ba3fa-9af3-4ba4-9155-abc592d68000","Type":"ContainerStarted","Data":"a791f2c4e874244fab6159ebda8114ec9b2dfab2381bee4df681c49016f7ceef"}
Sep 17 19:20:09.449904 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:09.449783    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng" event={"ID":"d47ba3fa-9af3-4ba4-9155-abc592d68000","Type":"ContainerStarted","Data":"8ac9053486261987059a875a1a88564de68034cd5a4bd513ead295688be311bc"}
Sep 17 19:20:12.469745 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:12.469704    2565 generic.go:356] "Generic (PLEG): container finished" podID="d47ba3fa-9af3-4ba4-9155-abc592d68000" containerID="a791f2c4e874244fab6159ebda8114ec9b2dfab2381bee4df681c49016f7ceef" exitCode=0
Sep 17 19:20:12.470119 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:12.469776    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng" event={"ID":"d47ba3fa-9af3-4ba4-9155-abc592d68000","Type":"ContainerDied","Data":"a791f2c4e874244fab6159ebda8114ec9b2dfab2381bee4df681c49016f7ceef"}
Sep 17 19:20:13.476639 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:13.476599    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng" event={"ID":"d47ba3fa-9af3-4ba4-9155-abc592d68000","Type":"ContainerStarted","Data":"96d95a18936d31c2b8b982707bebb8abcf0663934772b1d081e644373842cb8d"}
Sep 17 19:20:13.501747 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:13.501687    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng" podStartSLOduration=3.357019163 podStartE2EDuration="5.50166469s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.144645527s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:20:13.498901546 +0000 UTC m=+3071.431103217" watchObservedRunningTime="2026-09-17 19:20:13.50166469 +0000 UTC m=+3071.433866211"
Sep 17 19:20:18.989903 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:18.989867    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:20:18.990361 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:18.989932    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:20:18.991714 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:18.991687    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng" podUID="d47ba3fa-9af3-4ba4-9155-abc592d68000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 17 19:20:28.990118 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:28.990054    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng" podUID="d47ba3fa-9af3-4ba4-9155-abc592d68000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 17 19:20:38.989896 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:38.989843    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng" podUID="d47ba3fa-9af3-4ba4-9155-abc592d68000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 17 19:20:48.990111 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:48.990057    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng" podUID="d47ba3fa-9af3-4ba4-9155-abc592d68000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 17 19:20:51.740184 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:51.740150    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-5f848df9d5-76bkz_d240afea-ff4c-4dbb-9f1e-6ccc9717db22/storage-initializer/0.log"
Sep 17 19:20:51.749346 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:51.749311    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-5f848df9d5-76bkz_d240afea-ff4c-4dbb-9f1e-6ccc9717db22/main/0.log"
Sep 17 19:20:52.835279 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:52.835194    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"]
Sep 17 19:20:58.990763 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:20:58.990702    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng" podUID="d47ba3fa-9af3-4ba4-9155-abc592d68000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 17 19:21:07.857453 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:07.857406    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz" podUID="d240afea-ff4c-4dbb-9f1e-6ccc9717db22" containerName="main" containerID="cri-o://32883d8c0723ec1c582d2fc0efd5d60658147dccb6ca97b9a611b78dd3679580" gracePeriod=45
Sep 17 19:21:07.998693 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:07.998651    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz" podUID="d240afea-ff4c-4dbb-9f1e-6ccc9717db22" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 17 19:21:08.515845 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.515819    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:21:08.655343 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.655263    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-cabundle-cert\") pod \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") "
Sep 17 19:21:08.655495 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.655356    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjllp\" (UniqueName: \"kubernetes.io/projected/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-kube-api-access-vjllp\") pod \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") "
Sep 17 19:21:08.655495 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.655385    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-home\") pod \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") "
Sep 17 19:21:08.655495 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.655415    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-tmp-dir\") pod \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") "
Sep 17 19:21:08.655495 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.655439    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-tls-certs\") pod \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") "
Sep 17 19:21:08.655495 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.655468    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-model-cache\") pod \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") "
Sep 17 19:21:08.655759 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.655520    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-kserve-provision-location\") pod \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") "
Sep 17 19:21:08.655759 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.655546    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-dshm\") pod \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\" (UID: \"d240afea-ff4c-4dbb-9f1e-6ccc9717db22\") "
Sep 17 19:21:08.655861 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.655744    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d240afea-ff4c-4dbb-9f1e-6ccc9717db22" (UID: "d240afea-ff4c-4dbb-9f1e-6ccc9717db22"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:21:08.655861 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.655814    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-home" (OuterVolumeSpecName: "home") pod "d240afea-ff4c-4dbb-9f1e-6ccc9717db22" (UID: "d240afea-ff4c-4dbb-9f1e-6ccc9717db22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:21:08.655980 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.655893    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-model-cache" (OuterVolumeSpecName: "model-cache") pod "d240afea-ff4c-4dbb-9f1e-6ccc9717db22" (UID: "d240afea-ff4c-4dbb-9f1e-6ccc9717db22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:21:08.657925 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.657885    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-dshm" (OuterVolumeSpecName: "dshm") pod "d240afea-ff4c-4dbb-9f1e-6ccc9717db22" (UID: "d240afea-ff4c-4dbb-9f1e-6ccc9717db22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:21:08.658056 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.657990    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d240afea-ff4c-4dbb-9f1e-6ccc9717db22" (UID: "d240afea-ff4c-4dbb-9f1e-6ccc9717db22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:21:08.658189 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.658170    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-kube-api-access-vjllp" (OuterVolumeSpecName: "kube-api-access-vjllp") pod "d240afea-ff4c-4dbb-9f1e-6ccc9717db22" (UID: "d240afea-ff4c-4dbb-9f1e-6ccc9717db22"). InnerVolumeSpecName "kube-api-access-vjllp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:21:08.669390 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.669363    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d240afea-ff4c-4dbb-9f1e-6ccc9717db22" (UID: "d240afea-ff4c-4dbb-9f1e-6ccc9717db22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:21:08.684016 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.683986    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d240afea-ff4c-4dbb-9f1e-6ccc9717db22" (UID: "d240afea-ff4c-4dbb-9f1e-6ccc9717db22"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:21:08.756332 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.756307    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-kserve-provision-location\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:21:08.756416 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.756335    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:21:08.756416 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.756352    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-cabundle-cert\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:21:08.756416 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.756366    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjllp\" (UniqueName: \"kubernetes.io/projected/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-kube-api-access-vjllp\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:21:08.756416 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.756378    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:21:08.756416 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.756391    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:21:08.756416 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.756403    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:21:08.756416 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.756413    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d240afea-ff4c-4dbb-9f1e-6ccc9717db22-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:21:08.804902 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.804873    2565 generic.go:356] "Generic (PLEG): container finished" podID="d240afea-ff4c-4dbb-9f1e-6ccc9717db22" containerID="32883d8c0723ec1c582d2fc0efd5d60658147dccb6ca97b9a611b78dd3679580" exitCode=0
Sep 17 19:21:08.805018 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.804922    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"
Sep 17 19:21:08.805018 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.804964    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz" event={"ID":"d240afea-ff4c-4dbb-9f1e-6ccc9717db22","Type":"ContainerDied","Data":"32883d8c0723ec1c582d2fc0efd5d60658147dccb6ca97b9a611b78dd3679580"}
Sep 17 19:21:08.805095 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.805023    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz" event={"ID":"d240afea-ff4c-4dbb-9f1e-6ccc9717db22","Type":"ContainerDied","Data":"1a84b5fe95522a74c39cfa0b08ad95ae901a7409249bacc3fb58805f6d145239"}
Sep 17 19:21:08.805095 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.805046    2565 scope.go:117] "RemoveContainer" containerID="32883d8c0723ec1c582d2fc0efd5d60658147dccb6ca97b9a611b78dd3679580"
Sep 17 19:21:08.815614 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.815595    2565 scope.go:117] "RemoveContainer" containerID="4084ca9613228eb7096e8cd53db2468f772109409f360b46e86b51265942f432"
Sep 17 19:21:08.825409 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.825386    2565 scope.go:117] "RemoveContainer" containerID="32883d8c0723ec1c582d2fc0efd5d60658147dccb6ca97b9a611b78dd3679580"
Sep 17 19:21:08.825698 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:21:08.825673    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32883d8c0723ec1c582d2fc0efd5d60658147dccb6ca97b9a611b78dd3679580\": container with ID starting with 32883d8c0723ec1c582d2fc0efd5d60658147dccb6ca97b9a611b78dd3679580 not found: ID does not exist" containerID="32883d8c0723ec1c582d2fc0efd5d60658147dccb6ca97b9a611b78dd3679580"
Sep 17 19:21:08.825786 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.825707    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32883d8c0723ec1c582d2fc0efd5d60658147dccb6ca97b9a611b78dd3679580"} err="failed to get container status \"32883d8c0723ec1c582d2fc0efd5d60658147dccb6ca97b9a611b78dd3679580\": rpc error: code = NotFound desc = could not find container \"32883d8c0723ec1c582d2fc0efd5d60658147dccb6ca97b9a611b78dd3679580\": container with ID starting with 32883d8c0723ec1c582d2fc0efd5d60658147dccb6ca97b9a611b78dd3679580 not found: ID does not exist"
Sep 17 19:21:08.825786 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.825724    2565 scope.go:117] "RemoveContainer" containerID="4084ca9613228eb7096e8cd53db2468f772109409f360b46e86b51265942f432"
Sep 17 19:21:08.825996 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:21:08.825976    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4084ca9613228eb7096e8cd53db2468f772109409f360b46e86b51265942f432\": container with ID starting with 4084ca9613228eb7096e8cd53db2468f772109409f360b46e86b51265942f432 not found: ID does not exist" containerID="4084ca9613228eb7096e8cd53db2468f772109409f360b46e86b51265942f432"
Sep 17 19:21:08.826110 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.826007    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4084ca9613228eb7096e8cd53db2468f772109409f360b46e86b51265942f432"} err="failed to get container status \"4084ca9613228eb7096e8cd53db2468f772109409f360b46e86b51265942f432\": rpc error: code = NotFound desc = could not find container \"4084ca9613228eb7096e8cd53db2468f772109409f360b46e86b51265942f432\": container with ID starting with 4084ca9613228eb7096e8cd53db2468f772109409f360b46e86b51265942f432 not found: ID does not exist"
Sep 17 19:21:08.829690 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.829663    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"]
Sep 17 19:21:08.833062 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.833038    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-5f848df9d5-76bkz"]
Sep 17 19:21:08.989950 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:08.989912    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng" podUID="d47ba3fa-9af3-4ba4-9155-abc592d68000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 17 19:21:10.611680 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:10.611639    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d240afea-ff4c-4dbb-9f1e-6ccc9717db22" path="/var/lib/kubelet/pods/d240afea-ff4c-4dbb-9f1e-6ccc9717db22/volumes"
Sep 17 19:21:16.306015 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:16.305978    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-gbj6g"]
Sep 17 19:21:16.306583 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:16.306561    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d240afea-ff4c-4dbb-9f1e-6ccc9717db22" containerName="main"
Sep 17 19:21:16.306673 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:16.306587    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d240afea-ff4c-4dbb-9f1e-6ccc9717db22" containerName="main"
Sep 17 19:21:16.306673 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:16.306617    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d240afea-ff4c-4dbb-9f1e-6ccc9717db22" containerName="storage-initializer"
Sep 17 19:21:16.306673 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:16.306627    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d240afea-ff4c-4dbb-9f1e-6ccc9717db22" containerName="storage-initializer"
Sep 17 19:21:16.313873 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:16.307354    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d240afea-ff4c-4dbb-9f1e-6ccc9717db22" containerName="main"
Sep 17 19:21:16.314164 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:16.314124    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-gbj6g"
Sep 17 19:21:16.317124 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:16.317096    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 17 19:21:16.317294 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:16.317151    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 17 19:21:16.317969 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:16.317935    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 17 19:21:16.318117 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:16.317979    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-gbj6g"]
Sep 17 19:21:16.318336 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:16.318315    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-wmqdv\""
Sep 17 19:21:16.417582 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:16.417536    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bz2n\" (UniqueName: \"kubernetes.io/projected/f7adf157-103f-4bc6-bab8-e034195ad99c-kube-api-access-6bz2n\") pod \"e2e-pvc-lora-download-gbj6g\" (UID: \"f7adf157-103f-4bc6-bab8-e034195ad99c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-gbj6g"
Sep 17 19:21:16.518268 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:16.518232    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bz2n\" (UniqueName: \"kubernetes.io/projected/f7adf157-103f-4bc6-bab8-e034195ad99c-kube-api-access-6bz2n\") pod \"e2e-pvc-lora-download-gbj6g\" (UID: \"f7adf157-103f-4bc6-bab8-e034195ad99c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-gbj6g"
Sep 17 19:21:16.527394 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:16.527371    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bz2n\" (UniqueName: \"kubernetes.io/projected/f7adf157-103f-4bc6-bab8-e034195ad99c-kube-api-access-6bz2n\") pod \"e2e-pvc-lora-download-gbj6g\" (UID: \"f7adf157-103f-4bc6-bab8-e034195ad99c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-gbj6g"
Sep 17 19:21:18.990560 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:18.990520    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng" podUID="d47ba3fa-9af3-4ba4-9155-abc592d68000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 17 19:21:25.793789 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:25.793710    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-866c5d0e-37c4-4911-9ef1-61c18f9cfed3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09bcc94a89b73146d\") pod \"e2e-pvc-lora-download-gbj6g\" (UID: \"f7adf157-103f-4bc6-bab8-e034195ad99c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-gbj6g"
Sep 17 19:21:25.799429 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:25.799403    2565 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-866c5d0e-37c4-4911-9ef1-61c18f9cfed3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09bcc94a89b73146d\") pod \"e2e-pvc-lora-download-gbj6g\" (UID: \"f7adf157-103f-4bc6-bab8-e034195ad99c\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-gbj6g"
Sep 17 19:21:25.894349 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:25.894318    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-866c5d0e-37c4-4911-9ef1-61c18f9cfed3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09bcc94a89b73146d\") pod \"e2e-pvc-lora-download-gbj6g\" (UID: \"f7adf157-103f-4bc6-bab8-e034195ad99c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-gbj6g"
Sep 17 19:21:25.894500 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:25.894425    2565 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-866c5d0e-37c4-4911-9ef1-61c18f9cfed3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09bcc94a89b73146d\") pod \"e2e-pvc-lora-download-gbj6g\" (UID: \"f7adf157-103f-4bc6-bab8-e034195ad99c\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-gbj6g"
Sep 17 19:21:25.897427 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:25.897406    2565 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-866c5d0e-37c4-4911-9ef1-61c18f9cfed3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09bcc94a89b73146d\") pod \"e2e-pvc-lora-download-gbj6g\" (UID: \"f7adf157-103f-4bc6-bab8-e034195ad99c\") DevicePath \"csi-9ae73291776fc149187ee36bc333d7c7ca72b4e655d4fd9da7ac162c43021d0d\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-gbj6g"
Sep 17 19:21:26.196081 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:26.196049    2565 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-866c5d0e-37c4-4911-9ef1-61c18f9cfed3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09bcc94a89b73146d\") pod \"e2e-pvc-lora-download-gbj6g\" (UID: \"f7adf157-103f-4bc6-bab8-e034195ad99c\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b0b06c278443ccecf00d3e7e45f83f92e557e0fa0af3590fad2fa0391d025b6f/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-gbj6g"
Sep 17 19:21:26.210126 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:26.210100    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-866c5d0e-37c4-4911-9ef1-61c18f9cfed3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09bcc94a89b73146d\") pod \"e2e-pvc-lora-download-gbj6g\" (UID: \"f7adf157-103f-4bc6-bab8-e034195ad99c\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-gbj6g"
Sep 17 19:21:26.225458 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:26.225435    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-gbj6g"
Sep 17 19:21:26.496414 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:26.496387    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-gbj6g"]
Sep 17 19:21:26.499530 ip-10-0-139-75 kubenswrapper[2565]: W0917 19:21:26.499502    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7adf157_103f_4bc6_bab8_e034195ad99c.slice/crio-4fd3c6c6f01f7105096a7b86d99c5ba1b50a1fd8a3e277caca4857f71c860ac0 WatchSource:0}: Error finding container 4fd3c6c6f01f7105096a7b86d99c5ba1b50a1fd8a3e277caca4857f71c860ac0: Status 404 returned error can't find the container with id 4fd3c6c6f01f7105096a7b86d99c5ba1b50a1fd8a3e277caca4857f71c860ac0
Sep 17 19:21:26.908705 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:26.908668    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-gbj6g" event={"ID":"f7adf157-103f-4bc6-bab8-e034195ad99c","Type":"ContainerStarted","Data":"253aa2e15a953b76e1dbe5faf1df30deae72255ad7aac75795e5dd6b8829745c"}
Sep 17 19:21:26.909218 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:26.908712    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-gbj6g" event={"ID":"f7adf157-103f-4bc6-bab8-e034195ad99c","Type":"ContainerStarted","Data":"4fd3c6c6f01f7105096a7b86d99c5ba1b50a1fd8a3e277caca4857f71c860ac0"}
Sep 17 19:21:26.928164 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:26.928109    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-gbj6g" podStartSLOduration=13.928094439 podStartE2EDuration="13.928094439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:21:26.926005077 +0000 UTC m=+3144.858206597" watchObservedRunningTime="2026-09-17 19:21:26.928094439 +0000 UTC m=+3144.860295958"
Sep 17 19:21:28.926546 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:28.926508    2565 generic.go:356] "Generic (PLEG): container finished" podID="f7adf157-103f-4bc6-bab8-e034195ad99c" containerID="253aa2e15a953b76e1dbe5faf1df30deae72255ad7aac75795e5dd6b8829745c" exitCode=0
Sep 17 19:21:28.927050 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:28.926577    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-gbj6g" event={"ID":"f7adf157-103f-4bc6-bab8-e034195ad99c","Type":"ContainerDied","Data":"253aa2e15a953b76e1dbe5faf1df30deae72255ad7aac75795e5dd6b8829745c"}
Sep 17 19:21:28.990343 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:28.990308    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng" podUID="d47ba3fa-9af3-4ba4-9155-abc592d68000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 17 19:21:30.073783 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:30.073759    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-gbj6g"
Sep 17 19:21:30.227862 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:30.227792    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09bcc94a89b73146d\") pod \"f7adf157-103f-4bc6-bab8-e034195ad99c\" (UID: \"f7adf157-103f-4bc6-bab8-e034195ad99c\") "
Sep 17 19:21:30.227988 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:30.227900    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bz2n\" (UniqueName: \"kubernetes.io/projected/f7adf157-103f-4bc6-bab8-e034195ad99c-kube-api-access-6bz2n\") pod \"f7adf157-103f-4bc6-bab8-e034195ad99c\" (UID: \"f7adf157-103f-4bc6-bab8-e034195ad99c\") "
Sep 17 19:21:30.230172 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:30.230136    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7adf157-103f-4bc6-bab8-e034195ad99c-kube-api-access-6bz2n" (OuterVolumeSpecName: "kube-api-access-6bz2n") pod "f7adf157-103f-4bc6-bab8-e034195ad99c" (UID: "f7adf157-103f-4bc6-bab8-e034195ad99c"). InnerVolumeSpecName "kube-api-access-6bz2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:21:30.231660 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:30.231635    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-09bcc94a89b73146d" (OuterVolumeSpecName: "model-storage") pod "f7adf157-103f-4bc6-bab8-e034195ad99c" (UID: "f7adf157-103f-4bc6-bab8-e034195ad99c"). InnerVolumeSpecName "pvc-866c5d0e-37c4-4911-9ef1-61c18f9cfed3". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 19:21:30.329292 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:30.329256    2565 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-866c5d0e-37c4-4911-9ef1-61c18f9cfed3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09bcc94a89b73146d\") on node \"ip-10-0-139-75.ec2.internal\" "
Sep 17 19:21:30.329292 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:30.329288    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bz2n\" (UniqueName: \"kubernetes.io/projected/f7adf157-103f-4bc6-bab8-e034195ad99c-kube-api-access-6bz2n\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:21:30.938449 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:30.938421    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-gbj6g"
Sep 17 19:21:30.938449 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:30.938437    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-gbj6g" event={"ID":"f7adf157-103f-4bc6-bab8-e034195ad99c","Type":"ContainerDied","Data":"4fd3c6c6f01f7105096a7b86d99c5ba1b50a1fd8a3e277caca4857f71c860ac0"}
Sep 17 19:21:30.938627 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:30.938473    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fd3c6c6f01f7105096a7b86d99c5ba1b50a1fd8a3e277caca4857f71c860ac0"
Sep 17 19:21:31.299617 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:31.299596    2565 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-866c5d0e-37c4-4911-9ef1-61c18f9cfed3" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-09bcc94a89b73146d") on node "ip-10-0-139-75.ec2.internal"
Sep 17 19:21:31.337515 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:31.337489    2565 reconciler_common.go:299] "Volume detached for volume \"pvc-866c5d0e-37c4-4911-9ef1-61c18f9cfed3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09bcc94a89b73146d\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"csi-9ae73291776fc149187ee36bc333d7c7ca72b4e655d4fd9da7ac162c43021d0d\""
Sep 17 19:21:36.855061 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:36.855026    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"]
Sep 17 19:21:36.855468 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:36.855420    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7adf157-103f-4bc6-bab8-e034195ad99c" containerName="storage-initializer"
Sep 17 19:21:36.855468 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:36.855433    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7adf157-103f-4bc6-bab8-e034195ad99c" containerName="storage-initializer"
Sep 17 19:21:36.855545 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:36.855497    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7adf157-103f-4bc6-bab8-e034195ad99c" containerName="storage-initializer"
Sep 17 19:21:36.858853 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:36.858829    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:36.861590 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:36.861547    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"lora-pvc-single-adapter-test-kserve-self-signed-certs\""
Sep 17 19:21:36.861590 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:36.861557    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 17 19:21:36.861590 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:36.861570    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 17 19:21:36.861910 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:36.861893    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-wmqdv\""
Sep 17 19:21:36.862009 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:36.861989    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:21:36.862101 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:36.862087    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 17 19:21:36.876498 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:36.876289    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"]
Sep 17 19:21:36.984645 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:36.984617    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/895fcd89-1728-4eaa-9977-584c903239f1-dshm\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:36.984785 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:36.984660    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/895fcd89-1728-4eaa-9977-584c903239f1-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:36.984785 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:36.984717    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txx9w\" (UniqueName: \"kubernetes.io/projected/895fcd89-1728-4eaa-9977-584c903239f1-kube-api-access-txx9w\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:36.984785 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:36.984745    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/895fcd89-1728-4eaa-9977-584c903239f1-home\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:36.984785 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:36.984770    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/895fcd89-1728-4eaa-9977-584c903239f1-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:36.984937 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:36.984787    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/895fcd89-1728-4eaa-9977-584c903239f1-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:36.984937 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:36.984831    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/895fcd89-1728-4eaa-9977-584c903239f1-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:36.984937 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:36.984873    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/895fcd89-1728-4eaa-9977-584c903239f1-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:37.085279 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:37.085246    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/895fcd89-1728-4eaa-9977-584c903239f1-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:37.085432 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:37.085403    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txx9w\" (UniqueName: \"kubernetes.io/projected/895fcd89-1728-4eaa-9977-584c903239f1-kube-api-access-txx9w\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:37.085482 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:37.085463    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/895fcd89-1728-4eaa-9977-584c903239f1-home\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:37.085519 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:37.085505    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/895fcd89-1728-4eaa-9977-584c903239f1-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:37.085559 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:37.085537    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/895fcd89-1728-4eaa-9977-584c903239f1-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:37.085801 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:37.085650    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/895fcd89-1728-4eaa-9977-584c903239f1-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:37.085801 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:37.085695    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/895fcd89-1728-4eaa-9977-584c903239f1-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:37.085801 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:37.085740    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/895fcd89-1728-4eaa-9977-584c903239f1-dshm\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:37.086038 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:37.085898    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/895fcd89-1728-4eaa-9977-584c903239f1-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:37.086038 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:37.085953    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/895fcd89-1728-4eaa-9977-584c903239f1-home\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:37.086177 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:37.086074    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/895fcd89-1728-4eaa-9977-584c903239f1-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:37.086177 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:37.086157    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/895fcd89-1728-4eaa-9977-584c903239f1-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:37.086479 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:37.086458    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/895fcd89-1728-4eaa-9977-584c903239f1-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:37.087926 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:37.087899    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/895fcd89-1728-4eaa-9977-584c903239f1-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:37.088410 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:37.088390    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/895fcd89-1728-4eaa-9977-584c903239f1-dshm\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:37.097494 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:37.097475    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txx9w\" (UniqueName: \"kubernetes.io/projected/895fcd89-1728-4eaa-9977-584c903239f1-kube-api-access-txx9w\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:38.990559 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:38.990512    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng" podUID="d47ba3fa-9af3-4ba4-9155-abc592d68000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 17 19:21:46.273895 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:46.273861    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-866c5d0e-37c4-4911-9ef1-61c18f9cfed3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09bcc94a89b73146d\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:46.279566 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:46.279526    2565 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-866c5d0e-37c4-4911-9ef1-61c18f9cfed3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09bcc94a89b73146d\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:46.374960 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:46.374925    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-866c5d0e-37c4-4911-9ef1-61c18f9cfed3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09bcc94a89b73146d\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:46.375108 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:46.375024    2565 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-866c5d0e-37c4-4911-9ef1-61c18f9cfed3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09bcc94a89b73146d\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:46.378530 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:46.378506    2565 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-866c5d0e-37c4-4911-9ef1-61c18f9cfed3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09bcc94a89b73146d\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") DevicePath \"csi-9ae73291776fc149187ee36bc333d7c7ca72b4e655d4fd9da7ac162c43021d0d\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:46.471563 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:46.471532    2565 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-866c5d0e-37c4-4911-9ef1-61c18f9cfed3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09bcc94a89b73146d\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b0b06c278443ccecf00d3e7e45f83f92e557e0fa0af3590fad2fa0391d025b6f/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:46.491406 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:46.491377    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-866c5d0e-37c4-4911-9ef1-61c18f9cfed3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09bcc94a89b73146d\") pod \"lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:46.772288 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:46.772245    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:47.104332 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:47.104251    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"]
Sep 17 19:21:47.107372 ip-10-0-139-75 kubenswrapper[2565]: W0917 19:21:47.107342    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895fcd89_1728_4eaa_9977_584c903239f1.slice/crio-3c01a60078dd7fab266ea9a69d137deed353c3de3ea782d1660e28411763c7eb WatchSource:0}: Error finding container 3c01a60078dd7fab266ea9a69d137deed353c3de3ea782d1660e28411763c7eb: Status 404 returned error can't find the container with id 3c01a60078dd7fab266ea9a69d137deed353c3de3ea782d1660e28411763c7eb
Sep 17 19:21:48.039013 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:48.038974    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5" event={"ID":"895fcd89-1728-4eaa-9977-584c903239f1","Type":"ContainerStarted","Data":"678a1d27479a806101b0b1c7875bb6a6a2bc9c2203af2524d67c1f4deca2f25a"}
Sep 17 19:21:48.039558 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:48.039025    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5" event={"ID":"895fcd89-1728-4eaa-9977-584c903239f1","Type":"ContainerStarted","Data":"3c01a60078dd7fab266ea9a69d137deed353c3de3ea782d1660e28411763c7eb"}
Sep 17 19:21:48.990344 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:48.990300    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng" podUID="d47ba3fa-9af3-4ba4-9155-abc592d68000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 17 19:21:50.053649 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:50.053609    2565 generic.go:356] "Generic (PLEG): container finished" podID="895fcd89-1728-4eaa-9977-584c903239f1" containerID="678a1d27479a806101b0b1c7875bb6a6a2bc9c2203af2524d67c1f4deca2f25a" exitCode=0
Sep 17 19:21:50.054169 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:50.053682    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5" event={"ID":"895fcd89-1728-4eaa-9977-584c903239f1","Type":"ContainerDied","Data":"678a1d27479a806101b0b1c7875bb6a6a2bc9c2203af2524d67c1f4deca2f25a"}
Sep 17 19:21:51.061728 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:51.061692    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5" event={"ID":"895fcd89-1728-4eaa-9977-584c903239f1","Type":"ContainerStarted","Data":"32dceb80d18be382646821b67e030fabdb4da089953370e6a13ba60f680eead0"}
Sep 17 19:21:51.090922 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:51.090857    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5" podStartSLOduration=12.820037922000001 podStartE2EDuration="15.090842664s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.270804742s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:21:51.088357417 +0000 UTC m=+3169.020558938" watchObservedRunningTime="2026-09-17 19:21:51.090842664 +0000 UTC m=+3169.023044182"
Sep 17 19:21:51.773431 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:51.773384    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:56.773023 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:56.772969    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:21:56.775076 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:56.775044    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5" podUID="895fcd89-1728-4eaa-9977-584c903239f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 17 19:21:58.990409 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:21:58.990362    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng" podUID="d47ba3fa-9af3-4ba4-9155-abc592d68000" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 17 19:22:06.773211 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:06.773148    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5" podUID="895fcd89-1728-4eaa-9977-584c903239f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 17 19:22:08.999823 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:08.999788    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:22:09.007701 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:09.007671    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:22:16.773247 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:16.773173    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5" podUID="895fcd89-1728-4eaa-9977-584c903239f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 17 19:22:18.274136 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:18.274101    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"]
Sep 17 19:22:26.773020 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:26.772972    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5" podUID="895fcd89-1728-4eaa-9977-584c903239f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 17 19:22:33.297022 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:33.296956    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng" podUID="d47ba3fa-9af3-4ba4-9155-abc592d68000" containerName="main" containerID="cri-o://96d95a18936d31c2b8b982707bebb8abcf0663934772b1d081e644373842cb8d" gracePeriod=45
Sep 17 19:22:34.055209 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.055168    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:22:34.079933 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.079854    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d47ba3fa-9af3-4ba4-9155-abc592d68000-kserve-provision-location\") pod \"d47ba3fa-9af3-4ba4-9155-abc592d68000\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") "
Sep 17 19:22:34.079933 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.079923    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d47ba3fa-9af3-4ba4-9155-abc592d68000-tls-certs\") pod \"d47ba3fa-9af3-4ba4-9155-abc592d68000\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") "
Sep 17 19:22:34.080138 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.079956    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d47ba3fa-9af3-4ba4-9155-abc592d68000-model-cache\") pod \"d47ba3fa-9af3-4ba4-9155-abc592d68000\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") "
Sep 17 19:22:34.080138 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.080041    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d47ba3fa-9af3-4ba4-9155-abc592d68000-cabundle-cert\") pod \"d47ba3fa-9af3-4ba4-9155-abc592d68000\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") "
Sep 17 19:22:34.080138 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.080068    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-284jv\" (UniqueName: \"kubernetes.io/projected/d47ba3fa-9af3-4ba4-9155-abc592d68000-kube-api-access-284jv\") pod \"d47ba3fa-9af3-4ba4-9155-abc592d68000\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") "
Sep 17 19:22:34.080138 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.080096    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d47ba3fa-9af3-4ba4-9155-abc592d68000-home\") pod \"d47ba3fa-9af3-4ba4-9155-abc592d68000\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") "
Sep 17 19:22:34.080138 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.080135    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d47ba3fa-9af3-4ba4-9155-abc592d68000-dshm\") pod \"d47ba3fa-9af3-4ba4-9155-abc592d68000\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") "
Sep 17 19:22:34.080438 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.080162    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d47ba3fa-9af3-4ba4-9155-abc592d68000-tmp-dir\") pod \"d47ba3fa-9af3-4ba4-9155-abc592d68000\" (UID: \"d47ba3fa-9af3-4ba4-9155-abc592d68000\") "
Sep 17 19:22:34.081372 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.080577    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d47ba3fa-9af3-4ba4-9155-abc592d68000-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d47ba3fa-9af3-4ba4-9155-abc592d68000" (UID: "d47ba3fa-9af3-4ba4-9155-abc592d68000"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:22:34.081372 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.081160    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47ba3fa-9af3-4ba4-9155-abc592d68000-home" (OuterVolumeSpecName: "home") pod "d47ba3fa-9af3-4ba4-9155-abc592d68000" (UID: "d47ba3fa-9af3-4ba4-9155-abc592d68000"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:22:34.081372 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.081336    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47ba3fa-9af3-4ba4-9155-abc592d68000-model-cache" (OuterVolumeSpecName: "model-cache") pod "d47ba3fa-9af3-4ba4-9155-abc592d68000" (UID: "d47ba3fa-9af3-4ba4-9155-abc592d68000"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:22:34.085178 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.084531    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d47ba3fa-9af3-4ba4-9155-abc592d68000-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d47ba3fa-9af3-4ba4-9155-abc592d68000" (UID: "d47ba3fa-9af3-4ba4-9155-abc592d68000"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:22:34.085178 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.084921    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47ba3fa-9af3-4ba4-9155-abc592d68000-dshm" (OuterVolumeSpecName: "dshm") pod "d47ba3fa-9af3-4ba4-9155-abc592d68000" (UID: "d47ba3fa-9af3-4ba4-9155-abc592d68000"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:22:34.086154 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.086124    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d47ba3fa-9af3-4ba4-9155-abc592d68000-kube-api-access-284jv" (OuterVolumeSpecName: "kube-api-access-284jv") pod "d47ba3fa-9af3-4ba4-9155-abc592d68000" (UID: "d47ba3fa-9af3-4ba4-9155-abc592d68000"). InnerVolumeSpecName "kube-api-access-284jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:22:34.096868 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.096845    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47ba3fa-9af3-4ba4-9155-abc592d68000-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d47ba3fa-9af3-4ba4-9155-abc592d68000" (UID: "d47ba3fa-9af3-4ba4-9155-abc592d68000"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:22:34.111064 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.111035    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47ba3fa-9af3-4ba4-9155-abc592d68000-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d47ba3fa-9af3-4ba4-9155-abc592d68000" (UID: "d47ba3fa-9af3-4ba4-9155-abc592d68000"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:22:34.181125 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.181103    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d47ba3fa-9af3-4ba4-9155-abc592d68000-cabundle-cert\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:22:34.181125 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.181125    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-284jv\" (UniqueName: \"kubernetes.io/projected/d47ba3fa-9af3-4ba4-9155-abc592d68000-kube-api-access-284jv\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:22:34.181278 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.181134    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d47ba3fa-9af3-4ba4-9155-abc592d68000-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:22:34.181278 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.181143    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d47ba3fa-9af3-4ba4-9155-abc592d68000-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:22:34.181278 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.181150    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d47ba3fa-9af3-4ba4-9155-abc592d68000-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:22:34.181278 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.181159    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d47ba3fa-9af3-4ba4-9155-abc592d68000-kserve-provision-location\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:22:34.181278 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.181168    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d47ba3fa-9af3-4ba4-9155-abc592d68000-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:22:34.181278 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.181177    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d47ba3fa-9af3-4ba4-9155-abc592d68000-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:22:34.312184 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.312152    2565 generic.go:356] "Generic (PLEG): container finished" podID="d47ba3fa-9af3-4ba4-9155-abc592d68000" containerID="96d95a18936d31c2b8b982707bebb8abcf0663934772b1d081e644373842cb8d" exitCode=0
Sep 17 19:22:34.312571 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.312240    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"
Sep 17 19:22:34.312571 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.312236    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng" event={"ID":"d47ba3fa-9af3-4ba4-9155-abc592d68000","Type":"ContainerDied","Data":"96d95a18936d31c2b8b982707bebb8abcf0663934772b1d081e644373842cb8d"}
Sep 17 19:22:34.312571 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.312368    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng" event={"ID":"d47ba3fa-9af3-4ba4-9155-abc592d68000","Type":"ContainerDied","Data":"8ac9053486261987059a875a1a88564de68034cd5a4bd513ead295688be311bc"}
Sep 17 19:22:34.312571 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.312392    2565 scope.go:117] "RemoveContainer" containerID="96d95a18936d31c2b8b982707bebb8abcf0663934772b1d081e644373842cb8d"
Sep 17 19:22:34.321821 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.321805    2565 scope.go:117] "RemoveContainer" containerID="a791f2c4e874244fab6159ebda8114ec9b2dfab2381bee4df681c49016f7ceef"
Sep 17 19:22:34.330014 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.329973    2565 scope.go:117] "RemoveContainer" containerID="96d95a18936d31c2b8b982707bebb8abcf0663934772b1d081e644373842cb8d"
Sep 17 19:22:34.330307 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:22:34.330288    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96d95a18936d31c2b8b982707bebb8abcf0663934772b1d081e644373842cb8d\": container with ID starting with 96d95a18936d31c2b8b982707bebb8abcf0663934772b1d081e644373842cb8d not found: ID does not exist" containerID="96d95a18936d31c2b8b982707bebb8abcf0663934772b1d081e644373842cb8d"
Sep 17 19:22:34.330355 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.330317    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96d95a18936d31c2b8b982707bebb8abcf0663934772b1d081e644373842cb8d"} err="failed to get container status \"96d95a18936d31c2b8b982707bebb8abcf0663934772b1d081e644373842cb8d\": rpc error: code = NotFound desc = could not find container \"96d95a18936d31c2b8b982707bebb8abcf0663934772b1d081e644373842cb8d\": container with ID starting with 96d95a18936d31c2b8b982707bebb8abcf0663934772b1d081e644373842cb8d not found: ID does not exist"
Sep 17 19:22:34.330355 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.330333    2565 scope.go:117] "RemoveContainer" containerID="a791f2c4e874244fab6159ebda8114ec9b2dfab2381bee4df681c49016f7ceef"
Sep 17 19:22:34.330597 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:22:34.330578    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a791f2c4e874244fab6159ebda8114ec9b2dfab2381bee4df681c49016f7ceef\": container with ID starting with a791f2c4e874244fab6159ebda8114ec9b2dfab2381bee4df681c49016f7ceef not found: ID does not exist" containerID="a791f2c4e874244fab6159ebda8114ec9b2dfab2381bee4df681c49016f7ceef"
Sep 17 19:22:34.330645 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.330604    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a791f2c4e874244fab6159ebda8114ec9b2dfab2381bee4df681c49016f7ceef"} err="failed to get container status \"a791f2c4e874244fab6159ebda8114ec9b2dfab2381bee4df681c49016f7ceef\": rpc error: code = NotFound desc = could not find container \"a791f2c4e874244fab6159ebda8114ec9b2dfab2381bee4df681c49016f7ceef\": container with ID starting with a791f2c4e874244fab6159ebda8114ec9b2dfab2381bee4df681c49016f7ceef not found: ID does not exist"
Sep 17 19:22:34.341224 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.341186    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"]
Sep 17 19:22:34.348124 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.348104    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-8c4cb96dc-whlng"]
Sep 17 19:22:34.607385 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:34.607307    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d47ba3fa-9af3-4ba4-9155-abc592d68000" path="/var/lib/kubelet/pods/d47ba3fa-9af3-4ba4-9155-abc592d68000/volumes"
Sep 17 19:22:36.772939 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:36.772898    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5" podUID="895fcd89-1728-4eaa-9977-584c903239f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 17 19:22:46.773702 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:46.773656    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5" podUID="895fcd89-1728-4eaa-9977-584c903239f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 17 19:22:56.772990 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:22:56.772900    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5" podUID="895fcd89-1728-4eaa-9977-584c903239f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 17 19:23:06.783487 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:06.783453    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:23:06.791152 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:06.791121    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:23:32.513657 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:32.513623    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5_895fcd89-1728-4eaa-9977-584c903239f1/storage-initializer/0.log"
Sep 17 19:23:32.522263 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:32.522239    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5_895fcd89-1728-4eaa-9977-584c903239f1/main/0.log"
Sep 17 19:23:33.437752 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:33.437717    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"]
Sep 17 19:23:48.460045 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:48.459979    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5" podUID="895fcd89-1728-4eaa-9977-584c903239f1" containerName="main" containerID="cri-o://32dceb80d18be382646821b67e030fabdb4da089953370e6a13ba60f680eead0" gracePeriod=45
Sep 17 19:23:48.784552 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:48.784443    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5" podUID="895fcd89-1728-4eaa-9977-584c903239f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 17 19:23:49.120217 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.120181    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:23:49.185256 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.185228    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/895fcd89-1728-4eaa-9977-584c903239f1-home\") pod \"895fcd89-1728-4eaa-9977-584c903239f1\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") "
Sep 17 19:23:49.185395 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.185279    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/895fcd89-1728-4eaa-9977-584c903239f1-model-cache\") pod \"895fcd89-1728-4eaa-9977-584c903239f1\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") "
Sep 17 19:23:49.185395 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.185302    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/895fcd89-1728-4eaa-9977-584c903239f1-dshm\") pod \"895fcd89-1728-4eaa-9977-584c903239f1\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") "
Sep 17 19:23:49.185395 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.185320    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/895fcd89-1728-4eaa-9977-584c903239f1-kserve-provision-location\") pod \"895fcd89-1728-4eaa-9977-584c903239f1\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") "
Sep 17 19:23:49.185584 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.185459    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"lora-pvc-lora-adapter-1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09bcc94a89b73146d\") pod \"895fcd89-1728-4eaa-9977-584c903239f1\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") "
Sep 17 19:23:49.185584 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.185492    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/895fcd89-1728-4eaa-9977-584c903239f1-cabundle-cert\") pod \"895fcd89-1728-4eaa-9977-584c903239f1\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") "
Sep 17 19:23:49.185584 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.185532    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/895fcd89-1728-4eaa-9977-584c903239f1-tmp-dir\") pod \"895fcd89-1728-4eaa-9977-584c903239f1\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") "
Sep 17 19:23:49.185584 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.185532    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895fcd89-1728-4eaa-9977-584c903239f1-model-cache" (OuterVolumeSpecName: "model-cache") pod "895fcd89-1728-4eaa-9977-584c903239f1" (UID: "895fcd89-1728-4eaa-9977-584c903239f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:23:49.185584 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.185576    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/895fcd89-1728-4eaa-9977-584c903239f1-tls-certs\") pod \"895fcd89-1728-4eaa-9977-584c903239f1\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") "
Sep 17 19:23:49.185859 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.185587    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895fcd89-1728-4eaa-9977-584c903239f1-home" (OuterVolumeSpecName: "home") pod "895fcd89-1728-4eaa-9977-584c903239f1" (UID: "895fcd89-1728-4eaa-9977-584c903239f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:23:49.185859 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.185603    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txx9w\" (UniqueName: \"kubernetes.io/projected/895fcd89-1728-4eaa-9977-584c903239f1-kube-api-access-txx9w\") pod \"895fcd89-1728-4eaa-9977-584c903239f1\" (UID: \"895fcd89-1728-4eaa-9977-584c903239f1\") "
Sep 17 19:23:49.185859 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.185821    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/895fcd89-1728-4eaa-9977-584c903239f1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "895fcd89-1728-4eaa-9977-584c903239f1" (UID: "895fcd89-1728-4eaa-9977-584c903239f1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:23:49.186034 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.185861    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/895fcd89-1728-4eaa-9977-584c903239f1-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:23:49.186034 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.185879    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/895fcd89-1728-4eaa-9977-584c903239f1-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:23:49.188180 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.187976    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/895fcd89-1728-4eaa-9977-584c903239f1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "895fcd89-1728-4eaa-9977-584c903239f1" (UID: "895fcd89-1728-4eaa-9977-584c903239f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:23:49.188328 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.188301    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895fcd89-1728-4eaa-9977-584c903239f1-kube-api-access-txx9w" (OuterVolumeSpecName: "kube-api-access-txx9w") pod "895fcd89-1728-4eaa-9977-584c903239f1" (UID: "895fcd89-1728-4eaa-9977-584c903239f1"). InnerVolumeSpecName "kube-api-access-txx9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:23:49.188404 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.188386    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895fcd89-1728-4eaa-9977-584c903239f1-dshm" (OuterVolumeSpecName: "dshm") pod "895fcd89-1728-4eaa-9977-584c903239f1" (UID: "895fcd89-1728-4eaa-9977-584c903239f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:23:49.191074 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.191053    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-09bcc94a89b73146d" (OuterVolumeSpecName: "lora-pvc-lora-adapter-1") pod "895fcd89-1728-4eaa-9977-584c903239f1" (UID: "895fcd89-1728-4eaa-9977-584c903239f1"). InnerVolumeSpecName "pvc-866c5d0e-37c4-4911-9ef1-61c18f9cfed3". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 19:23:49.199933 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.199904    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895fcd89-1728-4eaa-9977-584c903239f1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "895fcd89-1728-4eaa-9977-584c903239f1" (UID: "895fcd89-1728-4eaa-9977-584c903239f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:23:49.209953 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.209927    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895fcd89-1728-4eaa-9977-584c903239f1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "895fcd89-1728-4eaa-9977-584c903239f1" (UID: "895fcd89-1728-4eaa-9977-584c903239f1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:23:49.286650 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.286624    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/895fcd89-1728-4eaa-9977-584c903239f1-cabundle-cert\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:23:49.286650 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.286649    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/895fcd89-1728-4eaa-9977-584c903239f1-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:23:49.286881 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.286663    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/895fcd89-1728-4eaa-9977-584c903239f1-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:23:49.286881 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.286675    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txx9w\" (UniqueName: \"kubernetes.io/projected/895fcd89-1728-4eaa-9977-584c903239f1-kube-api-access-txx9w\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:23:49.286881 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.286687    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/895fcd89-1728-4eaa-9977-584c903239f1-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:23:49.286881 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.286699    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/895fcd89-1728-4eaa-9977-584c903239f1-kserve-provision-location\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:23:49.286881 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.286736    2565 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-866c5d0e-37c4-4911-9ef1-61c18f9cfed3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09bcc94a89b73146d\") on node \"ip-10-0-139-75.ec2.internal\" "
Sep 17 19:23:49.297827 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.297811    2565 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-866c5d0e-37c4-4911-9ef1-61c18f9cfed3" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-09bcc94a89b73146d") on node "ip-10-0-139-75.ec2.internal"
Sep 17 19:23:49.387460 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.387393    2565 reconciler_common.go:299] "Volume detached for volume \"pvc-866c5d0e-37c4-4911-9ef1-61c18f9cfed3\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-09bcc94a89b73146d\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"csi-9ae73291776fc149187ee36bc333d7c7ca72b4e655d4fd9da7ac162c43021d0d\""
Sep 17 19:23:49.731245 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.731195    2565 generic.go:356] "Generic (PLEG): container finished" podID="895fcd89-1728-4eaa-9977-584c903239f1" containerID="32dceb80d18be382646821b67e030fabdb4da089953370e6a13ba60f680eead0" exitCode=0
Sep 17 19:23:49.731707 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.731268    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"
Sep 17 19:23:49.731707 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.731281    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5" event={"ID":"895fcd89-1728-4eaa-9977-584c903239f1","Type":"ContainerDied","Data":"32dceb80d18be382646821b67e030fabdb4da089953370e6a13ba60f680eead0"}
Sep 17 19:23:49.731707 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.731336    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5" event={"ID":"895fcd89-1728-4eaa-9977-584c903239f1","Type":"ContainerDied","Data":"3c01a60078dd7fab266ea9a69d137deed353c3de3ea782d1660e28411763c7eb"}
Sep 17 19:23:49.731707 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.731370    2565 scope.go:117] "RemoveContainer" containerID="32dceb80d18be382646821b67e030fabdb4da089953370e6a13ba60f680eead0"
Sep 17 19:23:49.744171 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.744150    2565 scope.go:117] "RemoveContainer" containerID="678a1d27479a806101b0b1c7875bb6a6a2bc9c2203af2524d67c1f4deca2f25a"
Sep 17 19:23:49.751902 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.751883    2565 scope.go:117] "RemoveContainer" containerID="32dceb80d18be382646821b67e030fabdb4da089953370e6a13ba60f680eead0"
Sep 17 19:23:49.752179 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:23:49.752160    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32dceb80d18be382646821b67e030fabdb4da089953370e6a13ba60f680eead0\": container with ID starting with 32dceb80d18be382646821b67e030fabdb4da089953370e6a13ba60f680eead0 not found: ID does not exist" containerID="32dceb80d18be382646821b67e030fabdb4da089953370e6a13ba60f680eead0"
Sep 17 19:23:49.752234 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.752187    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32dceb80d18be382646821b67e030fabdb4da089953370e6a13ba60f680eead0"} err="failed to get container status \"32dceb80d18be382646821b67e030fabdb4da089953370e6a13ba60f680eead0\": rpc error: code = NotFound desc = could not find container \"32dceb80d18be382646821b67e030fabdb4da089953370e6a13ba60f680eead0\": container with ID starting with 32dceb80d18be382646821b67e030fabdb4da089953370e6a13ba60f680eead0 not found: ID does not exist"
Sep 17 19:23:49.752234 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.752222    2565 scope.go:117] "RemoveContainer" containerID="678a1d27479a806101b0b1c7875bb6a6a2bc9c2203af2524d67c1f4deca2f25a"
Sep 17 19:23:49.752509 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:23:49.752492    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"678a1d27479a806101b0b1c7875bb6a6a2bc9c2203af2524d67c1f4deca2f25a\": container with ID starting with 678a1d27479a806101b0b1c7875bb6a6a2bc9c2203af2524d67c1f4deca2f25a not found: ID does not exist" containerID="678a1d27479a806101b0b1c7875bb6a6a2bc9c2203af2524d67c1f4deca2f25a"
Sep 17 19:23:49.752552 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.752517    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"678a1d27479a806101b0b1c7875bb6a6a2bc9c2203af2524d67c1f4deca2f25a"} err="failed to get container status \"678a1d27479a806101b0b1c7875bb6a6a2bc9c2203af2524d67c1f4deca2f25a\": rpc error: code = NotFound desc = could not find container \"678a1d27479a806101b0b1c7875bb6a6a2bc9c2203af2524d67c1f4deca2f25a\": container with ID starting with 678a1d27479a806101b0b1c7875bb6a6a2bc9c2203af2524d67c1f4deca2f25a not found: ID does not exist"
Sep 17 19:23:49.755938 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.755915    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"]
Sep 17 19:23:49.762670 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:49.762644    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-66459fff44-h7zf5"]
Sep 17 19:23:50.612244 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:50.612173    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="895fcd89-1728-4eaa-9977-584c903239f1" path="/var/lib/kubelet/pods/895fcd89-1728-4eaa-9977-584c903239f1/volumes"
Sep 17 19:23:56.883484 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:56.883445    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-7wb5f"]
Sep 17 19:23:56.883888 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:56.883776    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="895fcd89-1728-4eaa-9977-584c903239f1" containerName="main"
Sep 17 19:23:56.883888 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:56.883787    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="895fcd89-1728-4eaa-9977-584c903239f1" containerName="main"
Sep 17 19:23:56.883888 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:56.883798    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d47ba3fa-9af3-4ba4-9155-abc592d68000" containerName="main"
Sep 17 19:23:56.883888 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:56.883804    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d47ba3fa-9af3-4ba4-9155-abc592d68000" containerName="main"
Sep 17 19:23:56.883888 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:56.883822    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d47ba3fa-9af3-4ba4-9155-abc592d68000" containerName="storage-initializer"
Sep 17 19:23:56.883888 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:56.883828    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d47ba3fa-9af3-4ba4-9155-abc592d68000" containerName="storage-initializer"
Sep 17 19:23:56.883888 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:56.883838    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="895fcd89-1728-4eaa-9977-584c903239f1" containerName="storage-initializer"
Sep 17 19:23:56.883888 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:56.883844    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="895fcd89-1728-4eaa-9977-584c903239f1" containerName="storage-initializer"
Sep 17 19:23:56.884218 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:56.883894    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d47ba3fa-9af3-4ba4-9155-abc592d68000" containerName="main"
Sep 17 19:23:56.884218 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:56.883903    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="895fcd89-1728-4eaa-9977-584c903239f1" containerName="main"
Sep 17 19:23:56.888513 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:56.888495    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-7wb5f"
Sep 17 19:23:56.890911 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:56.890884    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"seaweedfs-s3-creds\""
Sep 17 19:23:56.891044 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:56.891002    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 17 19:23:56.891812 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:56.891788    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-ff6jg\""
Sep 17 19:23:56.892379 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:56.892360    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 17 19:23:56.894577 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:56.894551    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-7wb5f"]
Sep 17 19:23:56.937144 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:56.937113    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzp5h\" (UniqueName: \"kubernetes.io/projected/4b155042-f6c7-4cca-8e2a-a57e8c0093e4-kube-api-access-rzp5h\") pod \"e2e-pvc-lora-download-7wb5f\" (UID: \"4b155042-f6c7-4cca-8e2a-a57e8c0093e4\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-7wb5f"
Sep 17 19:23:57.038340 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:57.038314    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzp5h\" (UniqueName: \"kubernetes.io/projected/4b155042-f6c7-4cca-8e2a-a57e8c0093e4-kube-api-access-rzp5h\") pod \"e2e-pvc-lora-download-7wb5f\" (UID: \"4b155042-f6c7-4cca-8e2a-a57e8c0093e4\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-7wb5f"
Sep 17 19:23:57.046830 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:57.046804    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzp5h\" (UniqueName: \"kubernetes.io/projected/4b155042-f6c7-4cca-8e2a-a57e8c0093e4-kube-api-access-rzp5h\") pod \"e2e-pvc-lora-download-7wb5f\" (UID: \"4b155042-f6c7-4cca-8e2a-a57e8c0093e4\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-7wb5f"
Sep 17 19:23:58.512659 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:58.512627    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-gbj6g"]
Sep 17 19:23:58.517139 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:58.517114    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-gbj6g"]
Sep 17 19:23:58.607041 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:58.607010    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7adf157-103f-4bc6-bab8-e034195ad99c" path="/var/lib/kubelet/pods/f7adf157-103f-4bc6-bab8-e034195ad99c/volumes"
Sep 17 19:23:59.355796 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:59.355763    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-0edb74e0-7c56-4c0c-b632-efea22e2e212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fd777616949cb757\") pod \"e2e-pvc-lora-download-7wb5f\" (UID: \"4b155042-f6c7-4cca-8e2a-a57e8c0093e4\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-7wb5f"
Sep 17 19:23:59.361177 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:59.361154    2565 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-0edb74e0-7c56-4c0c-b632-efea22e2e212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fd777616949cb757\") pod \"e2e-pvc-lora-download-7wb5f\" (UID: \"4b155042-f6c7-4cca-8e2a-a57e8c0093e4\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-7wb5f"
Sep 17 19:23:59.456033 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:59.456003    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-0edb74e0-7c56-4c0c-b632-efea22e2e212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fd777616949cb757\") pod \"e2e-pvc-lora-download-7wb5f\" (UID: \"4b155042-f6c7-4cca-8e2a-a57e8c0093e4\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-7wb5f"
Sep 17 19:23:59.456159 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:59.456137    2565 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-0edb74e0-7c56-4c0c-b632-efea22e2e212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fd777616949cb757\") pod \"e2e-pvc-lora-download-7wb5f\" (UID: \"4b155042-f6c7-4cca-8e2a-a57e8c0093e4\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-7wb5f"
Sep 17 19:23:59.461107 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:59.461088    2565 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-0edb74e0-7c56-4c0c-b632-efea22e2e212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fd777616949cb757\") pod \"e2e-pvc-lora-download-7wb5f\" (UID: \"4b155042-f6c7-4cca-8e2a-a57e8c0093e4\") DevicePath \"csi-e2802f9e4170644ff76a1e3b9f990b020b7e7eef5e7cda62365fd963dcd8760d\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-7wb5f"
Sep 17 19:23:59.675606 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:59.675579    2565 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-0edb74e0-7c56-4c0c-b632-efea22e2e212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fd777616949cb757\") pod \"e2e-pvc-lora-download-7wb5f\" (UID: \"4b155042-f6c7-4cca-8e2a-a57e8c0093e4\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/fe733076528e64892e0c689cc5f1b549fd9f0b7606d2edb04c8c86d4f995baf9/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-7wb5f"
Sep 17 19:23:59.692794 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:59.692768    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-0edb74e0-7c56-4c0c-b632-efea22e2e212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fd777616949cb757\") pod \"e2e-pvc-lora-download-7wb5f\" (UID: \"4b155042-f6c7-4cca-8e2a-a57e8c0093e4\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-7wb5f"
Sep 17 19:23:59.898122 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:23:59.898097    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-7wb5f"
Sep 17 19:24:00.175642 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:00.175609    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-7wb5f"]
Sep 17 19:24:00.176178 ip-10-0-139-75 kubenswrapper[2565]: W0917 19:24:00.176155    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b155042_f6c7_4cca_8e2a_a57e8c0093e4.slice/crio-bc7ffb7b7f0eb49c070c57382e075ad0fa8fb861f595ae1eaa3b2c4adf0a6691 WatchSource:0}: Error finding container bc7ffb7b7f0eb49c070c57382e075ad0fa8fb861f595ae1eaa3b2c4adf0a6691: Status 404 returned error can't find the container with id bc7ffb7b7f0eb49c070c57382e075ad0fa8fb861f595ae1eaa3b2c4adf0a6691
Sep 17 19:24:00.791581 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:00.791539    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-7wb5f" event={"ID":"4b155042-f6c7-4cca-8e2a-a57e8c0093e4","Type":"ContainerStarted","Data":"78e64b5f7ec15ac947e7397f4ea5338af484565b71567d8a98f637dd2ee5cf9c"}
Sep 17 19:24:00.791581 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:00.791594    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-7wb5f" event={"ID":"4b155042-f6c7-4cca-8e2a-a57e8c0093e4","Type":"ContainerStarted","Data":"bc7ffb7b7f0eb49c070c57382e075ad0fa8fb861f595ae1eaa3b2c4adf0a6691"}
Sep 17 19:24:00.808293 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:00.808240    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-7wb5f" podStartSLOduration=7.808224757 podStartE2EDuration="7.808224757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:24:00.806220411 +0000 UTC m=+3298.738421931" watchObservedRunningTime="2026-09-17 19:24:00.808224757 +0000 UTC m=+3298.740426270"
Sep 17 19:24:02.804510 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:02.804477    2565 generic.go:356] "Generic (PLEG): container finished" podID="4b155042-f6c7-4cca-8e2a-a57e8c0093e4" containerID="78e64b5f7ec15ac947e7397f4ea5338af484565b71567d8a98f637dd2ee5cf9c" exitCode=0
Sep 17 19:24:02.804848 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:02.804545    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-7wb5f" event={"ID":"4b155042-f6c7-4cca-8e2a-a57e8c0093e4","Type":"ContainerDied","Data":"78e64b5f7ec15ac947e7397f4ea5338af484565b71567d8a98f637dd2ee5cf9c"}
Sep 17 19:24:03.944450 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:03.944426    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-7wb5f"
Sep 17 19:24:03.994957 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:03.994931    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzp5h\" (UniqueName: \"kubernetes.io/projected/4b155042-f6c7-4cca-8e2a-a57e8c0093e4-kube-api-access-rzp5h\") pod \"4b155042-f6c7-4cca-8e2a-a57e8c0093e4\" (UID: \"4b155042-f6c7-4cca-8e2a-a57e8c0093e4\") "
Sep 17 19:24:03.995066 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:03.995028    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fd777616949cb757\") pod \"4b155042-f6c7-4cca-8e2a-a57e8c0093e4\" (UID: \"4b155042-f6c7-4cca-8e2a-a57e8c0093e4\") "
Sep 17 19:24:03.997185 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:03.997155    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b155042-f6c7-4cca-8e2a-a57e8c0093e4-kube-api-access-rzp5h" (OuterVolumeSpecName: "kube-api-access-rzp5h") pod "4b155042-f6c7-4cca-8e2a-a57e8c0093e4" (UID: "4b155042-f6c7-4cca-8e2a-a57e8c0093e4"). InnerVolumeSpecName "kube-api-access-rzp5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:24:03.997577 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:03.997558    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0fd777616949cb757" (OuterVolumeSpecName: "model-storage") pod "4b155042-f6c7-4cca-8e2a-a57e8c0093e4" (UID: "4b155042-f6c7-4cca-8e2a-a57e8c0093e4"). InnerVolumeSpecName "pvc-0edb74e0-7c56-4c0c-b632-efea22e2e212". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 19:24:04.096035 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:04.095969    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzp5h\" (UniqueName: \"kubernetes.io/projected/4b155042-f6c7-4cca-8e2a-a57e8c0093e4-kube-api-access-rzp5h\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:24:04.096035 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:04.096017    2565 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-0edb74e0-7c56-4c0c-b632-efea22e2e212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fd777616949cb757\") on node \"ip-10-0-139-75.ec2.internal\" "
Sep 17 19:24:04.815837 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:04.815809    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-7wb5f"
Sep 17 19:24:04.815979 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:04.815807    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-7wb5f" event={"ID":"4b155042-f6c7-4cca-8e2a-a57e8c0093e4","Type":"ContainerDied","Data":"bc7ffb7b7f0eb49c070c57382e075ad0fa8fb861f595ae1eaa3b2c4adf0a6691"}
Sep 17 19:24:04.815979 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:04.815924    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc7ffb7b7f0eb49c070c57382e075ad0fa8fb861f595ae1eaa3b2c4adf0a6691"
Sep 17 19:24:05.067612 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:05.067591    2565 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-0edb74e0-7c56-4c0c-b632-efea22e2e212" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0fd777616949cb757") on node "ip-10-0-139-75.ec2.internal"
Sep 17 19:24:05.103276 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:05.103250    2565 reconciler_common.go:299] "Volume detached for volume \"pvc-0edb74e0-7c56-4c0c-b632-efea22e2e212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fd777616949cb757\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"csi-e2802f9e4170644ff76a1e3b9f990b020b7e7eef5e7cda62365fd963dcd8760d\""
Sep 17 19:24:27.319002 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:27.318903    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"]
Sep 17 19:24:27.319594 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:27.319396    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b155042-f6c7-4cca-8e2a-a57e8c0093e4" containerName="storage-initializer"
Sep 17 19:24:27.319594 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:27.319412    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b155042-f6c7-4cca-8e2a-a57e8c0093e4" containerName="storage-initializer"
Sep 17 19:24:27.319594 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:27.319478    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b155042-f6c7-4cca-8e2a-a57e8c0093e4" containerName="storage-initializer"
Sep 17 19:24:27.322825 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:27.322803    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:24:27.325459 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:27.325434    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 17 19:24:27.325606 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:27.325504    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-ff6jg\""
Sep 17 19:24:27.325606 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:27.325585    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"lora-pvc-adapters-test-kserve-self-signed-certs\""
Sep 17 19:24:27.325742 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:27.325440    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 17 19:24:27.333240 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:27.333175    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"]
Sep 17 19:24:27.373742 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:27.373715    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c77a87e-1833-455b-8a28-984040d5c5bf-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-s47vz\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:24:27.373870 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:27.373744    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v8kd\" (UniqueName: \"kubernetes.io/projected/4c77a87e-1833-455b-8a28-984040d5c5bf-kube-api-access-5v8kd\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-s47vz\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:24:27.373870 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:27.373763    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c77a87e-1833-455b-8a28-984040d5c5bf-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-s47vz\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:24:27.373979 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:27.373878    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c77a87e-1833-455b-8a28-984040d5c5bf-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-s47vz\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:24:27.373979 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:27.373962    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c77a87e-1833-455b-8a28-984040d5c5bf-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-s47vz\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:24:27.374062 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:27.373983    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c77a87e-1833-455b-8a28-984040d5c5bf-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-s47vz\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:24:27.474318 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:27.474289    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c77a87e-1833-455b-8a28-984040d5c5bf-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-s47vz\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:24:27.474443 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:27.474325    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c77a87e-1833-455b-8a28-984040d5c5bf-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-s47vz\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:24:27.474443 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:27.474342    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c77a87e-1833-455b-8a28-984040d5c5bf-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-s47vz\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:24:27.474443 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:27.474358    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v8kd\" (UniqueName: \"kubernetes.io/projected/4c77a87e-1833-455b-8a28-984040d5c5bf-kube-api-access-5v8kd\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-s47vz\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:24:27.474443 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:27.474375    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c77a87e-1833-455b-8a28-984040d5c5bf-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-s47vz\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:24:27.474443 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:27.474419    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c77a87e-1833-455b-8a28-984040d5c5bf-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-s47vz\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:24:27.474823 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:27.474800    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c77a87e-1833-455b-8a28-984040d5c5bf-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-s47vz\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:24:27.474889 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:27.474832    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c77a87e-1833-455b-8a28-984040d5c5bf-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-s47vz\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:24:27.474889 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:27.474858    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c77a87e-1833-455b-8a28-984040d5c5bf-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-s47vz\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:24:27.476525 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:27.476504    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c77a87e-1833-455b-8a28-984040d5c5bf-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-s47vz\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:24:27.476804 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:27.476786    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c77a87e-1833-455b-8a28-984040d5c5bf-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-s47vz\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:24:27.482839 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:27.482819    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v8kd\" (UniqueName: \"kubernetes.io/projected/4c77a87e-1833-455b-8a28-984040d5c5bf-kube-api-access-5v8kd\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-s47vz\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:24:29.793476 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:29.793439    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-0edb74e0-7c56-4c0c-b632-efea22e2e212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fd777616949cb757\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-s47vz\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:24:29.798654 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:29.798624    2565 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-0edb74e0-7c56-4c0c-b632-efea22e2e212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fd777616949cb757\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-s47vz\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:24:29.894437 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:29.894408    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-0edb74e0-7c56-4c0c-b632-efea22e2e212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fd777616949cb757\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-s47vz\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:24:29.894585 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:29.894542    2565 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-0edb74e0-7c56-4c0c-b632-efea22e2e212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fd777616949cb757\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-s47vz\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:24:29.901544 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:29.901517    2565 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-0edb74e0-7c56-4c0c-b632-efea22e2e212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fd777616949cb757\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-s47vz\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") DevicePath \"csi-e2802f9e4170644ff76a1e3b9f990b020b7e7eef5e7cda62365fd963dcd8760d\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:24:29.988310 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:29.988277    2565 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-0edb74e0-7c56-4c0c-b632-efea22e2e212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fd777616949cb757\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-s47vz\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/fe733076528e64892e0c689cc5f1b549fd9f0b7606d2edb04c8c86d4f995baf9/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:24:30.006465 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:30.006437    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-0edb74e0-7c56-4c0c-b632-efea22e2e212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fd777616949cb757\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-s47vz\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:24:30.033964 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:30.033937    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:24:30.320376 ip-10-0-139-75 kubenswrapper[2565]: W0917 19:24:30.320344    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c77a87e_1833_455b_8a28_984040d5c5bf.slice/crio-cdf9594e6be8ebb7006d0ea8b2b3cc86ebceddb87127001d17f5d4891ba46b58 WatchSource:0}: Error finding container cdf9594e6be8ebb7006d0ea8b2b3cc86ebceddb87127001d17f5d4891ba46b58: Status 404 returned error can't find the container with id cdf9594e6be8ebb7006d0ea8b2b3cc86ebceddb87127001d17f5d4891ba46b58
Sep 17 19:24:30.320504 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:30.320483    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"]
Sep 17 19:24:30.959624 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:30.959578    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz" event={"ID":"4c77a87e-1833-455b-8a28-984040d5c5bf","Type":"ContainerStarted","Data":"3f067ffde9c2e9fb75a43ee4eff79a41e605802134313bbc4597528ceaeae52e"}
Sep 17 19:24:30.960126 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:30.959649    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz" event={"ID":"4c77a87e-1833-455b-8a28-984040d5c5bf","Type":"ContainerStarted","Data":"cdf9594e6be8ebb7006d0ea8b2b3cc86ebceddb87127001d17f5d4891ba46b58"}
Sep 17 19:24:30.984392 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:30.984339    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz" podStartSLOduration=3.984319501 podStartE2EDuration="3.984319501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:24:30.982491557 +0000 UTC m=+3328.914693077" watchObservedRunningTime="2026-09-17 19:24:30.984319501 +0000 UTC m=+3328.916521021"
Sep 17 19:24:31.034893 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:31.034864    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:24:40.034389 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:40.034354    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:24:40.035842 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:40.035816    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz" podUID="4c77a87e-1833-455b-8a28-984040d5c5bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 17 19:24:50.035005 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:24:50.034956    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz" podUID="4c77a87e-1833-455b-8a28-984040d5c5bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 17 19:25:00.034998 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:00.034942    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz" podUID="4c77a87e-1833-455b-8a28-984040d5c5bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 17 19:25:10.034514 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:10.034463    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz" podUID="4c77a87e-1833-455b-8a28-984040d5c5bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 17 19:25:20.034370 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:20.034326    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz" podUID="4c77a87e-1833-455b-8a28-984040d5c5bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 17 19:25:30.034378 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:30.034329    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz" podUID="4c77a87e-1833-455b-8a28-984040d5c5bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 17 19:25:40.034509 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:40.034465    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz" podUID="4c77a87e-1833-455b-8a28-984040d5c5bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 17 19:25:50.045040 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:50.044999    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:25:50.053473 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:50.053445    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:25:56.789995 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.789918    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 19:25:56.793461 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.793441    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:25:56.797311 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.797290    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 17 19:25:56.797450 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.797373    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 17 19:25:56.797770 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.797751    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-kr2sj\""
Sep 17 19:25:56.798113 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.798094    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:25:56.798219 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.798117    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 17 19:25:56.798219 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.798094    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-vf24g\""
Sep 17 19:25:56.798385 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.798364    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 17 19:25:56.806087 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.806062    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 19:25:56.892035 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.892005    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe8d5419-ce53-4118-bf3c-0e3417f60327-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:25:56.892174 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.892042    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe8d5419-ce53-4118-bf3c-0e3417f60327-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:25:56.892174 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.892059    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe8d5419-ce53-4118-bf3c-0e3417f60327-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:25:56.892174 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.892156    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe8d5419-ce53-4118-bf3c-0e3417f60327-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:25:56.892392 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.892224    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8d5419-ce53-4118-bf3c-0e3417f60327-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:25:56.892392 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.892260    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe8d5419-ce53-4118-bf3c-0e3417f60327-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:25:56.892392 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.892340    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe8d5419-ce53-4118-bf3c-0e3417f60327-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:25:56.892519 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.892387    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prdqq\" (UniqueName: \"kubernetes.io/projected/fe8d5419-ce53-4118-bf3c-0e3417f60327-kube-api-access-prdqq\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:25:56.993443 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.993414    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe8d5419-ce53-4118-bf3c-0e3417f60327-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:25:56.993443 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.993442    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8d5419-ce53-4118-bf3c-0e3417f60327-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:25:56.993589 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.993464    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe8d5419-ce53-4118-bf3c-0e3417f60327-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:25:56.993589 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.993492    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe8d5419-ce53-4118-bf3c-0e3417f60327-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:25:56.993589 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.993524    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prdqq\" (UniqueName: \"kubernetes.io/projected/fe8d5419-ce53-4118-bf3c-0e3417f60327-kube-api-access-prdqq\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:25:56.993589 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.993551    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe8d5419-ce53-4118-bf3c-0e3417f60327-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:25:56.993589 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.993575    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe8d5419-ce53-4118-bf3c-0e3417f60327-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:25:56.993848 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.993653    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe8d5419-ce53-4118-bf3c-0e3417f60327-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:25:56.993848 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.993792    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe8d5419-ce53-4118-bf3c-0e3417f60327-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:25:56.994000 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.993977    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe8d5419-ce53-4118-bf3c-0e3417f60327-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:25:56.994059 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.994003    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe8d5419-ce53-4118-bf3c-0e3417f60327-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:25:56.994059 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.994031    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe8d5419-ce53-4118-bf3c-0e3417f60327-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:25:56.994246 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.994228    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe8d5419-ce53-4118-bf3c-0e3417f60327-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:25:56.996220 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.996182    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe8d5419-ce53-4118-bf3c-0e3417f60327-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:25:56.996439 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:56.996421    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8d5419-ce53-4118-bf3c-0e3417f60327-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:25:57.001300 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:57.001278    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prdqq\" (UniqueName: \"kubernetes.io/projected/fe8d5419-ce53-4118-bf3c-0e3417f60327-kube-api-access-prdqq\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:25:57.104629 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:57.104571    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:25:57.419358 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:57.419328    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 19:25:57.421652 ip-10-0-139-75 kubenswrapper[2565]: W0917 19:25:57.421628    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe8d5419_ce53_4118_bf3c_0e3417f60327.slice/crio-843cb36d0767509a435abc567689eb24a6ec1fed9c1f1e1062f7cd93abae719d WatchSource:0}: Error finding container 843cb36d0767509a435abc567689eb24a6ec1fed9c1f1e1062f7cd93abae719d: Status 404 returned error can't find the container with id 843cb36d0767509a435abc567689eb24a6ec1fed9c1f1e1062f7cd93abae719d
Sep 17 19:25:57.448155 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:57.448124    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"fe8d5419-ce53-4118-bf3c-0e3417f60327","Type":"ContainerStarted","Data":"843cb36d0767509a435abc567689eb24a6ec1fed9c1f1e1062f7cd93abae719d"}
Sep 17 19:25:58.459437 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:25:58.459383    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"fe8d5419-ce53-4118-bf3c-0e3417f60327","Type":"ContainerStarted","Data":"961229a9dfb14dbeeaa2472f8a6b829a56f1e5a27af7d271f3d69afcdc5a4d4c"}
Sep 17 19:26:00.473631 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:00.473598    2565 generic.go:356] "Generic (PLEG): container finished" podID="fe8d5419-ce53-4118-bf3c-0e3417f60327" containerID="961229a9dfb14dbeeaa2472f8a6b829a56f1e5a27af7d271f3d69afcdc5a4d4c" exitCode=0
Sep 17 19:26:00.474143 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:00.473673    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"fe8d5419-ce53-4118-bf3c-0e3417f60327","Type":"ContainerDied","Data":"961229a9dfb14dbeeaa2472f8a6b829a56f1e5a27af7d271f3d69afcdc5a4d4c"}
Sep 17 19:26:01.481128 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:01.481092    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"fe8d5419-ce53-4118-bf3c-0e3417f60327","Type":"ContainerStarted","Data":"1e8a9205719507a2bb94c5847d6b10d9c10a88f8606cd7eee9823fcbe6f86773"}
Sep 17 19:26:01.504964 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:01.504916    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=3.321997983 podStartE2EDuration="5.504900473s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.18290249s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:25:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:26:01.502000943 +0000 UTC m=+3419.434202462" watchObservedRunningTime="2026-09-17 19:26:01.504900473 +0000 UTC m=+3419.437101991"
Sep 17 19:26:34.725739 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:34.725707    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-a290d4b0_lora-pvc-adapters-test-kserve-8cfd5947f-s47vz_4c77a87e-1833-455b-8a28-984040d5c5bf/main/0.log"
Sep 17 19:26:35.737159 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:35.737122    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"]
Sep 17 19:26:50.760258 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:50.760180    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz" podUID="4c77a87e-1833-455b-8a28-984040d5c5bf" containerName="main" containerID="cri-o://3f067ffde9c2e9fb75a43ee4eff79a41e605802134313bbc4597528ceaeae52e" gracePeriod=45
Sep 17 19:26:51.045721 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.045617    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz" podUID="4c77a87e-1833-455b-8a28-984040d5c5bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Sep 17 19:26:51.415582 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.415558    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:26:51.472975 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.472942    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c77a87e-1833-455b-8a28-984040d5c5bf-model-cache\") pod \"4c77a87e-1833-455b-8a28-984040d5c5bf\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") "
Sep 17 19:26:51.473130 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.472987    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c77a87e-1833-455b-8a28-984040d5c5bf-tmp-dir\") pod \"4c77a87e-1833-455b-8a28-984040d5c5bf\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") "
Sep 17 19:26:51.473215 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.473186    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c77a87e-1833-455b-8a28-984040d5c5bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "4c77a87e-1833-455b-8a28-984040d5c5bf" (UID: "4c77a87e-1833-455b-8a28-984040d5c5bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:26:51.483538 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.483507    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c77a87e-1833-455b-8a28-984040d5c5bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4c77a87e-1833-455b-8a28-984040d5c5bf" (UID: "4c77a87e-1833-455b-8a28-984040d5c5bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:26:51.574002 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.573928    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c77a87e-1833-455b-8a28-984040d5c5bf-home\") pod \"4c77a87e-1833-455b-8a28-984040d5c5bf\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") "
Sep 17 19:26:51.574147 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.574037    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fd777616949cb757\") pod \"4c77a87e-1833-455b-8a28-984040d5c5bf\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") "
Sep 17 19:26:51.574147 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.574059    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c77a87e-1833-455b-8a28-984040d5c5bf-tls-certs\") pod \"4c77a87e-1833-455b-8a28-984040d5c5bf\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") "
Sep 17 19:26:51.574147 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.574077    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5v8kd\" (UniqueName: \"kubernetes.io/projected/4c77a87e-1833-455b-8a28-984040d5c5bf-kube-api-access-5v8kd\") pod \"4c77a87e-1833-455b-8a28-984040d5c5bf\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") "
Sep 17 19:26:51.574147 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.574093    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c77a87e-1833-455b-8a28-984040d5c5bf-dshm\") pod \"4c77a87e-1833-455b-8a28-984040d5c5bf\" (UID: \"4c77a87e-1833-455b-8a28-984040d5c5bf\") "
Sep 17 19:26:51.574421 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.574325    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c77a87e-1833-455b-8a28-984040d5c5bf-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:26:51.574421 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.574343    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c77a87e-1833-455b-8a28-984040d5c5bf-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:26:51.574421 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.574335    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c77a87e-1833-455b-8a28-984040d5c5bf-home" (OuterVolumeSpecName: "home") pod "4c77a87e-1833-455b-8a28-984040d5c5bf" (UID: "4c77a87e-1833-455b-8a28-984040d5c5bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:26:51.576318 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.576271    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c77a87e-1833-455b-8a28-984040d5c5bf-dshm" (OuterVolumeSpecName: "dshm") pod "4c77a87e-1833-455b-8a28-984040d5c5bf" (UID: "4c77a87e-1833-455b-8a28-984040d5c5bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:26:51.576449 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.576352    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c77a87e-1833-455b-8a28-984040d5c5bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4c77a87e-1833-455b-8a28-984040d5c5bf" (UID: "4c77a87e-1833-455b-8a28-984040d5c5bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:26:51.576706 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.576674    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c77a87e-1833-455b-8a28-984040d5c5bf-kube-api-access-5v8kd" (OuterVolumeSpecName: "kube-api-access-5v8kd") pod "4c77a87e-1833-455b-8a28-984040d5c5bf" (UID: "4c77a87e-1833-455b-8a28-984040d5c5bf"). InnerVolumeSpecName "kube-api-access-5v8kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:26:51.581876 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.581856    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0fd777616949cb757" (OuterVolumeSpecName: "kserve-pvc-source") pod "4c77a87e-1833-455b-8a28-984040d5c5bf" (UID: "4c77a87e-1833-455b-8a28-984040d5c5bf"). InnerVolumeSpecName "pvc-0edb74e0-7c56-4c0c-b632-efea22e2e212". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 19:26:51.674924 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.674893    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c77a87e-1833-455b-8a28-984040d5c5bf-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:26:51.675037 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.674946    2565 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-0edb74e0-7c56-4c0c-b632-efea22e2e212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fd777616949cb757\") on node \"ip-10-0-139-75.ec2.internal\" "
Sep 17 19:26:51.675037 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.674963    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c77a87e-1833-455b-8a28-984040d5c5bf-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:26:51.675037 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.674978    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v8kd\" (UniqueName: \"kubernetes.io/projected/4c77a87e-1833-455b-8a28-984040d5c5bf-kube-api-access-5v8kd\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:26:51.675037 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.674992    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c77a87e-1833-455b-8a28-984040d5c5bf-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:26:51.706621 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.706593    2565 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-0edb74e0-7c56-4c0c-b632-efea22e2e212" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0fd777616949cb757") on node "ip-10-0-139-75.ec2.internal"
Sep 17 19:26:51.776255 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.776176    2565 reconciler_common.go:299] "Volume detached for volume \"pvc-0edb74e0-7c56-4c0c-b632-efea22e2e212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0fd777616949cb757\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"csi-e2802f9e4170644ff76a1e3b9f990b020b7e7eef5e7cda62365fd963dcd8760d\""
Sep 17 19:26:51.799437 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.799410    2565 generic.go:356] "Generic (PLEG): container finished" podID="4c77a87e-1833-455b-8a28-984040d5c5bf" containerID="3f067ffde9c2e9fb75a43ee4eff79a41e605802134313bbc4597528ceaeae52e" exitCode=0
Sep 17 19:26:51.799570 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.799446    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz" event={"ID":"4c77a87e-1833-455b-8a28-984040d5c5bf","Type":"ContainerDied","Data":"3f067ffde9c2e9fb75a43ee4eff79a41e605802134313bbc4597528ceaeae52e"}
Sep 17 19:26:51.799570 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.799518    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz" event={"ID":"4c77a87e-1833-455b-8a28-984040d5c5bf","Type":"ContainerDied","Data":"cdf9594e6be8ebb7006d0ea8b2b3cc86ebceddb87127001d17f5d4891ba46b58"}
Sep 17 19:26:51.799570 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.799455    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"
Sep 17 19:26:51.799570 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.799552    2565 scope.go:117] "RemoveContainer" containerID="3f067ffde9c2e9fb75a43ee4eff79a41e605802134313bbc4597528ceaeae52e"
Sep 17 19:26:51.813667 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.813627    2565 scope.go:117] "RemoveContainer" containerID="3f067ffde9c2e9fb75a43ee4eff79a41e605802134313bbc4597528ceaeae52e"
Sep 17 19:26:51.814049 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:26:51.814015    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f067ffde9c2e9fb75a43ee4eff79a41e605802134313bbc4597528ceaeae52e\": container with ID starting with 3f067ffde9c2e9fb75a43ee4eff79a41e605802134313bbc4597528ceaeae52e not found: ID does not exist" containerID="3f067ffde9c2e9fb75a43ee4eff79a41e605802134313bbc4597528ceaeae52e"
Sep 17 19:26:51.814149 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.814060    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f067ffde9c2e9fb75a43ee4eff79a41e605802134313bbc4597528ceaeae52e"} err="failed to get container status \"3f067ffde9c2e9fb75a43ee4eff79a41e605802134313bbc4597528ceaeae52e\": rpc error: code = NotFound desc = could not find container \"3f067ffde9c2e9fb75a43ee4eff79a41e605802134313bbc4597528ceaeae52e\": container with ID starting with 3f067ffde9c2e9fb75a43ee4eff79a41e605802134313bbc4597528ceaeae52e not found: ID does not exist"
Sep 17 19:26:51.830729 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.830657    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"]
Sep 17 19:26:51.835648 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:51.835625    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-s47vz"]
Sep 17 19:26:52.608099 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:26:52.608018    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c77a87e-1833-455b-8a28-984040d5c5bf" path="/var/lib/kubelet/pods/4c77a87e-1833-455b-8a28-984040d5c5bf/volumes"
Sep 17 19:27:00.804666 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:00.804630    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-7wb5f"]
Sep 17 19:27:00.810116 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:00.810087    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-7wb5f"]
Sep 17 19:27:02.607346 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:02.607303    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b155042-f6c7-4cca-8e2a-a57e8c0093e4" path="/var/lib/kubelet/pods/4b155042-f6c7-4cca-8e2a-a57e8c0093e4/volumes"
Sep 17 19:27:04.209104 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.209065    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"]
Sep 17 19:27:04.209580 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.209564    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c77a87e-1833-455b-8a28-984040d5c5bf" containerName="main"
Sep 17 19:27:04.209630 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.209583    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c77a87e-1833-455b-8a28-984040d5c5bf" containerName="main"
Sep 17 19:27:04.209666 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.209661    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c77a87e-1833-455b-8a28-984040d5c5bf" containerName="main"
Sep 17 19:27:04.213906 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.213888    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:27:04.216368 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.216341    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Sep 17 19:27:04.216501 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.216340    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 17 19:27:04.216586 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.216340    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:27:04.217302 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.217272    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 17 19:27:04.217406 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.217271    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-dxtdh\""
Sep 17 19:27:04.217406 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.217272    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 17 19:27:04.226323 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.226299    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"]
Sep 17 19:27:04.284416 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.284388    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/285762e6-8c23-4c94-8b32-928a3d5edee4-cabundle-cert\") pod \"tls-verification-test-kserve-6dd587f89b-wst77\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:27:04.284557 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.284439    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/285762e6-8c23-4c94-8b32-928a3d5edee4-tmp-dir\") pod \"tls-verification-test-kserve-6dd587f89b-wst77\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:27:04.284607 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.284556    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/285762e6-8c23-4c94-8b32-928a3d5edee4-home\") pod \"tls-verification-test-kserve-6dd587f89b-wst77\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:27:04.284645 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.284604    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/285762e6-8c23-4c94-8b32-928a3d5edee4-model-cache\") pod \"tls-verification-test-kserve-6dd587f89b-wst77\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:27:04.284683 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.284644    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/285762e6-8c23-4c94-8b32-928a3d5edee4-kserve-provision-location\") pod \"tls-verification-test-kserve-6dd587f89b-wst77\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:27:04.284718 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.284692    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm8jc\" (UniqueName: \"kubernetes.io/projected/285762e6-8c23-4c94-8b32-928a3d5edee4-kube-api-access-jm8jc\") pod \"tls-verification-test-kserve-6dd587f89b-wst77\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:27:04.284767 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.284729    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/285762e6-8c23-4c94-8b32-928a3d5edee4-tls-certs\") pod \"tls-verification-test-kserve-6dd587f89b-wst77\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:27:04.284807 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.284794    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/285762e6-8c23-4c94-8b32-928a3d5edee4-dshm\") pod \"tls-verification-test-kserve-6dd587f89b-wst77\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:27:04.385757 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.385730    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/285762e6-8c23-4c94-8b32-928a3d5edee4-tmp-dir\") pod \"tls-verification-test-kserve-6dd587f89b-wst77\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:27:04.385865 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.385775    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/285762e6-8c23-4c94-8b32-928a3d5edee4-home\") pod \"tls-verification-test-kserve-6dd587f89b-wst77\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:27:04.385865 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.385799    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/285762e6-8c23-4c94-8b32-928a3d5edee4-model-cache\") pod \"tls-verification-test-kserve-6dd587f89b-wst77\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:27:04.385865 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.385823    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/285762e6-8c23-4c94-8b32-928a3d5edee4-kserve-provision-location\") pod \"tls-verification-test-kserve-6dd587f89b-wst77\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:27:04.386015 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.385863    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm8jc\" (UniqueName: \"kubernetes.io/projected/285762e6-8c23-4c94-8b32-928a3d5edee4-kube-api-access-jm8jc\") pod \"tls-verification-test-kserve-6dd587f89b-wst77\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:27:04.386015 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.385892    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/285762e6-8c23-4c94-8b32-928a3d5edee4-tls-certs\") pod \"tls-verification-test-kserve-6dd587f89b-wst77\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:27:04.386015 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.385931    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/285762e6-8c23-4c94-8b32-928a3d5edee4-dshm\") pod \"tls-verification-test-kserve-6dd587f89b-wst77\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:27:04.386015 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.385961    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/285762e6-8c23-4c94-8b32-928a3d5edee4-cabundle-cert\") pod \"tls-verification-test-kserve-6dd587f89b-wst77\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:27:04.386262 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.386125    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/285762e6-8c23-4c94-8b32-928a3d5edee4-tmp-dir\") pod \"tls-verification-test-kserve-6dd587f89b-wst77\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:27:04.386262 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.386159    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/285762e6-8c23-4c94-8b32-928a3d5edee4-home\") pod \"tls-verification-test-kserve-6dd587f89b-wst77\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:27:04.386262 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.386244    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/285762e6-8c23-4c94-8b32-928a3d5edee4-model-cache\") pod \"tls-verification-test-kserve-6dd587f89b-wst77\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:27:04.386417 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.386297    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/285762e6-8c23-4c94-8b32-928a3d5edee4-kserve-provision-location\") pod \"tls-verification-test-kserve-6dd587f89b-wst77\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:27:04.386684 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.386661    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/285762e6-8c23-4c94-8b32-928a3d5edee4-cabundle-cert\") pod \"tls-verification-test-kserve-6dd587f89b-wst77\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:27:04.388436 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.388411    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/285762e6-8c23-4c94-8b32-928a3d5edee4-dshm\") pod \"tls-verification-test-kserve-6dd587f89b-wst77\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:27:04.388567 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.388547    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/285762e6-8c23-4c94-8b32-928a3d5edee4-tls-certs\") pod \"tls-verification-test-kserve-6dd587f89b-wst77\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:27:04.394668 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.394641    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm8jc\" (UniqueName: \"kubernetes.io/projected/285762e6-8c23-4c94-8b32-928a3d5edee4-kube-api-access-jm8jc\") pod \"tls-verification-test-kserve-6dd587f89b-wst77\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:27:04.525435 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.525349    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:27:04.809703 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.809668    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"]
Sep 17 19:27:04.810292 ip-10-0-139-75 kubenswrapper[2565]: W0917 19:27:04.810259    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod285762e6_8c23_4c94_8b32_928a3d5edee4.slice/crio-e65ec73b6ea21e6bd961a1ca0065bc1e7a39e59c7318b9a298ae102808b82a50 WatchSource:0}: Error finding container e65ec73b6ea21e6bd961a1ca0065bc1e7a39e59c7318b9a298ae102808b82a50: Status 404 returned error can't find the container with id e65ec73b6ea21e6bd961a1ca0065bc1e7a39e59c7318b9a298ae102808b82a50
Sep 17 19:27:04.882362 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.882328    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77" event={"ID":"285762e6-8c23-4c94-8b32-928a3d5edee4","Type":"ContainerStarted","Data":"559f1e0066dfdbaa26120857a80da88ceccfeb6e1969e96240d38e8063a07d56"}
Sep 17 19:27:04.882451 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:04.882388    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77" event={"ID":"285762e6-8c23-4c94-8b32-928a3d5edee4","Type":"ContainerStarted","Data":"e65ec73b6ea21e6bd961a1ca0065bc1e7a39e59c7318b9a298ae102808b82a50"}
Sep 17 19:27:07.900779 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:07.900747    2565 generic.go:356] "Generic (PLEG): container finished" podID="285762e6-8c23-4c94-8b32-928a3d5edee4" containerID="559f1e0066dfdbaa26120857a80da88ceccfeb6e1969e96240d38e8063a07d56" exitCode=0
Sep 17 19:27:07.901188 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:07.900839    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77" event={"ID":"285762e6-8c23-4c94-8b32-928a3d5edee4","Type":"ContainerDied","Data":"559f1e0066dfdbaa26120857a80da88ceccfeb6e1969e96240d38e8063a07d56"}
Sep 17 19:27:08.908225 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:08.908169    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77" event={"ID":"285762e6-8c23-4c94-8b32-928a3d5edee4","Type":"ContainerStarted","Data":"e0accedac635ba4bd4499be6677b28d932d398ffe3803b4eadafd9035d6429d9"}
Sep 17 19:27:08.946910 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:08.946839    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77" podStartSLOduration=2.698933778 podStartE2EDuration="4.946819921s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.247886143s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:27:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:27:08.944284417 +0000 UTC m=+3486.876485938" watchObservedRunningTime="2026-09-17 19:27:08.946819921 +0000 UTC m=+3486.879021446"
Sep 17 19:27:09.525835 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:09.525802    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:27:14.526236 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:14.526175    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:27:14.527579 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:14.527539    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77" podUID="285762e6-8c23-4c94-8b32-928a3d5edee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 17 19:27:24.526074 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:24.526015    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77" podUID="285762e6-8c23-4c94-8b32-928a3d5edee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 17 19:27:34.526652 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:34.526566    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77" podUID="285762e6-8c23-4c94-8b32-928a3d5edee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 17 19:27:44.526448 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:44.526388    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77" podUID="285762e6-8c23-4c94-8b32-928a3d5edee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 17 19:27:54.526508 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:54.526457    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77" podUID="285762e6-8c23-4c94-8b32-928a3d5edee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 17 19:27:59.733075 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:27:59.733034    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 19:28:04.354467 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:04.354430    2565 scope.go:117] "RemoveContainer" containerID="253aa2e15a953b76e1dbe5faf1df30deae72255ad7aac75795e5dd6b8829745c"
Sep 17 19:28:04.526433 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:04.526391    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77" podUID="285762e6-8c23-4c94-8b32-928a3d5edee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 17 19:28:14.526035 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:14.525982    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77" podUID="285762e6-8c23-4c94-8b32-928a3d5edee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 17 19:28:14.755303 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:14.755263    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="fe8d5419-ce53-4118-bf3c-0e3417f60327" containerName="main" containerID="cri-o://1e8a9205719507a2bb94c5847d6b10d9c10a88f8606cd7eee9823fcbe6f86773" gracePeriod=45
Sep 17 19:28:15.509823 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:15.509797    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:28:15.587732 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:15.587654    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe8d5419-ce53-4118-bf3c-0e3417f60327-cabundle-cert\") pod \"fe8d5419-ce53-4118-bf3c-0e3417f60327\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") "
Sep 17 19:28:15.587732 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:15.587718    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe8d5419-ce53-4118-bf3c-0e3417f60327-model-cache\") pod \"fe8d5419-ce53-4118-bf3c-0e3417f60327\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") "
Sep 17 19:28:15.588240 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:15.587785    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe8d5419-ce53-4118-bf3c-0e3417f60327-kserve-provision-location\") pod \"fe8d5419-ce53-4118-bf3c-0e3417f60327\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") "
Sep 17 19:28:15.588240 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:15.587855    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8d5419-ce53-4118-bf3c-0e3417f60327-tls-certs\") pod \"fe8d5419-ce53-4118-bf3c-0e3417f60327\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") "
Sep 17 19:28:15.588240 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:15.587887    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe8d5419-ce53-4118-bf3c-0e3417f60327-dshm\") pod \"fe8d5419-ce53-4118-bf3c-0e3417f60327\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") "
Sep 17 19:28:15.588240 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:15.587910    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe8d5419-ce53-4118-bf3c-0e3417f60327-home\") pod \"fe8d5419-ce53-4118-bf3c-0e3417f60327\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") "
Sep 17 19:28:15.588240 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:15.587937    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prdqq\" (UniqueName: \"kubernetes.io/projected/fe8d5419-ce53-4118-bf3c-0e3417f60327-kube-api-access-prdqq\") pod \"fe8d5419-ce53-4118-bf3c-0e3417f60327\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") "
Sep 17 19:28:15.588240 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:15.587964    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe8d5419-ce53-4118-bf3c-0e3417f60327-tmp-dir\") pod \"fe8d5419-ce53-4118-bf3c-0e3417f60327\" (UID: \"fe8d5419-ce53-4118-bf3c-0e3417f60327\") "
Sep 17 19:28:15.588240 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:15.587957    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8d5419-ce53-4118-bf3c-0e3417f60327-model-cache" (OuterVolumeSpecName: "model-cache") pod "fe8d5419-ce53-4118-bf3c-0e3417f60327" (UID: "fe8d5419-ce53-4118-bf3c-0e3417f60327"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:28:15.588240 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:15.588002    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe8d5419-ce53-4118-bf3c-0e3417f60327-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fe8d5419-ce53-4118-bf3c-0e3417f60327" (UID: "fe8d5419-ce53-4118-bf3c-0e3417f60327"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:28:15.588689 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:15.588305    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe8d5419-ce53-4118-bf3c-0e3417f60327-cabundle-cert\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:28:15.588689 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:15.588327    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe8d5419-ce53-4118-bf3c-0e3417f60327-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:28:15.588806 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:15.588772    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8d5419-ce53-4118-bf3c-0e3417f60327-home" (OuterVolumeSpecName: "home") pod "fe8d5419-ce53-4118-bf3c-0e3417f60327" (UID: "fe8d5419-ce53-4118-bf3c-0e3417f60327"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:28:15.590454 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:15.590416    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe8d5419-ce53-4118-bf3c-0e3417f60327-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe8d5419-ce53-4118-bf3c-0e3417f60327" (UID: "fe8d5419-ce53-4118-bf3c-0e3417f60327"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:28:15.590618 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:15.590591    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe8d5419-ce53-4118-bf3c-0e3417f60327-kube-api-access-prdqq" (OuterVolumeSpecName: "kube-api-access-prdqq") pod "fe8d5419-ce53-4118-bf3c-0e3417f60327" (UID: "fe8d5419-ce53-4118-bf3c-0e3417f60327"). InnerVolumeSpecName "kube-api-access-prdqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:28:15.590853 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:15.590835    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8d5419-ce53-4118-bf3c-0e3417f60327-dshm" (OuterVolumeSpecName: "dshm") pod "fe8d5419-ce53-4118-bf3c-0e3417f60327" (UID: "fe8d5419-ce53-4118-bf3c-0e3417f60327"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:28:15.607561 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:15.607528    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8d5419-ce53-4118-bf3c-0e3417f60327-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fe8d5419-ce53-4118-bf3c-0e3417f60327" (UID: "fe8d5419-ce53-4118-bf3c-0e3417f60327"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:28:15.610423 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:15.610402    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8d5419-ce53-4118-bf3c-0e3417f60327-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fe8d5419-ce53-4118-bf3c-0e3417f60327" (UID: "fe8d5419-ce53-4118-bf3c-0e3417f60327"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:28:15.689177 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:15.689154    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe8d5419-ce53-4118-bf3c-0e3417f60327-kserve-provision-location\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:28:15.689302 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:15.689177    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe8d5419-ce53-4118-bf3c-0e3417f60327-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:28:15.689302 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:15.689193    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe8d5419-ce53-4118-bf3c-0e3417f60327-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:28:15.689302 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:15.689218    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe8d5419-ce53-4118-bf3c-0e3417f60327-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:28:15.689302 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:15.689231    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prdqq\" (UniqueName: \"kubernetes.io/projected/fe8d5419-ce53-4118-bf3c-0e3417f60327-kube-api-access-prdqq\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:28:15.689302 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:15.689243    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe8d5419-ce53-4118-bf3c-0e3417f60327-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:28:16.309654 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:16.309615    2565 generic.go:356] "Generic (PLEG): container finished" podID="fe8d5419-ce53-4118-bf3c-0e3417f60327" containerID="1e8a9205719507a2bb94c5847d6b10d9c10a88f8606cd7eee9823fcbe6f86773" exitCode=0
Sep 17 19:28:16.309862 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:16.309677    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 19:28:16.309862 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:16.309697    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"fe8d5419-ce53-4118-bf3c-0e3417f60327","Type":"ContainerDied","Data":"1e8a9205719507a2bb94c5847d6b10d9c10a88f8606cd7eee9823fcbe6f86773"}
Sep 17 19:28:16.309862 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:16.309755    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"fe8d5419-ce53-4118-bf3c-0e3417f60327","Type":"ContainerDied","Data":"843cb36d0767509a435abc567689eb24a6ec1fed9c1f1e1062f7cd93abae719d"}
Sep 17 19:28:16.309862 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:16.309781    2565 scope.go:117] "RemoveContainer" containerID="1e8a9205719507a2bb94c5847d6b10d9c10a88f8606cd7eee9823fcbe6f86773"
Sep 17 19:28:16.320963 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:16.320939    2565 scope.go:117] "RemoveContainer" containerID="961229a9dfb14dbeeaa2472f8a6b829a56f1e5a27af7d271f3d69afcdc5a4d4c"
Sep 17 19:28:16.330749 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:16.330732    2565 scope.go:117] "RemoveContainer" containerID="1e8a9205719507a2bb94c5847d6b10d9c10a88f8606cd7eee9823fcbe6f86773"
Sep 17 19:28:16.331023 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:28:16.330999    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e8a9205719507a2bb94c5847d6b10d9c10a88f8606cd7eee9823fcbe6f86773\": container with ID starting with 1e8a9205719507a2bb94c5847d6b10d9c10a88f8606cd7eee9823fcbe6f86773 not found: ID does not exist" containerID="1e8a9205719507a2bb94c5847d6b10d9c10a88f8606cd7eee9823fcbe6f86773"
Sep 17 19:28:16.331102 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:16.331037    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e8a9205719507a2bb94c5847d6b10d9c10a88f8606cd7eee9823fcbe6f86773"} err="failed to get container status \"1e8a9205719507a2bb94c5847d6b10d9c10a88f8606cd7eee9823fcbe6f86773\": rpc error: code = NotFound desc = could not find container \"1e8a9205719507a2bb94c5847d6b10d9c10a88f8606cd7eee9823fcbe6f86773\": container with ID starting with 1e8a9205719507a2bb94c5847d6b10d9c10a88f8606cd7eee9823fcbe6f86773 not found: ID does not exist"
Sep 17 19:28:16.331102 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:16.331062    2565 scope.go:117] "RemoveContainer" containerID="961229a9dfb14dbeeaa2472f8a6b829a56f1e5a27af7d271f3d69afcdc5a4d4c"
Sep 17 19:28:16.331334 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:28:16.331316    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"961229a9dfb14dbeeaa2472f8a6b829a56f1e5a27af7d271f3d69afcdc5a4d4c\": container with ID starting with 961229a9dfb14dbeeaa2472f8a6b829a56f1e5a27af7d271f3d69afcdc5a4d4c not found: ID does not exist" containerID="961229a9dfb14dbeeaa2472f8a6b829a56f1e5a27af7d271f3d69afcdc5a4d4c"
Sep 17 19:28:16.331396 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:16.331342    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"961229a9dfb14dbeeaa2472f8a6b829a56f1e5a27af7d271f3d69afcdc5a4d4c"} err="failed to get container status \"961229a9dfb14dbeeaa2472f8a6b829a56f1e5a27af7d271f3d69afcdc5a4d4c\": rpc error: code = NotFound desc = could not find container \"961229a9dfb14dbeeaa2472f8a6b829a56f1e5a27af7d271f3d69afcdc5a4d4c\": container with ID starting with 961229a9dfb14dbeeaa2472f8a6b829a56f1e5a27af7d271f3d69afcdc5a4d4c not found: ID does not exist"
Sep 17 19:28:16.336483 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:16.336459    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 19:28:16.341616 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:16.341595    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 19:28:16.608351 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:16.608274    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe8d5419-ce53-4118-bf3c-0e3417f60327" path="/var/lib/kubelet/pods/fe8d5419-ce53-4118-bf3c-0e3417f60327/volumes"
Sep 17 19:28:24.526166 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:24.526123    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77" podUID="285762e6-8c23-4c94-8b32-928a3d5edee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 17 19:28:34.526257 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:34.526183    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77" podUID="285762e6-8c23-4c94-8b32-928a3d5edee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 17 19:28:44.526821 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:44.526771    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77" podUID="285762e6-8c23-4c94-8b32-928a3d5edee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 17 19:28:48.734541 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:48.734499    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"]
Sep 17 19:28:48.735085 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:48.734903    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe8d5419-ce53-4118-bf3c-0e3417f60327" containerName="storage-initializer"
Sep 17 19:28:48.735085 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:48.734916    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe8d5419-ce53-4118-bf3c-0e3417f60327" containerName="storage-initializer"
Sep 17 19:28:48.735085 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:48.734935    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe8d5419-ce53-4118-bf3c-0e3417f60327" containerName="main"
Sep 17 19:28:48.735085 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:48.734941    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe8d5419-ce53-4118-bf3c-0e3417f60327" containerName="main"
Sep 17 19:28:48.735085 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:48.735008    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe8d5419-ce53-4118-bf3c-0e3417f60327" containerName="main"
Sep 17 19:28:48.738212 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:48.738173    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"
Sep 17 19:28:48.745645 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:48.745624    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 17 19:28:48.750019 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:48.749992    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 17 19:28:48.750136 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:48.750064    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 17 19:28:48.750136 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:48.750127    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-tqz5k\""
Sep 17 19:28:48.764321 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:48.764295    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58a685a8-a5f3-4d76-9ab1-e265065a9297-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-d2wcz\" (UID: \"58a685a8-a5f3-4d76-9ab1-e265065a9297\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"
Sep 17 19:28:48.764432 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:48.764359    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58a685a8-a5f3-4d76-9ab1-e265065a9297-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-d2wcz\" (UID: \"58a685a8-a5f3-4d76-9ab1-e265065a9297\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"
Sep 17 19:28:48.764432 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:48.764407    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58a685a8-a5f3-4d76-9ab1-e265065a9297-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-d2wcz\" (UID: \"58a685a8-a5f3-4d76-9ab1-e265065a9297\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"
Sep 17 19:28:48.764535 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:48.764492    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58a685a8-a5f3-4d76-9ab1-e265065a9297-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-d2wcz\" (UID: \"58a685a8-a5f3-4d76-9ab1-e265065a9297\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"
Sep 17 19:28:48.764535 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:48.764514    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58a685a8-a5f3-4d76-9ab1-e265065a9297-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-d2wcz\" (UID: \"58a685a8-a5f3-4d76-9ab1-e265065a9297\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"
Sep 17 19:28:48.764621 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:48.764532    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvkp5\" (UniqueName: \"kubernetes.io/projected/58a685a8-a5f3-4d76-9ab1-e265065a9297-kube-api-access-jvkp5\") pod \"scheduler-inline-config-test-kserve-8655bc757-d2wcz\" (UID: \"58a685a8-a5f3-4d76-9ab1-e265065a9297\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"
Sep 17 19:28:48.787564 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:48.787534    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"]
Sep 17 19:28:48.865366 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:48.865316    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58a685a8-a5f3-4d76-9ab1-e265065a9297-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-d2wcz\" (UID: \"58a685a8-a5f3-4d76-9ab1-e265065a9297\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"
Sep 17 19:28:48.865561 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:48.865389    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58a685a8-a5f3-4d76-9ab1-e265065a9297-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-d2wcz\" (UID: \"58a685a8-a5f3-4d76-9ab1-e265065a9297\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"
Sep 17 19:28:48.865561 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:48.865462    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58a685a8-a5f3-4d76-9ab1-e265065a9297-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-d2wcz\" (UID: \"58a685a8-a5f3-4d76-9ab1-e265065a9297\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"
Sep 17 19:28:48.865561 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:48.865494    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58a685a8-a5f3-4d76-9ab1-e265065a9297-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-d2wcz\" (UID: \"58a685a8-a5f3-4d76-9ab1-e265065a9297\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"
Sep 17 19:28:48.865561 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:48.865529    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvkp5\" (UniqueName: \"kubernetes.io/projected/58a685a8-a5f3-4d76-9ab1-e265065a9297-kube-api-access-jvkp5\") pod \"scheduler-inline-config-test-kserve-8655bc757-d2wcz\" (UID: \"58a685a8-a5f3-4d76-9ab1-e265065a9297\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"
Sep 17 19:28:48.865774 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:48.865562    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58a685a8-a5f3-4d76-9ab1-e265065a9297-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-d2wcz\" (UID: \"58a685a8-a5f3-4d76-9ab1-e265065a9297\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"
Sep 17 19:28:48.865895 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:48.865867    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58a685a8-a5f3-4d76-9ab1-e265065a9297-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-d2wcz\" (UID: \"58a685a8-a5f3-4d76-9ab1-e265065a9297\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"
Sep 17 19:28:48.866026 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:48.865932    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58a685a8-a5f3-4d76-9ab1-e265065a9297-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-d2wcz\" (UID: \"58a685a8-a5f3-4d76-9ab1-e265065a9297\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"
Sep 17 19:28:48.866093 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:48.866073    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58a685a8-a5f3-4d76-9ab1-e265065a9297-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-d2wcz\" (UID: \"58a685a8-a5f3-4d76-9ab1-e265065a9297\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"
Sep 17 19:28:48.867815 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:48.867793    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58a685a8-a5f3-4d76-9ab1-e265065a9297-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-d2wcz\" (UID: \"58a685a8-a5f3-4d76-9ab1-e265065a9297\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"
Sep 17 19:28:48.868037 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:48.868016    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58a685a8-a5f3-4d76-9ab1-e265065a9297-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-d2wcz\" (UID: \"58a685a8-a5f3-4d76-9ab1-e265065a9297\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"
Sep 17 19:28:48.876256 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:48.876227    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvkp5\" (UniqueName: \"kubernetes.io/projected/58a685a8-a5f3-4d76-9ab1-e265065a9297-kube-api-access-jvkp5\") pod \"scheduler-inline-config-test-kserve-8655bc757-d2wcz\" (UID: \"58a685a8-a5f3-4d76-9ab1-e265065a9297\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"
Sep 17 19:28:49.052757 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:49.052645    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"
Sep 17 19:28:49.334085 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:49.334059    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"]
Sep 17 19:28:49.336419 ip-10-0-139-75 kubenswrapper[2565]: W0917 19:28:49.336386    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a685a8_a5f3_4d76_9ab1_e265065a9297.slice/crio-5091ee47ecc75350bff834e73736cdb312158ad31cd66a0ad1e42bbdc67b81b7 WatchSource:0}: Error finding container 5091ee47ecc75350bff834e73736cdb312158ad31cd66a0ad1e42bbdc67b81b7: Status 404 returned error can't find the container with id 5091ee47ecc75350bff834e73736cdb312158ad31cd66a0ad1e42bbdc67b81b7
Sep 17 19:28:49.503678 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:49.503639    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz" event={"ID":"58a685a8-a5f3-4d76-9ab1-e265065a9297","Type":"ContainerStarted","Data":"b106937d7ec58d23cebd8216781dfa4cdc50d4db8dff98c236b46570b12dd767"}
Sep 17 19:28:49.503841 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:49.503710    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz" event={"ID":"58a685a8-a5f3-4d76-9ab1-e265065a9297","Type":"ContainerStarted","Data":"5091ee47ecc75350bff834e73736cdb312158ad31cd66a0ad1e42bbdc67b81b7"}
Sep 17 19:28:49.523382 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:49.523338    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz" podStartSLOduration=1.523322395 podStartE2EDuration="1.523322395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:28:49.521501959 +0000 UTC m=+3587.453703478" watchObservedRunningTime="2026-09-17 19:28:49.523322395 +0000 UTC m=+3587.455523914"
Sep 17 19:28:50.052774 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:50.052735    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"
Sep 17 19:28:54.526464 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:54.526415    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77" podUID="285762e6-8c23-4c94-8b32-928a3d5edee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 17 19:28:59.052873 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:59.052785    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"
Sep 17 19:28:59.065583 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:59.065546    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"
Sep 17 19:28:59.570111 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:28:59.570080    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"
Sep 17 19:29:04.536631 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:04.536595    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:29:04.544690 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:04.544657    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:29:09.181530 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:09.181491    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"]
Sep 17 19:29:23.891766 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:23.891728    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"]
Sep 17 19:29:23.901030 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:29:23.900986    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:29:23.901030 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:29:23.901030 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 19:29:23.901030 ip-10-0-139-75 kubenswrapper[2565]:  > containerID="b106937d7ec58d23cebd8216781dfa4cdc50d4db8dff98c236b46570b12dd767" cmd=["/bin/sleep","15"]
Sep 17 19:29:23.901308 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:23.901042    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz" message="" err=<
Sep 17 19:29:23.901308 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:29:23.901308 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 19:29:23.901308 ip-10-0-139-75 kubenswrapper[2565]:  >
Sep 17 19:29:23.901308 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:29:23.901055    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:29:23.901308 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:29:23.901308 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 19:29:23.901308 ip-10-0-139-75 kubenswrapper[2565]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz" podUID="58a685a8-a5f3-4d76-9ab1-e265065a9297" containerName="main" containerID="cri-o://b106937d7ec58d23cebd8216781dfa4cdc50d4db8dff98c236b46570b12dd767"
Sep 17 19:29:23.901308 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:23.901092    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz" podUID="58a685a8-a5f3-4d76-9ab1-e265065a9297" containerName="main" containerID="cri-o://b106937d7ec58d23cebd8216781dfa4cdc50d4db8dff98c236b46570b12dd767" gracePeriod=60
Sep 17 19:29:24.204474 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.204432    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77" podUID="285762e6-8c23-4c94-8b32-928a3d5edee4" containerName="main" containerID="cri-o://e0accedac635ba4bd4499be6677b28d932d398ffe3803b4eadafd9035d6429d9" gracePeriod=45
Sep 17 19:29:24.262122 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.262095    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"
Sep 17 19:29:24.354161 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.354130    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58a685a8-a5f3-4d76-9ab1-e265065a9297-home\") pod \"58a685a8-a5f3-4d76-9ab1-e265065a9297\" (UID: \"58a685a8-a5f3-4d76-9ab1-e265065a9297\") "
Sep 17 19:29:24.354341 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.354170    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58a685a8-a5f3-4d76-9ab1-e265065a9297-tls-certs\") pod \"58a685a8-a5f3-4d76-9ab1-e265065a9297\" (UID: \"58a685a8-a5f3-4d76-9ab1-e265065a9297\") "
Sep 17 19:29:24.354341 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.354188    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58a685a8-a5f3-4d76-9ab1-e265065a9297-model-cache\") pod \"58a685a8-a5f3-4d76-9ab1-e265065a9297\" (UID: \"58a685a8-a5f3-4d76-9ab1-e265065a9297\") "
Sep 17 19:29:24.354341 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.354240    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58a685a8-a5f3-4d76-9ab1-e265065a9297-tmp-dir\") pod \"58a685a8-a5f3-4d76-9ab1-e265065a9297\" (UID: \"58a685a8-a5f3-4d76-9ab1-e265065a9297\") "
Sep 17 19:29:24.354341 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.354314    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58a685a8-a5f3-4d76-9ab1-e265065a9297-dshm\") pod \"58a685a8-a5f3-4d76-9ab1-e265065a9297\" (UID: \"58a685a8-a5f3-4d76-9ab1-e265065a9297\") "
Sep 17 19:29:24.354557 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.354355    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvkp5\" (UniqueName: \"kubernetes.io/projected/58a685a8-a5f3-4d76-9ab1-e265065a9297-kube-api-access-jvkp5\") pod \"58a685a8-a5f3-4d76-9ab1-e265065a9297\" (UID: \"58a685a8-a5f3-4d76-9ab1-e265065a9297\") "
Sep 17 19:29:24.354557 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.354441    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a685a8-a5f3-4d76-9ab1-e265065a9297-model-cache" (OuterVolumeSpecName: "model-cache") pod "58a685a8-a5f3-4d76-9ab1-e265065a9297" (UID: "58a685a8-a5f3-4d76-9ab1-e265065a9297"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:29:24.354557 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.354470    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a685a8-a5f3-4d76-9ab1-e265065a9297-home" (OuterVolumeSpecName: "home") pod "58a685a8-a5f3-4d76-9ab1-e265065a9297" (UID: "58a685a8-a5f3-4d76-9ab1-e265065a9297"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:29:24.354557 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.354532    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a685a8-a5f3-4d76-9ab1-e265065a9297-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "58a685a8-a5f3-4d76-9ab1-e265065a9297" (UID: "58a685a8-a5f3-4d76-9ab1-e265065a9297"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:29:24.354722 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.354682    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58a685a8-a5f3-4d76-9ab1-e265065a9297-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:29:24.354722 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.354698    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58a685a8-a5f3-4d76-9ab1-e265065a9297-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:29:24.354813 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.354726    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58a685a8-a5f3-4d76-9ab1-e265065a9297-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:29:24.356575 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.356552    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a685a8-a5f3-4d76-9ab1-e265065a9297-kube-api-access-jvkp5" (OuterVolumeSpecName: "kube-api-access-jvkp5") pod "58a685a8-a5f3-4d76-9ab1-e265065a9297" (UID: "58a685a8-a5f3-4d76-9ab1-e265065a9297"). InnerVolumeSpecName "kube-api-access-jvkp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:29:24.356671 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.356620    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58a685a8-a5f3-4d76-9ab1-e265065a9297-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "58a685a8-a5f3-4d76-9ab1-e265065a9297" (UID: "58a685a8-a5f3-4d76-9ab1-e265065a9297"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:29:24.356671 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.356658    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a685a8-a5f3-4d76-9ab1-e265065a9297-dshm" (OuterVolumeSpecName: "dshm") pod "58a685a8-a5f3-4d76-9ab1-e265065a9297" (UID: "58a685a8-a5f3-4d76-9ab1-e265065a9297"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:29:24.456152 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.456061    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58a685a8-a5f3-4d76-9ab1-e265065a9297-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:29:24.456152 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.456102    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58a685a8-a5f3-4d76-9ab1-e265065a9297-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:29:24.456152 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.456118    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvkp5\" (UniqueName: \"kubernetes.io/projected/58a685a8-a5f3-4d76-9ab1-e265065a9297-kube-api-access-jvkp5\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:29:24.537480 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.537431    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77" podUID="285762e6-8c23-4c94-8b32-928a3d5edee4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Sep 17 19:29:24.708045 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.707955    2565 generic.go:356] "Generic (PLEG): container finished" podID="58a685a8-a5f3-4d76-9ab1-e265065a9297" containerID="b106937d7ec58d23cebd8216781dfa4cdc50d4db8dff98c236b46570b12dd767" exitCode=0
Sep 17 19:29:24.708045 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.708013    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"
Sep 17 19:29:24.708249 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.708044    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz" event={"ID":"58a685a8-a5f3-4d76-9ab1-e265065a9297","Type":"ContainerDied","Data":"b106937d7ec58d23cebd8216781dfa4cdc50d4db8dff98c236b46570b12dd767"}
Sep 17 19:29:24.708249 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.708096    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz" event={"ID":"58a685a8-a5f3-4d76-9ab1-e265065a9297","Type":"ContainerDied","Data":"5091ee47ecc75350bff834e73736cdb312158ad31cd66a0ad1e42bbdc67b81b7"}
Sep 17 19:29:24.708249 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.708122    2565 scope.go:117] "RemoveContainer" containerID="b106937d7ec58d23cebd8216781dfa4cdc50d4db8dff98c236b46570b12dd767"
Sep 17 19:29:24.717693 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.717672    2565 scope.go:117] "RemoveContainer" containerID="b106937d7ec58d23cebd8216781dfa4cdc50d4db8dff98c236b46570b12dd767"
Sep 17 19:29:24.718005 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:29:24.717980    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b106937d7ec58d23cebd8216781dfa4cdc50d4db8dff98c236b46570b12dd767\": container with ID starting with b106937d7ec58d23cebd8216781dfa4cdc50d4db8dff98c236b46570b12dd767 not found: ID does not exist" containerID="b106937d7ec58d23cebd8216781dfa4cdc50d4db8dff98c236b46570b12dd767"
Sep 17 19:29:24.718063 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.718016    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b106937d7ec58d23cebd8216781dfa4cdc50d4db8dff98c236b46570b12dd767"} err="failed to get container status \"b106937d7ec58d23cebd8216781dfa4cdc50d4db8dff98c236b46570b12dd767\": rpc error: code = NotFound desc = could not find container \"b106937d7ec58d23cebd8216781dfa4cdc50d4db8dff98c236b46570b12dd767\": container with ID starting with b106937d7ec58d23cebd8216781dfa4cdc50d4db8dff98c236b46570b12dd767 not found: ID does not exist"
Sep 17 19:29:24.723923 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.723899    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"]
Sep 17 19:29:24.728084 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.728062    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-d2wcz"]
Sep 17 19:29:24.940729 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:24.940704    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:29:25.060876 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.060850    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/285762e6-8c23-4c94-8b32-928a3d5edee4-dshm\") pod \"285762e6-8c23-4c94-8b32-928a3d5edee4\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") "
Sep 17 19:29:25.060986 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.060896    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/285762e6-8c23-4c94-8b32-928a3d5edee4-kserve-provision-location\") pod \"285762e6-8c23-4c94-8b32-928a3d5edee4\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") "
Sep 17 19:29:25.060986 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.060920    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/285762e6-8c23-4c94-8b32-928a3d5edee4-home\") pod \"285762e6-8c23-4c94-8b32-928a3d5edee4\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") "
Sep 17 19:29:25.060986 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.060940    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/285762e6-8c23-4c94-8b32-928a3d5edee4-cabundle-cert\") pod \"285762e6-8c23-4c94-8b32-928a3d5edee4\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") "
Sep 17 19:29:25.061167 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.060997    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/285762e6-8c23-4c94-8b32-928a3d5edee4-model-cache\") pod \"285762e6-8c23-4c94-8b32-928a3d5edee4\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") "
Sep 17 19:29:25.061167 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.061024    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/285762e6-8c23-4c94-8b32-928a3d5edee4-tls-certs\") pod \"285762e6-8c23-4c94-8b32-928a3d5edee4\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") "
Sep 17 19:29:25.061167 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.061054    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jm8jc\" (UniqueName: \"kubernetes.io/projected/285762e6-8c23-4c94-8b32-928a3d5edee4-kube-api-access-jm8jc\") pod \"285762e6-8c23-4c94-8b32-928a3d5edee4\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") "
Sep 17 19:29:25.061167 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.061091    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/285762e6-8c23-4c94-8b32-928a3d5edee4-tmp-dir\") pod \"285762e6-8c23-4c94-8b32-928a3d5edee4\" (UID: \"285762e6-8c23-4c94-8b32-928a3d5edee4\") "
Sep 17 19:29:25.061404 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.061317    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285762e6-8c23-4c94-8b32-928a3d5edee4-model-cache" (OuterVolumeSpecName: "model-cache") pod "285762e6-8c23-4c94-8b32-928a3d5edee4" (UID: "285762e6-8c23-4c94-8b32-928a3d5edee4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:29:25.061644 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.061616    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/285762e6-8c23-4c94-8b32-928a3d5edee4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "285762e6-8c23-4c94-8b32-928a3d5edee4" (UID: "285762e6-8c23-4c94-8b32-928a3d5edee4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:29:25.061784 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.061761    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285762e6-8c23-4c94-8b32-928a3d5edee4-home" (OuterVolumeSpecName: "home") pod "285762e6-8c23-4c94-8b32-928a3d5edee4" (UID: "285762e6-8c23-4c94-8b32-928a3d5edee4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:29:25.063399 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.063374    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285762e6-8c23-4c94-8b32-928a3d5edee4-dshm" (OuterVolumeSpecName: "dshm") pod "285762e6-8c23-4c94-8b32-928a3d5edee4" (UID: "285762e6-8c23-4c94-8b32-928a3d5edee4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:29:25.063990 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.063963    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/285762e6-8c23-4c94-8b32-928a3d5edee4-kube-api-access-jm8jc" (OuterVolumeSpecName: "kube-api-access-jm8jc") pod "285762e6-8c23-4c94-8b32-928a3d5edee4" (UID: "285762e6-8c23-4c94-8b32-928a3d5edee4"). InnerVolumeSpecName "kube-api-access-jm8jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:29:25.064090 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.064011    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/285762e6-8c23-4c94-8b32-928a3d5edee4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "285762e6-8c23-4c94-8b32-928a3d5edee4" (UID: "285762e6-8c23-4c94-8b32-928a3d5edee4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:29:25.075745 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.075713    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285762e6-8c23-4c94-8b32-928a3d5edee4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "285762e6-8c23-4c94-8b32-928a3d5edee4" (UID: "285762e6-8c23-4c94-8b32-928a3d5edee4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:29:25.085482 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.085453    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/285762e6-8c23-4c94-8b32-928a3d5edee4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "285762e6-8c23-4c94-8b32-928a3d5edee4" (UID: "285762e6-8c23-4c94-8b32-928a3d5edee4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:29:25.162162 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.162137    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/285762e6-8c23-4c94-8b32-928a3d5edee4-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:29:25.162162 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.162159    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/285762e6-8c23-4c94-8b32-928a3d5edee4-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:29:25.162308 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.162169    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/285762e6-8c23-4c94-8b32-928a3d5edee4-kserve-provision-location\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:29:25.162308 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.162179    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/285762e6-8c23-4c94-8b32-928a3d5edee4-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:29:25.162308 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.162187    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/285762e6-8c23-4c94-8b32-928a3d5edee4-cabundle-cert\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:29:25.162308 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.162195    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/285762e6-8c23-4c94-8b32-928a3d5edee4-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:29:25.162308 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.162222    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/285762e6-8c23-4c94-8b32-928a3d5edee4-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:29:25.162308 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.162230    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm8jc\" (UniqueName: \"kubernetes.io/projected/285762e6-8c23-4c94-8b32-928a3d5edee4-kube-api-access-jm8jc\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:29:25.715732 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.715688    2565 generic.go:356] "Generic (PLEG): container finished" podID="285762e6-8c23-4c94-8b32-928a3d5edee4" containerID="e0accedac635ba4bd4499be6677b28d932d398ffe3803b4eadafd9035d6429d9" exitCode=0
Sep 17 19:29:25.715916 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.715748    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"
Sep 17 19:29:25.715916 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.715778    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77" event={"ID":"285762e6-8c23-4c94-8b32-928a3d5edee4","Type":"ContainerDied","Data":"e0accedac635ba4bd4499be6677b28d932d398ffe3803b4eadafd9035d6429d9"}
Sep 17 19:29:25.715916 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.715839    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77" event={"ID":"285762e6-8c23-4c94-8b32-928a3d5edee4","Type":"ContainerDied","Data":"e65ec73b6ea21e6bd961a1ca0065bc1e7a39e59c7318b9a298ae102808b82a50"}
Sep 17 19:29:25.715916 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.715868    2565 scope.go:117] "RemoveContainer" containerID="e0accedac635ba4bd4499be6677b28d932d398ffe3803b4eadafd9035d6429d9"
Sep 17 19:29:25.727435 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.727406    2565 scope.go:117] "RemoveContainer" containerID="559f1e0066dfdbaa26120857a80da88ceccfeb6e1969e96240d38e8063a07d56"
Sep 17 19:29:25.743018 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.742991    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"]
Sep 17 19:29:25.743100 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.743077    2565 scope.go:117] "RemoveContainer" containerID="e0accedac635ba4bd4499be6677b28d932d398ffe3803b4eadafd9035d6429d9"
Sep 17 19:29:25.743427 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:29:25.743404    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0accedac635ba4bd4499be6677b28d932d398ffe3803b4eadafd9035d6429d9\": container with ID starting with e0accedac635ba4bd4499be6677b28d932d398ffe3803b4eadafd9035d6429d9 not found: ID does not exist" containerID="e0accedac635ba4bd4499be6677b28d932d398ffe3803b4eadafd9035d6429d9"
Sep 17 19:29:25.743509 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.743432    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0accedac635ba4bd4499be6677b28d932d398ffe3803b4eadafd9035d6429d9"} err="failed to get container status \"e0accedac635ba4bd4499be6677b28d932d398ffe3803b4eadafd9035d6429d9\": rpc error: code = NotFound desc = could not find container \"e0accedac635ba4bd4499be6677b28d932d398ffe3803b4eadafd9035d6429d9\": container with ID starting with e0accedac635ba4bd4499be6677b28d932d398ffe3803b4eadafd9035d6429d9 not found: ID does not exist"
Sep 17 19:29:25.743509 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.743451    2565 scope.go:117] "RemoveContainer" containerID="559f1e0066dfdbaa26120857a80da88ceccfeb6e1969e96240d38e8063a07d56"
Sep 17 19:29:25.743723 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:29:25.743707    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"559f1e0066dfdbaa26120857a80da88ceccfeb6e1969e96240d38e8063a07d56\": container with ID starting with 559f1e0066dfdbaa26120857a80da88ceccfeb6e1969e96240d38e8063a07d56 not found: ID does not exist" containerID="559f1e0066dfdbaa26120857a80da88ceccfeb6e1969e96240d38e8063a07d56"
Sep 17 19:29:25.743765 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.743726    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"559f1e0066dfdbaa26120857a80da88ceccfeb6e1969e96240d38e8063a07d56"} err="failed to get container status \"559f1e0066dfdbaa26120857a80da88ceccfeb6e1969e96240d38e8063a07d56\": rpc error: code = NotFound desc = could not find container \"559f1e0066dfdbaa26120857a80da88ceccfeb6e1969e96240d38e8063a07d56\": container with ID starting with 559f1e0066dfdbaa26120857a80da88ceccfeb6e1969e96240d38e8063a07d56 not found: ID does not exist"
Sep 17 19:29:25.747019 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:25.746993    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6dd587f89b-wst77"]
Sep 17 19:29:26.607916 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:26.607882    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="285762e6-8c23-4c94-8b32-928a3d5edee4" path="/var/lib/kubelet/pods/285762e6-8c23-4c94-8b32-928a3d5edee4/volumes"
Sep 17 19:29:26.608352 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:29:26.608336    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58a685a8-a5f3-4d76-9ab1-e265065a9297" path="/var/lib/kubelet/pods/58a685a8-a5f3-4d76-9ab1-e265065a9297/volumes"
Sep 17 19:30:02.298226 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.295917    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"]
Sep 17 19:30:02.298226 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.296741    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58a685a8-a5f3-4d76-9ab1-e265065a9297" containerName="main"
Sep 17 19:30:02.298226 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.296767    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="58a685a8-a5f3-4d76-9ab1-e265065a9297" containerName="main"
Sep 17 19:30:02.298226 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.296781    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="285762e6-8c23-4c94-8b32-928a3d5edee4" containerName="storage-initializer"
Sep 17 19:30:02.298226 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.296790    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="285762e6-8c23-4c94-8b32-928a3d5edee4" containerName="storage-initializer"
Sep 17 19:30:02.298226 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.296817    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="285762e6-8c23-4c94-8b32-928a3d5edee4" containerName="main"
Sep 17 19:30:02.298226 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.296826    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="285762e6-8c23-4c94-8b32-928a3d5edee4" containerName="main"
Sep 17 19:30:02.298226 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.297003    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="58a685a8-a5f3-4d76-9ab1-e265065a9297" containerName="main"
Sep 17 19:30:02.298226 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.297019    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="285762e6-8c23-4c94-8b32-928a3d5edee4" containerName="main"
Sep 17 19:30:02.303671 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.303610    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:30:02.306255 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.306225    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 17 19:30:02.306427 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.306235    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-8h8z6\""
Sep 17 19:30:02.306427 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.306380    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 17 19:30:02.306427 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.306394    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 17 19:30:02.306619 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.306400    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:30:02.306619 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.306502    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 17 19:30:02.311022 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.310997    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"]
Sep 17 19:30:02.436600 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.436570    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-dshm\") pod \"prestop-hook-test-kserve-6894dfb696-xh7wt\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:30:02.436600 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.436602    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-home\") pod \"prestop-hook-test-kserve-6894dfb696-xh7wt\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:30:02.436793 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.436626    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-model-cache\") pod \"prestop-hook-test-kserve-6894dfb696-xh7wt\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:30:02.436793 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.436690    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-cabundle-cert\") pod \"prestop-hook-test-kserve-6894dfb696-xh7wt\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:30:02.436793 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.436719    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-kserve-provision-location\") pod \"prestop-hook-test-kserve-6894dfb696-xh7wt\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:30:02.436793 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.436740    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-tmp-dir\") pod \"prestop-hook-test-kserve-6894dfb696-xh7wt\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:30:02.436793 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.436763    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-tls-certs\") pod \"prestop-hook-test-kserve-6894dfb696-xh7wt\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:30:02.436793 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.436783    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5qtk\" (UniqueName: \"kubernetes.io/projected/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-kube-api-access-z5qtk\") pod \"prestop-hook-test-kserve-6894dfb696-xh7wt\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:30:02.537516 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.537485    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-model-cache\") pod \"prestop-hook-test-kserve-6894dfb696-xh7wt\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:30:02.537685 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.537526    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-cabundle-cert\") pod \"prestop-hook-test-kserve-6894dfb696-xh7wt\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:30:02.537685 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.537545    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-kserve-provision-location\") pod \"prestop-hook-test-kserve-6894dfb696-xh7wt\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:30:02.537685 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.537567    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-tmp-dir\") pod \"prestop-hook-test-kserve-6894dfb696-xh7wt\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:30:02.537685 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.537590    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-tls-certs\") pod \"prestop-hook-test-kserve-6894dfb696-xh7wt\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:30:02.537685 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.537610    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5qtk\" (UniqueName: \"kubernetes.io/projected/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-kube-api-access-z5qtk\") pod \"prestop-hook-test-kserve-6894dfb696-xh7wt\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:30:02.537685 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.537658    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-dshm\") pod \"prestop-hook-test-kserve-6894dfb696-xh7wt\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:30:02.537685 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.537681    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-home\") pod \"prestop-hook-test-kserve-6894dfb696-xh7wt\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:30:02.538071 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.537980    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-tmp-dir\") pod \"prestop-hook-test-kserve-6894dfb696-xh7wt\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:30:02.538071 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.537989    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-kserve-provision-location\") pod \"prestop-hook-test-kserve-6894dfb696-xh7wt\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:30:02.538071 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.538037    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-model-cache\") pod \"prestop-hook-test-kserve-6894dfb696-xh7wt\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:30:02.538071 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.538067    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-home\") pod \"prestop-hook-test-kserve-6894dfb696-xh7wt\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:30:02.539949 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.539927    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-dshm\") pod \"prestop-hook-test-kserve-6894dfb696-xh7wt\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:30:02.540173 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.540153    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:30:02.540296 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.540271    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 17 19:30:02.548399 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.548316    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-cabundle-cert\") pod \"prestop-hook-test-kserve-6894dfb696-xh7wt\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:30:02.550148 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.550127    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-tls-certs\") pod \"prestop-hook-test-kserve-6894dfb696-xh7wt\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:30:02.550451 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.550434    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 17 19:30:02.560757 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.560737    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 17 19:30:02.570843 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.570818    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5qtk\" (UniqueName: \"kubernetes.io/projected/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-kube-api-access-z5qtk\") pod \"prestop-hook-test-kserve-6894dfb696-xh7wt\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:30:02.621298 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.621274    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-8h8z6\""
Sep 17 19:30:02.627336 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.627321    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:30:02.904479 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.904451    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"]
Sep 17 19:30:02.906129 ip-10-0-139-75 kubenswrapper[2565]: W0917 19:30:02.906107    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc9bbd9_f47e_47ea_9ab2_bf4d387b47f5.slice/crio-027a8b24aa7a8c456b9335a819acb576b0958abab0f663e8ec5d173ae0d0af94 WatchSource:0}: Error finding container 027a8b24aa7a8c456b9335a819acb576b0958abab0f663e8ec5d173ae0d0af94: Status 404 returned error can't find the container with id 027a8b24aa7a8c456b9335a819acb576b0958abab0f663e8ec5d173ae0d0af94
Sep 17 19:30:02.909954 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.909934    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 17 19:30:02.917187 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:02.917158    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt" event={"ID":"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5","Type":"ContainerStarted","Data":"027a8b24aa7a8c456b9335a819acb576b0958abab0f663e8ec5d173ae0d0af94"}
Sep 17 19:30:03.924049 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:03.924010    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt" event={"ID":"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5","Type":"ContainerStarted","Data":"012387cdef41d8f355bb310626c317f2ca50a27c23966b8725f56501f4826032"}
Sep 17 19:30:04.389027 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:04.388939    2565 scope.go:117] "RemoveContainer" containerID="78e64b5f7ec15ac947e7397f4ea5338af484565b71567d8a98f637dd2ee5cf9c"
Sep 17 19:30:05.936379 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:05.936347    2565 generic.go:356] "Generic (PLEG): container finished" podID="9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5" containerID="012387cdef41d8f355bb310626c317f2ca50a27c23966b8725f56501f4826032" exitCode=0
Sep 17 19:30:05.936793 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:05.936392    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt" event={"ID":"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5","Type":"ContainerDied","Data":"012387cdef41d8f355bb310626c317f2ca50a27c23966b8725f56501f4826032"}
Sep 17 19:30:06.944095 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:06.944053    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt" event={"ID":"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5","Type":"ContainerStarted","Data":"9606f2766a7cc7e11464e929578236c5cebc7d7e8ad7b2adda94377b6d1e9084"}
Sep 17 19:30:06.967472 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:06.967415    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt" podStartSLOduration=2.813725669 podStartE2EDuration="4.96740096s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.153675291s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:30:06.966069114 +0000 UTC m=+3664.898270634" watchObservedRunningTime="2026-09-17 19:30:06.96740096 +0000 UTC m=+3664.899602479"
Sep 17 19:30:07.627904 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:07.627858    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:30:12.627543 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:12.627509    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:30:12.629276 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:12.629236    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt" podUID="9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 19:30:22.628008 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:22.627961    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt" podUID="9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 19:30:32.628474 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:32.628382    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt" podUID="9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 19:30:42.628621 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:42.628569    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt" podUID="9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 19:30:52.628128 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:30:52.628080    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt" podUID="9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 19:31:02.628396 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:31:02.628355    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt" podUID="9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 19:31:12.628549 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:31:12.628508    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt" podUID="9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 19:31:22.628703 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:31:22.628665    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt" podUID="9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 19:31:32.628693 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:31:32.628646    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt" podUID="9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 19:31:42.628077 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:31:42.628028    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt" podUID="9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 19:31:52.627981 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:31:52.627935    2565 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt" podUID="9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Sep 17 19:32:00.473286 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:00.473178    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"]
Sep 17 19:32:00.477249 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:00.477226    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"
Sep 17 19:32:00.482027 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:00.482004    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 17 19:32:00.482027 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:00.482014    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 17 19:32:00.482183 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:00.482004    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 17 19:32:00.482183 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:00.482027    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-jwjdt\""
Sep 17 19:32:00.515304 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:00.515278    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"]
Sep 17 19:32:00.586784 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:00.586753    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77e84e92-235d-4bad-a61d-d3f2e82013c7-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8\" (UID: \"77e84e92-235d-4bad-a61d-d3f2e82013c7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"
Sep 17 19:32:00.586893 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:00.586789    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77e84e92-235d-4bad-a61d-d3f2e82013c7-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8\" (UID: \"77e84e92-235d-4bad-a61d-d3f2e82013c7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"
Sep 17 19:32:00.586893 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:00.586857    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77e84e92-235d-4bad-a61d-d3f2e82013c7-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8\" (UID: \"77e84e92-235d-4bad-a61d-d3f2e82013c7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"
Sep 17 19:32:00.586893 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:00.586877    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77e84e92-235d-4bad-a61d-d3f2e82013c7-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8\" (UID: \"77e84e92-235d-4bad-a61d-d3f2e82013c7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"
Sep 17 19:32:00.587009 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:00.586897    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77e84e92-235d-4bad-a61d-d3f2e82013c7-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8\" (UID: \"77e84e92-235d-4bad-a61d-d3f2e82013c7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"
Sep 17 19:32:00.587009 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:00.586983    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhdvj\" (UniqueName: \"kubernetes.io/projected/77e84e92-235d-4bad-a61d-d3f2e82013c7-kube-api-access-jhdvj\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8\" (UID: \"77e84e92-235d-4bad-a61d-d3f2e82013c7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"
Sep 17 19:32:00.687704 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:00.687678    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77e84e92-235d-4bad-a61d-d3f2e82013c7-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8\" (UID: \"77e84e92-235d-4bad-a61d-d3f2e82013c7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"
Sep 17 19:32:00.687829 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:00.687711    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77e84e92-235d-4bad-a61d-d3f2e82013c7-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8\" (UID: \"77e84e92-235d-4bad-a61d-d3f2e82013c7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"
Sep 17 19:32:00.687829 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:00.687729    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77e84e92-235d-4bad-a61d-d3f2e82013c7-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8\" (UID: \"77e84e92-235d-4bad-a61d-d3f2e82013c7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"
Sep 17 19:32:00.687829 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:00.687754    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhdvj\" (UniqueName: \"kubernetes.io/projected/77e84e92-235d-4bad-a61d-d3f2e82013c7-kube-api-access-jhdvj\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8\" (UID: \"77e84e92-235d-4bad-a61d-d3f2e82013c7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"
Sep 17 19:32:00.687829 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:00.687782    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77e84e92-235d-4bad-a61d-d3f2e82013c7-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8\" (UID: \"77e84e92-235d-4bad-a61d-d3f2e82013c7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"
Sep 17 19:32:00.688042 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:00.687832    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77e84e92-235d-4bad-a61d-d3f2e82013c7-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8\" (UID: \"77e84e92-235d-4bad-a61d-d3f2e82013c7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"
Sep 17 19:32:00.688185 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:00.688158    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77e84e92-235d-4bad-a61d-d3f2e82013c7-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8\" (UID: \"77e84e92-235d-4bad-a61d-d3f2e82013c7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"
Sep 17 19:32:00.688302 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:00.688177    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77e84e92-235d-4bad-a61d-d3f2e82013c7-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8\" (UID: \"77e84e92-235d-4bad-a61d-d3f2e82013c7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"
Sep 17 19:32:00.688302 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:00.688254    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77e84e92-235d-4bad-a61d-d3f2e82013c7-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8\" (UID: \"77e84e92-235d-4bad-a61d-d3f2e82013c7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"
Sep 17 19:32:00.690089 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:00.690068    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77e84e92-235d-4bad-a61d-d3f2e82013c7-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8\" (UID: \"77e84e92-235d-4bad-a61d-d3f2e82013c7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"
Sep 17 19:32:00.690552 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:00.690530    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77e84e92-235d-4bad-a61d-d3f2e82013c7-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8\" (UID: \"77e84e92-235d-4bad-a61d-d3f2e82013c7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"
Sep 17 19:32:00.697003 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:00.696981    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhdvj\" (UniqueName: \"kubernetes.io/projected/77e84e92-235d-4bad-a61d-d3f2e82013c7-kube-api-access-jhdvj\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8\" (UID: \"77e84e92-235d-4bad-a61d-d3f2e82013c7\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"
Sep 17 19:32:00.787268 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:00.787186    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"
Sep 17 19:32:01.069407 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:01.069376    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"]
Sep 17 19:32:01.070361 ip-10-0-139-75 kubenswrapper[2565]: W0917 19:32:01.070335    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e84e92_235d_4bad_a61d_d3f2e82013c7.slice/crio-3e8eda3fb6417d1907af20d9c68ed30ba1fc193b03a47158e8cc6adab4c27a36 WatchSource:0}: Error finding container 3e8eda3fb6417d1907af20d9c68ed30ba1fc193b03a47158e8cc6adab4c27a36: Status 404 returned error can't find the container with id 3e8eda3fb6417d1907af20d9c68ed30ba1fc193b03a47158e8cc6adab4c27a36
Sep 17 19:32:01.573680 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:01.573642    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8" event={"ID":"77e84e92-235d-4bad-a61d-d3f2e82013c7","Type":"ContainerStarted","Data":"4cc6e30dfe4d1506aa4f861f5c9587b8c6ad159e918842050c3b9dad18907ed3"}
Sep 17 19:32:01.574121 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:01.573707    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8" event={"ID":"77e84e92-235d-4bad-a61d-d3f2e82013c7","Type":"ContainerStarted","Data":"3e8eda3fb6417d1907af20d9c68ed30ba1fc193b03a47158e8cc6adab4c27a36"}
Sep 17 19:32:01.594676 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:01.594622    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8" podStartSLOduration=1.594604306 podStartE2EDuration="1.594604306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:32:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:32:01.592484559 +0000 UTC m=+3779.524686079" watchObservedRunningTime="2026-09-17 19:32:01.594604306 +0000 UTC m=+3779.526805825"
Sep 17 19:32:01.787868 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:01.787834    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"
Sep 17 19:32:02.638025 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:02.637990    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:32:02.647344 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:02.647318    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:32:05.013022 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:05.012981    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"]
Sep 17 19:32:10.788258 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:10.788192    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"
Sep 17 19:32:10.800627 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:10.800603    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"
Sep 17 19:32:11.638233 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:11.638183    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"
Sep 17 19:32:20.035359 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:20.035289    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt" podUID="9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5" containerName="main" containerID="cri-o://9606f2766a7cc7e11464e929578236c5cebc7d7e8ad7b2adda94377b6d1e9084" gracePeriod=45
Sep 17 19:32:20.675213 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:20.675175    2565 generic.go:356] "Generic (PLEG): container finished" podID="9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5" containerID="9606f2766a7cc7e11464e929578236c5cebc7d7e8ad7b2adda94377b6d1e9084" exitCode=0
Sep 17 19:32:20.675341 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:20.675235    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt" event={"ID":"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5","Type":"ContainerDied","Data":"9606f2766a7cc7e11464e929578236c5cebc7d7e8ad7b2adda94377b6d1e9084"}
Sep 17 19:32:20.675341 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:20.675293    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt" event={"ID":"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5","Type":"ContainerDied","Data":"027a8b24aa7a8c456b9335a819acb576b0958abab0f663e8ec5d173ae0d0af94"}
Sep 17 19:32:20.675341 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:20.675309    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="027a8b24aa7a8c456b9335a819acb576b0958abab0f663e8ec5d173ae0d0af94"
Sep 17 19:32:20.685063 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:20.685043    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:32:20.837509 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:20.837437    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-home\") pod \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") "
Sep 17 19:32:20.837509 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:20.837488    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-tls-certs\") pod \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") "
Sep 17 19:32:20.837719 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:20.837548    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-kserve-provision-location\") pod \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") "
Sep 17 19:32:20.837719 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:20.837568    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-dshm\") pod \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") "
Sep 17 19:32:20.837719 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:20.837587    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-tmp-dir\") pod \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") "
Sep 17 19:32:20.837719 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:20.837613    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-cabundle-cert\") pod \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") "
Sep 17 19:32:20.837719 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:20.837636    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5qtk\" (UniqueName: \"kubernetes.io/projected/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-kube-api-access-z5qtk\") pod \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") "
Sep 17 19:32:20.837719 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:20.837664    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-model-cache\") pod \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\" (UID: \"9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5\") "
Sep 17 19:32:20.838046 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:20.838003    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5" (UID: "9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:32:20.838189 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:20.838166    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-model-cache" (OuterVolumeSpecName: "model-cache") pod "9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5" (UID: "9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:32:20.838290 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:20.838265    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-home" (OuterVolumeSpecName: "home") pod "9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5" (UID: "9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:32:20.839998 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:20.839958    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-dshm" (OuterVolumeSpecName: "dshm") pod "9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5" (UID: "9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:32:20.840123 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:20.840013    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5" (UID: "9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:32:20.840273 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:20.840250    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-kube-api-access-z5qtk" (OuterVolumeSpecName: "kube-api-access-z5qtk") pod "9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5" (UID: "9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5"). InnerVolumeSpecName "kube-api-access-z5qtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:32:20.851801 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:20.851768    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5" (UID: "9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:32:20.862557 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:20.862532    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5" (UID: "9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:32:20.938726 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:20.938697    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:32:20.938726 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:20.938723    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:32:20.938876 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:20.938736    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-kserve-provision-location\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:32:20.938876 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:20.938746    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:32:20.938876 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:20.938759    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:32:20.938876 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:20.938768    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-cabundle-cert\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:32:20.938876 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:20.938777    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5qtk\" (UniqueName: \"kubernetes.io/projected/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-kube-api-access-z5qtk\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:32:20.938876 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:20.938785    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:32:21.638828 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:21.638796    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt" podUID="9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": context deadline exceeded"
Sep 17 19:32:21.681295 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:21.681268    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"
Sep 17 19:32:21.705671 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:21.705647    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"]
Sep 17 19:32:21.714015 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:21.713988    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6894dfb696-xh7wt"]
Sep 17 19:32:22.607905 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:22.607875    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5" path="/var/lib/kubelet/pods/9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5/volumes"
Sep 17 19:32:39.278339 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:39.278293    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"]
Sep 17 19:32:39.287956 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:32:39.287910    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:32:39.287956 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:32:39.287956 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 19:32:39.287956 ip-10-0-139-75 kubenswrapper[2565]:  > containerID="4cc6e30dfe4d1506aa4f861f5c9587b8c6ad159e918842050c3b9dad18907ed3" cmd=["/bin/sleep","15"]
Sep 17 19:32:39.288350 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:39.287981    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8" message="" err=<
Sep 17 19:32:39.288350 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:32:39.288350 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 19:32:39.288350 ip-10-0-139-75 kubenswrapper[2565]:  >
Sep 17 19:32:39.288350 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:32:39.287999    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:32:39.288350 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:32:39.288350 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 19:32:39.288350 ip-10-0-139-75 kubenswrapper[2565]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8" podUID="77e84e92-235d-4bad-a61d-d3f2e82013c7" containerName="main" containerID="cri-o://4cc6e30dfe4d1506aa4f861f5c9587b8c6ad159e918842050c3b9dad18907ed3"
Sep 17 19:32:39.288350 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:39.288047    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8" podUID="77e84e92-235d-4bad-a61d-d3f2e82013c7" containerName="main" containerID="cri-o://4cc6e30dfe4d1506aa4f861f5c9587b8c6ad159e918842050c3b9dad18907ed3" gracePeriod=60
Sep 17 19:32:39.643022 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:39.642996    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"
Sep 17 19:32:39.773029 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:39.772993    2565 generic.go:356] "Generic (PLEG): container finished" podID="77e84e92-235d-4bad-a61d-d3f2e82013c7" containerID="4cc6e30dfe4d1506aa4f861f5c9587b8c6ad159e918842050c3b9dad18907ed3" exitCode=0
Sep 17 19:32:39.773164 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:39.773020    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8" event={"ID":"77e84e92-235d-4bad-a61d-d3f2e82013c7","Type":"ContainerDied","Data":"4cc6e30dfe4d1506aa4f861f5c9587b8c6ad159e918842050c3b9dad18907ed3"}
Sep 17 19:32:39.773164 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:39.773037    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"
Sep 17 19:32:39.773164 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:39.773102    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8" event={"ID":"77e84e92-235d-4bad-a61d-d3f2e82013c7","Type":"ContainerDied","Data":"3e8eda3fb6417d1907af20d9c68ed30ba1fc193b03a47158e8cc6adab4c27a36"}
Sep 17 19:32:39.773164 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:39.773128    2565 scope.go:117] "RemoveContainer" containerID="4cc6e30dfe4d1506aa4f861f5c9587b8c6ad159e918842050c3b9dad18907ed3"
Sep 17 19:32:39.777018 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:39.776994    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77e84e92-235d-4bad-a61d-d3f2e82013c7-dshm\") pod \"77e84e92-235d-4bad-a61d-d3f2e82013c7\" (UID: \"77e84e92-235d-4bad-a61d-d3f2e82013c7\") "
Sep 17 19:32:39.777236 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:39.777035    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhdvj\" (UniqueName: \"kubernetes.io/projected/77e84e92-235d-4bad-a61d-d3f2e82013c7-kube-api-access-jhdvj\") pod \"77e84e92-235d-4bad-a61d-d3f2e82013c7\" (UID: \"77e84e92-235d-4bad-a61d-d3f2e82013c7\") "
Sep 17 19:32:39.777236 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:39.777092    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77e84e92-235d-4bad-a61d-d3f2e82013c7-home\") pod \"77e84e92-235d-4bad-a61d-d3f2e82013c7\" (UID: \"77e84e92-235d-4bad-a61d-d3f2e82013c7\") "
Sep 17 19:32:39.777236 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:39.777154    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77e84e92-235d-4bad-a61d-d3f2e82013c7-model-cache\") pod \"77e84e92-235d-4bad-a61d-d3f2e82013c7\" (UID: \"77e84e92-235d-4bad-a61d-d3f2e82013c7\") "
Sep 17 19:32:39.777236 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:39.777227    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77e84e92-235d-4bad-a61d-d3f2e82013c7-tls-certs\") pod \"77e84e92-235d-4bad-a61d-d3f2e82013c7\" (UID: \"77e84e92-235d-4bad-a61d-d3f2e82013c7\") "
Sep 17 19:32:39.777502 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:39.777255    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77e84e92-235d-4bad-a61d-d3f2e82013c7-tmp-dir\") pod \"77e84e92-235d-4bad-a61d-d3f2e82013c7\" (UID: \"77e84e92-235d-4bad-a61d-d3f2e82013c7\") "
Sep 17 19:32:39.777502 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:39.777420    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77e84e92-235d-4bad-a61d-d3f2e82013c7-home" (OuterVolumeSpecName: "home") pod "77e84e92-235d-4bad-a61d-d3f2e82013c7" (UID: "77e84e92-235d-4bad-a61d-d3f2e82013c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:32:39.777617 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:39.777528    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77e84e92-235d-4bad-a61d-d3f2e82013c7-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:32:39.777681 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:39.777653    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77e84e92-235d-4bad-a61d-d3f2e82013c7-model-cache" (OuterVolumeSpecName: "model-cache") pod "77e84e92-235d-4bad-a61d-d3f2e82013c7" (UID: "77e84e92-235d-4bad-a61d-d3f2e82013c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:32:39.777738 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:39.777692    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77e84e92-235d-4bad-a61d-d3f2e82013c7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "77e84e92-235d-4bad-a61d-d3f2e82013c7" (UID: "77e84e92-235d-4bad-a61d-d3f2e82013c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:32:39.779596 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:39.779554    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77e84e92-235d-4bad-a61d-d3f2e82013c7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "77e84e92-235d-4bad-a61d-d3f2e82013c7" (UID: "77e84e92-235d-4bad-a61d-d3f2e82013c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:32:39.779787 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:39.779767    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77e84e92-235d-4bad-a61d-d3f2e82013c7-dshm" (OuterVolumeSpecName: "dshm") pod "77e84e92-235d-4bad-a61d-d3f2e82013c7" (UID: "77e84e92-235d-4bad-a61d-d3f2e82013c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:32:39.779901 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:39.779877    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77e84e92-235d-4bad-a61d-d3f2e82013c7-kube-api-access-jhdvj" (OuterVolumeSpecName: "kube-api-access-jhdvj") pod "77e84e92-235d-4bad-a61d-d3f2e82013c7" (UID: "77e84e92-235d-4bad-a61d-d3f2e82013c7"). InnerVolumeSpecName "kube-api-access-jhdvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:32:39.783070 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:39.783050    2565 scope.go:117] "RemoveContainer" containerID="4cc6e30dfe4d1506aa4f861f5c9587b8c6ad159e918842050c3b9dad18907ed3"
Sep 17 19:32:39.783344 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:32:39.783314    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cc6e30dfe4d1506aa4f861f5c9587b8c6ad159e918842050c3b9dad18907ed3\": container with ID starting with 4cc6e30dfe4d1506aa4f861f5c9587b8c6ad159e918842050c3b9dad18907ed3 not found: ID does not exist" containerID="4cc6e30dfe4d1506aa4f861f5c9587b8c6ad159e918842050c3b9dad18907ed3"
Sep 17 19:32:39.783399 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:39.783343    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cc6e30dfe4d1506aa4f861f5c9587b8c6ad159e918842050c3b9dad18907ed3"} err="failed to get container status \"4cc6e30dfe4d1506aa4f861f5c9587b8c6ad159e918842050c3b9dad18907ed3\": rpc error: code = NotFound desc = could not find container \"4cc6e30dfe4d1506aa4f861f5c9587b8c6ad159e918842050c3b9dad18907ed3\": container with ID starting with 4cc6e30dfe4d1506aa4f861f5c9587b8c6ad159e918842050c3b9dad18907ed3 not found: ID does not exist"
Sep 17 19:32:39.884303 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:39.878352    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhdvj\" (UniqueName: \"kubernetes.io/projected/77e84e92-235d-4bad-a61d-d3f2e82013c7-kube-api-access-jhdvj\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:32:39.884303 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:39.878388    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77e84e92-235d-4bad-a61d-d3f2e82013c7-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:32:39.884303 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:39.878406    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77e84e92-235d-4bad-a61d-d3f2e82013c7-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:32:39.884303 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:39.878421    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77e84e92-235d-4bad-a61d-d3f2e82013c7-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:32:39.884303 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:39.878442    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77e84e92-235d-4bad-a61d-d3f2e82013c7-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:32:40.095941 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:40.095911    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"]
Sep 17 19:32:40.099441 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:40.099416    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8"]
Sep 17 19:32:40.607106 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:40.607071    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77e84e92-235d-4bad-a61d-d3f2e82013c7" path="/var/lib/kubelet/pods/77e84e92-235d-4bad-a61d-d3f2e82013c7/volumes"
Sep 17 19:32:40.628253 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:32:40.628222    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-v8gz8" podUID="77e84e92-235d-4bad-a61d-d3f2e82013c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 19:33:41.315372 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.315296    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"]
Sep 17 19:33:41.315884 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.315675    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5" containerName="main"
Sep 17 19:33:41.315884 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.315691    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5" containerName="main"
Sep 17 19:33:41.315884 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.315717    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5" containerName="storage-initializer"
Sep 17 19:33:41.315884 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.315723    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5" containerName="storage-initializer"
Sep 17 19:33:41.315884 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.315733    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77e84e92-235d-4bad-a61d-d3f2e82013c7" containerName="main"
Sep 17 19:33:41.315884 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.315739    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="77e84e92-235d-4bad-a61d-d3f2e82013c7" containerName="main"
Sep 17 19:33:41.315884 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.315803    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fc9bbd9-f47e-47ea-9ab2-bf4d387b47f5" containerName="main"
Sep 17 19:33:41.315884 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.315812    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="77e84e92-235d-4bad-a61d-d3f2e82013c7" containerName="main"
Sep 17 19:33:41.319004 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.318982    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"
Sep 17 19:33:41.321894 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.321873    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 17 19:33:41.322896 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.322874    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 17 19:33:41.323014 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.322915    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-9psbc\""
Sep 17 19:33:41.323014 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.322934    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 17 19:33:41.338251 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.338219    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"]
Sep 17 19:33:41.426427 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.426398    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/886d84ae-3b4b-4047-a616-6731adf1935f-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4qksn\" (UID: \"886d84ae-3b4b-4047-a616-6731adf1935f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"
Sep 17 19:33:41.426533 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.426446    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/886d84ae-3b4b-4047-a616-6731adf1935f-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4qksn\" (UID: \"886d84ae-3b4b-4047-a616-6731adf1935f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"
Sep 17 19:33:41.426533 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.426469    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp6s8\" (UniqueName: \"kubernetes.io/projected/886d84ae-3b4b-4047-a616-6731adf1935f-kube-api-access-tp6s8\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4qksn\" (UID: \"886d84ae-3b4b-4047-a616-6731adf1935f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"
Sep 17 19:33:41.426533 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.426516    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/886d84ae-3b4b-4047-a616-6731adf1935f-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4qksn\" (UID: \"886d84ae-3b4b-4047-a616-6731adf1935f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"
Sep 17 19:33:41.426641 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.426542    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/886d84ae-3b4b-4047-a616-6731adf1935f-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4qksn\" (UID: \"886d84ae-3b4b-4047-a616-6731adf1935f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"
Sep 17 19:33:41.426641 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.426600    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/886d84ae-3b4b-4047-a616-6731adf1935f-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4qksn\" (UID: \"886d84ae-3b4b-4047-a616-6731adf1935f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"
Sep 17 19:33:41.527809 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.527782    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp6s8\" (UniqueName: \"kubernetes.io/projected/886d84ae-3b4b-4047-a616-6731adf1935f-kube-api-access-tp6s8\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4qksn\" (UID: \"886d84ae-3b4b-4047-a616-6731adf1935f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"
Sep 17 19:33:41.527916 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.527814    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/886d84ae-3b4b-4047-a616-6731adf1935f-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4qksn\" (UID: \"886d84ae-3b4b-4047-a616-6731adf1935f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"
Sep 17 19:33:41.527916 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.527832    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/886d84ae-3b4b-4047-a616-6731adf1935f-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4qksn\" (UID: \"886d84ae-3b4b-4047-a616-6731adf1935f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"
Sep 17 19:33:41.527916 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.527851    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/886d84ae-3b4b-4047-a616-6731adf1935f-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4qksn\" (UID: \"886d84ae-3b4b-4047-a616-6731adf1935f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"
Sep 17 19:33:41.527916 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.527886    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/886d84ae-3b4b-4047-a616-6731adf1935f-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4qksn\" (UID: \"886d84ae-3b4b-4047-a616-6731adf1935f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"
Sep 17 19:33:41.528189 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.527928    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/886d84ae-3b4b-4047-a616-6731adf1935f-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4qksn\" (UID: \"886d84ae-3b4b-4047-a616-6731adf1935f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"
Sep 17 19:33:41.528285 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.528255    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/886d84ae-3b4b-4047-a616-6731adf1935f-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4qksn\" (UID: \"886d84ae-3b4b-4047-a616-6731adf1935f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"
Sep 17 19:33:41.528347 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.528292    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/886d84ae-3b4b-4047-a616-6731adf1935f-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4qksn\" (UID: \"886d84ae-3b4b-4047-a616-6731adf1935f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"
Sep 17 19:33:41.528347 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.528328    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/886d84ae-3b4b-4047-a616-6731adf1935f-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4qksn\" (UID: \"886d84ae-3b4b-4047-a616-6731adf1935f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"
Sep 17 19:33:41.530210 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.530176    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/886d84ae-3b4b-4047-a616-6731adf1935f-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4qksn\" (UID: \"886d84ae-3b4b-4047-a616-6731adf1935f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"
Sep 17 19:33:41.531006 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.530972    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/886d84ae-3b4b-4047-a616-6731adf1935f-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4qksn\" (UID: \"886d84ae-3b4b-4047-a616-6731adf1935f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"
Sep 17 19:33:41.537289 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.537259    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp6s8\" (UniqueName: \"kubernetes.io/projected/886d84ae-3b4b-4047-a616-6731adf1935f-kube-api-access-tp6s8\") pod \"rolling-upgrade-test-kserve-74b79d7f77-4qksn\" (UID: \"886d84ae-3b4b-4047-a616-6731adf1935f\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"
Sep 17 19:33:41.629246 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.629178    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"
Sep 17 19:33:41.903345 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:41.903310    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"]
Sep 17 19:33:41.904285 ip-10-0-139-75 kubenswrapper[2565]: W0917 19:33:41.904257    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod886d84ae_3b4b_4047_a616_6731adf1935f.slice/crio-5598c343028407d07a26782cca9bada0099439770b81fb94ef506cea8e09e865 WatchSource:0}: Error finding container 5598c343028407d07a26782cca9bada0099439770b81fb94ef506cea8e09e865: Status 404 returned error can't find the container with id 5598c343028407d07a26782cca9bada0099439770b81fb94ef506cea8e09e865
Sep 17 19:33:42.108460 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:42.108415    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn" event={"ID":"886d84ae-3b4b-4047-a616-6731adf1935f","Type":"ContainerStarted","Data":"b37b073787f509a0f796ac600a9b761b7ba0b3915ae892e6c4fffadd8e3478b0"}
Sep 17 19:33:42.108600 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:42.108468    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn" event={"ID":"886d84ae-3b4b-4047-a616-6731adf1935f","Type":"ContainerStarted","Data":"5598c343028407d07a26782cca9bada0099439770b81fb94ef506cea8e09e865"}
Sep 17 19:33:42.128509 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:42.128454    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn" podStartSLOduration=1.128439749 podStartE2EDuration="1.128439749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:33:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:33:42.126381238 +0000 UTC m=+3880.058582758" watchObservedRunningTime="2026-09-17 19:33:42.128439749 +0000 UTC m=+3880.060641262"
Sep 17 19:33:42.630187 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:42.630157    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"
Sep 17 19:33:51.629892 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:51.629797    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"
Sep 17 19:33:51.642354 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:51.642326    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"
Sep 17 19:33:52.172437 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:33:52.172402    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"
Sep 17 19:34:26.294529 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:26.294487    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"]
Sep 17 19:34:26.303627 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:34:26.303582    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:34:26.303627 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:34:26.303627 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 19:34:26.303627 ip-10-0-139-75 kubenswrapper[2565]:  > containerID="b37b073787f509a0f796ac600a9b761b7ba0b3915ae892e6c4fffadd8e3478b0" cmd=["/bin/sleep","15"]
Sep 17 19:34:26.303874 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:26.303650    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn" message="" err=<
Sep 17 19:34:26.303874 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:34:26.303874 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 19:34:26.303874 ip-10-0-139-75 kubenswrapper[2565]:  >
Sep 17 19:34:26.303874 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:34:26.303666    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:34:26.303874 ip-10-0-139-75 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:34:26.303874 ip-10-0-139-75 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Sep 17 19:34:26.303874 ip-10-0-139-75 kubenswrapper[2565]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn" podUID="886d84ae-3b4b-4047-a616-6731adf1935f" containerName="main" containerID="cri-o://b37b073787f509a0f796ac600a9b761b7ba0b3915ae892e6c4fffadd8e3478b0"
Sep 17 19:34:26.303874 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:26.303720    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn" podUID="886d84ae-3b4b-4047-a616-6731adf1935f" containerName="main" containerID="cri-o://b37b073787f509a0f796ac600a9b761b7ba0b3915ae892e6c4fffadd8e3478b0" gracePeriod=60
Sep 17 19:34:26.660328 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:26.660303    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"
Sep 17 19:34:26.754496 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:26.754471    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/886d84ae-3b4b-4047-a616-6731adf1935f-model-cache\") pod \"886d84ae-3b4b-4047-a616-6731adf1935f\" (UID: \"886d84ae-3b4b-4047-a616-6731adf1935f\") "
Sep 17 19:34:26.754655 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:26.754535    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tp6s8\" (UniqueName: \"kubernetes.io/projected/886d84ae-3b4b-4047-a616-6731adf1935f-kube-api-access-tp6s8\") pod \"886d84ae-3b4b-4047-a616-6731adf1935f\" (UID: \"886d84ae-3b4b-4047-a616-6731adf1935f\") "
Sep 17 19:34:26.754655 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:26.754576    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/886d84ae-3b4b-4047-a616-6731adf1935f-dshm\") pod \"886d84ae-3b4b-4047-a616-6731adf1935f\" (UID: \"886d84ae-3b4b-4047-a616-6731adf1935f\") "
Sep 17 19:34:26.754655 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:26.754602    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/886d84ae-3b4b-4047-a616-6731adf1935f-tls-certs\") pod \"886d84ae-3b4b-4047-a616-6731adf1935f\" (UID: \"886d84ae-3b4b-4047-a616-6731adf1935f\") "
Sep 17 19:34:26.754655 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:26.754638    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/886d84ae-3b4b-4047-a616-6731adf1935f-home\") pod \"886d84ae-3b4b-4047-a616-6731adf1935f\" (UID: \"886d84ae-3b4b-4047-a616-6731adf1935f\") "
Sep 17 19:34:26.754891 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:26.754684    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/886d84ae-3b4b-4047-a616-6731adf1935f-tmp-dir\") pod \"886d84ae-3b4b-4047-a616-6731adf1935f\" (UID: \"886d84ae-3b4b-4047-a616-6731adf1935f\") "
Sep 17 19:34:26.754891 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:26.754765    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/886d84ae-3b4b-4047-a616-6731adf1935f-model-cache" (OuterVolumeSpecName: "model-cache") pod "886d84ae-3b4b-4047-a616-6731adf1935f" (UID: "886d84ae-3b4b-4047-a616-6731adf1935f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:34:26.755015 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:26.754973    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/886d84ae-3b4b-4047-a616-6731adf1935f-home" (OuterVolumeSpecName: "home") pod "886d84ae-3b4b-4047-a616-6731adf1935f" (UID: "886d84ae-3b4b-4047-a616-6731adf1935f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:34:26.755015 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:26.754966    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/886d84ae-3b4b-4047-a616-6731adf1935f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "886d84ae-3b4b-4047-a616-6731adf1935f" (UID: "886d84ae-3b4b-4047-a616-6731adf1935f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:34:26.755415 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:26.755388    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/886d84ae-3b4b-4047-a616-6731adf1935f-model-cache\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:34:26.756905 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:26.756882    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/886d84ae-3b4b-4047-a616-6731adf1935f-kube-api-access-tp6s8" (OuterVolumeSpecName: "kube-api-access-tp6s8") pod "886d84ae-3b4b-4047-a616-6731adf1935f" (UID: "886d84ae-3b4b-4047-a616-6731adf1935f"). InnerVolumeSpecName "kube-api-access-tp6s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:34:26.757130 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:26.757105    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/886d84ae-3b4b-4047-a616-6731adf1935f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "886d84ae-3b4b-4047-a616-6731adf1935f" (UID: "886d84ae-3b4b-4047-a616-6731adf1935f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:34:26.757130 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:26.757117    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/886d84ae-3b4b-4047-a616-6731adf1935f-dshm" (OuterVolumeSpecName: "dshm") pod "886d84ae-3b4b-4047-a616-6731adf1935f" (UID: "886d84ae-3b4b-4047-a616-6731adf1935f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:34:26.856560 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:26.856503    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/886d84ae-3b4b-4047-a616-6731adf1935f-tmp-dir\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:34:26.856560 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:26.856526    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp6s8\" (UniqueName: \"kubernetes.io/projected/886d84ae-3b4b-4047-a616-6731adf1935f-kube-api-access-tp6s8\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:34:26.856560 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:26.856536    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/886d84ae-3b4b-4047-a616-6731adf1935f-dshm\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:34:26.856560 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:26.856545    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/886d84ae-3b4b-4047-a616-6731adf1935f-tls-certs\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:34:26.856560 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:26.856552    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/886d84ae-3b4b-4047-a616-6731adf1935f-home\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:34:27.346777 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:27.346749    2565 generic.go:356] "Generic (PLEG): container finished" podID="886d84ae-3b4b-4047-a616-6731adf1935f" containerID="b37b073787f509a0f796ac600a9b761b7ba0b3915ae892e6c4fffadd8e3478b0" exitCode=0
Sep 17 19:34:27.347162 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:27.346795    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"
Sep 17 19:34:27.347162 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:27.346826    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn" event={"ID":"886d84ae-3b4b-4047-a616-6731adf1935f","Type":"ContainerDied","Data":"b37b073787f509a0f796ac600a9b761b7ba0b3915ae892e6c4fffadd8e3478b0"}
Sep 17 19:34:27.347162 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:27.346883    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn" event={"ID":"886d84ae-3b4b-4047-a616-6731adf1935f","Type":"ContainerDied","Data":"5598c343028407d07a26782cca9bada0099439770b81fb94ef506cea8e09e865"}
Sep 17 19:34:27.347162 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:27.346915    2565 scope.go:117] "RemoveContainer" containerID="b37b073787f509a0f796ac600a9b761b7ba0b3915ae892e6c4fffadd8e3478b0"
Sep 17 19:34:27.360090 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:27.360070    2565 scope.go:117] "RemoveContainer" containerID="b37b073787f509a0f796ac600a9b761b7ba0b3915ae892e6c4fffadd8e3478b0"
Sep 17 19:34:27.360370 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:34:27.360348    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b37b073787f509a0f796ac600a9b761b7ba0b3915ae892e6c4fffadd8e3478b0\": container with ID starting with b37b073787f509a0f796ac600a9b761b7ba0b3915ae892e6c4fffadd8e3478b0 not found: ID does not exist" containerID="b37b073787f509a0f796ac600a9b761b7ba0b3915ae892e6c4fffadd8e3478b0"
Sep 17 19:34:27.360437 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:27.360379    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b37b073787f509a0f796ac600a9b761b7ba0b3915ae892e6c4fffadd8e3478b0"} err="failed to get container status \"b37b073787f509a0f796ac600a9b761b7ba0b3915ae892e6c4fffadd8e3478b0\": rpc error: code = NotFound desc = could not find container \"b37b073787f509a0f796ac600a9b761b7ba0b3915ae892e6c4fffadd8e3478b0\": container with ID starting with b37b073787f509a0f796ac600a9b761b7ba0b3915ae892e6c4fffadd8e3478b0 not found: ID does not exist"
Sep 17 19:34:27.370851 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:27.370829    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"]
Sep 17 19:34:27.377836 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:27.377811    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-4qksn"]
Sep 17 19:34:28.607524 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:34:28.607490    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="886d84ae-3b4b-4047-a616-6731adf1935f" path="/var/lib/kubelet/pods/886d84ae-3b4b-4047-a616-6731adf1935f/volumes"
Sep 17 19:35:15.022626 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:15.022584    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vbzp8/must-gather-tt6l9"]
Sep 17 19:35:15.023045 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:15.022929    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="886d84ae-3b4b-4047-a616-6731adf1935f" containerName="main"
Sep 17 19:35:15.023045 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:15.022940    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="886d84ae-3b4b-4047-a616-6731adf1935f" containerName="main"
Sep 17 19:35:15.023045 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:15.022994    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="886d84ae-3b4b-4047-a616-6731adf1935f" containerName="main"
Sep 17 19:35:15.027250 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:15.027226    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vbzp8/must-gather-tt6l9"
Sep 17 19:35:15.029510 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:15.029490    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vbzp8\"/\"default-dockercfg-5zshk\""
Sep 17 19:35:15.030653 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:15.030631    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vbzp8\"/\"openshift-service-ca.crt\""
Sep 17 19:35:15.030821 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:15.030801    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vbzp8\"/\"kube-root-ca.crt\""
Sep 17 19:35:15.041278 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:15.041253    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vbzp8/must-gather-tt6l9"]
Sep 17 19:35:15.101950 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:15.101925    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpxpc\" (UniqueName: \"kubernetes.io/projected/e6568bfb-02ab-4129-a955-9879b3fd3510-kube-api-access-hpxpc\") pod \"must-gather-tt6l9\" (UID: \"e6568bfb-02ab-4129-a955-9879b3fd3510\") " pod="openshift-must-gather-vbzp8/must-gather-tt6l9"
Sep 17 19:35:15.102048 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:15.101970    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e6568bfb-02ab-4129-a955-9879b3fd3510-must-gather-output\") pod \"must-gather-tt6l9\" (UID: \"e6568bfb-02ab-4129-a955-9879b3fd3510\") " pod="openshift-must-gather-vbzp8/must-gather-tt6l9"
Sep 17 19:35:15.203028 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:15.203003    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e6568bfb-02ab-4129-a955-9879b3fd3510-must-gather-output\") pod \"must-gather-tt6l9\" (UID: \"e6568bfb-02ab-4129-a955-9879b3fd3510\") " pod="openshift-must-gather-vbzp8/must-gather-tt6l9"
Sep 17 19:35:15.203165 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:15.203060    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpxpc\" (UniqueName: \"kubernetes.io/projected/e6568bfb-02ab-4129-a955-9879b3fd3510-kube-api-access-hpxpc\") pod \"must-gather-tt6l9\" (UID: \"e6568bfb-02ab-4129-a955-9879b3fd3510\") " pod="openshift-must-gather-vbzp8/must-gather-tt6l9"
Sep 17 19:35:15.203424 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:15.203398    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e6568bfb-02ab-4129-a955-9879b3fd3510-must-gather-output\") pod \"must-gather-tt6l9\" (UID: \"e6568bfb-02ab-4129-a955-9879b3fd3510\") " pod="openshift-must-gather-vbzp8/must-gather-tt6l9"
Sep 17 19:35:15.211951 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:15.211925    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpxpc\" (UniqueName: \"kubernetes.io/projected/e6568bfb-02ab-4129-a955-9879b3fd3510-kube-api-access-hpxpc\") pod \"must-gather-tt6l9\" (UID: \"e6568bfb-02ab-4129-a955-9879b3fd3510\") " pod="openshift-must-gather-vbzp8/must-gather-tt6l9"
Sep 17 19:35:15.336341 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:15.336276    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vbzp8/must-gather-tt6l9"
Sep 17 19:35:15.613262 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:15.613193    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vbzp8/must-gather-tt6l9"]
Sep 17 19:35:15.615274 ip-10-0-139-75 kubenswrapper[2565]: W0917 19:35:15.615245    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6568bfb_02ab_4129_a955_9879b3fd3510.slice/crio-10789057016ddf2c69c179a8ec16d8a7853e503258adbc928cba313bb416ca51 WatchSource:0}: Error finding container 10789057016ddf2c69c179a8ec16d8a7853e503258adbc928cba313bb416ca51: Status 404 returned error can't find the container with id 10789057016ddf2c69c179a8ec16d8a7853e503258adbc928cba313bb416ca51
Sep 17 19:35:15.616997 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:15.616977    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:35:16.623331 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:16.623270    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vbzp8/must-gather-tt6l9" event={"ID":"e6568bfb-02ab-4129-a955-9879b3fd3510","Type":"ContainerStarted","Data":"10789057016ddf2c69c179a8ec16d8a7853e503258adbc928cba313bb416ca51"}
Sep 17 19:35:22.661584 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:22.661538    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vbzp8/must-gather-tt6l9" event={"ID":"e6568bfb-02ab-4129-a955-9879b3fd3510","Type":"ContainerStarted","Data":"3898df4ee60e4e98bc5c0e7aba8b85ec13928d913238de544417e5ba15b6ce82"}
Sep 17 19:35:22.661584 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:22.661590    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vbzp8/must-gather-tt6l9" event={"ID":"e6568bfb-02ab-4129-a955-9879b3fd3510","Type":"ContainerStarted","Data":"321fbb8ab57785de98ee5c26050f04e91bec0a9ea52bd14e741fe89516251596"}
Sep 17 19:35:22.679747 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:22.679694    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vbzp8/must-gather-tt6l9" podStartSLOduration=1.24965777 podStartE2EDuration="7.679677512s" totalImagesPullingTime="6.430019742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:35:22.678151723 +0000 UTC m=+3980.610353243" watchObservedRunningTime="2026-09-17 19:35:22.679677512 +0000 UTC m=+3980.611879082"
Sep 17 19:35:38.173795 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:38.173767    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b4db969fb-dbb5b_a9afa96b-b391-4db4-9312-20c884e90da3/router/0.log"
Sep 17 19:35:39.196931 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:39.196896    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b4db969fb-dbb5b_a9afa96b-b391-4db4-9312-20c884e90da3/router/0.log"
Sep 17 19:35:40.027814 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:40.027778    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-m6nk9_19ce5dd0-8789-4072-b3b1-b934e727585f/manager/0.log"
Sep 17 19:35:40.219098 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:40.219068    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-njxjg_f483ed7b-343f-4d14-9ed4-0b9c138621b9/limitador/0.log"
Sep 17 19:35:40.235954 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:40.235924    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-zhm82_f74c2e58-f318-4563-8c9a-d6d695ad70cf/manager/0.log"
Sep 17 19:35:44.795839 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:44.795807    2565 generic.go:356] "Generic (PLEG): container finished" podID="e6568bfb-02ab-4129-a955-9879b3fd3510" containerID="321fbb8ab57785de98ee5c26050f04e91bec0a9ea52bd14e741fe89516251596" exitCode=0
Sep 17 19:35:44.796394 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:44.795867    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vbzp8/must-gather-tt6l9" event={"ID":"e6568bfb-02ab-4129-a955-9879b3fd3510","Type":"ContainerDied","Data":"321fbb8ab57785de98ee5c26050f04e91bec0a9ea52bd14e741fe89516251596"}
Sep 17 19:35:44.796394 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:44.796235    2565 scope.go:117] "RemoveContainer" containerID="321fbb8ab57785de98ee5c26050f04e91bec0a9ea52bd14e741fe89516251596"
Sep 17 19:35:45.326287 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:45.326255    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-vbzp8_must-gather-tt6l9_e6568bfb-02ab-4129-a955-9879b3fd3510/gather/0.log"
Sep 17 19:35:45.868992 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:45.868956    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4lb2x/must-gather-6wxc8"]
Sep 17 19:35:45.873472 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:45.873449    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4lb2x/must-gather-6wxc8"
Sep 17 19:35:45.875695 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:45.875665    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4lb2x\"/\"openshift-service-ca.crt\""
Sep 17 19:35:45.875806 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:45.875734    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4lb2x\"/\"default-dockercfg-zzdfx\""
Sep 17 19:35:45.876493 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:45.876477    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4lb2x\"/\"kube-root-ca.crt\""
Sep 17 19:35:45.885656 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:45.885629    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4lb2x/must-gather-6wxc8"]
Sep 17 19:35:45.956938 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:45.956916    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bclls\" (UniqueName: \"kubernetes.io/projected/85d77414-5ad7-4f91-a7c6-6839cea5596f-kube-api-access-bclls\") pod \"must-gather-6wxc8\" (UID: \"85d77414-5ad7-4f91-a7c6-6839cea5596f\") " pod="openshift-must-gather-4lb2x/must-gather-6wxc8"
Sep 17 19:35:45.957060 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:45.956975    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/85d77414-5ad7-4f91-a7c6-6839cea5596f-must-gather-output\") pod \"must-gather-6wxc8\" (UID: \"85d77414-5ad7-4f91-a7c6-6839cea5596f\") " pod="openshift-must-gather-4lb2x/must-gather-6wxc8"
Sep 17 19:35:46.057637 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:46.057609    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bclls\" (UniqueName: \"kubernetes.io/projected/85d77414-5ad7-4f91-a7c6-6839cea5596f-kube-api-access-bclls\") pod \"must-gather-6wxc8\" (UID: \"85d77414-5ad7-4f91-a7c6-6839cea5596f\") " pod="openshift-must-gather-4lb2x/must-gather-6wxc8"
Sep 17 19:35:46.057778 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:46.057682    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/85d77414-5ad7-4f91-a7c6-6839cea5596f-must-gather-output\") pod \"must-gather-6wxc8\" (UID: \"85d77414-5ad7-4f91-a7c6-6839cea5596f\") " pod="openshift-must-gather-4lb2x/must-gather-6wxc8"
Sep 17 19:35:46.058029 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:46.058009    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/85d77414-5ad7-4f91-a7c6-6839cea5596f-must-gather-output\") pod \"must-gather-6wxc8\" (UID: \"85d77414-5ad7-4f91-a7c6-6839cea5596f\") " pod="openshift-must-gather-4lb2x/must-gather-6wxc8"
Sep 17 19:35:46.070563 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:46.070539    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bclls\" (UniqueName: \"kubernetes.io/projected/85d77414-5ad7-4f91-a7c6-6839cea5596f-kube-api-access-bclls\") pod \"must-gather-6wxc8\" (UID: \"85d77414-5ad7-4f91-a7c6-6839cea5596f\") " pod="openshift-must-gather-4lb2x/must-gather-6wxc8"
Sep 17 19:35:46.190953 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:46.190930    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4lb2x/must-gather-6wxc8"
Sep 17 19:35:46.462864 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:46.462834    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4lb2x/must-gather-6wxc8"]
Sep 17 19:35:46.464593 ip-10-0-139-75 kubenswrapper[2565]: W0917 19:35:46.464564    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85d77414_5ad7_4f91_a7c6_6839cea5596f.slice/crio-2a9f22d27a246bdca78b72fb7128771e983cabac415153c00758736b400b9bbd WatchSource:0}: Error finding container 2a9f22d27a246bdca78b72fb7128771e983cabac415153c00758736b400b9bbd: Status 404 returned error can't find the container with id 2a9f22d27a246bdca78b72fb7128771e983cabac415153c00758736b400b9bbd
Sep 17 19:35:46.807437 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:46.807366    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4lb2x/must-gather-6wxc8" event={"ID":"85d77414-5ad7-4f91-a7c6-6839cea5596f","Type":"ContainerStarted","Data":"2a9f22d27a246bdca78b72fb7128771e983cabac415153c00758736b400b9bbd"}
Sep 17 19:35:47.818238 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:47.818179    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4lb2x/must-gather-6wxc8" event={"ID":"85d77414-5ad7-4f91-a7c6-6839cea5596f","Type":"ContainerStarted","Data":"9f6392c264bf8177793082c2511515fcadb63dbe10cae0b2033567c54e77eff3"}
Sep 17 19:35:47.818672 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:47.818253    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4lb2x/must-gather-6wxc8" event={"ID":"85d77414-5ad7-4f91-a7c6-6839cea5596f","Type":"ContainerStarted","Data":"c88853a2337d06a6c639b4da6abc2adacdc0d33f25073af04464241f0dd4c4b3"}
Sep 17 19:35:47.838095 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:47.838030    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4lb2x/must-gather-6wxc8" podStartSLOduration=1.991953411 podStartE2EDuration="2.838006962s" totalImagesPullingTime="846.053551ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:35:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:35:47.83581358 +0000 UTC m=+4005.768015098" watchObservedRunningTime="2026-09-17 19:35:47.838006962 +0000 UTC m=+4005.770208483"
Sep 17 19:35:48.923358 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:48.923321    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4zwdx_43264d48-807c-43de-8b8f-52db0225d1c9/global-pull-secret-syncer/0.log"
Sep 17 19:35:49.164374 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:49.164334    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zttbn_2da5dc7d-7056-497b-a8e3-01deda7b2134/konnectivity-agent/0.log"
Sep 17 19:35:49.255113 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:49.255028    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-75.ec2.internal_7ef83900f88d178fe684eab642563132/haproxy/0.log"
Sep 17 19:35:50.715026 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:50.714986    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-vbzp8/must-gather-tt6l9"]
Sep 17 19:35:50.715645 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:50.715234    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-vbzp8/must-gather-tt6l9" podUID="e6568bfb-02ab-4129-a955-9879b3fd3510" containerName="copy" containerID="cri-o://3898df4ee60e4e98bc5c0e7aba8b85ec13928d913238de544417e5ba15b6ce82" gracePeriod=2
Sep 17 19:35:50.717385 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:35:50.717335    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-tt6l9\" is forbidden: User \"system:node:ip-10-0-139-75.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-vbzp8\": no relationship found between node 'ip-10-0-139-75.ec2.internal' and this object" podUID="e6568bfb-02ab-4129-a955-9879b3fd3510" pod="openshift-must-gather-vbzp8/must-gather-tt6l9"
Sep 17 19:35:50.744603 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:50.723413    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-vbzp8/must-gather-tt6l9"]
Sep 17 19:35:50.844189 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:50.844160    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-vbzp8_must-gather-tt6l9_e6568bfb-02ab-4129-a955-9879b3fd3510/copy/0.log"
Sep 17 19:35:50.844619 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:50.844590    2565 generic.go:356] "Generic (PLEG): container finished" podID="e6568bfb-02ab-4129-a955-9879b3fd3510" containerID="3898df4ee60e4e98bc5c0e7aba8b85ec13928d913238de544417e5ba15b6ce82" exitCode=143
Sep 17 19:35:51.073813 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:51.073516    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-vbzp8_must-gather-tt6l9_e6568bfb-02ab-4129-a955-9879b3fd3510/copy/0.log"
Sep 17 19:35:51.079450 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:51.079364    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vbzp8/must-gather-tt6l9"
Sep 17 19:35:51.081587 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:35:51.081543    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-tt6l9\" is forbidden: User \"system:node:ip-10-0-139-75.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-vbzp8\": no relationship found between node 'ip-10-0-139-75.ec2.internal' and this object" podUID="e6568bfb-02ab-4129-a955-9879b3fd3510" pod="openshift-must-gather-vbzp8/must-gather-tt6l9"
Sep 17 19:35:51.109224 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:51.108794    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpxpc\" (UniqueName: \"kubernetes.io/projected/e6568bfb-02ab-4129-a955-9879b3fd3510-kube-api-access-hpxpc\") pod \"e6568bfb-02ab-4129-a955-9879b3fd3510\" (UID: \"e6568bfb-02ab-4129-a955-9879b3fd3510\") "
Sep 17 19:35:51.109224 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:51.108851    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e6568bfb-02ab-4129-a955-9879b3fd3510-must-gather-output\") pod \"e6568bfb-02ab-4129-a955-9879b3fd3510\" (UID: \"e6568bfb-02ab-4129-a955-9879b3fd3510\") "
Sep 17 19:35:51.118736 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:51.117769    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6568bfb-02ab-4129-a955-9879b3fd3510-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "e6568bfb-02ab-4129-a955-9879b3fd3510" (UID: "e6568bfb-02ab-4129-a955-9879b3fd3510"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:35:51.128570 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:51.128525    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6568bfb-02ab-4129-a955-9879b3fd3510-kube-api-access-hpxpc" (OuterVolumeSpecName: "kube-api-access-hpxpc") pod "e6568bfb-02ab-4129-a955-9879b3fd3510" (UID: "e6568bfb-02ab-4129-a955-9879b3fd3510"). InnerVolumeSpecName "kube-api-access-hpxpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:35:51.210117 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:51.210070    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpxpc\" (UniqueName: \"kubernetes.io/projected/e6568bfb-02ab-4129-a955-9879b3fd3510-kube-api-access-hpxpc\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:35:51.210117 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:51.210117    2565 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e6568bfb-02ab-4129-a955-9879b3fd3510-must-gather-output\") on node \"ip-10-0-139-75.ec2.internal\" DevicePath \"\""
Sep 17 19:35:51.854659 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:51.854575    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-vbzp8_must-gather-tt6l9_e6568bfb-02ab-4129-a955-9879b3fd3510/copy/0.log"
Sep 17 19:35:51.859272 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:51.855835    2565 scope.go:117] "RemoveContainer" containerID="3898df4ee60e4e98bc5c0e7aba8b85ec13928d913238de544417e5ba15b6ce82"
Sep 17 19:35:51.859272 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:51.856020    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vbzp8/must-gather-tt6l9"
Sep 17 19:35:51.874610 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:35:51.874410    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-tt6l9\" is forbidden: User \"system:node:ip-10-0-139-75.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-vbzp8\": no relationship found between node 'ip-10-0-139-75.ec2.internal' and this object" podUID="e6568bfb-02ab-4129-a955-9879b3fd3510" pod="openshift-must-gather-vbzp8/must-gather-tt6l9"
Sep 17 19:35:51.881403 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:35:51.881362    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-tt6l9\" is forbidden: User \"system:node:ip-10-0-139-75.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-vbzp8\": no relationship found between node 'ip-10-0-139-75.ec2.internal' and this object" podUID="e6568bfb-02ab-4129-a955-9879b3fd3510" pod="openshift-must-gather-vbzp8/must-gather-tt6l9"
Sep 17 19:35:51.891276 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:51.889790    2565 scope.go:117] "RemoveContainer" containerID="321fbb8ab57785de98ee5c26050f04e91bec0a9ea52bd14e741fe89516251596"
Sep 17 19:35:52.614276 ip-10-0-139-75 kubenswrapper[2565]: E0917 19:35:52.614222    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-tt6l9\" is forbidden: User \"system:node:ip-10-0-139-75.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-vbzp8\": no relationship found between node 'ip-10-0-139-75.ec2.internal' and this object" podUID="e6568bfb-02ab-4129-a955-9879b3fd3510" pod="openshift-must-gather-vbzp8/must-gather-tt6l9"
Sep 17 19:35:52.633028 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:52.632989    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6568bfb-02ab-4129-a955-9879b3fd3510" path="/var/lib/kubelet/pods/e6568bfb-02ab-4129-a955-9879b3fd3510/volumes"
Sep 17 19:35:53.716272 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:53.716238    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-m6nk9_19ce5dd0-8789-4072-b3b1-b934e727585f/manager/0.log"
Sep 17 19:35:54.031583 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:54.031502    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-njxjg_f483ed7b-343f-4d14-9ed4-0b9c138621b9/limitador/0.log"
Sep 17 19:35:54.072638 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:54.072590    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-zhm82_f74c2e58-f318-4563-8c9a-d6d695ad70cf/manager/0.log"
Sep 17 19:35:55.749000 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:55.748964    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-fc4f47d85-v9gh9_ae28078b-ffbf-401b-b4ff-73b634a168c5/metrics-server/0.log"
Sep 17 19:35:55.989647 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:55.989606    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qhfss_1aea236a-8063-4c52-bda5-8c22fe1a166c/node-exporter/0.log"
Sep 17 19:35:56.022600 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:56.022516    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qhfss_1aea236a-8063-4c52-bda5-8c22fe1a166c/kube-rbac-proxy/0.log"
Sep 17 19:35:56.046600 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:56.046403    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qhfss_1aea236a-8063-4c52-bda5-8c22fe1a166c/init-textfile/0.log"
Sep 17 19:35:56.566384 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:56.566347    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dcb4d49c-6fkxm_8905b946-3f63-4c73-b657-cc2c937b09e3/telemeter-client/0.log"
Sep 17 19:35:56.594544 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:56.594515    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dcb4d49c-6fkxm_8905b946-3f63-4c73-b657-cc2c937b09e3/reload/0.log"
Sep 17 19:35:56.624884 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:56.624795    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dcb4d49c-6fkxm_8905b946-3f63-4c73-b657-cc2c937b09e3/kube-rbac-proxy/0.log"
Sep 17 19:35:58.001273 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:58.001235    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-66kd4_6e2745b5-c8e7-4fcf-8414-94b1e315a38b/networking-console-plugin/0.log"
Sep 17 19:35:59.001149 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:59.001104    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-hlwgt_172edcc2-26ce-48f5-b8a6-20c54c0537ea/download-server/0.log"
Sep 17 19:35:59.366699 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:59.366614    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4lb2x/perf-node-gather-daemonset-wrfsc"]
Sep 17 19:35:59.367191 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:59.367170    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6568bfb-02ab-4129-a955-9879b3fd3510" containerName="copy"
Sep 17 19:35:59.367273 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:59.367214    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6568bfb-02ab-4129-a955-9879b3fd3510" containerName="copy"
Sep 17 19:35:59.367273 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:59.367245    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6568bfb-02ab-4129-a955-9879b3fd3510" containerName="gather"
Sep 17 19:35:59.367273 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:59.367255    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6568bfb-02ab-4129-a955-9879b3fd3510" containerName="gather"
Sep 17 19:35:59.367405 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:59.367369    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6568bfb-02ab-4129-a955-9879b3fd3510" containerName="gather"
Sep 17 19:35:59.367405 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:59.367382    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6568bfb-02ab-4129-a955-9879b3fd3510" containerName="copy"
Sep 17 19:35:59.372494 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:59.372463    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-wrfsc"
Sep 17 19:35:59.386002 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:59.385972    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4lb2x/perf-node-gather-daemonset-wrfsc"]
Sep 17 19:35:59.402256 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:59.402231    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fabd2cfe-9807-409e-8eec-290f0bec964a-proc\") pod \"perf-node-gather-daemonset-wrfsc\" (UID: \"fabd2cfe-9807-409e-8eec-290f0bec964a\") " pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-wrfsc"
Sep 17 19:35:59.402401 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:59.402316    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fabd2cfe-9807-409e-8eec-290f0bec964a-sys\") pod \"perf-node-gather-daemonset-wrfsc\" (UID: \"fabd2cfe-9807-409e-8eec-290f0bec964a\") " pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-wrfsc"
Sep 17 19:35:59.402401 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:59.402366    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fabd2cfe-9807-409e-8eec-290f0bec964a-lib-modules\") pod \"perf-node-gather-daemonset-wrfsc\" (UID: \"fabd2cfe-9807-409e-8eec-290f0bec964a\") " pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-wrfsc"
Sep 17 19:35:59.402401 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:59.402396    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh76q\" (UniqueName: \"kubernetes.io/projected/fabd2cfe-9807-409e-8eec-290f0bec964a-kube-api-access-wh76q\") pod \"perf-node-gather-daemonset-wrfsc\" (UID: \"fabd2cfe-9807-409e-8eec-290f0bec964a\") " pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-wrfsc"
Sep 17 19:35:59.402579 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:59.402436    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fabd2cfe-9807-409e-8eec-290f0bec964a-podres\") pod \"perf-node-gather-daemonset-wrfsc\" (UID: \"fabd2cfe-9807-409e-8eec-290f0bec964a\") " pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-wrfsc"
Sep 17 19:35:59.503131 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:59.503086    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fabd2cfe-9807-409e-8eec-290f0bec964a-sys\") pod \"perf-node-gather-daemonset-wrfsc\" (UID: \"fabd2cfe-9807-409e-8eec-290f0bec964a\") " pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-wrfsc"
Sep 17 19:35:59.503340 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:59.503154    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fabd2cfe-9807-409e-8eec-290f0bec964a-lib-modules\") pod \"perf-node-gather-daemonset-wrfsc\" (UID: \"fabd2cfe-9807-409e-8eec-290f0bec964a\") " pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-wrfsc"
Sep 17 19:35:59.503340 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:59.503188    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh76q\" (UniqueName: \"kubernetes.io/projected/fabd2cfe-9807-409e-8eec-290f0bec964a-kube-api-access-wh76q\") pod \"perf-node-gather-daemonset-wrfsc\" (UID: \"fabd2cfe-9807-409e-8eec-290f0bec964a\") " pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-wrfsc"
Sep 17 19:35:59.503340 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:59.503247    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fabd2cfe-9807-409e-8eec-290f0bec964a-podres\") pod \"perf-node-gather-daemonset-wrfsc\" (UID: \"fabd2cfe-9807-409e-8eec-290f0bec964a\") " pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-wrfsc"
Sep 17 19:35:59.503340 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:59.503250    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fabd2cfe-9807-409e-8eec-290f0bec964a-sys\") pod \"perf-node-gather-daemonset-wrfsc\" (UID: \"fabd2cfe-9807-409e-8eec-290f0bec964a\") " pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-wrfsc"
Sep 17 19:35:59.503585 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:59.503342    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fabd2cfe-9807-409e-8eec-290f0bec964a-proc\") pod \"perf-node-gather-daemonset-wrfsc\" (UID: \"fabd2cfe-9807-409e-8eec-290f0bec964a\") " pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-wrfsc"
Sep 17 19:35:59.503585 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:59.503356    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fabd2cfe-9807-409e-8eec-290f0bec964a-lib-modules\") pod \"perf-node-gather-daemonset-wrfsc\" (UID: \"fabd2cfe-9807-409e-8eec-290f0bec964a\") " pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-wrfsc"
Sep 17 19:35:59.503585 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:59.503377    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fabd2cfe-9807-409e-8eec-290f0bec964a-podres\") pod \"perf-node-gather-daemonset-wrfsc\" (UID: \"fabd2cfe-9807-409e-8eec-290f0bec964a\") " pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-wrfsc"
Sep 17 19:35:59.503585 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:59.503448    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fabd2cfe-9807-409e-8eec-290f0bec964a-proc\") pod \"perf-node-gather-daemonset-wrfsc\" (UID: \"fabd2cfe-9807-409e-8eec-290f0bec964a\") " pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-wrfsc"
Sep 17 19:35:59.514818 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:59.514785    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh76q\" (UniqueName: \"kubernetes.io/projected/fabd2cfe-9807-409e-8eec-290f0bec964a-kube-api-access-wh76q\") pod \"perf-node-gather-daemonset-wrfsc\" (UID: \"fabd2cfe-9807-409e-8eec-290f0bec964a\") " pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-wrfsc"
Sep 17 19:35:59.684509 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:59.684470    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-wrfsc"
Sep 17 19:35:59.978816 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:35:59.978780    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4lb2x/perf-node-gather-daemonset-wrfsc"]
Sep 17 19:35:59.979834 ip-10-0-139-75 kubenswrapper[2565]: W0917 19:35:59.979794    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfabd2cfe_9807_409e_8eec_290f0bec964a.slice/crio-4997e958f4f80e4b99bb028b22f66d30727ac6358260264904f097d5534bc64a WatchSource:0}: Error finding container 4997e958f4f80e4b99bb028b22f66d30727ac6358260264904f097d5534bc64a: Status 404 returned error can't find the container with id 4997e958f4f80e4b99bb028b22f66d30727ac6358260264904f097d5534bc64a
Sep 17 19:36:00.780148 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:00.780117    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x9pxw_ba1caac4-a2a1-4917-9598-4b2003401a84/dns/0.log"
Sep 17 19:36:00.801343 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:00.801307    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x9pxw_ba1caac4-a2a1-4917-9598-4b2003401a84/kube-rbac-proxy/0.log"
Sep 17 19:36:00.875348 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:00.875320    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mkdtq_0058f4db-dbaa-4b8b-942b-06ecbd1301ef/dns-node-resolver/0.log"
Sep 17 19:36:00.931242 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:00.931156    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-wrfsc" event={"ID":"fabd2cfe-9807-409e-8eec-290f0bec964a","Type":"ContainerStarted","Data":"6d12b2544670cfdb998059d89e77cc7b05981156a56e4c2a0a7c1377c33ee4ea"}
Sep 17 19:36:00.931242 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:00.931243    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-wrfsc" event={"ID":"fabd2cfe-9807-409e-8eec-290f0bec964a","Type":"ContainerStarted","Data":"4997e958f4f80e4b99bb028b22f66d30727ac6358260264904f097d5534bc64a"}
Sep 17 19:36:00.932085 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:00.932055    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-wrfsc"
Sep 17 19:36:00.950639 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:00.950591    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-wrfsc" podStartSLOduration=1.9505781660000001 podStartE2EDuration="1.950578166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:36:00.947720112 +0000 UTC m=+4018.879921621" watchObservedRunningTime="2026-09-17 19:36:00.950578166 +0000 UTC m=+4018.882779685"
Sep 17 19:36:01.441915 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:01.441875    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-glcw2_822d9b1e-1a38-49ac-900a-6ad946e6078f/node-ca/0.log"
Sep 17 19:36:02.561446 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:02.561412    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b4db969fb-dbb5b_a9afa96b-b391-4db4-9312-20c884e90da3/router/0.log"
Sep 17 19:36:03.132974 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:03.132934    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ltdzb_3158326a-19ea-455b-8737-8f01b5d51373/serve-healthcheck-canary/0.log"
Sep 17 19:36:03.575708 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:03.575676    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r2mgv_46404667-f42b-4c16-a119-2e4f5dfc8d41/kube-rbac-proxy/0.log"
Sep 17 19:36:03.596557 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:03.596527    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r2mgv_46404667-f42b-4c16-a119-2e4f5dfc8d41/exporter/0.log"
Sep 17 19:36:03.618696 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:03.618662    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r2mgv_46404667-f42b-4c16-a119-2e4f5dfc8d41/extractor/0.log"
Sep 17 19:36:04.475491 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:04.475453    2565 scope.go:117] "RemoveContainer" containerID="012387cdef41d8f355bb310626c317f2ca50a27c23966b8725f56501f4826032"
Sep 17 19:36:07.451783 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:07.451736    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-69d59d7cdd-5hq8f_c12116ee-ac6f-4448-ab40-3643615402da/manager/0.log"
Sep 17 19:36:07.960152 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:07.960109    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4lb2x/perf-node-gather-daemonset-wrfsc"
Sep 17 19:36:07.964599 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:07.964568    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-5c49f86564-cdnq9_b520ade1-0e10-41c3-813e-c079b247e5eb/manager/0.log"
Sep 17 19:36:07.990480 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:07.990449    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-f7d8g_b48875aa-e67f-4c77-8e0d-aa2051d34499/server/0.log"
Sep 17 19:36:08.989403 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:08.989352    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b4db969fb-dbb5b_a9afa96b-b391-4db4-9312-20c884e90da3/router/0.log"
Sep 17 19:36:09.093456 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:09.093415    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-zgn4g_50a1c0f1-dd09-416b-be08-875f33721b5e/manager/0.log"
Sep 17 19:36:09.146810 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:09.146768    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-bp7px_38583224-6640-45bf-9a02-1dc45f709cb1/s3-init/0.log"
Sep 17 19:36:09.173098 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:09.173069    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-bp7px_38583224-6640-45bf-9a02-1dc45f709cb1/download-hf-model/0.log"
Sep 17 19:36:09.220733 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:09.220699    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-47bvz_84f28508-030d-404b-a15f-a693c52537d4/seaweedfs/0.log"
Sep 17 19:36:10.383273 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:10.383167    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b4db969fb-dbb5b_a9afa96b-b391-4db4-9312-20c884e90da3/router/0.log"
Sep 17 19:36:14.107494 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:14.107455    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-cqjfb_3b433190-01d1-492c-af7a-2579e21e03ff/migrator/0.log"
Sep 17 19:36:14.131471 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:14.131444    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-cqjfb_3b433190-01d1-492c-af7a-2579e21e03ff/graceful-termination/0.log"
Sep 17 19:36:14.489558 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:14.489511    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-lnf27_5e907c64-1699-4fa4-902b-b90ba5d7b8e6/kube-storage-version-migrator-operator/1.log"
Sep 17 19:36:14.492118 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:14.492089    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-lnf27_5e907c64-1699-4fa4-902b-b90ba5d7b8e6/kube-storage-version-migrator-operator/0.log"
Sep 17 19:36:15.689057 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:15.689022    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-stwlh_42102f16-c1e3-4791-9aea-61a4b415e2ab/kube-multus-additional-cni-plugins/0.log"
Sep 17 19:36:15.725348 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:15.725322    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-stwlh_42102f16-c1e3-4791-9aea-61a4b415e2ab/egress-router-binary-copy/0.log"
Sep 17 19:36:15.761334 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:15.761300    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-stwlh_42102f16-c1e3-4791-9aea-61a4b415e2ab/cni-plugins/0.log"
Sep 17 19:36:15.794306 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:15.794271    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-stwlh_42102f16-c1e3-4791-9aea-61a4b415e2ab/bond-cni-plugin/0.log"
Sep 17 19:36:15.832220 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:15.832188    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-stwlh_42102f16-c1e3-4791-9aea-61a4b415e2ab/routeoverride-cni/0.log"
Sep 17 19:36:15.873267 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:15.873251    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-stwlh_42102f16-c1e3-4791-9aea-61a4b415e2ab/whereabouts-cni-bincopy/0.log"
Sep 17 19:36:15.907176 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:15.907153    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-stwlh_42102f16-c1e3-4791-9aea-61a4b415e2ab/whereabouts-cni/0.log"
Sep 17 19:36:16.188515 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:16.188475    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-mfxhw_0eeeae49-526f-41f9-8438-3eebbee372f6/kube-multus/0.log"
Sep 17 19:36:16.388614 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:16.388577    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-w67kg_db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4/network-metrics-daemon/0.log"
Sep 17 19:36:16.415837 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:16.415809    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-w67kg_db00c9db-dbf0-4d5b-a9e9-7708ec49dcf4/kube-rbac-proxy/0.log"
Sep 17 19:36:17.811296 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:17.811264    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-grgmp_9d09fbd1-01dd-4ec2-8dec-f05ee9a28709/ovn-controller/0.log"
Sep 17 19:36:17.829265 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:17.829224    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-grgmp_9d09fbd1-01dd-4ec2-8dec-f05ee9a28709/ovn-acl-logging/0.log"
Sep 17 19:36:17.855845 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:17.855778    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-grgmp_9d09fbd1-01dd-4ec2-8dec-f05ee9a28709/kube-rbac-proxy-node/0.log"
Sep 17 19:36:17.877564 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:17.877544    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-grgmp_9d09fbd1-01dd-4ec2-8dec-f05ee9a28709/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 19:36:17.896068 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:17.896047    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-grgmp_9d09fbd1-01dd-4ec2-8dec-f05ee9a28709/northd/0.log"
Sep 17 19:36:17.916966 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:17.916938    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-grgmp_9d09fbd1-01dd-4ec2-8dec-f05ee9a28709/nbdb/0.log"
Sep 17 19:36:17.938771 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:17.938750    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-grgmp_9d09fbd1-01dd-4ec2-8dec-f05ee9a28709/sbdb/0.log"
Sep 17 19:36:18.039217 ip-10-0-139-75 kubenswrapper[2565]: I0917 19:36:18.039161    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-grgmp_9d09fbd1-01dd-4ec2-8dec-f05ee9a28709/ovnkube-controller/0.log"