Aug 05 17:10:18.992556 ip-10-0-135-151 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Aug 05 17:10:18.992565 ip-10-0-135-151 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Aug 05 17:10:18.992572 ip-10-0-135-151 systemd[1]: kubelet.service: Failed with result 'resources'.
Aug 05 17:10:18.992792 ip-10-0-135-151 systemd[1]: Failed to start Kubernetes Kubelet.
Aug 05 17:10:29.241733 ip-10-0-135-151 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Aug 05 17:10:29.241753 ip-10-0-135-151 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 654416afde44477fbd79db1f04442c97 --
Aug 05 17:12:22.846226 ip-10-0-135-151 systemd[1]: Starting Kubernetes Kubelet...
Aug 05 17:12:23.218755 ip-10-0-135-151 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 05 17:12:23.218755 ip-10-0-135-151 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Aug 05 17:12:23.218755 ip-10-0-135-151 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 05 17:12:23.218755 ip-10-0-135-151 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Aug 05 17:12:23.218755 ip-10-0-135-151 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 05 17:12:23.221173 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.221073    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Aug 05 17:12:23.226198 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226180    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 05 17:12:23.226198 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226196    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 05 17:12:23.226198 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226201    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 05 17:12:23.226387 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226209    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 05 17:12:23.226387 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226215    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 05 17:12:23.226387 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226219    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 05 17:12:23.226387 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226223    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 05 17:12:23.226387 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226228    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 05 17:12:23.226387 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226232    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 05 17:12:23.226387 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226237    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 05 17:12:23.226387 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226241    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 05 17:12:23.226387 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226245    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 05 17:12:23.226387 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226249    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 05 17:12:23.226387 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226253    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 05 17:12:23.226387 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226256    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 05 17:12:23.226387 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226260    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 05 17:12:23.226387 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226265    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 05 17:12:23.226387 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226269    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 05 17:12:23.226387 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226273    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 05 17:12:23.226387 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226277    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 05 17:12:23.226387 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226281    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 05 17:12:23.226387 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226291    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 05 17:12:23.227187 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226295    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 05 17:12:23.227187 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226299    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 05 17:12:23.227187 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226303    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 05 17:12:23.227187 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226307    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 05 17:12:23.227187 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226311    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 05 17:12:23.227187 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226317    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 05 17:12:23.227187 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226324    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 05 17:12:23.227187 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226329    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 05 17:12:23.227187 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226334    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 05 17:12:23.227187 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226338    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 05 17:12:23.227187 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226343    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 05 17:12:23.227187 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226347    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 05 17:12:23.227187 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226351    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 05 17:12:23.227187 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226356    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 05 17:12:23.227187 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226361    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 05 17:12:23.227187 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226366    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 05 17:12:23.227187 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226372    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 05 17:12:23.227187 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226376    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 05 17:12:23.227187 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226381    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 05 17:12:23.228051 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226385    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 05 17:12:23.228051 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226389    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 05 17:12:23.228051 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226393    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 05 17:12:23.228051 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226398    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 05 17:12:23.228051 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226402    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 05 17:12:23.228051 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226406    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 05 17:12:23.228051 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226411    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 05 17:12:23.228051 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226415    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 05 17:12:23.228051 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226420    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 05 17:12:23.228051 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226424    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 05 17:12:23.228051 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226429    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 05 17:12:23.228051 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226433    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 05 17:12:23.228051 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226437    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 05 17:12:23.228051 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226443    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 05 17:12:23.228051 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226448    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 05 17:12:23.228051 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226452    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 05 17:12:23.228051 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226457    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 05 17:12:23.228051 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226461    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 05 17:12:23.228051 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226465    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 05 17:12:23.228051 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226470    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 05 17:12:23.228742 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226476    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 05 17:12:23.228742 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226480    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 05 17:12:23.228742 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226485    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 05 17:12:23.228742 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226489    2561 feature_gate.go:328] unrecognized feature gate: Example2
Aug 05 17:12:23.228742 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226493    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 05 17:12:23.228742 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226498    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 05 17:12:23.228742 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226506    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 05 17:12:23.228742 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226512    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 05 17:12:23.228742 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226517    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 05 17:12:23.228742 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226522    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 05 17:12:23.228742 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226526    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 05 17:12:23.228742 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226531    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 05 17:12:23.228742 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226535    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 05 17:12:23.228742 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226540    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 05 17:12:23.228742 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226545    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 05 17:12:23.228742 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226549    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 05 17:12:23.228742 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226553    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 05 17:12:23.228742 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226558    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 05 17:12:23.228742 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226562    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 05 17:12:23.229234 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226566    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 05 17:12:23.229234 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226570    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 05 17:12:23.229234 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226575    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 05 17:12:23.229234 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226580    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 05 17:12:23.229234 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226585    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 05 17:12:23.229234 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226589    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 05 17:12:23.229234 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226594    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 05 17:12:23.229234 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226606    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 05 17:12:23.229234 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226610    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 05 17:12:23.229234 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226615    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 05 17:12:23.229234 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226619    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 05 17:12:23.229234 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226623    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 05 17:12:23.229234 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226628    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 05 17:12:23.229234 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226633    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 05 17:12:23.229234 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226638    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 05 17:12:23.229234 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226642    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 05 17:12:23.229234 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226646    2561 feature_gate.go:328] unrecognized feature gate: Example
Aug 05 17:12:23.229234 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226650    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 05 17:12:23.229234 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.226654    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 05 17:12:23.229234 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228082    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 05 17:12:23.229721 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228094    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 05 17:12:23.229721 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228099    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 05 17:12:23.229721 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228103    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 05 17:12:23.229721 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228108    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 05 17:12:23.229721 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228112    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 05 17:12:23.229721 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228116    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 05 17:12:23.229721 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228121    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 05 17:12:23.229721 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228126    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 05 17:12:23.229721 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228130    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 05 17:12:23.229721 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228135    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 05 17:12:23.229721 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228139    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 05 17:12:23.229721 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228143    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 05 17:12:23.229721 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228147    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 05 17:12:23.229721 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228151    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 05 17:12:23.229721 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228156    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 05 17:12:23.229721 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228159    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 05 17:12:23.229721 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228163    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 05 17:12:23.229721 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228168    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 05 17:12:23.229721 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228172    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 05 17:12:23.229721 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228177    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 05 17:12:23.230237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228189    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 05 17:12:23.230237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228194    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 05 17:12:23.230237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228198    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 05 17:12:23.230237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228202    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 05 17:12:23.230237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228206    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 05 17:12:23.230237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228211    2561 feature_gate.go:328] unrecognized feature gate: Example2
Aug 05 17:12:23.230237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228216    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 05 17:12:23.230237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228220    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 05 17:12:23.230237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228224    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 05 17:12:23.230237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228228    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 05 17:12:23.230237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228233    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 05 17:12:23.230237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228237    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 05 17:12:23.230237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228241    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 05 17:12:23.230237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228246    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 05 17:12:23.230237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228250    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 05 17:12:23.230237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228255    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 05 17:12:23.230237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228259    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 05 17:12:23.230237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228263    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 05 17:12:23.230237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228267    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 05 17:12:23.230237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228271    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 05 17:12:23.230718 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228275    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 05 17:12:23.230718 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228284    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 05 17:12:23.230718 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228289    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 05 17:12:23.230718 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228294    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 05 17:12:23.230718 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228300    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 05 17:12:23.230718 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228304    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 05 17:12:23.230718 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228309    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 05 17:12:23.230718 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228313    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 05 17:12:23.230718 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228318    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 05 17:12:23.230718 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228322    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 05 17:12:23.230718 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228327    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 05 17:12:23.230718 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228331    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 05 17:12:23.230718 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228335    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 05 17:12:23.230718 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228344    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 05 17:12:23.230718 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228350    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 05 17:12:23.230718 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228354    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 05 17:12:23.230718 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228361    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 05 17:12:23.230718 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228366    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 05 17:12:23.230718 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228371    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 05 17:12:23.231249 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228375    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 05 17:12:23.231249 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228379    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 05 17:12:23.231249 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228383    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 05 17:12:23.231249 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228387    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 05 17:12:23.231249 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228391    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 05 17:12:23.231249 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228396    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 05 17:12:23.231249 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228400    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 05 17:12:23.231249 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228404    2561 feature_gate.go:328] unrecognized feature gate: Example
Aug 05 17:12:23.231249 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228409    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 05 17:12:23.231249 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228413    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 05 17:12:23.231249 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228417    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 05 17:12:23.231249 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228421    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 05 17:12:23.231249 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228426    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 05 17:12:23.231249 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228430    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 05 17:12:23.231249 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228434    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 05 17:12:23.231249 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228439    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 05 17:12:23.231249 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228442    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 05 17:12:23.231249 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228448    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 05 17:12:23.231249 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228452    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 05 17:12:23.231249 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228457    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 05 17:12:23.231734 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228461    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 05 17:12:23.231734 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228465    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 05 17:12:23.231734 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228469    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 05 17:12:23.231734 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228473    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 05 17:12:23.231734 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228478    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 05 17:12:23.231734 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228482    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 05 17:12:23.231734 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228486    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 05 17:12:23.231734 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228491    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 05 17:12:23.231734 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228495    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 05 17:12:23.231734 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228499    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 05 17:12:23.231734 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228504    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 05 17:12:23.231734 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228508    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 05 17:12:23.231734 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228514    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 05 17:12:23.231734 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228519    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 05 17:12:23.231734 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228551    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 05 17:12:23.231734 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228566    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 05 17:12:23.231734 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228571    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 05 17:12:23.231734 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228575    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 05 17:12:23.231734 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.228578    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 05 17:12:23.232237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229054    2561 feature_gate.go:328] unrecognized feature gate: Example2
Aug 05 17:12:23.232237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229062    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 05 17:12:23.232237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229067    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 05 17:12:23.232237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229070    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 05 17:12:23.232237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229074    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 05 17:12:23.232237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229077    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 05 17:12:23.232237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229079    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 05 17:12:23.232237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229082    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 05 17:12:23.232237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229085    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 05 17:12:23.232237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229088    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 05 17:12:23.232237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229090    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 05 17:12:23.232237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229093    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 05 17:12:23.232237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229096    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 05 17:12:23.232237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229098    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 05 17:12:23.232237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229102    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 05 17:12:23.232237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229104    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 05 17:12:23.232237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229107    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 05 17:12:23.232237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229109    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 05 17:12:23.232237 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229112    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 05 17:12:23.232703 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229114    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 05 17:12:23.232703 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229117    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 05 17:12:23.232703 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229119    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 05 17:12:23.232703 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229122    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 05 17:12:23.232703 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229125    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 05 17:12:23.232703 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229128    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 05 17:12:23.232703 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229130    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 05 17:12:23.232703 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229133    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 05 17:12:23.232703 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229136    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 05 17:12:23.232703 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229139    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 05 17:12:23.232703 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229142    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 05 17:12:23.232703 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229146    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 05 17:12:23.232703 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229150    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 05 17:12:23.232703 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229153    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 05 17:12:23.232703 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229155    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 05 17:12:23.232703 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229158    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 05 17:12:23.232703 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229162    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 05 17:12:23.232703 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229164    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 05 17:12:23.232703 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229167    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 05 17:12:23.233172 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229170    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 05 17:12:23.233172 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229174    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 05 17:12:23.233172 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229176    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 05 17:12:23.233172 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229179    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 05 17:12:23.233172 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229182    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 05 17:12:23.233172 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229184    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 05 17:12:23.233172 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229187    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 05 17:12:23.233172 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229190    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 05 17:12:23.233172 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229192    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 05 17:12:23.233172 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229194    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 05 17:12:23.233172 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229197    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 05 17:12:23.233172 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229199    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 05 17:12:23.233172 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229202    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 05 17:12:23.233172 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229204    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 05 17:12:23.233172 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229207    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 05 17:12:23.233172 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229209    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 05 17:12:23.233172 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229212    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 05 17:12:23.233172 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229214    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 05 17:12:23.233172 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229217    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 05 17:12:23.233172 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229219    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 05 17:12:23.233666 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229224    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 05 17:12:23.233666 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229228    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 05 17:12:23.233666 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229235    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 05 17:12:23.233666 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229237    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 05 17:12:23.233666 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229240    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 05 17:12:23.233666 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229243    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 05 17:12:23.233666 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229245    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 05 17:12:23.233666 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229248    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 05 17:12:23.233666 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229251    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 05 17:12:23.233666 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229253    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 05 17:12:23.233666 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229256    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 05 17:12:23.233666 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229258    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 05 17:12:23.233666 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229261    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 05 17:12:23.233666 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229263    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 05 17:12:23.233666 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229266    2561 feature_gate.go:328] unrecognized feature gate: Example
Aug 05 17:12:23.233666 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229268    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 05 17:12:23.233666 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229271    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 05 17:12:23.233666 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229273    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 05 17:12:23.233666 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229275    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 05 17:12:23.233666 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229278    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 05 17:12:23.234196 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229280    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 05 17:12:23.234196 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229283    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 05 17:12:23.234196 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229285    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 05 17:12:23.234196 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229288    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 05 17:12:23.234196 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229290    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 05 17:12:23.234196 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229293    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 05 17:12:23.234196 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229295    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 05 17:12:23.234196 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229298    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 05 17:12:23.234196 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229300    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 05 17:12:23.234196 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229303    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 05 17:12:23.234196 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229306    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 05 17:12:23.234196 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229308    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 05 17:12:23.234196 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229311    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 05 17:12:23.234196 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229313    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 05 17:12:23.234196 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229317    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 05 17:12:23.234196 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229321    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 05 17:12:23.234196 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229323    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 05 17:12:23.234196 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229326    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 05 17:12:23.234196 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229328    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 05 17:12:23.234196 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229331    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 05 17:12:23.234708 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.229333    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 05 17:12:23.234708 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229404    2561 flags.go:64] FLAG: --address="0.0.0.0"
Aug 05 17:12:23.234708 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229415    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Aug 05 17:12:23.234708 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229424    2561 flags.go:64] FLAG: --anonymous-auth="true"
Aug 05 17:12:23.234708 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229429    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Aug 05 17:12:23.234708 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229434    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Aug 05 17:12:23.234708 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229437    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Aug 05 17:12:23.234708 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229441    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Aug 05 17:12:23.234708 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229445    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Aug 05 17:12:23.234708 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229449    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Aug 05 17:12:23.234708 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229452    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Aug 05 17:12:23.234708 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229455    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Aug 05 17:12:23.234708 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229459    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Aug 05 17:12:23.234708 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229462    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Aug 05 17:12:23.234708 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229465    2561 flags.go:64] FLAG: --cgroup-root=""
Aug 05 17:12:23.234708 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229468    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Aug 05 17:12:23.234708 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229471    2561 flags.go:64] FLAG: --client-ca-file=""
Aug 05 17:12:23.234708 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229474    2561 flags.go:64] FLAG: --cloud-provider="external"
Aug 05 17:12:23.234708 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229477    2561 flags.go:64] FLAG: --cluster-dns="[]"
Aug 05 17:12:23.234708 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229481    2561 flags.go:64] FLAG: --cluster-domain=""
Aug 05 17:12:23.234708 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229484    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Aug 05 17:12:23.234708 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229487    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Aug 05 17:12:23.234708 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229490    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Aug 05 17:12:23.234708 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229496    2561 flags.go:64] FLAG: --container-log-max-files="5"
Aug 05 17:12:23.235314 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229500    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Aug 05 17:12:23.235314 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229503    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Aug 05 17:12:23.235314 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229508    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Aug 05 17:12:23.235314 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229511    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Aug 05 17:12:23.235314 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229516    2561 flags.go:64] FLAG: --contention-profiling="false"
Aug 05 17:12:23.235314 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229519    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Aug 05 17:12:23.235314 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229522    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Aug 05 17:12:23.235314 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229525    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Aug 05 17:12:23.235314 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229528    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Aug 05 17:12:23.235314 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229532    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Aug 05 17:12:23.235314 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229535    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Aug 05 17:12:23.235314 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229538    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Aug 05 17:12:23.235314 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229541    2561 flags.go:64] FLAG: --enable-load-reader="false"
Aug 05 17:12:23.235314 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229544    2561 flags.go:64] FLAG: --enable-server="true"
Aug 05 17:12:23.235314 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229547    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Aug 05 17:12:23.235314 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229552    2561 flags.go:64] FLAG: --event-burst="100"
Aug 05 17:12:23.235314 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229555    2561 flags.go:64] FLAG: --event-qps="50"
Aug 05 17:12:23.235314 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229558    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Aug 05 17:12:23.235314 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229562    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Aug 05 17:12:23.235314 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229564    2561 flags.go:64] FLAG: --eviction-hard=""
Aug 05 17:12:23.235314 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229568    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Aug 05 17:12:23.235314 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229571    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Aug 05 17:12:23.235314 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229574    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Aug 05 17:12:23.235314 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229578    2561 flags.go:64] FLAG: --eviction-soft=""
Aug 05 17:12:23.235314 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229581    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Aug 05 17:12:23.235939 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229584    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Aug 05 17:12:23.235939 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229587    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Aug 05 17:12:23.235939 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229590    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Aug 05 17:12:23.235939 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229593    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Aug 05 17:12:23.235939 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229595    2561 flags.go:64] FLAG: --fail-swap-on="true"
Aug 05 17:12:23.235939 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229598    2561 flags.go:64] FLAG: --feature-gates=""
Aug 05 17:12:23.235939 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229603    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Aug 05 17:12:23.235939 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229606    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Aug 05 17:12:23.235939 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229609    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Aug 05 17:12:23.235939 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229614    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Aug 05 17:12:23.235939 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229617    2561 flags.go:64] FLAG: --healthz-port="10248"
Aug 05 17:12:23.235939 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229620    2561 flags.go:64] FLAG: --help="false"
Aug 05 17:12:23.235939 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229624    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-135-151.ec2.internal"
Aug 05 17:12:23.235939 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229627    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Aug 05 17:12:23.235939 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229630    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Aug 05 17:12:23.235939 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229633    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Aug 05 17:12:23.235939 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229637    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Aug 05 17:12:23.235939 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229640    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Aug 05 17:12:23.235939 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229643    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Aug 05 17:12:23.235939 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229646    2561 flags.go:64] FLAG: --image-service-endpoint=""
Aug 05 17:12:23.235939 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229649    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Aug 05 17:12:23.235939 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229652    2561 flags.go:64] FLAG: --kube-api-burst="100"
Aug 05 17:12:23.235939 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229655    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Aug 05 17:12:23.236578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229658    2561 flags.go:64] FLAG: --kube-api-qps="50"
Aug 05 17:12:23.236578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229661    2561 flags.go:64] FLAG: --kube-reserved=""
Aug 05 17:12:23.236578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229664    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Aug 05 17:12:23.236578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229667    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Aug 05 17:12:23.236578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229670    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Aug 05 17:12:23.236578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229672    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Aug 05 17:12:23.236578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229675    2561 flags.go:64] FLAG: --lock-file=""
Aug 05 17:12:23.236578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229678    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Aug 05 17:12:23.236578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229681    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Aug 05 17:12:23.236578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229684    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Aug 05 17:12:23.236578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229690    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Aug 05 17:12:23.236578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229693    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Aug 05 17:12:23.236578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229695    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Aug 05 17:12:23.236578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229698    2561 flags.go:64] FLAG: --logging-format="text"
Aug 05 17:12:23.236578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229701    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Aug 05 17:12:23.236578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229705    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Aug 05 17:12:23.236578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229708    2561 flags.go:64] FLAG: --manifest-url=""
Aug 05 17:12:23.236578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229711    2561 flags.go:64] FLAG: --manifest-url-header=""
Aug 05 17:12:23.236578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229717    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Aug 05 17:12:23.236578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229720    2561 flags.go:64] FLAG: --max-open-files="1000000"
Aug 05 17:12:23.236578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229725    2561 flags.go:64] FLAG: --max-pods="110"
Aug 05 17:12:23.236578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229728    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Aug 05 17:12:23.236578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229732    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Aug 05 17:12:23.236578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229735    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Aug 05 17:12:23.236578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229738    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Aug 05 17:12:23.237236 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229741    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Aug 05 17:12:23.237236 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229744    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Aug 05 17:12:23.237236 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229747    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Aug 05 17:12:23.237236 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229754    2561 flags.go:64] FLAG: --node-status-max-images="50"
Aug 05 17:12:23.237236 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229757    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Aug 05 17:12:23.237236 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229760    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Aug 05 17:12:23.237236 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229763    2561 flags.go:64] FLAG: --pod-cidr=""
Aug 05 17:12:23.237236 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229766    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Aug 05 17:12:23.237236 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229772    2561 flags.go:64] FLAG: --pod-manifest-path=""
Aug 05 17:12:23.237236 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229775    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Aug 05 17:12:23.237236 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229778    2561 flags.go:64] FLAG: --pods-per-core="0"
Aug 05 17:12:23.237236 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229781    2561 flags.go:64] FLAG: --port="10250"
Aug 05 17:12:23.237236 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229784    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Aug 05 17:12:23.237236 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229787    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0eb6e14b541c16752"
Aug 05 17:12:23.237236 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229790    2561 flags.go:64] FLAG: --qos-reserved=""
Aug 05 17:12:23.237236 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229793    2561 flags.go:64] FLAG: --read-only-port="10255"
Aug 05 17:12:23.237236 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229796    2561 flags.go:64] FLAG: --register-node="true"
Aug 05 17:12:23.237236 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229799    2561 flags.go:64] FLAG: --register-with-taints=""
Aug 05 17:12:23.237236 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229802    2561 flags.go:64] FLAG: --registry-burst="10"
Aug 05 17:12:23.237236 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229805    2561 flags.go:64] FLAG: --registry-qps="5"
Aug 05 17:12:23.237236 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229808    2561 flags.go:64] FLAG: --reserved-cpus=""
Aug 05 17:12:23.237236 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229811    2561 flags.go:64] FLAG: --reserved-memory=""
Aug 05 17:12:23.237236 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229815    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Aug 05 17:12:23.237236 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229819    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Aug 05 17:12:23.237236 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229822    2561 flags.go:64] FLAG: --rotate-certificates="false"
Aug 05 17:12:23.238155 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229827    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Aug 05 17:12:23.238155 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229830    2561 flags.go:64] FLAG: --runonce="false"
Aug 05 17:12:23.238155 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229833    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Aug 05 17:12:23.238155 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229836    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Aug 05 17:12:23.238155 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229839    2561 flags.go:64] FLAG: --seccomp-default="false"
Aug 05 17:12:23.238155 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229842    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Aug 05 17:12:23.238155 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229845    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Aug 05 17:12:23.238155 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229848    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Aug 05 17:12:23.238155 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229851    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Aug 05 17:12:23.238155 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229854    2561 flags.go:64] FLAG: --storage-driver-password="root"
Aug 05 17:12:23.238155 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229857    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Aug 05 17:12:23.238155 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229861    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Aug 05 17:12:23.238155 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229863    2561 flags.go:64] FLAG: --storage-driver-user="root"
Aug 05 17:12:23.238155 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229881    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Aug 05 17:12:23.238155 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229885    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Aug 05 17:12:23.238155 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229888    2561 flags.go:64] FLAG: --system-cgroups=""
Aug 05 17:12:23.238155 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229891    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Aug 05 17:12:23.238155 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229896    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Aug 05 17:12:23.238155 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229899    2561 flags.go:64] FLAG: --tls-cert-file=""
Aug 05 17:12:23.238155 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229902    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Aug 05 17:12:23.238155 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229906    2561 flags.go:64] FLAG: --tls-min-version=""
Aug 05 17:12:23.238155 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229909    2561 flags.go:64] FLAG: --tls-private-key-file=""
Aug 05 17:12:23.238155 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229912    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Aug 05 17:12:23.238155 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229915    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Aug 05 17:12:23.238155 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229918    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Aug 05 17:12:23.238762 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229921    2561 flags.go:64] FLAG: --v="2"
Aug 05 17:12:23.238762 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229924    2561 flags.go:64] FLAG: --version="false"
Aug 05 17:12:23.238762 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229928    2561 flags.go:64] FLAG: --vmodule=""
Aug 05 17:12:23.238762 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229932    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Aug 05 17:12:23.238762 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.229935    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Aug 05 17:12:23.238762 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230065    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 05 17:12:23.238762 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230069    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 05 17:12:23.238762 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230073    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 05 17:12:23.238762 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230077    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 05 17:12:23.238762 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230079    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 05 17:12:23.238762 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230082    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 05 17:12:23.238762 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230085    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 05 17:12:23.238762 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230088    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 05 17:12:23.238762 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230091    2561 feature_gate.go:328] unrecognized feature gate: Example2
Aug 05 17:12:23.238762 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230094    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 05 17:12:23.238762 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230097    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 05 17:12:23.238762 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230099    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 05 17:12:23.238762 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230102    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 05 17:12:23.238762 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230105    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 05 17:12:23.238762 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230107    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 05 17:12:23.238762 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230110    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 05 17:12:23.239318 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230113    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 05 17:12:23.239318 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230115    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 05 17:12:23.239318 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230118    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 05 17:12:23.239318 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230121    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 05 17:12:23.239318 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230123    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 05 17:12:23.239318 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230126    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 05 17:12:23.239318 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230129    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 05 17:12:23.239318 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230133    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 05 17:12:23.239318 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230137    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 05 17:12:23.239318 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230140    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 05 17:12:23.239318 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230142    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 05 17:12:23.239318 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230145    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 05 17:12:23.239318 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230148    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 05 17:12:23.239318 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230151    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 05 17:12:23.239318 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230153    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 05 17:12:23.239318 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230156    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 05 17:12:23.239318 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230160    2561 feature_gate.go:328] unrecognized feature gate: Example
Aug 05 17:12:23.239318 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230162    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 05 17:12:23.239318 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230167    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 05 17:12:23.239318 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230170    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 05 17:12:23.239813 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230173    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 05 17:12:23.239813 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230175    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 05 17:12:23.239813 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230178    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 05 17:12:23.239813 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230180    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 05 17:12:23.239813 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230183    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 05 17:12:23.239813 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230186    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 05 17:12:23.239813 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230189    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 05 17:12:23.239813 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230192    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 05 17:12:23.239813 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230195    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 05 17:12:23.239813 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230198    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 05 17:12:23.239813 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230201    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 05 17:12:23.239813 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230203    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 05 17:12:23.239813 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230206    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 05 17:12:23.239813 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230208    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 05 17:12:23.239813 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230211    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 05 17:12:23.239813 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230214    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 05 17:12:23.239813 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230217    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 05 17:12:23.239813 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230219    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 05 17:12:23.239813 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230221    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 05 17:12:23.239813 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230224    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 05 17:12:23.240359 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230227    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 05 17:12:23.240359 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230229    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 05 17:12:23.240359 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230232    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 05 17:12:23.240359 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230235    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 05 17:12:23.240359 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230238    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 05 17:12:23.240359 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230241    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 05 17:12:23.240359 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230245    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 05 17:12:23.240359 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230249    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 05 17:12:23.240359 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230253    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 05 17:12:23.240359 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230256    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 05 17:12:23.240359 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230260    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 05 17:12:23.240359 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230264    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 05 17:12:23.240359 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230266    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 05 17:12:23.240359 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230269    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 05 17:12:23.240359 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230272    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 05 17:12:23.240359 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230275    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 05 17:12:23.240359 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230277    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 05 17:12:23.240359 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230280    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 05 17:12:23.240359 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230283    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 05 17:12:23.240810 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230286    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 05 17:12:23.240810 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230288    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 05 17:12:23.240810 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230291    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 05 17:12:23.240810 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230294    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 05 17:12:23.240810 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230297    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 05 17:12:23.240810 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230301    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 05 17:12:23.240810 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230304    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 05 17:12:23.240810 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230307    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 05 17:12:23.240810 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230310    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 05 17:12:23.240810 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230313    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 05 17:12:23.240810 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230316    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 05 17:12:23.240810 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230318    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 05 17:12:23.240810 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230321    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 05 17:12:23.240810 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230323    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 05 17:12:23.240810 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230326    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 05 17:12:23.240810 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230328    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 05 17:12:23.240810 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230331    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 05 17:12:23.240810 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230334    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 05 17:12:23.240810 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230337    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 05 17:12:23.241289 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230339    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 05 17:12:23.241289 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230342    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 05 17:12:23.241289 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230345    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 05 17:12:23.241289 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230348    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 05 17:12:23.241289 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.230352    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 05 17:12:23.241289 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.231031    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 05 17:12:23.241289 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.240092    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Aug 05 17:12:23.241289 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.240113    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Aug 05 17:12:23.241289 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.240134    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Aug 05 17:12:23.241289 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.240140    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Aug 05 17:12:23.241289 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240252    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 05 17:12:23.241289 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240257    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 05 17:12:23.241289 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240261    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 05 17:12:23.241289 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240264    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 05 17:12:23.241698 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240270    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 05 17:12:23.241698 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240272    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 05 17:12:23.241698 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240281    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 05 17:12:23.241698 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240285    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 05 17:12:23.241698 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240288    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 05 17:12:23.241698 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240291    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 05 17:12:23.241698 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240294    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 05 17:12:23.241698 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240297    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 05 17:12:23.241698 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240300    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 05 17:12:23.241698 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240304    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 05 17:12:23.241698 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240308    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 05 17:12:23.241698 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240311    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 05 17:12:23.241698 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240317    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 05 17:12:23.241698 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240320    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 05 17:12:23.241698 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240323    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 05 17:12:23.241698 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240326    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 05 17:12:23.241698 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240329    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 05 17:12:23.241698 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240331    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 05 17:12:23.241698 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240334    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 05 17:12:23.242189 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240337    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 05 17:12:23.242189 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240340    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 05 17:12:23.242189 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240344    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 05 17:12:23.242189 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240347    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 05 17:12:23.242189 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240350    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 05 17:12:23.242189 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240353    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 05 17:12:23.242189 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240359    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 05 17:12:23.242189 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240362    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 05 17:12:23.242189 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240365    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 05 17:12:23.242189 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240368    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 05 17:12:23.242189 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240371    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 05 17:12:23.242189 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240374    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 05 17:12:23.242189 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240376    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 05 17:12:23.242189 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240379    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 05 17:12:23.242189 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240382    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 05 17:12:23.242189 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240384    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 05 17:12:23.242189 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240393    2561 feature_gate.go:328] unrecognized feature gate: Example
Aug 05 17:12:23.242189 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240396    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 05 17:12:23.242189 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240401    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 05 17:12:23.242189 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240403    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 05 17:12:23.242664 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240406    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 05 17:12:23.242664 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240409    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 05 17:12:23.242664 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240412    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 05 17:12:23.242664 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240414    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 05 17:12:23.242664 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240417    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 05 17:12:23.242664 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240421    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 05 17:12:23.242664 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240425    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 05 17:12:23.242664 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240429    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 05 17:12:23.242664 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240432    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 05 17:12:23.242664 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240435    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 05 17:12:23.242664 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240438    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 05 17:12:23.242664 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240443    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 05 17:12:23.242664 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240446    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 05 17:12:23.242664 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240449    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 05 17:12:23.242664 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240451    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 05 17:12:23.242664 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240455    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 05 17:12:23.242664 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240458    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 05 17:12:23.242664 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240460    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 05 17:12:23.242664 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240463    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 05 17:12:23.242664 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240466    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 05 17:12:23.243254 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240469    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 05 17:12:23.243254 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240472    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 05 17:12:23.243254 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240475    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 05 17:12:23.243254 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240480    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 05 17:12:23.243254 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240483    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 05 17:12:23.243254 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240486    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 05 17:12:23.243254 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240488    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 05 17:12:23.243254 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240491    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 05 17:12:23.243254 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240494    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 05 17:12:23.243254 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240502    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 05 17:12:23.243254 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240505    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 05 17:12:23.243254 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240507    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 05 17:12:23.243254 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240510    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 05 17:12:23.243254 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240512    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 05 17:12:23.243254 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240516    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 05 17:12:23.243254 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240522    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 05 17:12:23.243254 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240525    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 05 17:12:23.243254 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240527    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 05 17:12:23.243254 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240530    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 05 17:12:23.243713 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240533    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 05 17:12:23.243713 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240535    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 05 17:12:23.243713 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240538    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 05 17:12:23.243713 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240540    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 05 17:12:23.243713 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240543    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 05 17:12:23.243713 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240546    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 05 17:12:23.243713 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240548    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 05 17:12:23.243713 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240551    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 05 17:12:23.243713 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240554    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 05 17:12:23.243713 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240559    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 05 17:12:23.243713 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240562    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 05 17:12:23.243713 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240565    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 05 17:12:23.243713 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240567    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 05 17:12:23.243713 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240570    2561 feature_gate.go:328] unrecognized feature gate: Example2
Aug 05 17:12:23.243713 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240573    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 05 17:12:23.243713 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240575    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 05 17:12:23.243713 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240578    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 05 17:12:23.244180 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.240583    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 05 17:12:23.244180 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240842    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 05 17:12:23.244180 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240847    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 05 17:12:23.244180 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240850    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 05 17:12:23.244180 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240853    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 05 17:12:23.244180 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240856    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 05 17:12:23.244180 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240864    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 05 17:12:23.244180 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240882    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 05 17:12:23.244180 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240886    2561 feature_gate.go:328] unrecognized feature gate: Example
Aug 05 17:12:23.244180 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240889    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 05 17:12:23.244180 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240891    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 05 17:12:23.244180 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240894    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 05 17:12:23.244180 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240900    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 05 17:12:23.244180 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240903    2561 feature_gate.go:328] unrecognized feature gate: Example2
Aug 05 17:12:23.244180 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240906    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 05 17:12:23.244546 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240908    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 05 17:12:23.244546 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240911    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 05 17:12:23.244546 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240914    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 05 17:12:23.244546 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240916    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 05 17:12:23.244546 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240919    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 05 17:12:23.244546 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240922    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 05 17:12:23.244546 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240925    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 05 17:12:23.244546 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240927    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 05 17:12:23.244546 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240930    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 05 17:12:23.244546 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240933    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 05 17:12:23.244546 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240938    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 05 17:12:23.244546 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240941    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 05 17:12:23.244546 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240944    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 05 17:12:23.244546 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240947    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 05 17:12:23.244546 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240950    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 05 17:12:23.244546 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240953    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 05 17:12:23.244546 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240955    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 05 17:12:23.244546 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240958    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 05 17:12:23.244546 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240961    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 05 17:12:23.244546 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240964    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 05 17:12:23.245040 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240966    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 05 17:12:23.245040 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240969    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 05 17:12:23.245040 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240972    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 05 17:12:23.245040 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240977    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 05 17:12:23.245040 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240986    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 05 17:12:23.245040 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240989    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 05 17:12:23.245040 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240991    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 05 17:12:23.245040 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240994    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 05 17:12:23.245040 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240997    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 05 17:12:23.245040 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.240999    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 05 17:12:23.245040 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241002    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 05 17:12:23.245040 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241005    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 05 17:12:23.245040 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241007    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 05 17:12:23.245040 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241010    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 05 17:12:23.245040 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241012    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 05 17:12:23.245040 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241017    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 05 17:12:23.245040 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241020    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 05 17:12:23.245040 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241023    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 05 17:12:23.245040 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241026    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 05 17:12:23.245493 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241029    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 05 17:12:23.245493 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241032    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 05 17:12:23.245493 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241035    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 05 17:12:23.245493 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241038    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 05 17:12:23.245493 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241040    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 05 17:12:23.245493 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241043    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 05 17:12:23.245493 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241046    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 05 17:12:23.245493 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241049    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 05 17:12:23.245493 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241051    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 05 17:12:23.245493 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241056    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 05 17:12:23.245493 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241059    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 05 17:12:23.245493 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241062    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 05 17:12:23.245493 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241064    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 05 17:12:23.245493 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241067    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 05 17:12:23.245493 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241070    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 05 17:12:23.245493 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241072    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 05 17:12:23.245493 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241076    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 05 17:12:23.245493 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241080    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 05 17:12:23.245493 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241090    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 05 17:12:23.245493 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241093    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 05 17:12:23.245999 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241097    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 05 17:12:23.245999 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241103    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 05 17:12:23.245999 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241106    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 05 17:12:23.245999 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241109    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 05 17:12:23.245999 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241112    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 05 17:12:23.245999 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241115    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 05 17:12:23.245999 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241124    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 05 17:12:23.245999 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241127    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 05 17:12:23.245999 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241130    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 05 17:12:23.245999 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241133    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 05 17:12:23.245999 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241138    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 05 17:12:23.245999 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241141    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 05 17:12:23.245999 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241144    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 05 17:12:23.245999 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241148    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 05 17:12:23.245999 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241154    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 05 17:12:23.245999 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241156    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 05 17:12:23.245999 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241159    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 05 17:12:23.245999 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241162    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 05 17:12:23.245999 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241165    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 05 17:12:23.245999 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241167    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 05 17:12:23.246480 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241170    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 05 17:12:23.246480 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241173    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 05 17:12:23.246480 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241176    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 05 17:12:23.246480 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241179    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 05 17:12:23.246480 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241182    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 05 17:12:23.246480 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:23.241186    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 05 17:12:23.246480 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.241194    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 05 17:12:23.246480 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.241369    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Aug 05 17:12:23.246480 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.245480    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Aug 05 17:12:23.247150 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.247134    2561 server.go:1019] "Starting client certificate rotation"
Aug 05 17:12:23.247257 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.247238    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 05 17:12:23.247287 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.247279    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 05 17:12:23.249535 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.249520    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 05 17:12:23.250740 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.250726    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 05 17:12:23.260846 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.260829    2561 log.go:25] "Validated CRI v1 runtime API"
Aug 05 17:12:23.265265 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.265250    2561 log.go:25] "Validated CRI v1 image API"
Aug 05 17:12:23.266803 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.266729    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Aug 05 17:12:23.269179 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.269160    2561 fs.go:135] Filesystem UUIDs: map[6685749f-782d-4bed-938f-636cd2c7c878:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 dff020ef-f368-42bb-beb6-d1a94597d926:/dev/nvme0n1p3]
Aug 05 17:12:23.269248 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.269180    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Aug 05 17:12:23.274215 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.274090    2561 manager.go:216] Machine: {Timestamp:2026-08-05 17:12:23.272832127 +0000 UTC m=+0.315550221 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3027105 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26bf0947a4af475cf5eee4d586be56 SystemUUID:ec26bf09-47a4-af47-5cf5-eee4d586be56 BootID:654416af-de44-477f-bd79-db1f04442c97 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c7:fb:0d:ce:bb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c7:fb:0d:ce:bb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0a:9a:16:d0:82:e8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Aug 05 17:12:23.274215 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.274206    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Aug 05 17:12:23.274349 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.274287    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Aug 05 17:12:23.275343 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.275316    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Aug 05 17:12:23.275544 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.275346    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-151.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Aug 05 17:12:23.276089 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.276076    2561 topology_manager.go:138] "Creating topology manager with none policy"
Aug 05 17:12:23.276143 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.276095    2561 container_manager_linux.go:309] "Creating device plugin manager"
Aug 05 17:12:23.276143 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.276107    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 05 17:12:23.276737 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.276724    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 05 17:12:23.277529 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.277518    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Aug 05 17:12:23.279802 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.279786    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 05 17:12:23.281136 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.281122    2561 state_mem.go:36] "Initialized new in-memory state store"
Aug 05 17:12:23.281279 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.281268    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Aug 05 17:12:23.281345 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.281309    2561 kubelet.go:486] "Attempting to sync node with API server"
Aug 05 17:12:23.281345 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.281323    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Aug 05 17:12:23.281345 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.281344    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Aug 05 17:12:23.281465 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.281358    2561 kubelet.go:398] "Adding apiserver pod source"
Aug 05 17:12:23.281465 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.281370    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Aug 05 17:12:23.281758 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.281743    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 05 17:12:23.281830 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.281767    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 05 17:12:23.284140 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.284121    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Aug 05 17:12:23.286075 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.286058    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Aug 05 17:12:23.286173 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.286091    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Aug 05 17:12:23.287223 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.287210    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Aug 05 17:12:23.287268 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.287228    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Aug 05 17:12:23.287268 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.287235    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Aug 05 17:12:23.287268 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.287240    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Aug 05 17:12:23.287268 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.287249    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Aug 05 17:12:23.287268 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.287255    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Aug 05 17:12:23.287268 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.287263    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Aug 05 17:12:23.287268 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.287269    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Aug 05 17:12:23.287457 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.287276    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Aug 05 17:12:23.287457 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.287282    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Aug 05 17:12:23.287457 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.287296    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Aug 05 17:12:23.287457 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.287302    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Aug 05 17:12:23.287923 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.287911    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Aug 05 17:12:23.287923 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.287921    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Aug 05 17:12:23.291019 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.291002    2561 server.go:1268] "Started kubelet"
Aug 05 17:12:23.291116 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.291056    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Aug 05 17:12:23.291171 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.291119    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Aug 05 17:12:23.291224 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.291191    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Aug 05 17:12:23.291800 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:23.291773    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Aug 05 17:12:23.291800 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.291793    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-151.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Aug 05 17:12:23.291964 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:23.291798    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-151.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Aug 05 17:12:23.291978 ip-10-0-135-151 systemd[1]: Started Kubernetes Kubelet.
Aug 05 17:12:23.292317 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.292249    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Aug 05 17:12:23.293267 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.293248    2561 server.go:310] "Adding debug handlers to kubelet server"
Aug 05 17:12:23.295718 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:23.295700    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Aug 05 17:12:23.296988 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:23.295295    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-151.ec2.internal.18c8f802d47cd543  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-151.ec2.internal,UID:ip-10-0-135-151.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-151.ec2.internal,},FirstTimestamp:2026-08-05 17:12:23.290975555 +0000 UTC m=+0.333693650,LastTimestamp:2026-08-05 17:12:23.290975555 +0000 UTC m=+0.333693650,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-151.ec2.internal,}"
Aug 05 17:12:23.297093 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.296997    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Aug 05 17:12:23.297093 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.297068    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Aug 05 17:12:23.297199 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:23.297094    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-151.ec2.internal\" not found"
Aug 05 17:12:23.297607 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.297588    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Aug 05 17:12:23.297607 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.297588    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Aug 05 17:12:23.297723 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.297620    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Aug 05 17:12:23.297781 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.297737    2561 reconstruct.go:97] "Volume reconstruction finished"
Aug 05 17:12:23.297781 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.297748    2561 reconciler.go:29] "Reconciler: start to sync state"
Aug 05 17:12:23.298019 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.298003    2561 factory.go:153] Registering CRI-O factory
Aug 05 17:12:23.298019 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.298021    2561 factory.go:223] Registration of the crio container factory successfully
Aug 05 17:12:23.298170 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.298069    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Aug 05 17:12:23.298170 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.298076    2561 factory.go:55] Registering systemd factory
Aug 05 17:12:23.298170 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.298082    2561 factory.go:223] Registration of the systemd container factory successfully
Aug 05 17:12:23.298170 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.298099    2561 factory.go:103] Registering Raw factory
Aug 05 17:12:23.298170 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.298111    2561 manager.go:1196] Started watching for new ooms in manager
Aug 05 17:12:23.298701 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.298686    2561 manager.go:349] Starting recovery of all containers
Aug 05 17:12:23.299568 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:23.299531    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-151.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Aug 05 17:12:23.299675 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:23.299656    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Aug 05 17:12:23.305176 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.305150    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fg6gj"
Aug 05 17:12:23.305654 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.305616    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Aug 05 17:12:23.309339 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.309313    2561 manager.go:354] Recovery completed
Aug 05 17:12:23.313715 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.313696    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fg6gj"
Aug 05 17:12:23.316182 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.316168    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 05 17:12:23.318038 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.318021    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-151.ec2.internal" event="NodeHasSufficientMemory"
Aug 05 17:12:23.318100 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.318048    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-151.ec2.internal" event="NodeHasNoDiskPressure"
Aug 05 17:12:23.318100 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.318057    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-151.ec2.internal" event="NodeHasSufficientPID"
Aug 05 17:12:23.318589 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.318576    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Aug 05 17:12:23.318589 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.318588    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Aug 05 17:12:23.318682 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.318604    2561 state_mem.go:36] "Initialized new in-memory state store"
Aug 05 17:12:23.318682 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.318589    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 05 17:12:23.319376 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.319360    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-151.ec2.internal" event="NodeHasSufficientMemory"
Aug 05 17:12:23.319447 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.319392    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-151.ec2.internal" event="NodeHasNoDiskPressure"
Aug 05 17:12:23.319447 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.319407    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-151.ec2.internal" event="NodeHasSufficientPID"
Aug 05 17:12:23.319699 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.319686    2561 policy_none.go:49] "None policy: Start"
Aug 05 17:12:23.319737 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.319703    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Aug 05 17:12:23.319737 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.319713    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Aug 05 17:12:23.320360 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:23.320290    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-151.ec2.internal.18c8f802d619b92a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-151.ec2.internal,UID:ip-10-0-135-151.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-151.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-151.ec2.internal,},FirstTimestamp:2026-08-05 17:12:23.31803473 +0000 UTC m=+0.360752827,LastTimestamp:2026-08-05 17:12:23.31803473 +0000 UTC m=+0.360752827,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-151.ec2.internal,}"
Aug 05 17:12:23.320676 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:23.320664    2561 policy_none.go:47] "Start"
Aug 05 17:12:24.655171 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.654986    2561 manager.go:337] "Starting Device Plugin manager"
Aug 05 17:12:24.655274 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:24.655199    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Aug 05 17:12:24.655274 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.655214    2561 server.go:87] "Starting device plugin registration server"
Aug 05 17:12:24.655488 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.655474    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Aug 05 17:12:24.655570 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.655492    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Aug 05 17:12:24.655633 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.655606    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Aug 05 17:12:24.655721 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.655707    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Aug 05 17:12:24.655797 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.655724    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Aug 05 17:12:24.656198 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:24.656181    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Aug 05 17:12:24.656288 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:24.656233    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-151.ec2.internal\" not found"
Aug 05 17:12:24.723275 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.723243    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Aug 05 17:12:24.724336 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.723292    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Aug 05 17:12:24.724336 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.723318    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Aug 05 17:12:24.724336 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:24.723360    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Aug 05 17:12:24.725602 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.725582    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 05 17:12:24.755736 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.755707    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 05 17:12:24.756695 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.756676    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-151.ec2.internal" event="NodeHasSufficientMemory"
Aug 05 17:12:24.756779 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.756710    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-151.ec2.internal" event="NodeHasNoDiskPressure"
Aug 05 17:12:24.756779 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.756724    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-151.ec2.internal" event="NodeHasSufficientPID"
Aug 05 17:12:24.756779 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.756755    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-151.ec2.internal"
Aug 05 17:12:24.767754 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.767666    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-151.ec2.internal"
Aug 05 17:12:24.767754 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:24.767692    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-151.ec2.internal\": node \"ip-10-0-135-151.ec2.internal\" not found"
Aug 05 17:12:24.779630 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:24.779607    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-151.ec2.internal\" not found"
Aug 05 17:12:24.823998 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.823943    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-151.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-151.ec2.internal"]
Aug 05 17:12:24.824136 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.824028    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 05 17:12:24.824933 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.824916    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-151.ec2.internal" event="NodeHasSufficientMemory"
Aug 05 17:12:24.825060 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.824946    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-151.ec2.internal" event="NodeHasNoDiskPressure"
Aug 05 17:12:24.825060 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.824970    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-151.ec2.internal" event="NodeHasSufficientPID"
Aug 05 17:12:24.826139 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.826127    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 05 17:12:24.826275 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.826263    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-151.ec2.internal"
Aug 05 17:12:24.826320 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.826287    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 05 17:12:24.826793 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.826768    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-151.ec2.internal" event="NodeHasSufficientMemory"
Aug 05 17:12:24.826890 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.826771    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-151.ec2.internal" event="NodeHasSufficientMemory"
Aug 05 17:12:24.826890 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.826823    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-151.ec2.internal" event="NodeHasNoDiskPressure"
Aug 05 17:12:24.826890 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.826833    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-151.ec2.internal" event="NodeHasSufficientPID"
Aug 05 17:12:24.826890 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.826801    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-151.ec2.internal" event="NodeHasNoDiskPressure"
Aug 05 17:12:24.826890 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.826871    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-151.ec2.internal" event="NodeHasSufficientPID"
Aug 05 17:12:24.828012 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.827996    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-151.ec2.internal"
Aug 05 17:12:24.828111 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.828021    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 05 17:12:24.829423 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.829396    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-151.ec2.internal" event="NodeHasSufficientMemory"
Aug 05 17:12:24.829501 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.829432    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-151.ec2.internal" event="NodeHasNoDiskPressure"
Aug 05 17:12:24.829501 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.829442    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-151.ec2.internal" event="NodeHasSufficientPID"
Aug 05 17:12:24.858739 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:24.858713    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-151.ec2.internal\" not found" node="ip-10-0-135-151.ec2.internal"
Aug 05 17:12:24.863124 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:24.863106    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-151.ec2.internal\" not found" node="ip-10-0-135-151.ec2.internal"
Aug 05 17:12:24.880608 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:24.880582    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-151.ec2.internal\" not found"
Aug 05 17:12:24.897545 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.897517    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc8290ea05f84958b6e4f1826fe71dbb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-151.ec2.internal\" (UID: \"fc8290ea05f84958b6e4f1826fe71dbb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-151.ec2.internal"
Aug 05 17:12:24.897669 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.897553    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/37213c9548f92e588dc6c041ff6110a8-config\") pod \"kube-apiserver-proxy-ip-10-0-135-151.ec2.internal\" (UID: \"37213c9548f92e588dc6c041ff6110a8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-151.ec2.internal"
Aug 05 17:12:24.897669 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.897569    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fc8290ea05f84958b6e4f1826fe71dbb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-151.ec2.internal\" (UID: \"fc8290ea05f84958b6e4f1826fe71dbb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-151.ec2.internal"
Aug 05 17:12:24.980881 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:24.980831    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-151.ec2.internal\" not found"
Aug 05 17:12:24.998249 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.998211    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/37213c9548f92e588dc6c041ff6110a8-config\") pod \"kube-apiserver-proxy-ip-10-0-135-151.ec2.internal\" (UID: \"37213c9548f92e588dc6c041ff6110a8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-151.ec2.internal"
Aug 05 17:12:24.998343 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.998172    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/37213c9548f92e588dc6c041ff6110a8-config\") pod \"kube-apiserver-proxy-ip-10-0-135-151.ec2.internal\" (UID: \"37213c9548f92e588dc6c041ff6110a8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-151.ec2.internal"
Aug 05 17:12:24.998343 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.998302    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fc8290ea05f84958b6e4f1826fe71dbb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-151.ec2.internal\" (UID: \"fc8290ea05f84958b6e4f1826fe71dbb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-151.ec2.internal"
Aug 05 17:12:24.998343 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.998319    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc8290ea05f84958b6e4f1826fe71dbb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-151.ec2.internal\" (UID: \"fc8290ea05f84958b6e4f1826fe71dbb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-151.ec2.internal"
Aug 05 17:12:24.998435 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.998348    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc8290ea05f84958b6e4f1826fe71dbb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-151.ec2.internal\" (UID: \"fc8290ea05f84958b6e4f1826fe71dbb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-151.ec2.internal"
Aug 05 17:12:24.998435 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:24.998378    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fc8290ea05f84958b6e4f1826fe71dbb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-151.ec2.internal\" (UID: \"fc8290ea05f84958b6e4f1826fe71dbb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-151.ec2.internal"
Aug 05 17:12:25.081589 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:25.081520    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-151.ec2.internal\" not found"
Aug 05 17:12:25.162171 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.162139    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-151.ec2.internal"
Aug 05 17:12:25.165159 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.165131    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-151.ec2.internal"
Aug 05 17:12:25.182168 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:25.182145    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-151.ec2.internal\" not found"
Aug 05 17:12:25.282707 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:25.282671    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-151.ec2.internal\" not found"
Aug 05 17:12:25.383266 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:25.383179    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-151.ec2.internal\" not found"
Aug 05 17:12:25.407232 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.407206    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 05 17:12:25.496333 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.496297    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-151.ec2.internal"
Aug 05 17:12:25.507153 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.507121    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 05 17:12:25.508899 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.508880    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-151.ec2.internal"
Aug 05 17:12:25.519863 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.519841    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 05 17:12:25.545042 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.544996    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Aug 05 17:12:25.545196 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.545170    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Aug 05 17:12:25.545259 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.545202    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Aug 05 17:12:25.580348 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.580320    2561 apiserver.go:52] "Watching apiserver"
Aug 05 17:12:25.595867 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.595840    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Aug 05 17:12:25.597441 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.597424    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Aug 05 17:12:25.598211 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.598192    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-8vtq4","kube-system/kube-apiserver-proxy-ip-10-0-135-151.ec2.internal","openshift-cluster-node-tuning-operator/tuned-bd6hd","openshift-image-registry/node-ca-tg9l4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-151.ec2.internal","openshift-multus/multus-additional-cni-plugins-d7kqb","openshift-multus/multus-r4gx5","openshift-network-diagnostics/network-check-target-bbcvm","kube-system/global-pull-secret-syncer-gfsf2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qtd","openshift-multus/network-metrics-daemon-v6tb4","openshift-network-operator/iptables-alerter-thg2c","openshift-ovn-kubernetes/ovnkube-node-87v9q"]
Aug 05 17:12:25.599884 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.599868    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.602594 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.602570    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.603692 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.603672    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tg9l4"
Aug 05 17:12:25.604752 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.604731    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d7kqb"
Aug 05 17:12:25.605800 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.605786    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8vtq4"
Aug 05 17:12:25.606394 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.606378    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Aug 05 17:12:25.606454 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.606446    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Aug 05 17:12:25.606528 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.606515    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rm5jf\""
Aug 05 17:12:25.606868 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.606854    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Aug 05 17:12:25.607179 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.607167    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:12:25.607240 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:25.607223    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbcvm" podUID="1cbe388d-927f-4384-9abd-49d8864afe2c"
Aug 05 17:12:25.608148 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.608133    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:25.608225 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:25.608198    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfsf2" podUID="25132135-9fbe-49c4-9f6d-0e9db2cce6b8"
Aug 05 17:12:25.608573 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.608551    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Aug 05 17:12:25.608714 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.608586    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Aug 05 17:12:25.608714 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.608670    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Aug 05 17:12:25.608845 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.608721    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tk49d\""
Aug 05 17:12:25.608845 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.608754    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Aug 05 17:12:25.608845 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.608808    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Aug 05 17:12:25.608845 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.608754    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tbcff\""
Aug 05 17:12:25.609173 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.609144    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qtd"
Aug 05 17:12:25.609663 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.609645    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Aug 05 17:12:25.609737 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.609645    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Aug 05 17:12:25.609737 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.609704    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ktxc7\""
Aug 05 17:12:25.609737 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.609712    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Aug 05 17:12:25.609898 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.609840    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dz5h7\""
Aug 05 17:12:25.610038 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.610024    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Aug 05 17:12:25.610038 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.610032    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Aug 05 17:12:25.610440 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.610425    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:12:25.610517 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:25.610496    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6tb4" podUID="ba7276d8-e168-437d-adcd-3ea703e97ab1"
Aug 05 17:12:25.611631 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.611614    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-thg2c"
Aug 05 17:12:25.612767 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.612744    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.614583 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.614563    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pzdzw\""
Aug 05 17:12:25.614668 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.614584    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 05 17:12:25.614816 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.614798    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Aug 05 17:12:25.614926 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.614908    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Aug 05 17:12:25.615122 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.615104    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Aug 05 17:12:25.615179 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.615151    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Aug 05 17:12:25.615222 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.615194    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Aug 05 17:12:25.615361 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.615335    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-04 17:07:24 +0000 UTC" deadline="2028-05-19 23:27:00.852148254 +0000 UTC"
Aug 05 17:12:25.615409 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.615363    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15678h14m35.236787966s"
Aug 05 17:12:25.616917 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.615690    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pxqls\""
Aug 05 17:12:25.616917 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.615873    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Aug 05 17:12:25.616917 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.616063    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5fh8h\""
Aug 05 17:12:25.617230 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.617208    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Aug 05 17:12:25.617303 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.617265    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Aug 05 17:12:25.617483 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.617464    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Aug 05 17:12:25.617563 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.617500    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Aug 05 17:12:25.617684 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.617667    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Aug 05 17:12:25.617896 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.617853    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Aug 05 17:12:25.621816 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.621798    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 05 17:12:25.666063 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.666039    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fq8zr"
Aug 05 17:12:25.679220 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.679200    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fq8zr"
Aug 05 17:12:25.697488 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.697468    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Aug 05 17:12:25.703137 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703117    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-host-var-lib-cni-multus\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.703243 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703145    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5d7aaa3e-c19a-4838-97af-90e196626895-multus-daemon-config\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.703243 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703166    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-host\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.703243 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703191    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e32781be-78b9-42c2-b56c-ac2c3f919e1b-iptables-alerter-script\") pod \"iptables-alerter-thg2c\" (UID: \"e32781be-78b9-42c2-b56c-ac2c3f919e1b\") " pod="openshift-network-operator/iptables-alerter-thg2c"
Aug 05 17:12:25.703243 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703213    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-host-run-ovn-kubernetes\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.703243 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703233    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5d7aaa3e-c19a-4838-97af-90e196626895-cni-binary-copy\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.703417 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703252    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-host-var-lib-kubelet\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.703417 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703284    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-host-run-multus-certs\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.703417 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703326    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvslc\" (UniqueName: \"kubernetes.io/projected/5d7aaa3e-c19a-4838-97af-90e196626895-kube-api-access-xvslc\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.703417 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703357    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-etc-modprobe-d\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.703417 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703381    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-log-socket\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.703417 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703401    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/57e503bf-db91-4091-b8de-0b7d65219906-ovnkube-config\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.703613 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703426    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-system-cni-dir\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.703613 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703455    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6d6169f-097e-40e1-b788-dc230acf07c5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j5qtd\" (UID: \"c6d6169f-097e-40e1-b788-dc230acf07c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qtd"
Aug 05 17:12:25.703613 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703480    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-etc-kubernetes\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.703613 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703540    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-etc-openvswitch\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.703613 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703574    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-node-log\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.703613 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703598    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/57e503bf-db91-4091-b8de-0b7d65219906-ovnkube-script-lib\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.703811 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703622    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6d6169f-097e-40e1-b788-dc230acf07c5-etc-selinux\") pod \"aws-ebs-csi-driver-node-j5qtd\" (UID: \"c6d6169f-097e-40e1-b788-dc230acf07c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qtd"
Aug 05 17:12:25.703811 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703647    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k46qc\" (UniqueName: \"kubernetes.io/projected/66f685fa-db6d-474f-b836-d0d9768e2d09-kube-api-access-k46qc\") pod \"multus-additional-cni-plugins-d7kqb\" (UID: \"66f685fa-db6d-474f-b836-d0d9768e2d09\") " pod="openshift-multus/multus-additional-cni-plugins-d7kqb"
Aug 05 17:12:25.703811 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703677    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25132135-9fbe-49c4-9f6d-0e9db2cce6b8-original-pull-secret\") pod \"global-pull-secret-syncer-gfsf2\" (UID: \"25132135-9fbe-49c4-9f6d-0e9db2cce6b8\") " pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:25.703811 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703700    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnzg5\" (UniqueName: \"kubernetes.io/projected/e32781be-78b9-42c2-b56c-ac2c3f919e1b-kube-api-access-dnzg5\") pod \"iptables-alerter-thg2c\" (UID: \"e32781be-78b9-42c2-b56c-ac2c3f919e1b\") " pod="openshift-network-operator/iptables-alerter-thg2c"
Aug 05 17:12:25.703811 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703730    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-host-cni-bin\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.703811 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703754    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-os-release\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.703811 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703778    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-hostroot\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.703811 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703804    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n5v2\" (UniqueName: \"kubernetes.io/projected/eacfb8d3-1760-45ee-a594-7656a55c9d93-kube-api-access-2n5v2\") pod \"node-ca-tg9l4\" (UID: \"eacfb8d3-1760-45ee-a594-7656a55c9d93\") " pod="openshift-image-registry/node-ca-tg9l4"
Aug 05 17:12:25.704141 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703826    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppwsq\" (UniqueName: \"kubernetes.io/projected/c6d6169f-097e-40e1-b788-dc230acf07c5-kube-api-access-ppwsq\") pod \"aws-ebs-csi-driver-node-j5qtd\" (UID: \"c6d6169f-097e-40e1-b788-dc230acf07c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qtd"
Aug 05 17:12:25.704141 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703848    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8be1b0df-b531-4179-918f-1cae637ce3f7-konnectivity-ca\") pod \"konnectivity-agent-8vtq4\" (UID: \"8be1b0df-b531-4179-918f-1cae637ce3f7\") " pod="kube-system/konnectivity-agent-8vtq4"
Aug 05 17:12:25.704141 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703870    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-systemd-units\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.704141 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703892    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-run-systemd\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.704141 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703914    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-run-ovn\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.704141 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.703939    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-etc-kubernetes\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.704141 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704002    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-etc-sysctl-conf\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.704141 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704040    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/66f685fa-db6d-474f-b836-d0d9768e2d09-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d7kqb\" (UID: \"66f685fa-db6d-474f-b836-d0d9768e2d09\") " pod="openshift-multus/multus-additional-cni-plugins-d7kqb"
Aug 05 17:12:25.704141 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704070    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/25132135-9fbe-49c4-9f6d-0e9db2cce6b8-kubelet-config\") pod \"global-pull-secret-syncer-gfsf2\" (UID: \"25132135-9fbe-49c4-9f6d-0e9db2cce6b8\") " pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:25.704141 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704093    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-host-slash\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.704141 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704117    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/57e503bf-db91-4091-b8de-0b7d65219906-env-overrides\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.704634 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704151    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba7276d8-e168-437d-adcd-3ea703e97ab1-metrics-certs\") pod \"network-metrics-daemon-v6tb4\" (UID: \"ba7276d8-e168-437d-adcd-3ea703e97ab1\") " pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:12:25.704634 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704182    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-run-openvswitch\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.704634 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704205    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-host-var-lib-cni-bin\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.704634 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704257    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-multus-conf-dir\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.704634 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704283    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-etc-sysctl-d\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.704634 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704305    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-etc-systemd\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.704634 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704329    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66f685fa-db6d-474f-b836-d0d9768e2d09-cnibin\") pod \"multus-additional-cni-plugins-d7kqb\" (UID: \"66f685fa-db6d-474f-b836-d0d9768e2d09\") " pod="openshift-multus/multus-additional-cni-plugins-d7kqb"
Aug 05 17:12:25.704634 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704352    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8be1b0df-b531-4179-918f-1cae637ce3f7-agent-certs\") pod \"konnectivity-agent-8vtq4\" (UID: \"8be1b0df-b531-4179-918f-1cae637ce3f7\") " pod="kube-system/konnectivity-agent-8vtq4"
Aug 05 17:12:25.704634 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704377    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-etc-sysconfig\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.704634 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704401    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7nsq\" (UniqueName: \"kubernetes.io/projected/72ee4123-b8c2-486a-8635-96ffd003756d-kube-api-access-q7nsq\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.704634 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704426    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-host-run-netns\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.704634 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704452    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-host-run-netns\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.704634 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704477    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrpwv\" (UniqueName: \"kubernetes.io/projected/1cbe388d-927f-4384-9abd-49d8864afe2c-kube-api-access-qrpwv\") pod \"network-check-target-bbcvm\" (UID: \"1cbe388d-927f-4384-9abd-49d8864afe2c\") " pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:12:25.704634 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704499    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6d6169f-097e-40e1-b788-dc230acf07c5-sys-fs\") pod \"aws-ebs-csi-driver-node-j5qtd\" (UID: \"c6d6169f-097e-40e1-b788-dc230acf07c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qtd"
Aug 05 17:12:25.704634 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704524    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6d6169f-097e-40e1-b788-dc230acf07c5-registration-dir\") pod \"aws-ebs-csi-driver-node-j5qtd\" (UID: \"c6d6169f-097e-40e1-b788-dc230acf07c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qtd"
Aug 05 17:12:25.704634 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704547    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/72ee4123-b8c2-486a-8635-96ffd003756d-etc-tuned\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.705146 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704568    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66f685fa-db6d-474f-b836-d0d9768e2d09-system-cni-dir\") pod \"multus-additional-cni-plugins-d7kqb\" (UID: \"66f685fa-db6d-474f-b836-d0d9768e2d09\") " pod="openshift-multus/multus-additional-cni-plugins-d7kqb"
Aug 05 17:12:25.705146 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704592    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdgnw\" (UniqueName: \"kubernetes.io/projected/ba7276d8-e168-437d-adcd-3ea703e97ab1-kube-api-access-jdgnw\") pod \"network-metrics-daemon-v6tb4\" (UID: \"ba7276d8-e168-437d-adcd-3ea703e97ab1\") " pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:12:25.705146 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704620    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/57e503bf-db91-4091-b8de-0b7d65219906-ovn-node-metrics-cert\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.705146 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704652    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kftk\" (UniqueName: \"kubernetes.io/projected/57e503bf-db91-4091-b8de-0b7d65219906-kube-api-access-6kftk\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.705146 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704679    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-multus-cni-dir\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.705146 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704696    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eacfb8d3-1760-45ee-a594-7656a55c9d93-host\") pod \"node-ca-tg9l4\" (UID: \"eacfb8d3-1760-45ee-a594-7656a55c9d93\") " pod="openshift-image-registry/node-ca-tg9l4"
Aug 05 17:12:25.705146 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704715    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-lib-modules\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.705146 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704738    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66f685fa-db6d-474f-b836-d0d9768e2d09-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d7kqb\" (UID: \"66f685fa-db6d-474f-b836-d0d9768e2d09\") " pod="openshift-multus/multus-additional-cni-plugins-d7kqb"
Aug 05 17:12:25.705146 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704754    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.705146 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704769    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eacfb8d3-1760-45ee-a594-7656a55c9d93-serviceca\") pod \"node-ca-tg9l4\" (UID: \"eacfb8d3-1760-45ee-a594-7656a55c9d93\") " pod="openshift-image-registry/node-ca-tg9l4"
Aug 05 17:12:25.705146 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704782    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-run\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.705146 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704803    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-var-lib-kubelet\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.705146 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704822    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72ee4123-b8c2-486a-8635-96ffd003756d-tmp\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.705146 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704844    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/66f685fa-db6d-474f-b836-d0d9768e2d09-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d7kqb\" (UID: \"66f685fa-db6d-474f-b836-d0d9768e2d09\") " pod="openshift-multus/multus-additional-cni-plugins-d7kqb"
Aug 05 17:12:25.705146 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704866    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-var-lib-openvswitch\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.705146 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704896    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-multus-socket-dir-parent\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.705577 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704917    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-host-run-k8s-cni-cncf-io\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.705577 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704932    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6d6169f-097e-40e1-b788-dc230acf07c5-socket-dir\") pod \"aws-ebs-csi-driver-node-j5qtd\" (UID: \"c6d6169f-097e-40e1-b788-dc230acf07c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qtd"
Aug 05 17:12:25.705577 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.704985    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6d6169f-097e-40e1-b788-dc230acf07c5-device-dir\") pod \"aws-ebs-csi-driver-node-j5qtd\" (UID: \"c6d6169f-097e-40e1-b788-dc230acf07c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qtd"
Aug 05 17:12:25.705577 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.705007    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-sys\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.705577 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.705033    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/66f685fa-db6d-474f-b836-d0d9768e2d09-os-release\") pod \"multus-additional-cni-plugins-d7kqb\" (UID: \"66f685fa-db6d-474f-b836-d0d9768e2d09\") " pod="openshift-multus/multus-additional-cni-plugins-d7kqb"
Aug 05 17:12:25.705577 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.705060    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66f685fa-db6d-474f-b836-d0d9768e2d09-cni-binary-copy\") pod \"multus-additional-cni-plugins-d7kqb\" (UID: \"66f685fa-db6d-474f-b836-d0d9768e2d09\") " pod="openshift-multus/multus-additional-cni-plugins-d7kqb"
Aug 05 17:12:25.705577 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.705082    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e32781be-78b9-42c2-b56c-ac2c3f919e1b-host-slash\") pod \"iptables-alerter-thg2c\" (UID: \"e32781be-78b9-42c2-b56c-ac2c3f919e1b\") " pod="openshift-network-operator/iptables-alerter-thg2c"
Aug 05 17:12:25.705577 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.705100    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-cnibin\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.705577 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.705121    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/25132135-9fbe-49c4-9f6d-0e9db2cce6b8-dbus\") pod \"global-pull-secret-syncer-gfsf2\" (UID: \"25132135-9fbe-49c4-9f6d-0e9db2cce6b8\") " pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:25.705577 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.705137    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-host-kubelet\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.705577 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.705152    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-host-cni-netd\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.710168 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:25.710141    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8290ea05f84958b6e4f1826fe71dbb.slice/crio-6d94b4aea05b389eabf55da1765baa81716c00f97521d3fbe71136ff01428dab WatchSource:0}: Error finding container 6d94b4aea05b389eabf55da1765baa81716c00f97521d3fbe71136ff01428dab: Status 404 returned error can't find the container with id 6d94b4aea05b389eabf55da1765baa81716c00f97521d3fbe71136ff01428dab
Aug 05 17:12:25.718488 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.718468    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 05 17:12:25.728409 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.728369    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-151.ec2.internal" event={"ID":"fc8290ea05f84958b6e4f1826fe71dbb","Type":"ContainerStarted","Data":"6d94b4aea05b389eabf55da1765baa81716c00f97521d3fbe71136ff01428dab"}
Aug 05 17:12:25.747772 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:25.747751    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37213c9548f92e588dc6c041ff6110a8.slice/crio-355b7eb144b7b00a3b43359794fb3314fd0b60caa665bd31895d2755471b257e WatchSource:0}: Error finding container 355b7eb144b7b00a3b43359794fb3314fd0b60caa665bd31895d2755471b257e: Status 404 returned error can't find the container with id 355b7eb144b7b00a3b43359794fb3314fd0b60caa665bd31895d2755471b257e
Aug 05 17:12:25.806237 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.806194    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6d6169f-097e-40e1-b788-dc230acf07c5-registration-dir\") pod \"aws-ebs-csi-driver-node-j5qtd\" (UID: \"c6d6169f-097e-40e1-b788-dc230acf07c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qtd"
Aug 05 17:12:25.806237 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.806236    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/72ee4123-b8c2-486a-8635-96ffd003756d-etc-tuned\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.806436 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.806261    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66f685fa-db6d-474f-b836-d0d9768e2d09-system-cni-dir\") pod \"multus-additional-cni-plugins-d7kqb\" (UID: \"66f685fa-db6d-474f-b836-d0d9768e2d09\") " pod="openshift-multus/multus-additional-cni-plugins-d7kqb"
Aug 05 17:12:25.806436 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.806306    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66f685fa-db6d-474f-b836-d0d9768e2d09-system-cni-dir\") pod \"multus-additional-cni-plugins-d7kqb\" (UID: \"66f685fa-db6d-474f-b836-d0d9768e2d09\") " pod="openshift-multus/multus-additional-cni-plugins-d7kqb"
Aug 05 17:12:25.806436 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.806311    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6d6169f-097e-40e1-b788-dc230acf07c5-registration-dir\") pod \"aws-ebs-csi-driver-node-j5qtd\" (UID: \"c6d6169f-097e-40e1-b788-dc230acf07c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qtd"
Aug 05 17:12:25.806436 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.806335    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdgnw\" (UniqueName: \"kubernetes.io/projected/ba7276d8-e168-437d-adcd-3ea703e97ab1-kube-api-access-jdgnw\") pod \"network-metrics-daemon-v6tb4\" (UID: \"ba7276d8-e168-437d-adcd-3ea703e97ab1\") " pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:12:25.806436 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.806354    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/57e503bf-db91-4091-b8de-0b7d65219906-ovn-node-metrics-cert\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.806436 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.806370    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kftk\" (UniqueName: \"kubernetes.io/projected/57e503bf-db91-4091-b8de-0b7d65219906-kube-api-access-6kftk\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.806436 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.806389    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-multus-cni-dir\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.806436 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.806405    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eacfb8d3-1760-45ee-a594-7656a55c9d93-host\") pod \"node-ca-tg9l4\" (UID: \"eacfb8d3-1760-45ee-a594-7656a55c9d93\") " pod="openshift-image-registry/node-ca-tg9l4"
Aug 05 17:12:25.806436 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.806425    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-lib-modules\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.806809 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.806447    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66f685fa-db6d-474f-b836-d0d9768e2d09-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d7kqb\" (UID: \"66f685fa-db6d-474f-b836-d0d9768e2d09\") " pod="openshift-multus/multus-additional-cni-plugins-d7kqb"
Aug 05 17:12:25.806809 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.806476    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.806809 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.806493    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eacfb8d3-1760-45ee-a594-7656a55c9d93-host\") pod \"node-ca-tg9l4\" (UID: \"eacfb8d3-1760-45ee-a594-7656a55c9d93\") " pod="openshift-image-registry/node-ca-tg9l4"
Aug 05 17:12:25.806809 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.806501    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eacfb8d3-1760-45ee-a594-7656a55c9d93-serviceca\") pod \"node-ca-tg9l4\" (UID: \"eacfb8d3-1760-45ee-a594-7656a55c9d93\") " pod="openshift-image-registry/node-ca-tg9l4"
Aug 05 17:12:25.806809 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.806525    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-run\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.806809 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.806549    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-var-lib-kubelet\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.806809 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.806572    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72ee4123-b8c2-486a-8635-96ffd003756d-tmp\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.806809 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.806594    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/66f685fa-db6d-474f-b836-d0d9768e2d09-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d7kqb\" (UID: \"66f685fa-db6d-474f-b836-d0d9768e2d09\") " pod="openshift-multus/multus-additional-cni-plugins-d7kqb"
Aug 05 17:12:25.806809 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.806607    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-lib-modules\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.806809 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.806621    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-var-lib-openvswitch\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.807275 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.806647    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-multus-socket-dir-parent\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.807275 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.806932    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-var-lib-kubelet\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.807275 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.806972    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66f685fa-db6d-474f-b836-d0d9768e2d09-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d7kqb\" (UID: \"66f685fa-db6d-474f-b836-d0d9768e2d09\") " pod="openshift-multus/multus-additional-cni-plugins-d7kqb"
Aug 05 17:12:25.807275 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.806575    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Aug 05 17:12:25.807275 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.807020    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-host-run-k8s-cni-cncf-io\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.807275 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.807043    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.807275 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.807058    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6d6169f-097e-40e1-b788-dc230acf07c5-socket-dir\") pod \"aws-ebs-csi-driver-node-j5qtd\" (UID: \"c6d6169f-097e-40e1-b788-dc230acf07c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qtd"
Aug 05 17:12:25.807275 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.807113    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-host-run-k8s-cni-cncf-io\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.807275 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.807162    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6d6169f-097e-40e1-b788-dc230acf07c5-device-dir\") pod \"aws-ebs-csi-driver-node-j5qtd\" (UID: \"c6d6169f-097e-40e1-b788-dc230acf07c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qtd"
Aug 05 17:12:25.807275 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.807200    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6d6169f-097e-40e1-b788-dc230acf07c5-socket-dir\") pod \"aws-ebs-csi-driver-node-j5qtd\" (UID: \"c6d6169f-097e-40e1-b788-dc230acf07c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qtd"
Aug 05 17:12:25.807275 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.807213    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-sys\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.807275 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.807247    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/66f685fa-db6d-474f-b836-d0d9768e2d09-os-release\") pod \"multus-additional-cni-plugins-d7kqb\" (UID: \"66f685fa-db6d-474f-b836-d0d9768e2d09\") " pod="openshift-multus/multus-additional-cni-plugins-d7kqb"
Aug 05 17:12:25.807275 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.807277    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-var-lib-openvswitch\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.807275 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.807279    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66f685fa-db6d-474f-b836-d0d9768e2d09-cni-binary-copy\") pod \"multus-additional-cni-plugins-d7kqb\" (UID: \"66f685fa-db6d-474f-b836-d0d9768e2d09\") " pod="openshift-multus/multus-additional-cni-plugins-d7kqb"
Aug 05 17:12:25.807867 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.807317    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e32781be-78b9-42c2-b56c-ac2c3f919e1b-host-slash\") pod \"iptables-alerter-thg2c\" (UID: \"e32781be-78b9-42c2-b56c-ac2c3f919e1b\") " pod="openshift-network-operator/iptables-alerter-thg2c"
Aug 05 17:12:25.807867 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.807379    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e32781be-78b9-42c2-b56c-ac2c3f919e1b-host-slash\") pod \"iptables-alerter-thg2c\" (UID: \"e32781be-78b9-42c2-b56c-ac2c3f919e1b\") " pod="openshift-network-operator/iptables-alerter-thg2c"
Aug 05 17:12:25.807867 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.807434    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6d6169f-097e-40e1-b788-dc230acf07c5-device-dir\") pod \"aws-ebs-csi-driver-node-j5qtd\" (UID: \"c6d6169f-097e-40e1-b788-dc230acf07c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qtd"
Aug 05 17:12:25.807867 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.807481    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-sys\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.807867 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.807537    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/66f685fa-db6d-474f-b836-d0d9768e2d09-os-release\") pod \"multus-additional-cni-plugins-d7kqb\" (UID: \"66f685fa-db6d-474f-b836-d0d9768e2d09\") " pod="openshift-multus/multus-additional-cni-plugins-d7kqb"
Aug 05 17:12:25.807867 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.807701    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eacfb8d3-1760-45ee-a594-7656a55c9d93-serviceca\") pod \"node-ca-tg9l4\" (UID: \"eacfb8d3-1760-45ee-a594-7656a55c9d93\") " pod="openshift-image-registry/node-ca-tg9l4"
Aug 05 17:12:25.807867 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.807743    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/66f685fa-db6d-474f-b836-d0d9768e2d09-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d7kqb\" (UID: \"66f685fa-db6d-474f-b836-d0d9768e2d09\") " pod="openshift-multus/multus-additional-cni-plugins-d7kqb"
Aug 05 17:12:25.807867 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.807786    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-multus-cni-dir\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.807867 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.807811    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-multus-socket-dir-parent\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.807867 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.806649    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-run\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.808324 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.807929    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-cnibin\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.808324 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.807984    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/25132135-9fbe-49c4-9f6d-0e9db2cce6b8-dbus\") pod \"global-pull-secret-syncer-gfsf2\" (UID: \"25132135-9fbe-49c4-9f6d-0e9db2cce6b8\") " pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:25.808324 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808021    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-host-kubelet\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.808324 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808035    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66f685fa-db6d-474f-b836-d0d9768e2d09-cni-binary-copy\") pod \"multus-additional-cni-plugins-d7kqb\" (UID: \"66f685fa-db6d-474f-b836-d0d9768e2d09\") " pod="openshift-multus/multus-additional-cni-plugins-d7kqb"
Aug 05 17:12:25.808324 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808054    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-host-cni-netd\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.808324 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808089    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-host-var-lib-cni-multus\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.808324 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808115    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/25132135-9fbe-49c4-9f6d-0e9db2cce6b8-dbus\") pod \"global-pull-secret-syncer-gfsf2\" (UID: \"25132135-9fbe-49c4-9f6d-0e9db2cce6b8\") " pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:25.808324 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808124    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5d7aaa3e-c19a-4838-97af-90e196626895-multus-daemon-config\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.808324 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808220    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-host-cni-netd\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.808324 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808251    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-host-kubelet\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.808324 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808278    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-host-var-lib-cni-multus\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.808324 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808299    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-host\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.808826 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808337    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-cnibin\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.808826 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808337    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e32781be-78b9-42c2-b56c-ac2c3f919e1b-iptables-alerter-script\") pod \"iptables-alerter-thg2c\" (UID: \"e32781be-78b9-42c2-b56c-ac2c3f919e1b\") " pod="openshift-network-operator/iptables-alerter-thg2c"
Aug 05 17:12:25.808826 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808377    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-host-run-ovn-kubernetes\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.808826 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808403    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-host\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.808826 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808409    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5d7aaa3e-c19a-4838-97af-90e196626895-cni-binary-copy\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.808826 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808441    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-host-var-lib-kubelet\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.808826 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808468    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-host-run-multus-certs\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.808826 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808504    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvslc\" (UniqueName: \"kubernetes.io/projected/5d7aaa3e-c19a-4838-97af-90e196626895-kube-api-access-xvslc\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.808826 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808535    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-etc-modprobe-d\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.808826 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808566    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-log-socket\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.808826 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808596    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/57e503bf-db91-4091-b8de-0b7d65219906-ovnkube-config\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.808826 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808622    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-system-cni-dir\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.808826 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808656    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6d6169f-097e-40e1-b788-dc230acf07c5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j5qtd\" (UID: \"c6d6169f-097e-40e1-b788-dc230acf07c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qtd"
Aug 05 17:12:25.808826 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808688    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-etc-kubernetes\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.808826 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808691    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5d7aaa3e-c19a-4838-97af-90e196626895-multus-daemon-config\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.808826 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808721    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-etc-openvswitch\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.808826 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808765    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-node-log\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.808826 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808782    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-etc-openvswitch\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.809679 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808808    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/57e503bf-db91-4091-b8de-0b7d65219906-ovnkube-script-lib\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.809679 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808846    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6d6169f-097e-40e1-b788-dc230acf07c5-etc-selinux\") pod \"aws-ebs-csi-driver-node-j5qtd\" (UID: \"c6d6169f-097e-40e1-b788-dc230acf07c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qtd"
Aug 05 17:12:25.809679 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808881    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-etc-modprobe-d\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.809679 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808884    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k46qc\" (UniqueName: \"kubernetes.io/projected/66f685fa-db6d-474f-b836-d0d9768e2d09-kube-api-access-k46qc\") pod \"multus-additional-cni-plugins-d7kqb\" (UID: \"66f685fa-db6d-474f-b836-d0d9768e2d09\") " pod="openshift-multus/multus-additional-cni-plugins-d7kqb"
Aug 05 17:12:25.809679 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808907    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e32781be-78b9-42c2-b56c-ac2c3f919e1b-iptables-alerter-script\") pod \"iptables-alerter-thg2c\" (UID: \"e32781be-78b9-42c2-b56c-ac2c3f919e1b\") " pod="openshift-network-operator/iptables-alerter-thg2c"
Aug 05 17:12:25.809679 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808927    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25132135-9fbe-49c4-9f6d-0e9db2cce6b8-original-pull-secret\") pod \"global-pull-secret-syncer-gfsf2\" (UID: \"25132135-9fbe-49c4-9f6d-0e9db2cce6b8\") " pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:25.809679 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808981    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnzg5\" (UniqueName: \"kubernetes.io/projected/e32781be-78b9-42c2-b56c-ac2c3f919e1b-kube-api-access-dnzg5\") pod \"iptables-alerter-thg2c\" (UID: \"e32781be-78b9-42c2-b56c-ac2c3f919e1b\") " pod="openshift-network-operator/iptables-alerter-thg2c"
Aug 05 17:12:25.809679 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.808984    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-host-run-ovn-kubernetes\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.809679 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809024    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-host-cni-bin\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.809679 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809073    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-os-release\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.809679 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809105    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-hostroot\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.809679 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809139    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n5v2\" (UniqueName: \"kubernetes.io/projected/eacfb8d3-1760-45ee-a594-7656a55c9d93-kube-api-access-2n5v2\") pod \"node-ca-tg9l4\" (UID: \"eacfb8d3-1760-45ee-a594-7656a55c9d93\") " pod="openshift-image-registry/node-ca-tg9l4"
Aug 05 17:12:25.809679 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809173    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppwsq\" (UniqueName: \"kubernetes.io/projected/c6d6169f-097e-40e1-b788-dc230acf07c5-kube-api-access-ppwsq\") pod \"aws-ebs-csi-driver-node-j5qtd\" (UID: \"c6d6169f-097e-40e1-b788-dc230acf07c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qtd"
Aug 05 17:12:25.809679 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809201    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8be1b0df-b531-4179-918f-1cae637ce3f7-konnectivity-ca\") pod \"konnectivity-agent-8vtq4\" (UID: \"8be1b0df-b531-4179-918f-1cae637ce3f7\") " pod="kube-system/konnectivity-agent-8vtq4"
Aug 05 17:12:25.809679 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809233    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-systemd-units\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.809679 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809263    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-run-systemd\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.809679 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809294    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-run-ovn\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.810452 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809321    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-etc-kubernetes\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.810452 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809355    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-etc-sysctl-conf\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.810452 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809389    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/66f685fa-db6d-474f-b836-d0d9768e2d09-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d7kqb\" (UID: \"66f685fa-db6d-474f-b836-d0d9768e2d09\") " pod="openshift-multus/multus-additional-cni-plugins-d7kqb"
Aug 05 17:12:25.810452 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809394    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-node-log\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.810452 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809431    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/25132135-9fbe-49c4-9f6d-0e9db2cce6b8-kubelet-config\") pod \"global-pull-secret-syncer-gfsf2\" (UID: \"25132135-9fbe-49c4-9f6d-0e9db2cce6b8\") " pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:25.810452 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809465    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-host-slash\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.810452 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809494    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/57e503bf-db91-4091-b8de-0b7d65219906-ovnkube-config\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.810452 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809499    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/57e503bf-db91-4091-b8de-0b7d65219906-env-overrides\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.810452 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809545    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba7276d8-e168-437d-adcd-3ea703e97ab1-metrics-certs\") pod \"network-metrics-daemon-v6tb4\" (UID: \"ba7276d8-e168-437d-adcd-3ea703e97ab1\") " pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:12:25.810452 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809578    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-run-openvswitch\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.810452 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809611    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-host-var-lib-cni-bin\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.810452 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809643    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-multus-conf-dir\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.810452 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809676    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-etc-sysctl-d\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.810452 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809703    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-etc-systemd\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.810452 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809735    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66f685fa-db6d-474f-b836-d0d9768e2d09-cnibin\") pod \"multus-additional-cni-plugins-d7kqb\" (UID: \"66f685fa-db6d-474f-b836-d0d9768e2d09\") " pod="openshift-multus/multus-additional-cni-plugins-d7kqb"
Aug 05 17:12:25.810452 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809767    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8be1b0df-b531-4179-918f-1cae637ce3f7-agent-certs\") pod \"konnectivity-agent-8vtq4\" (UID: \"8be1b0df-b531-4179-918f-1cae637ce3f7\") " pod="kube-system/konnectivity-agent-8vtq4"
Aug 05 17:12:25.810452 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809799    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-etc-sysconfig\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.811265 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809830    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7nsq\" (UniqueName: \"kubernetes.io/projected/72ee4123-b8c2-486a-8635-96ffd003756d-kube-api-access-q7nsq\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.811265 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809859    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-host-run-netns\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.811265 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809890    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/57e503bf-db91-4091-b8de-0b7d65219906-env-overrides\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.811265 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809978    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-host-run-netns\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.811265 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809976    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-system-cni-dir\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.811265 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.810030    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6d6169f-097e-40e1-b788-dc230acf07c5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j5qtd\" (UID: \"c6d6169f-097e-40e1-b788-dc230acf07c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qtd"
Aug 05 17:12:25.811265 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.810077    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-host-var-lib-kubelet\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.811265 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.810157    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-etc-kubernetes\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.811265 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:25.810249    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:25.811265 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:25.810378    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25132135-9fbe-49c4-9f6d-0e9db2cce6b8-original-pull-secret podName:25132135-9fbe-49c4-9f6d-0e9db2cce6b8 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:26.310308381 +0000 UTC m=+2.054942703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25132135-9fbe-49c4-9f6d-0e9db2cce6b8-original-pull-secret") pod "global-pull-secret-syncer-gfsf2" (UID: "25132135-9fbe-49c4-9f6d-0e9db2cce6b8") : object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:25.811265 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.810418    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/57e503bf-db91-4091-b8de-0b7d65219906-ovnkube-script-lib\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.811265 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.810533    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6d6169f-097e-40e1-b788-dc230acf07c5-etc-selinux\") pod \"aws-ebs-csi-driver-node-j5qtd\" (UID: \"c6d6169f-097e-40e1-b788-dc230acf07c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qtd"
Aug 05 17:12:25.811265 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.810602    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-etc-systemd\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.811265 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.810634    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66f685fa-db6d-474f-b836-d0d9768e2d09-cnibin\") pod \"multus-additional-cni-plugins-d7kqb\" (UID: \"66f685fa-db6d-474f-b836-d0d9768e2d09\") " pod="openshift-multus/multus-additional-cni-plugins-d7kqb"
Aug 05 17:12:25.811265 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.810714    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-etc-sysconfig\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.811265 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.810825    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72ee4123-b8c2-486a-8635-96ffd003756d-tmp\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.811265 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.810898    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-run-openvswitch\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.812070 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.810898    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/57e503bf-db91-4091-b8de-0b7d65219906-ovn-node-metrics-cert\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.812070 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:25.810937    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:25.812070 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.811002    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-host-var-lib-cni-bin\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.812070 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.811007    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5d7aaa3e-c19a-4838-97af-90e196626895-cni-binary-copy\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.812070 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:25.811022    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba7276d8-e168-437d-adcd-3ea703e97ab1-metrics-certs podName:ba7276d8-e168-437d-adcd-3ea703e97ab1 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:26.311003209 +0000 UTC m=+2.055637533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba7276d8-e168-437d-adcd-3ea703e97ab1-metrics-certs") pod "network-metrics-daemon-v6tb4" (UID: "ba7276d8-e168-437d-adcd-3ea703e97ab1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:25.812070 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.811058    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-multus-conf-dir\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.812070 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.811106    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-etc-kubernetes\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.812070 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.811127    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-host-cni-bin\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.812070 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.811143    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-etc-sysctl-d\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.812070 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.811162    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-os-release\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.812070 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.809892    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-host-run-netns\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.812070 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.811131    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/25132135-9fbe-49c4-9f6d-0e9db2cce6b8-kubelet-config\") pod \"global-pull-secret-syncer-gfsf2\" (UID: \"25132135-9fbe-49c4-9f6d-0e9db2cce6b8\") " pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:25.812070 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.811225    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrpwv\" (UniqueName: \"kubernetes.io/projected/1cbe388d-927f-4384-9abd-49d8864afe2c-kube-api-access-qrpwv\") pod \"network-check-target-bbcvm\" (UID: \"1cbe388d-927f-4384-9abd-49d8864afe2c\") " pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:12:25.812070 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.811262    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6d6169f-097e-40e1-b788-dc230acf07c5-sys-fs\") pod \"aws-ebs-csi-driver-node-j5qtd\" (UID: \"c6d6169f-097e-40e1-b788-dc230acf07c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qtd"
Aug 05 17:12:25.812070 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.811266    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-log-socket\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.812070 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.811351    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-host-run-multus-certs\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.812070 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.811462    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/72ee4123-b8c2-486a-8635-96ffd003756d-etc-sysctl-conf\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.812820 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.811557    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6d6169f-097e-40e1-b788-dc230acf07c5-sys-fs\") pod \"aws-ebs-csi-driver-node-j5qtd\" (UID: \"c6d6169f-097e-40e1-b788-dc230acf07c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qtd"
Aug 05 17:12:25.812820 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.811620    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/66f685fa-db6d-474f-b836-d0d9768e2d09-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d7kqb\" (UID: \"66f685fa-db6d-474f-b836-d0d9768e2d09\") " pod="openshift-multus/multus-additional-cni-plugins-d7kqb"
Aug 05 17:12:25.812820 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.811686    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-host-slash\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.812820 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.811717    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-run-systemd\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.812820 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.811742    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-run-ovn\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.812820 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.811776    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/57e503bf-db91-4091-b8de-0b7d65219906-systemd-units\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.812820 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.811822    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-host-run-netns\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.812820 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.811890    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5d7aaa3e-c19a-4838-97af-90e196626895-hostroot\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.812820 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.812176    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8be1b0df-b531-4179-918f-1cae637ce3f7-konnectivity-ca\") pod \"konnectivity-agent-8vtq4\" (UID: \"8be1b0df-b531-4179-918f-1cae637ce3f7\") " pod="kube-system/konnectivity-agent-8vtq4"
Aug 05 17:12:25.813591 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.813561    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/72ee4123-b8c2-486a-8635-96ffd003756d-etc-tuned\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.813830 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.813812    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8be1b0df-b531-4179-918f-1cae637ce3f7-agent-certs\") pod \"konnectivity-agent-8vtq4\" (UID: \"8be1b0df-b531-4179-918f-1cae637ce3f7\") " pod="kube-system/konnectivity-agent-8vtq4"
Aug 05 17:12:25.816060 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.816036    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdgnw\" (UniqueName: \"kubernetes.io/projected/ba7276d8-e168-437d-adcd-3ea703e97ab1-kube-api-access-jdgnw\") pod \"network-metrics-daemon-v6tb4\" (UID: \"ba7276d8-e168-437d-adcd-3ea703e97ab1\") " pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:12:25.816139 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.816115    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kftk\" (UniqueName: \"kubernetes.io/projected/57e503bf-db91-4091-b8de-0b7d65219906-kube-api-access-6kftk\") pod \"ovnkube-node-87v9q\" (UID: \"57e503bf-db91-4091-b8de-0b7d65219906\") " pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:25.817252 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.817232    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k46qc\" (UniqueName: \"kubernetes.io/projected/66f685fa-db6d-474f-b836-d0d9768e2d09-kube-api-access-k46qc\") pod \"multus-additional-cni-plugins-d7kqb\" (UID: \"66f685fa-db6d-474f-b836-d0d9768e2d09\") " pod="openshift-multus/multus-additional-cni-plugins-d7kqb"
Aug 05 17:12:25.817503 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.817487    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvslc\" (UniqueName: \"kubernetes.io/projected/5d7aaa3e-c19a-4838-97af-90e196626895-kube-api-access-xvslc\") pod \"multus-r4gx5\" (UID: \"5d7aaa3e-c19a-4838-97af-90e196626895\") " pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.819134 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:25.819114    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 05 17:12:25.819226 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:25.819167    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 05 17:12:25.819226 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:25.819183    2561 projected.go:196] Error preparing data for projected volume kube-api-access-qrpwv for pod openshift-network-diagnostics/network-check-target-bbcvm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:25.819297 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:25.819252    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1cbe388d-927f-4384-9abd-49d8864afe2c-kube-api-access-qrpwv podName:1cbe388d-927f-4384-9abd-49d8864afe2c nodeName:}" failed. No retries permitted until 2026-08-05 17:12:26.319238761 +0000 UTC m=+2.063873082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qrpwv" (UniqueName: "kubernetes.io/projected/1cbe388d-927f-4384-9abd-49d8864afe2c-kube-api-access-qrpwv") pod "network-check-target-bbcvm" (UID: "1cbe388d-927f-4384-9abd-49d8864afe2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:25.820870 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.820846    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnzg5\" (UniqueName: \"kubernetes.io/projected/e32781be-78b9-42c2-b56c-ac2c3f919e1b-kube-api-access-dnzg5\") pod \"iptables-alerter-thg2c\" (UID: \"e32781be-78b9-42c2-b56c-ac2c3f919e1b\") " pod="openshift-network-operator/iptables-alerter-thg2c"
Aug 05 17:12:25.821005 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.820983    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7nsq\" (UniqueName: \"kubernetes.io/projected/72ee4123-b8c2-486a-8635-96ffd003756d-kube-api-access-q7nsq\") pod \"tuned-bd6hd\" (UID: \"72ee4123-b8c2-486a-8635-96ffd003756d\") " pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.822511 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.822495    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n5v2\" (UniqueName: \"kubernetes.io/projected/eacfb8d3-1760-45ee-a594-7656a55c9d93-kube-api-access-2n5v2\") pod \"node-ca-tg9l4\" (UID: \"eacfb8d3-1760-45ee-a594-7656a55c9d93\") " pod="openshift-image-registry/node-ca-tg9l4"
Aug 05 17:12:25.822792 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.822776    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppwsq\" (UniqueName: \"kubernetes.io/projected/c6d6169f-097e-40e1-b788-dc230acf07c5-kube-api-access-ppwsq\") pod \"aws-ebs-csi-driver-node-j5qtd\" (UID: \"c6d6169f-097e-40e1-b788-dc230acf07c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qtd"
Aug 05 17:12:25.924360 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.924281    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r4gx5"
Aug 05 17:12:25.933047 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:25.933024    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7aaa3e_c19a_4838_97af_90e196626895.slice/crio-c4ebeffe348b87dac51d66f1c6f88bfcc8b53c559ee6fd058fc2e0aa734249a3 WatchSource:0}: Error finding container c4ebeffe348b87dac51d66f1c6f88bfcc8b53c559ee6fd058fc2e0aa734249a3: Status 404 returned error can't find the container with id c4ebeffe348b87dac51d66f1c6f88bfcc8b53c559ee6fd058fc2e0aa734249a3
Aug 05 17:12:25.942075 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.942053    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bd6hd"
Aug 05 17:12:25.948634 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:25.948612    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ee4123_b8c2_486a_8635_96ffd003756d.slice/crio-5dbfcf5b62d770c815d834a35d91142b7af49717c750c0f09c11faa6c4e1b573 WatchSource:0}: Error finding container 5dbfcf5b62d770c815d834a35d91142b7af49717c750c0f09c11faa6c4e1b573: Status 404 returned error can't find the container with id 5dbfcf5b62d770c815d834a35d91142b7af49717c750c0f09c11faa6c4e1b573
Aug 05 17:12:25.957791 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.957774    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tg9l4"
Aug 05 17:12:25.964768 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:25.964745    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeacfb8d3_1760_45ee_a594_7656a55c9d93.slice/crio-e96849a3654c069378f98f5d0d68a761b29128af74616b314480a5fc3758a7a1 WatchSource:0}: Error finding container e96849a3654c069378f98f5d0d68a761b29128af74616b314480a5fc3758a7a1: Status 404 returned error can't find the container with id e96849a3654c069378f98f5d0d68a761b29128af74616b314480a5fc3758a7a1
Aug 05 17:12:25.972793 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.972774    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d7kqb"
Aug 05 17:12:25.979914 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:25.979889    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f685fa_db6d_474f_b836_d0d9768e2d09.slice/crio-36990bdc4caba4939aa6164b198d5e49881c09f77a0954d573ff0d192f6abbf9 WatchSource:0}: Error finding container 36990bdc4caba4939aa6164b198d5e49881c09f77a0954d573ff0d192f6abbf9: Status 404 returned error can't find the container with id 36990bdc4caba4939aa6164b198d5e49881c09f77a0954d573ff0d192f6abbf9
Aug 05 17:12:25.999412 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:25.999385    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8vtq4"
Aug 05 17:12:26.006552 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:26.006526    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be1b0df_b531_4179_918f_1cae637ce3f7.slice/crio-35e9536eddac4e0f415b9b284484d307069300433eaa83995f066d31ece53444 WatchSource:0}: Error finding container 35e9536eddac4e0f415b9b284484d307069300433eaa83995f066d31ece53444: Status 404 returned error can't find the container with id 35e9536eddac4e0f415b9b284484d307069300433eaa83995f066d31ece53444
Aug 05 17:12:26.015555 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:26.015534    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qtd"
Aug 05 17:12:26.022044 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:26.022018    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-thg2c"
Aug 05 17:12:26.022628 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:26.022599    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d6169f_097e_40e1_b788_dc230acf07c5.slice/crio-2bbb7f43b206b1db5e993882f1f76a080e2b62effa7553a3b5bf7674289fa825 WatchSource:0}: Error finding container 2bbb7f43b206b1db5e993882f1f76a080e2b62effa7553a3b5bf7674289fa825: Status 404 returned error can't find the container with id 2bbb7f43b206b1db5e993882f1f76a080e2b62effa7553a3b5bf7674289fa825
Aug 05 17:12:26.026693 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:26.026677    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:26.030444 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:26.030422    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode32781be_78b9_42c2_b56c_ac2c3f919e1b.slice/crio-b8739b50ba9fa681f94e649e7e430f7a45801218a89fe07d64fa508066abeed5 WatchSource:0}: Error finding container b8739b50ba9fa681f94e649e7e430f7a45801218a89fe07d64fa508066abeed5: Status 404 returned error can't find the container with id b8739b50ba9fa681f94e649e7e430f7a45801218a89fe07d64fa508066abeed5
Aug 05 17:12:26.034725 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:26.034703    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e503bf_db91_4091_b8de_0b7d65219906.slice/crio-e5333fd65f6d3d5d0768e20bd05c346481db833a9cf52d189fb6643e75ed04ed WatchSource:0}: Error finding container e5333fd65f6d3d5d0768e20bd05c346481db833a9cf52d189fb6643e75ed04ed: Status 404 returned error can't find the container with id e5333fd65f6d3d5d0768e20bd05c346481db833a9cf52d189fb6643e75ed04ed
Aug 05 17:12:26.130125 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:26.130092    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 05 17:12:26.315053 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:26.314967    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25132135-9fbe-49c4-9f6d-0e9db2cce6b8-original-pull-secret\") pod \"global-pull-secret-syncer-gfsf2\" (UID: \"25132135-9fbe-49c4-9f6d-0e9db2cce6b8\") " pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:26.315053 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:26.315019    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba7276d8-e168-437d-adcd-3ea703e97ab1-metrics-certs\") pod \"network-metrics-daemon-v6tb4\" (UID: \"ba7276d8-e168-437d-adcd-3ea703e97ab1\") " pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:12:26.315258 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:26.315179    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:26.315258 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:26.315242    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba7276d8-e168-437d-adcd-3ea703e97ab1-metrics-certs podName:ba7276d8-e168-437d-adcd-3ea703e97ab1 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:27.315224122 +0000 UTC m=+3.059858431 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba7276d8-e168-437d-adcd-3ea703e97ab1-metrics-certs") pod "network-metrics-daemon-v6tb4" (UID: "ba7276d8-e168-437d-adcd-3ea703e97ab1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:26.315418 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:26.315392    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:26.315543 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:26.315459    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25132135-9fbe-49c4-9f6d-0e9db2cce6b8-original-pull-secret podName:25132135-9fbe-49c4-9f6d-0e9db2cce6b8 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:27.315439533 +0000 UTC m=+3.060073852 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25132135-9fbe-49c4-9f6d-0e9db2cce6b8-original-pull-secret") pod "global-pull-secret-syncer-gfsf2" (UID: "25132135-9fbe-49c4-9f6d-0e9db2cce6b8") : object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:26.415507 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:26.415476    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrpwv\" (UniqueName: \"kubernetes.io/projected/1cbe388d-927f-4384-9abd-49d8864afe2c-kube-api-access-qrpwv\") pod \"network-check-target-bbcvm\" (UID: \"1cbe388d-927f-4384-9abd-49d8864afe2c\") " pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:12:26.415682 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:26.415663    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 05 17:12:26.415759 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:26.415690    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 05 17:12:26.415759 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:26.415705    2561 projected.go:196] Error preparing data for projected volume kube-api-access-qrpwv for pod openshift-network-diagnostics/network-check-target-bbcvm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:26.415857 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:26.415765    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1cbe388d-927f-4384-9abd-49d8864afe2c-kube-api-access-qrpwv podName:1cbe388d-927f-4384-9abd-49d8864afe2c nodeName:}" failed. No retries permitted until 2026-08-05 17:12:27.415743947 +0000 UTC m=+3.160378277 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qrpwv" (UniqueName: "kubernetes.io/projected/1cbe388d-927f-4384-9abd-49d8864afe2c-kube-api-access-qrpwv") pod "network-check-target-bbcvm" (UID: "1cbe388d-927f-4384-9abd-49d8864afe2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:26.432982 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:26.432939    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 05 17:12:26.680697 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:26.680547    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-04 17:07:25 +0000 UTC" deadline="2028-05-06 21:35:52.172155222 +0000 UTC"
Aug 05 17:12:26.680697 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:26.680584    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15364h23m25.49157594s"
Aug 05 17:12:26.733741 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:26.733702    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-87v9q" event={"ID":"57e503bf-db91-4091-b8de-0b7d65219906","Type":"ContainerStarted","Data":"e5333fd65f6d3d5d0768e20bd05c346481db833a9cf52d189fb6643e75ed04ed"}
Aug 05 17:12:26.736946 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:26.736916    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-thg2c" event={"ID":"e32781be-78b9-42c2-b56c-ac2c3f919e1b","Type":"ContainerStarted","Data":"b8739b50ba9fa681f94e649e7e430f7a45801218a89fe07d64fa508066abeed5"}
Aug 05 17:12:26.743205 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:26.743166    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8vtq4" event={"ID":"8be1b0df-b531-4179-918f-1cae637ce3f7","Type":"ContainerStarted","Data":"35e9536eddac4e0f415b9b284484d307069300433eaa83995f066d31ece53444"}
Aug 05 17:12:26.747518 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:26.747443    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tg9l4" event={"ID":"eacfb8d3-1760-45ee-a594-7656a55c9d93","Type":"ContainerStarted","Data":"e96849a3654c069378f98f5d0d68a761b29128af74616b314480a5fc3758a7a1"}
Aug 05 17:12:26.750709 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:26.750630    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-151.ec2.internal" event={"ID":"37213c9548f92e588dc6c041ff6110a8","Type":"ContainerStarted","Data":"355b7eb144b7b00a3b43359794fb3314fd0b60caa665bd31895d2755471b257e"}
Aug 05 17:12:26.757499 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:26.757413    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qtd" event={"ID":"c6d6169f-097e-40e1-b788-dc230acf07c5","Type":"ContainerStarted","Data":"2bbb7f43b206b1db5e993882f1f76a080e2b62effa7553a3b5bf7674289fa825"}
Aug 05 17:12:26.759808 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:26.759772    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7kqb" event={"ID":"66f685fa-db6d-474f-b836-d0d9768e2d09","Type":"ContainerStarted","Data":"36990bdc4caba4939aa6164b198d5e49881c09f77a0954d573ff0d192f6abbf9"}
Aug 05 17:12:26.761469 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:26.761408    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bd6hd" event={"ID":"72ee4123-b8c2-486a-8635-96ffd003756d","Type":"ContainerStarted","Data":"5dbfcf5b62d770c815d834a35d91142b7af49717c750c0f09c11faa6c4e1b573"}
Aug 05 17:12:26.765358 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:26.765332    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r4gx5" event={"ID":"5d7aaa3e-c19a-4838-97af-90e196626895","Type":"ContainerStarted","Data":"c4ebeffe348b87dac51d66f1c6f88bfcc8b53c559ee6fd058fc2e0aa734249a3"}
Aug 05 17:12:27.150447 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:27.150178    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 05 17:12:27.321291 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:27.321203    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25132135-9fbe-49c4-9f6d-0e9db2cce6b8-original-pull-secret\") pod \"global-pull-secret-syncer-gfsf2\" (UID: \"25132135-9fbe-49c4-9f6d-0e9db2cce6b8\") " pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:27.321291 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:27.321259    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba7276d8-e168-437d-adcd-3ea703e97ab1-metrics-certs\") pod \"network-metrics-daemon-v6tb4\" (UID: \"ba7276d8-e168-437d-adcd-3ea703e97ab1\") " pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:12:27.321535 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:27.321392    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:27.321535 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:27.321458    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba7276d8-e168-437d-adcd-3ea703e97ab1-metrics-certs podName:ba7276d8-e168-437d-adcd-3ea703e97ab1 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:29.321440781 +0000 UTC m=+5.066075090 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba7276d8-e168-437d-adcd-3ea703e97ab1-metrics-certs") pod "network-metrics-daemon-v6tb4" (UID: "ba7276d8-e168-437d-adcd-3ea703e97ab1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:27.322143 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:27.321838    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:27.322143 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:27.321890    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25132135-9fbe-49c4-9f6d-0e9db2cce6b8-original-pull-secret podName:25132135-9fbe-49c4-9f6d-0e9db2cce6b8 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:29.321875292 +0000 UTC m=+5.066509600 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25132135-9fbe-49c4-9f6d-0e9db2cce6b8-original-pull-secret") pod "global-pull-secret-syncer-gfsf2" (UID: "25132135-9fbe-49c4-9f6d-0e9db2cce6b8") : object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:27.421839 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:27.421752    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrpwv\" (UniqueName: \"kubernetes.io/projected/1cbe388d-927f-4384-9abd-49d8864afe2c-kube-api-access-qrpwv\") pod \"network-check-target-bbcvm\" (UID: \"1cbe388d-927f-4384-9abd-49d8864afe2c\") " pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:12:27.422039 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:27.421917    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 05 17:12:27.422039 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:27.421935    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 05 17:12:27.422039 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:27.421948    2561 projected.go:196] Error preparing data for projected volume kube-api-access-qrpwv for pod openshift-network-diagnostics/network-check-target-bbcvm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:27.422039 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:27.422028    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1cbe388d-927f-4384-9abd-49d8864afe2c-kube-api-access-qrpwv podName:1cbe388d-927f-4384-9abd-49d8864afe2c nodeName:}" failed. No retries permitted until 2026-08-05 17:12:29.422008852 +0000 UTC m=+5.166643178 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qrpwv" (UniqueName: "kubernetes.io/projected/1cbe388d-927f-4384-9abd-49d8864afe2c-kube-api-access-qrpwv") pod "network-check-target-bbcvm" (UID: "1cbe388d-927f-4384-9abd-49d8864afe2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:27.680964 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:27.680861    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-04 17:07:25 +0000 UTC" deadline="2028-03-21 23:32:03.005177474 +0000 UTC"
Aug 05 17:12:27.680964 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:27.680901    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14262h19m35.324281259s"
Aug 05 17:12:27.724898 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:27.724848    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:27.725129 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:27.725000    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfsf2" podUID="25132135-9fbe-49c4-9f6d-0e9db2cce6b8"
Aug 05 17:12:27.725129 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:27.725089    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:12:27.725273 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:27.725173    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6tb4" podUID="ba7276d8-e168-437d-adcd-3ea703e97ab1"
Aug 05 17:12:27.725273 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:27.725226    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:12:27.725367 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:27.725308    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbcvm" podUID="1cbe388d-927f-4384-9abd-49d8864afe2c"
Aug 05 17:12:29.339609 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:29.339563    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25132135-9fbe-49c4-9f6d-0e9db2cce6b8-original-pull-secret\") pod \"global-pull-secret-syncer-gfsf2\" (UID: \"25132135-9fbe-49c4-9f6d-0e9db2cce6b8\") " pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:29.340197 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:29.339633    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba7276d8-e168-437d-adcd-3ea703e97ab1-metrics-certs\") pod \"network-metrics-daemon-v6tb4\" (UID: \"ba7276d8-e168-437d-adcd-3ea703e97ab1\") " pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:12:29.340197 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:29.339723    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:29.340197 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:29.339798    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25132135-9fbe-49c4-9f6d-0e9db2cce6b8-original-pull-secret podName:25132135-9fbe-49c4-9f6d-0e9db2cce6b8 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:33.339778905 +0000 UTC m=+9.084413234 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25132135-9fbe-49c4-9f6d-0e9db2cce6b8-original-pull-secret") pod "global-pull-secret-syncer-gfsf2" (UID: "25132135-9fbe-49c4-9f6d-0e9db2cce6b8") : object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:29.340197 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:29.339825    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:29.340197 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:29.339864    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba7276d8-e168-437d-adcd-3ea703e97ab1-metrics-certs podName:ba7276d8-e168-437d-adcd-3ea703e97ab1 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:33.339852505 +0000 UTC m=+9.084486825 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba7276d8-e168-437d-adcd-3ea703e97ab1-metrics-certs") pod "network-metrics-daemon-v6tb4" (UID: "ba7276d8-e168-437d-adcd-3ea703e97ab1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:29.440861 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:29.440823    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrpwv\" (UniqueName: \"kubernetes.io/projected/1cbe388d-927f-4384-9abd-49d8864afe2c-kube-api-access-qrpwv\") pod \"network-check-target-bbcvm\" (UID: \"1cbe388d-927f-4384-9abd-49d8864afe2c\") " pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:12:29.441062 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:29.441011    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 05 17:12:29.441062 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:29.441039    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 05 17:12:29.441062 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:29.441056    2561 projected.go:196] Error preparing data for projected volume kube-api-access-qrpwv for pod openshift-network-diagnostics/network-check-target-bbcvm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:29.441229 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:29.441126    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1cbe388d-927f-4384-9abd-49d8864afe2c-kube-api-access-qrpwv podName:1cbe388d-927f-4384-9abd-49d8864afe2c nodeName:}" failed. No retries permitted until 2026-08-05 17:12:33.441104772 +0000 UTC m=+9.185739077 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qrpwv" (UniqueName: "kubernetes.io/projected/1cbe388d-927f-4384-9abd-49d8864afe2c-kube-api-access-qrpwv") pod "network-check-target-bbcvm" (UID: "1cbe388d-927f-4384-9abd-49d8864afe2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:29.723877 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:29.723804    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:12:29.724049 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:29.723973    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6tb4" podUID="ba7276d8-e168-437d-adcd-3ea703e97ab1"
Aug 05 17:12:29.724121 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:29.724047    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:12:29.724172 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:29.724152    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbcvm" podUID="1cbe388d-927f-4384-9abd-49d8864afe2c"
Aug 05 17:12:29.724227 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:29.724052    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:29.724276 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:29.724259    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfsf2" podUID="25132135-9fbe-49c4-9f6d-0e9db2cce6b8"
Aug 05 17:12:31.724893 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:31.724105    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:12:31.724893 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:31.724242    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbcvm" podUID="1cbe388d-927f-4384-9abd-49d8864afe2c"
Aug 05 17:12:31.724893 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:31.724631    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:31.724893 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:31.724724    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfsf2" podUID="25132135-9fbe-49c4-9f6d-0e9db2cce6b8"
Aug 05 17:12:31.724893 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:31.724772    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:12:31.724893 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:31.724851    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6tb4" podUID="ba7276d8-e168-437d-adcd-3ea703e97ab1"
Aug 05 17:12:33.374578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:33.374528    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25132135-9fbe-49c4-9f6d-0e9db2cce6b8-original-pull-secret\") pod \"global-pull-secret-syncer-gfsf2\" (UID: \"25132135-9fbe-49c4-9f6d-0e9db2cce6b8\") " pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:33.375058 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:33.374601    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba7276d8-e168-437d-adcd-3ea703e97ab1-metrics-certs\") pod \"network-metrics-daemon-v6tb4\" (UID: \"ba7276d8-e168-437d-adcd-3ea703e97ab1\") " pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:12:33.375058 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:33.374761    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:33.375058 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:33.374830    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba7276d8-e168-437d-adcd-3ea703e97ab1-metrics-certs podName:ba7276d8-e168-437d-adcd-3ea703e97ab1 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:41.374809036 +0000 UTC m=+17.119443342 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba7276d8-e168-437d-adcd-3ea703e97ab1-metrics-certs") pod "network-metrics-daemon-v6tb4" (UID: "ba7276d8-e168-437d-adcd-3ea703e97ab1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:33.375058 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:33.374890    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:33.375058 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:33.374971    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25132135-9fbe-49c4-9f6d-0e9db2cce6b8-original-pull-secret podName:25132135-9fbe-49c4-9f6d-0e9db2cce6b8 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:41.374940034 +0000 UTC m=+17.119574358 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25132135-9fbe-49c4-9f6d-0e9db2cce6b8-original-pull-secret") pod "global-pull-secret-syncer-gfsf2" (UID: "25132135-9fbe-49c4-9f6d-0e9db2cce6b8") : object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:33.475918 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:33.475870    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrpwv\" (UniqueName: \"kubernetes.io/projected/1cbe388d-927f-4384-9abd-49d8864afe2c-kube-api-access-qrpwv\") pod \"network-check-target-bbcvm\" (UID: \"1cbe388d-927f-4384-9abd-49d8864afe2c\") " pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:12:33.476143 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:33.476114    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 05 17:12:33.476143 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:33.476139    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 05 17:12:33.476251 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:33.476155    2561 projected.go:196] Error preparing data for projected volume kube-api-access-qrpwv for pod openshift-network-diagnostics/network-check-target-bbcvm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:33.476251 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:33.476222    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1cbe388d-927f-4384-9abd-49d8864afe2c-kube-api-access-qrpwv podName:1cbe388d-927f-4384-9abd-49d8864afe2c nodeName:}" failed. No retries permitted until 2026-08-05 17:12:41.476202821 +0000 UTC m=+17.220837131 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qrpwv" (UniqueName: "kubernetes.io/projected/1cbe388d-927f-4384-9abd-49d8864afe2c-kube-api-access-qrpwv") pod "network-check-target-bbcvm" (UID: "1cbe388d-927f-4384-9abd-49d8864afe2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:33.723895 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:33.723822    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:12:33.724056 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:33.723983    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:33.724056 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:33.723995    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbcvm" podUID="1cbe388d-927f-4384-9abd-49d8864afe2c"
Aug 05 17:12:33.724189 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:33.724160    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfsf2" podUID="25132135-9fbe-49c4-9f6d-0e9db2cce6b8"
Aug 05 17:12:33.724327 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:33.724228    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:12:33.724394 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:33.724321    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6tb4" podUID="ba7276d8-e168-437d-adcd-3ea703e97ab1"
Aug 05 17:12:35.723906 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:35.723870    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:12:35.723906 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:35.723898    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:12:35.724414 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:35.724013    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:35.724414 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:35.724035    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6tb4" podUID="ba7276d8-e168-437d-adcd-3ea703e97ab1"
Aug 05 17:12:35.724414 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:35.724114    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfsf2" podUID="25132135-9fbe-49c4-9f6d-0e9db2cce6b8"
Aug 05 17:12:35.724414 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:35.724250    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbcvm" podUID="1cbe388d-927f-4384-9abd-49d8864afe2c"
Aug 05 17:12:37.723653 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:37.723611    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:37.723653 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:37.723643    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:12:37.724240 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:37.723666    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:12:37.724240 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:37.723748    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfsf2" podUID="25132135-9fbe-49c4-9f6d-0e9db2cce6b8"
Aug 05 17:12:37.724240 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:37.723838    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6tb4" podUID="ba7276d8-e168-437d-adcd-3ea703e97ab1"
Aug 05 17:12:37.724240 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:37.723915    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbcvm" podUID="1cbe388d-927f-4384-9abd-49d8864afe2c"
Aug 05 17:12:39.723969 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:39.723754    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:12:39.724437 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:39.723772    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:12:39.724437 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:39.724084    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6tb4" podUID="ba7276d8-e168-437d-adcd-3ea703e97ab1"
Aug 05 17:12:39.724437 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:39.723772    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:39.724437 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:39.724133    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbcvm" podUID="1cbe388d-927f-4384-9abd-49d8864afe2c"
Aug 05 17:12:39.724437 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:39.724222    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfsf2" podUID="25132135-9fbe-49c4-9f6d-0e9db2cce6b8"
Aug 05 17:12:41.439529 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:41.439487    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25132135-9fbe-49c4-9f6d-0e9db2cce6b8-original-pull-secret\") pod \"global-pull-secret-syncer-gfsf2\" (UID: \"25132135-9fbe-49c4-9f6d-0e9db2cce6b8\") " pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:41.439999 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:41.439553    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba7276d8-e168-437d-adcd-3ea703e97ab1-metrics-certs\") pod \"network-metrics-daemon-v6tb4\" (UID: \"ba7276d8-e168-437d-adcd-3ea703e97ab1\") " pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:12:41.439999 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:41.439640    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:41.439999 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:41.439694    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:41.439999 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:41.439709    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25132135-9fbe-49c4-9f6d-0e9db2cce6b8-original-pull-secret podName:25132135-9fbe-49c4-9f6d-0e9db2cce6b8 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:57.43968843 +0000 UTC m=+33.184322758 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25132135-9fbe-49c4-9f6d-0e9db2cce6b8-original-pull-secret") pod "global-pull-secret-syncer-gfsf2" (UID: "25132135-9fbe-49c4-9f6d-0e9db2cce6b8") : object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:41.439999 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:41.439745    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba7276d8-e168-437d-adcd-3ea703e97ab1-metrics-certs podName:ba7276d8-e168-437d-adcd-3ea703e97ab1 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:57.439728768 +0000 UTC m=+33.184363081 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba7276d8-e168-437d-adcd-3ea703e97ab1-metrics-certs") pod "network-metrics-daemon-v6tb4" (UID: "ba7276d8-e168-437d-adcd-3ea703e97ab1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:41.540386 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:41.540351    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrpwv\" (UniqueName: \"kubernetes.io/projected/1cbe388d-927f-4384-9abd-49d8864afe2c-kube-api-access-qrpwv\") pod \"network-check-target-bbcvm\" (UID: \"1cbe388d-927f-4384-9abd-49d8864afe2c\") " pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:12:41.540564 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:41.540504    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 05 17:12:41.540564 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:41.540523    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 05 17:12:41.540564 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:41.540537    2561 projected.go:196] Error preparing data for projected volume kube-api-access-qrpwv for pod openshift-network-diagnostics/network-check-target-bbcvm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:41.540700 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:41.540601    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1cbe388d-927f-4384-9abd-49d8864afe2c-kube-api-access-qrpwv podName:1cbe388d-927f-4384-9abd-49d8864afe2c nodeName:}" failed. No retries permitted until 2026-08-05 17:12:57.5405803 +0000 UTC m=+33.285214609 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qrpwv" (UniqueName: "kubernetes.io/projected/1cbe388d-927f-4384-9abd-49d8864afe2c-kube-api-access-qrpwv") pod "network-check-target-bbcvm" (UID: "1cbe388d-927f-4384-9abd-49d8864afe2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:41.723630 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:41.723546    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:12:41.723832 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:41.723554    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:41.723832 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:41.723684    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbcvm" podUID="1cbe388d-927f-4384-9abd-49d8864afe2c"
Aug 05 17:12:41.723832 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:41.723554    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:12:41.723832 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:41.723756    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfsf2" podUID="25132135-9fbe-49c4-9f6d-0e9db2cce6b8"
Aug 05 17:12:41.724061 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:41.723859    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6tb4" podUID="ba7276d8-e168-437d-adcd-3ea703e97ab1"
Aug 05 17:12:43.723893 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:43.723850    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:12:43.724430 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:43.723860    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:12:43.724430 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:43.724022    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:43.724430 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:43.724015    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6tb4" podUID="ba7276d8-e168-437d-adcd-3ea703e97ab1"
Aug 05 17:12:43.724430 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:43.724136    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbcvm" podUID="1cbe388d-927f-4384-9abd-49d8864afe2c"
Aug 05 17:12:43.724430 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:43.724208    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfsf2" podUID="25132135-9fbe-49c4-9f6d-0e9db2cce6b8"
Aug 05 17:12:45.724043 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:45.723843    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:12:45.724699 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:45.723854    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:12:45.724699 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:45.724114    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbcvm" podUID="1cbe388d-927f-4384-9abd-49d8864afe2c"
Aug 05 17:12:45.724699 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:45.724197    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6tb4" podUID="ba7276d8-e168-437d-adcd-3ea703e97ab1"
Aug 05 17:12:45.724699 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:45.723893    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:45.724699 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:45.724276    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfsf2" podUID="25132135-9fbe-49c4-9f6d-0e9db2cce6b8"
Aug 05 17:12:45.808895 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:45.808862    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-151.ec2.internal" event={"ID":"37213c9548f92e588dc6c041ff6110a8","Type":"ContainerStarted","Data":"fba8d73979fb8ebfce375f573d768adc53cd7d316e38194c972abaa1ca77239f"}
Aug 05 17:12:45.810326 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:45.810300    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qtd" event={"ID":"c6d6169f-097e-40e1-b788-dc230acf07c5","Type":"ContainerStarted","Data":"1f44dde8032bf3fa4099581f3822f455e3fd5b4d2f64601096c1808e65376d5e"}
Aug 05 17:12:45.811786 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:45.811761    2561 generic.go:356] "Generic (PLEG): container finished" podID="66f685fa-db6d-474f-b836-d0d9768e2d09" containerID="599afb720c1488c7ae64230e87ba9d91c68ecb789216e32155c995161cf8ac8f" exitCode=0
Aug 05 17:12:45.811889 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:45.811790    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7kqb" event={"ID":"66f685fa-db6d-474f-b836-d0d9768e2d09","Type":"ContainerDied","Data":"599afb720c1488c7ae64230e87ba9d91c68ecb789216e32155c995161cf8ac8f"}
Aug 05 17:12:45.813369 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:45.813322    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bd6hd" event={"ID":"72ee4123-b8c2-486a-8635-96ffd003756d","Type":"ContainerStarted","Data":"8718a9c2f19a1ebd73b2e6f3f218e56865d48a17ca2e924da9414f9f8c531a18"}
Aug 05 17:12:45.814810 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:45.814789    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r4gx5" event={"ID":"5d7aaa3e-c19a-4838-97af-90e196626895","Type":"ContainerStarted","Data":"159e5522ac7aaf0c918f6109250e018abbdd193fe604eba5d48cedba95292d28"}
Aug 05 17:12:45.816301 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:45.816261    2561 generic.go:356] "Generic (PLEG): container finished" podID="fc8290ea05f84958b6e4f1826fe71dbb" containerID="b6975ac410d020b1dc75f4059f8126c7e7c5cdc8ebb977df7f117c964a91346c" exitCode=0
Aug 05 17:12:45.816397 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:45.816345    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-151.ec2.internal" event={"ID":"fc8290ea05f84958b6e4f1826fe71dbb","Type":"ContainerDied","Data":"b6975ac410d020b1dc75f4059f8126c7e7c5cdc8ebb977df7f117c964a91346c"}
Aug 05 17:12:45.819720 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:45.819699    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-87v9q" event={"ID":"57e503bf-db91-4091-b8de-0b7d65219906","Type":"ContainerStarted","Data":"8c8470f14d864b20d5e8acfd6412bf685cf639accbe07d56896f8b7c66c506eb"}
Aug 05 17:12:45.819812 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:45.819766    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-87v9q" event={"ID":"57e503bf-db91-4091-b8de-0b7d65219906","Type":"ContainerStarted","Data":"0dde3f9d5f880d85b0a87a1f12d9ed821ccd591d908d638751bb89352f69e972"}
Aug 05 17:12:45.819863 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:45.819815    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-87v9q" event={"ID":"57e503bf-db91-4091-b8de-0b7d65219906","Type":"ContainerStarted","Data":"b68bb6d3c600020f9237b31ad761626f77f607f7997ce9aa74e0f145acc6ec08"}
Aug 05 17:12:45.819905 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:45.819864    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-87v9q" event={"ID":"57e503bf-db91-4091-b8de-0b7d65219906","Type":"ContainerStarted","Data":"c3e5c2b79d05db92c20e0ade635bb0588c3c9d91f58f0a43eb4cb4b860a4e615"}
Aug 05 17:12:45.819941 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:45.819908    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-87v9q" event={"ID":"57e503bf-db91-4091-b8de-0b7d65219906","Type":"ContainerStarted","Data":"90194c48ec647bdf0c9a3a3cf8acaaee89b12d05940a9afaff8464c38aaa26b9"}
Aug 05 17:12:45.820017 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:45.819945    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-87v9q" event={"ID":"57e503bf-db91-4091-b8de-0b7d65219906","Type":"ContainerStarted","Data":"678782965c2df5923653d2fb15609248a6196883a04ae6631797c15bd3281fcc"}
Aug 05 17:12:45.821070 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:45.821050    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8vtq4" event={"ID":"8be1b0df-b531-4179-918f-1cae637ce3f7","Type":"ContainerStarted","Data":"d249f578172daa2dac9d6063c4840799b484b3c464721996594c0005f462c66d"}
Aug 05 17:12:45.822475 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:45.822455    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tg9l4" event={"ID":"eacfb8d3-1760-45ee-a594-7656a55c9d93","Type":"ContainerStarted","Data":"c3ae90864d87542adadc16eba24dd9366a0b4e95b273f17efd747da295ef010b"}
Aug 05 17:12:45.828506 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:45.828463    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-151.ec2.internal" podStartSLOduration=20.828450965000002 podStartE2EDuration="20.828450965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:12:45.828197018 +0000 UTC m=+21.572831348" watchObservedRunningTime="2026-08-05 17:12:45.828450965 +0000 UTC m=+21.573085284"
Aug 05 17:12:45.850672 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:45.850627    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-r4gx5" podStartSLOduration=2.6815144760000003 podStartE2EDuration="21.850612893s" totalImagesPullingTime="19.169098417s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:12:45.850452556 +0000 UTC m=+21.595086884" watchObservedRunningTime="2026-08-05 17:12:45.850612893 +0000 UTC m=+21.595247220"
Aug 05 17:12:45.894763 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:45.894486    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tg9l4" podStartSLOduration=2.856166342 podStartE2EDuration="21.894468632s" totalImagesPullingTime="19.03830229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:12:45.877939898 +0000 UTC m=+21.622574299" watchObservedRunningTime="2026-08-05 17:12:45.894468632 +0000 UTC m=+21.639102959"
Aug 05 17:12:45.943699 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:45.943646    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bd6hd" podStartSLOduration=2.887784153 podStartE2EDuration="21.943628977s" totalImagesPullingTime="19.055844824s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:12:45.913132254 +0000 UTC m=+21.657766581" watchObservedRunningTime="2026-08-05 17:12:45.943628977 +0000 UTC m=+21.688263306"
Aug 05 17:12:45.960216 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:45.960172    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8vtq4" podStartSLOduration=2.963836248 podStartE2EDuration="21.960157307s" totalImagesPullingTime="18.996321059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:12:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:12:45.959840809 +0000 UTC m=+21.704475141" watchObservedRunningTime="2026-08-05 17:12:45.960157307 +0000 UTC m=+21.704791634"
Aug 05 17:12:46.825820 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:46.825785    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-thg2c" event={"ID":"e32781be-78b9-42c2-b56c-ac2c3f919e1b","Type":"ContainerStarted","Data":"d2b5a1c07d1596972a0bf32dd0c4bed9ea3cd56e738b4c4b057ecd570dc73e74"}
Aug 05 17:12:46.827845 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:46.827816    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-151.ec2.internal" event={"ID":"fc8290ea05f84958b6e4f1826fe71dbb","Type":"ContainerStarted","Data":"c86319f51b5f4e44d0dca458517fa56bfb4de2e3626c818ad6c03e53b16b1c14"}
Aug 05 17:12:46.842916 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:46.842855    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-thg2c" podStartSLOduration=3.890588957 podStartE2EDuration="22.842836473s" totalImagesPullingTime="18.952247516s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:12:46.84230556 +0000 UTC m=+22.586939891" watchObservedRunningTime="2026-08-05 17:12:46.842836473 +0000 UTC m=+22.587470802"
Aug 05 17:12:46.866097 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:46.866067    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Aug 05 17:12:47.666856 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:47.666686    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-05T17:12:46.866087791Z","UUID":"b562b0b4-90ad-4515-950f-69f383252cb9","Handler":null,"Name":"","Endpoint":""}
Aug 05 17:12:47.668863 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:47.668744    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Aug 05 17:12:47.668863 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:47.668788    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Aug 05 17:12:47.724301 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:47.724263    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:12:47.724301 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:47.724298    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:47.724557 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:47.724367    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:12:47.724557 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:47.724381    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbcvm" podUID="1cbe388d-927f-4384-9abd-49d8864afe2c"
Aug 05 17:12:47.724557 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:47.724437    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfsf2" podUID="25132135-9fbe-49c4-9f6d-0e9db2cce6b8"
Aug 05 17:12:47.724694 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:47.724559    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6tb4" podUID="ba7276d8-e168-437d-adcd-3ea703e97ab1"
Aug 05 17:12:47.832474 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:47.832435    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qtd" event={"ID":"c6d6169f-097e-40e1-b788-dc230acf07c5","Type":"ContainerStarted","Data":"00755d8991580c70a287225f47b4586973726ba16eb236cb38ac5371d4a6e8b9"}
Aug 05 17:12:48.376726 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:48.376689    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8vtq4"
Aug 05 17:12:48.377263 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:48.377246    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8vtq4"
Aug 05 17:12:48.396241 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:48.396195    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-151.ec2.internal" podStartSLOduration=23.39617907 podStartE2EDuration="23.39617907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:12:46.86356711 +0000 UTC m=+22.608201440" watchObservedRunningTime="2026-08-05 17:12:48.39617907 +0000 UTC m=+24.140813396"
Aug 05 17:12:48.838846 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:48.838813    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-87v9q" event={"ID":"57e503bf-db91-4091-b8de-0b7d65219906","Type":"ContainerStarted","Data":"01116ae36d734a9cfb32751ae2381919dec1382f8c87f969a6c3686918af856f"}
Aug 05 17:12:48.839336 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:48.838919    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8vtq4"
Aug 05 17:12:48.839612 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:48.839595    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8vtq4"
Aug 05 17:12:49.724469 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:49.724433    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:12:49.724469 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:49.724458    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:12:49.724755 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:49.724570    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:49.724755 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:49.724589    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbcvm" podUID="1cbe388d-927f-4384-9abd-49d8864afe2c"
Aug 05 17:12:49.724755 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:49.724653    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfsf2" podUID="25132135-9fbe-49c4-9f6d-0e9db2cce6b8"
Aug 05 17:12:49.724755 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:49.724739    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6tb4" podUID="ba7276d8-e168-437d-adcd-3ea703e97ab1"
Aug 05 17:12:50.844686 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:50.844466    2561 generic.go:356] "Generic (PLEG): container finished" podID="66f685fa-db6d-474f-b836-d0d9768e2d09" containerID="e0f0c941c0b020627c526ac008ca5290a492c8c3ccd691fde7c480beae3c441d" exitCode=0
Aug 05 17:12:50.845356 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:50.844542    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7kqb" event={"ID":"66f685fa-db6d-474f-b836-d0d9768e2d09","Type":"ContainerDied","Data":"e0f0c941c0b020627c526ac008ca5290a492c8c3ccd691fde7c480beae3c441d"}
Aug 05 17:12:50.848771 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:50.848747    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-87v9q" event={"ID":"57e503bf-db91-4091-b8de-0b7d65219906","Type":"ContainerStarted","Data":"e3f1bcd8be61e126f38e487acbf2a2a5eb9b3cc12642b98bc8fb6a96c7beb877"}
Aug 05 17:12:50.849082 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:50.849055    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:50.849219 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:50.849125    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:50.851440 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:50.851412    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qtd" event={"ID":"c6d6169f-097e-40e1-b788-dc230acf07c5","Type":"ContainerStarted","Data":"846265dfa93088a74068d4a676250917ff0d8c3f07710d3a412afd608b93e8f5"}
Aug 05 17:12:50.864526 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:50.864503    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:50.864624 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:50.864596    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:50.892524 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:50.892477    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-87v9q" podStartSLOduration=7.880128022 podStartE2EDuration="26.892462736s" totalImagesPullingTime="19.012334714s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:12:50.891094292 +0000 UTC m=+26.635728619" watchObservedRunningTime="2026-08-05 17:12:50.892462736 +0000 UTC m=+26.637097113"
Aug 05 17:12:50.908288 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:50.908246    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j5qtd" podStartSLOduration=3.3057621409999998 podStartE2EDuration="26.908232799s" totalImagesPullingTime="23.602470658s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:12:50.907982592 +0000 UTC m=+26.652616920" watchObservedRunningTime="2026-08-05 17:12:50.908232799 +0000 UTC m=+26.652867139"
Aug 05 17:12:51.259354 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:51.259268    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:12:51.724488 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:51.724449    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:12:51.724655 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:51.724449    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:12:51.724655 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:51.724567    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6tb4" podUID="ba7276d8-e168-437d-adcd-3ea703e97ab1"
Aug 05 17:12:51.724655 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:51.724462    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:51.724749 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:51.724616    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbcvm" podUID="1cbe388d-927f-4384-9abd-49d8864afe2c"
Aug 05 17:12:51.724749 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:51.724703    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfsf2" podUID="25132135-9fbe-49c4-9f6d-0e9db2cce6b8"
Aug 05 17:12:52.357510 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:52.357270    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bbcvm"]
Aug 05 17:12:52.358095 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:52.357602    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:12:52.358095 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:52.357705    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbcvm" podUID="1cbe388d-927f-4384-9abd-49d8864afe2c"
Aug 05 17:12:52.360707 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:52.360680    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gfsf2"]
Aug 05 17:12:52.360818 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:52.360773    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:52.360895 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:52.360874    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfsf2" podUID="25132135-9fbe-49c4-9f6d-0e9db2cce6b8"
Aug 05 17:12:52.361265 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:52.361243    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v6tb4"]
Aug 05 17:12:52.361370 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:52.361330    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:12:52.361444 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:52.361410    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6tb4" podUID="ba7276d8-e168-437d-adcd-3ea703e97ab1"
Aug 05 17:12:52.857867 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:52.857834    2561 generic.go:356] "Generic (PLEG): container finished" podID="66f685fa-db6d-474f-b836-d0d9768e2d09" containerID="c54c457520f3c88d8bd2e12e7e24bb0e5754565576d6cc888e161b779693e065" exitCode=0
Aug 05 17:12:52.858086 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:52.857923    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7kqb" event={"ID":"66f685fa-db6d-474f-b836-d0d9768e2d09","Type":"ContainerDied","Data":"c54c457520f3c88d8bd2e12e7e24bb0e5754565576d6cc888e161b779693e065"}
Aug 05 17:12:53.723900 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:53.723865    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:53.724282 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:53.723863    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:12:53.724282 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:53.724003    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfsf2" podUID="25132135-9fbe-49c4-9f6d-0e9db2cce6b8"
Aug 05 17:12:53.724282 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:53.724033    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbcvm" podUID="1cbe388d-927f-4384-9abd-49d8864afe2c"
Aug 05 17:12:53.862932 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:53.862896    2561 generic.go:356] "Generic (PLEG): container finished" podID="66f685fa-db6d-474f-b836-d0d9768e2d09" containerID="6295f3bd72dae1c116111f4b685de98d779d0082c74f5f2e3961d309d755b95a" exitCode=0
Aug 05 17:12:53.863087 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:53.862986    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7kqb" event={"ID":"66f685fa-db6d-474f-b836-d0d9768e2d09","Type":"ContainerDied","Data":"6295f3bd72dae1c116111f4b685de98d779d0082c74f5f2e3961d309d755b95a"}
Aug 05 17:12:54.725216 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:54.725130    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:12:54.725694 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:54.725247    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6tb4" podUID="ba7276d8-e168-437d-adcd-3ea703e97ab1"
Aug 05 17:12:55.724246 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:55.724214    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:12:55.724246 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:55.724255    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:55.724444 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:55.724342    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbcvm" podUID="1cbe388d-927f-4384-9abd-49d8864afe2c"
Aug 05 17:12:55.724552 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:55.724502    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfsf2" podUID="25132135-9fbe-49c4-9f6d-0e9db2cce6b8"
Aug 05 17:12:55.815407 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:55.815374    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-vjnk7"]
Aug 05 17:12:55.817412 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:55.817380    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vjnk7"
Aug 05 17:12:55.820985 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:55.820931    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cfcdk\""
Aug 05 17:12:55.821138 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:55.821113    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Aug 05 17:12:55.821362 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:55.821348    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Aug 05 17:12:55.946470 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:55.946434    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2zv5\" (UniqueName: \"kubernetes.io/projected/47a3441a-cffb-4919-95b8-6393fd63c14b-kube-api-access-n2zv5\") pod \"node-resolver-vjnk7\" (UID: \"47a3441a-cffb-4919-95b8-6393fd63c14b\") " pod="openshift-dns/node-resolver-vjnk7"
Aug 05 17:12:55.946625 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:55.946537    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/47a3441a-cffb-4919-95b8-6393fd63c14b-hosts-file\") pod \"node-resolver-vjnk7\" (UID: \"47a3441a-cffb-4919-95b8-6393fd63c14b\") " pod="openshift-dns/node-resolver-vjnk7"
Aug 05 17:12:55.946625 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:55.946588    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47a3441a-cffb-4919-95b8-6393fd63c14b-tmp-dir\") pod \"node-resolver-vjnk7\" (UID: \"47a3441a-cffb-4919-95b8-6393fd63c14b\") " pod="openshift-dns/node-resolver-vjnk7"
Aug 05 17:12:56.047896 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:56.047864    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2zv5\" (UniqueName: \"kubernetes.io/projected/47a3441a-cffb-4919-95b8-6393fd63c14b-kube-api-access-n2zv5\") pod \"node-resolver-vjnk7\" (UID: \"47a3441a-cffb-4919-95b8-6393fd63c14b\") " pod="openshift-dns/node-resolver-vjnk7"
Aug 05 17:12:56.048087 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:56.047949    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/47a3441a-cffb-4919-95b8-6393fd63c14b-hosts-file\") pod \"node-resolver-vjnk7\" (UID: \"47a3441a-cffb-4919-95b8-6393fd63c14b\") " pod="openshift-dns/node-resolver-vjnk7"
Aug 05 17:12:56.048087 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:56.048016    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47a3441a-cffb-4919-95b8-6393fd63c14b-tmp-dir\") pod \"node-resolver-vjnk7\" (UID: \"47a3441a-cffb-4919-95b8-6393fd63c14b\") " pod="openshift-dns/node-resolver-vjnk7"
Aug 05 17:12:56.048087 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:56.048051    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/47a3441a-cffb-4919-95b8-6393fd63c14b-hosts-file\") pod \"node-resolver-vjnk7\" (UID: \"47a3441a-cffb-4919-95b8-6393fd63c14b\") " pod="openshift-dns/node-resolver-vjnk7"
Aug 05 17:12:56.048515 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:56.048495    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47a3441a-cffb-4919-95b8-6393fd63c14b-tmp-dir\") pod \"node-resolver-vjnk7\" (UID: \"47a3441a-cffb-4919-95b8-6393fd63c14b\") " pod="openshift-dns/node-resolver-vjnk7"
Aug 05 17:12:56.059429 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:56.059399    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2zv5\" (UniqueName: \"kubernetes.io/projected/47a3441a-cffb-4919-95b8-6393fd63c14b-kube-api-access-n2zv5\") pod \"node-resolver-vjnk7\" (UID: \"47a3441a-cffb-4919-95b8-6393fd63c14b\") " pod="openshift-dns/node-resolver-vjnk7"
Aug 05 17:12:56.128620 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:56.128540    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vjnk7"
Aug 05 17:12:56.139306 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:56.139042    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a3441a_cffb_4919_95b8_6393fd63c14b.slice/crio-d0c00f3b92864db43bdb0e6cc7cdb71f92f948d7b372caac4dc6cca9f561b036 WatchSource:0}: Error finding container d0c00f3b92864db43bdb0e6cc7cdb71f92f948d7b372caac4dc6cca9f561b036: Status 404 returned error can't find the container with id d0c00f3b92864db43bdb0e6cc7cdb71f92f948d7b372caac4dc6cca9f561b036
Aug 05 17:12:56.724031 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:56.723999    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:12:56.724202 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:56.724143    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6tb4" podUID="ba7276d8-e168-437d-adcd-3ea703e97ab1"
Aug 05 17:12:56.872711 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:56.872662    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vjnk7" event={"ID":"47a3441a-cffb-4919-95b8-6393fd63c14b","Type":"ContainerStarted","Data":"bd7269592973d0c852711e3cecd0f2a93b0ad9118769c30ca6833cfcf4989188"}
Aug 05 17:12:56.873250 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:56.872742    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vjnk7" event={"ID":"47a3441a-cffb-4919-95b8-6393fd63c14b","Type":"ContainerStarted","Data":"d0c00f3b92864db43bdb0e6cc7cdb71f92f948d7b372caac4dc6cca9f561b036"}
Aug 05 17:12:57.458845 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:57.458813    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba7276d8-e168-437d-adcd-3ea703e97ab1-metrics-certs\") pod \"network-metrics-daemon-v6tb4\" (UID: \"ba7276d8-e168-437d-adcd-3ea703e97ab1\") " pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:12:57.459047 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:57.458918    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25132135-9fbe-49c4-9f6d-0e9db2cce6b8-original-pull-secret\") pod \"global-pull-secret-syncer-gfsf2\" (UID: \"25132135-9fbe-49c4-9f6d-0e9db2cce6b8\") " pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:57.459047 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:57.459039    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:57.459167 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:57.459122    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba7276d8-e168-437d-adcd-3ea703e97ab1-metrics-certs podName:ba7276d8-e168-437d-adcd-3ea703e97ab1 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:29.459100508 +0000 UTC m=+65.203734814 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba7276d8-e168-437d-adcd-3ea703e97ab1-metrics-certs") pod "network-metrics-daemon-v6tb4" (UID: "ba7276d8-e168-437d-adcd-3ea703e97ab1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 05 17:12:57.459167 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:57.459043    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:57.459272 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:57.459220    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25132135-9fbe-49c4-9f6d-0e9db2cce6b8-original-pull-secret podName:25132135-9fbe-49c4-9f6d-0e9db2cce6b8 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:29.45919937 +0000 UTC m=+65.203833679 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/25132135-9fbe-49c4-9f6d-0e9db2cce6b8-original-pull-secret") pod "global-pull-secret-syncer-gfsf2" (UID: "25132135-9fbe-49c4-9f6d-0e9db2cce6b8") : object "kube-system"/"original-pull-secret" not registered
Aug 05 17:12:57.559801 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:57.559761    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrpwv\" (UniqueName: \"kubernetes.io/projected/1cbe388d-927f-4384-9abd-49d8864afe2c-kube-api-access-qrpwv\") pod \"network-check-target-bbcvm\" (UID: \"1cbe388d-927f-4384-9abd-49d8864afe2c\") " pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:12:57.560316 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:57.559888    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 05 17:12:57.560316 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:57.559910    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 05 17:12:57.560316 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:57.559920    2561 projected.go:196] Error preparing data for projected volume kube-api-access-qrpwv for pod openshift-network-diagnostics/network-check-target-bbcvm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:57.560316 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:57.559998    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1cbe388d-927f-4384-9abd-49d8864afe2c-kube-api-access-qrpwv podName:1cbe388d-927f-4384-9abd-49d8864afe2c nodeName:}" failed. No retries permitted until 2026-08-05 17:13:29.559980022 +0000 UTC m=+65.304614348 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qrpwv" (UniqueName: "kubernetes.io/projected/1cbe388d-927f-4384-9abd-49d8864afe2c-kube-api-access-qrpwv") pod "network-check-target-bbcvm" (UID: "1cbe388d-927f-4384-9abd-49d8864afe2c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 05 17:12:57.723994 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:57.723886    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:57.723994 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:57.723886    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:12:57.724256 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:57.724024    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbcvm" podUID="1cbe388d-927f-4384-9abd-49d8864afe2c"
Aug 05 17:12:57.724256 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:57.724174    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfsf2" podUID="25132135-9fbe-49c4-9f6d-0e9db2cce6b8"
Aug 05 17:12:58.109593 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.109571    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-151.ec2.internal" event="NodeReady"
Aug 05 17:12:58.110025 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.109704    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Aug 05 17:12:58.149706 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.149596    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vjnk7" podStartSLOduration=3.149578929 podStartE2EDuration="3.149578929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:12:56.891871926 +0000 UTC m=+32.636506252" watchObservedRunningTime="2026-08-05 17:12:58.149578929 +0000 UTC m=+33.894213326"
Aug 05 17:12:58.149895 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.149780    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76b686c878-267wj"]
Aug 05 17:12:58.172913 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.172888    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4bxkx"]
Aug 05 17:12:58.173051 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.173024    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:12:58.177097 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.176994    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Aug 05 17:12:58.177097 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.177007    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Aug 05 17:12:58.177313 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.177293    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Aug 05 17:12:58.177391 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.177376    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9s4jc\""
Aug 05 17:12:58.184163 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.184141    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Aug 05 17:12:58.189261 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.189235    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pcknb"]
Aug 05 17:12:58.189390 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.189359    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4bxkx"
Aug 05 17:12:58.192195 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.192175    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Aug 05 17:12:58.192375 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.192179    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mrwwg\""
Aug 05 17:12:58.192640 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.192624    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Aug 05 17:12:58.207638 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.207617    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76b686c878-267wj"]
Aug 05 17:12:58.207752 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.207647    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4bxkx"]
Aug 05 17:12:58.207752 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.207665    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pcknb"]
Aug 05 17:12:58.207752 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.207723    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pcknb"
Aug 05 17:12:58.212631 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.212585    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Aug 05 17:12:58.213603 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.213585    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Aug 05 17:12:58.213771 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.213747    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Aug 05 17:12:58.213873 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.213775    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tvn52\""
Aug 05 17:12:58.265099 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.265065    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aba3d093-7afd-4f0e-8407-895e98526798-trusted-ca\") pod \"image-registry-76b686c878-267wj\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") " pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:12:58.265337 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.265106    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aba3d093-7afd-4f0e-8407-895e98526798-image-registry-private-configuration\") pod \"image-registry-76b686c878-267wj\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") " pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:12:58.265337 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.265131    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aba3d093-7afd-4f0e-8407-895e98526798-installation-pull-secrets\") pod \"image-registry-76b686c878-267wj\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") " pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:12:58.265337 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.265257    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00d8f640-e14d-4601-b45d-81ef6fbaf47d-metrics-tls\") pod \"dns-default-4bxkx\" (UID: \"00d8f640-e14d-4601-b45d-81ef6fbaf47d\") " pod="openshift-dns/dns-default-4bxkx"
Aug 05 17:12:58.265337 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.265293    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-registry-tls\") pod \"image-registry-76b686c878-267wj\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") " pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:12:58.265534 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.265345    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aba3d093-7afd-4f0e-8407-895e98526798-registry-certificates\") pod \"image-registry-76b686c878-267wj\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") " pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:12:58.265534 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.265372    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-bound-sa-token\") pod \"image-registry-76b686c878-267wj\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") " pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:12:58.265534 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.265422    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00d8f640-e14d-4601-b45d-81ef6fbaf47d-tmp-dir\") pod \"dns-default-4bxkx\" (UID: \"00d8f640-e14d-4601-b45d-81ef6fbaf47d\") " pod="openshift-dns/dns-default-4bxkx"
Aug 05 17:12:58.265534 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.265453    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/00d8f640-e14d-4601-b45d-81ef6fbaf47d-config-volume\") pod \"dns-default-4bxkx\" (UID: \"00d8f640-e14d-4601-b45d-81ef6fbaf47d\") " pod="openshift-dns/dns-default-4bxkx"
Aug 05 17:12:58.265534 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.265468    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aba3d093-7afd-4f0e-8407-895e98526798-ca-trust-extracted\") pod \"image-registry-76b686c878-267wj\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") " pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:12:58.265534 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.265519    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znwg2\" (UniqueName: \"kubernetes.io/projected/00d8f640-e14d-4601-b45d-81ef6fbaf47d-kube-api-access-znwg2\") pod \"dns-default-4bxkx\" (UID: \"00d8f640-e14d-4601-b45d-81ef6fbaf47d\") " pod="openshift-dns/dns-default-4bxkx"
Aug 05 17:12:58.265746 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.265552    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7qdd\" (UniqueName: \"kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-kube-api-access-b7qdd\") pod \"image-registry-76b686c878-267wj\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") " pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:12:58.366494 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.366417    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aba3d093-7afd-4f0e-8407-895e98526798-image-registry-private-configuration\") pod \"image-registry-76b686c878-267wj\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") " pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:12:58.366494 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.366450    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aba3d093-7afd-4f0e-8407-895e98526798-installation-pull-secrets\") pod \"image-registry-76b686c878-267wj\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") " pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:12:58.366494 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.366476    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00d8f640-e14d-4601-b45d-81ef6fbaf47d-metrics-tls\") pod \"dns-default-4bxkx\" (UID: \"00d8f640-e14d-4601-b45d-81ef6fbaf47d\") " pod="openshift-dns/dns-default-4bxkx"
Aug 05 17:12:58.366772 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.366584    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-registry-tls\") pod \"image-registry-76b686c878-267wj\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") " pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:12:58.366772 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.366633    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aba3d093-7afd-4f0e-8407-895e98526798-registry-certificates\") pod \"image-registry-76b686c878-267wj\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") " pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:12:58.366772 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.366658    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-bound-sa-token\") pod \"image-registry-76b686c878-267wj\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") " pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:12:58.366772 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.366695    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00d8f640-e14d-4601-b45d-81ef6fbaf47d-tmp-dir\") pod \"dns-default-4bxkx\" (UID: \"00d8f640-e14d-4601-b45d-81ef6fbaf47d\") " pod="openshift-dns/dns-default-4bxkx"
Aug 05 17:12:58.366772 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:58.366729    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 05 17:12:58.366772 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:58.366741    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 05 17:12:58.366772 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:58.366760    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76b686c878-267wj: secret "image-registry-tls" not found
Aug 05 17:12:58.367140 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:58.366802    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00d8f640-e14d-4601-b45d-81ef6fbaf47d-metrics-tls podName:00d8f640-e14d-4601-b45d-81ef6fbaf47d nodeName:}" failed. No retries permitted until 2026-08-05 17:12:58.866780462 +0000 UTC m=+34.611414775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/00d8f640-e14d-4601-b45d-81ef6fbaf47d-metrics-tls") pod "dns-default-4bxkx" (UID: "00d8f640-e14d-4601-b45d-81ef6fbaf47d") : secret "dns-default-metrics-tls" not found
Aug 05 17:12:58.367140 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.366736    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/00d8f640-e14d-4601-b45d-81ef6fbaf47d-config-volume\") pod \"dns-default-4bxkx\" (UID: \"00d8f640-e14d-4601-b45d-81ef6fbaf47d\") " pod="openshift-dns/dns-default-4bxkx"
Aug 05 17:12:58.367140 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:58.366821    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-registry-tls podName:aba3d093-7afd-4f0e-8407-895e98526798 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:58.86681176 +0000 UTC m=+34.611446079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-registry-tls") pod "image-registry-76b686c878-267wj" (UID: "aba3d093-7afd-4f0e-8407-895e98526798") : secret "image-registry-tls" not found
Aug 05 17:12:58.367140 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.366853    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz727\" (UniqueName: \"kubernetes.io/projected/50a0dce8-3a32-4b5e-a07a-7a4492cd5166-kube-api-access-gz727\") pod \"ingress-canary-pcknb\" (UID: \"50a0dce8-3a32-4b5e-a07a-7a4492cd5166\") " pod="openshift-ingress-canary/ingress-canary-pcknb"
Aug 05 17:12:58.367140 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.366887    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aba3d093-7afd-4f0e-8407-895e98526798-ca-trust-extracted\") pod \"image-registry-76b686c878-267wj\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") " pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:12:58.367140 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.366942    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znwg2\" (UniqueName: \"kubernetes.io/projected/00d8f640-e14d-4601-b45d-81ef6fbaf47d-kube-api-access-znwg2\") pod \"dns-default-4bxkx\" (UID: \"00d8f640-e14d-4601-b45d-81ef6fbaf47d\") " pod="openshift-dns/dns-default-4bxkx"
Aug 05 17:12:58.367140 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.367007    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7qdd\" (UniqueName: \"kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-kube-api-access-b7qdd\") pod \"image-registry-76b686c878-267wj\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") " pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:12:58.367140 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.367054    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aba3d093-7afd-4f0e-8407-895e98526798-trusted-ca\") pod \"image-registry-76b686c878-267wj\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") " pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:12:58.367140 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.367079    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50a0dce8-3a32-4b5e-a07a-7a4492cd5166-cert\") pod \"ingress-canary-pcknb\" (UID: \"50a0dce8-3a32-4b5e-a07a-7a4492cd5166\") " pod="openshift-ingress-canary/ingress-canary-pcknb"
Aug 05 17:12:58.367140 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.367119    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00d8f640-e14d-4601-b45d-81ef6fbaf47d-tmp-dir\") pod \"dns-default-4bxkx\" (UID: \"00d8f640-e14d-4601-b45d-81ef6fbaf47d\") " pod="openshift-dns/dns-default-4bxkx"
Aug 05 17:12:58.367657 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.367283    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aba3d093-7afd-4f0e-8407-895e98526798-registry-certificates\") pod \"image-registry-76b686c878-267wj\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") " pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:12:58.367657 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.367475    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/00d8f640-e14d-4601-b45d-81ef6fbaf47d-config-volume\") pod \"dns-default-4bxkx\" (UID: \"00d8f640-e14d-4601-b45d-81ef6fbaf47d\") " pod="openshift-dns/dns-default-4bxkx"
Aug 05 17:12:58.367657 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.367530    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aba3d093-7afd-4f0e-8407-895e98526798-ca-trust-extracted\") pod \"image-registry-76b686c878-267wj\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") " pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:12:58.367903 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.367885    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aba3d093-7afd-4f0e-8407-895e98526798-trusted-ca\") pod \"image-registry-76b686c878-267wj\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") " pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:12:58.369431 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.369414    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aba3d093-7afd-4f0e-8407-895e98526798-installation-pull-secrets\") pod \"image-registry-76b686c878-267wj\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") " pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:12:58.369495 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.369468    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aba3d093-7afd-4f0e-8407-895e98526798-image-registry-private-configuration\") pod \"image-registry-76b686c878-267wj\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") " pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:12:58.377175 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.377150    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-bound-sa-token\") pod \"image-registry-76b686c878-267wj\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") " pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:12:58.377991 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.377973    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7qdd\" (UniqueName: \"kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-kube-api-access-b7qdd\") pod \"image-registry-76b686c878-267wj\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") " pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:12:58.378079 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.377975    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znwg2\" (UniqueName: \"kubernetes.io/projected/00d8f640-e14d-4601-b45d-81ef6fbaf47d-kube-api-access-znwg2\") pod \"dns-default-4bxkx\" (UID: \"00d8f640-e14d-4601-b45d-81ef6fbaf47d\") " pod="openshift-dns/dns-default-4bxkx"
Aug 05 17:12:58.467631 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.467598    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz727\" (UniqueName: \"kubernetes.io/projected/50a0dce8-3a32-4b5e-a07a-7a4492cd5166-kube-api-access-gz727\") pod \"ingress-canary-pcknb\" (UID: \"50a0dce8-3a32-4b5e-a07a-7a4492cd5166\") " pod="openshift-ingress-canary/ingress-canary-pcknb"
Aug 05 17:12:58.467816 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.467648    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50a0dce8-3a32-4b5e-a07a-7a4492cd5166-cert\") pod \"ingress-canary-pcknb\" (UID: \"50a0dce8-3a32-4b5e-a07a-7a4492cd5166\") " pod="openshift-ingress-canary/ingress-canary-pcknb"
Aug 05 17:12:58.475200 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.475176    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50a0dce8-3a32-4b5e-a07a-7a4492cd5166-cert\") pod \"ingress-canary-pcknb\" (UID: \"50a0dce8-3a32-4b5e-a07a-7a4492cd5166\") " pod="openshift-ingress-canary/ingress-canary-pcknb"
Aug 05 17:12:58.478292 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.478274    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz727\" (UniqueName: \"kubernetes.io/projected/50a0dce8-3a32-4b5e-a07a-7a4492cd5166-kube-api-access-gz727\") pod \"ingress-canary-pcknb\" (UID: \"50a0dce8-3a32-4b5e-a07a-7a4492cd5166\") " pod="openshift-ingress-canary/ingress-canary-pcknb"
Aug 05 17:12:58.516430 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.516397    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pcknb"
Aug 05 17:12:58.686231 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.686197    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pcknb"]
Aug 05 17:12:58.690338 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:12:58.690311    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a0dce8_3a32_4b5e_a07a_7a4492cd5166.slice/crio-141ba504cc16a971623e31fd4cd7965c50cedf1c5437dd9bfa1ae2a8de7d2798 WatchSource:0}: Error finding container 141ba504cc16a971623e31fd4cd7965c50cedf1c5437dd9bfa1ae2a8de7d2798: Status 404 returned error can't find the container with id 141ba504cc16a971623e31fd4cd7965c50cedf1c5437dd9bfa1ae2a8de7d2798
Aug 05 17:12:58.724532 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.724497    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:12:58.727183 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.727161    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vwprk\""
Aug 05 17:12:58.727183 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.727172    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 05 17:12:58.871650 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.871614    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00d8f640-e14d-4601-b45d-81ef6fbaf47d-metrics-tls\") pod \"dns-default-4bxkx\" (UID: \"00d8f640-e14d-4601-b45d-81ef6fbaf47d\") " pod="openshift-dns/dns-default-4bxkx"
Aug 05 17:12:58.871891 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.871654    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-registry-tls\") pod \"image-registry-76b686c878-267wj\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") " pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:12:58.871891 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:58.871779    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 05 17:12:58.871891 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:58.871842    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00d8f640-e14d-4601-b45d-81ef6fbaf47d-metrics-tls podName:00d8f640-e14d-4601-b45d-81ef6fbaf47d nodeName:}" failed. No retries permitted until 2026-08-05 17:12:59.871821906 +0000 UTC m=+35.616456231 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/00d8f640-e14d-4601-b45d-81ef6fbaf47d-metrics-tls") pod "dns-default-4bxkx" (UID: "00d8f640-e14d-4601-b45d-81ef6fbaf47d") : secret "dns-default-metrics-tls" not found
Aug 05 17:12:58.871891 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:58.871841    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 05 17:12:58.871891 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:58.871858    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76b686c878-267wj: secret "image-registry-tls" not found
Aug 05 17:12:58.871891 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:58.871887    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-registry-tls podName:aba3d093-7afd-4f0e-8407-895e98526798 nodeName:}" failed. No retries permitted until 2026-08-05 17:12:59.871878015 +0000 UTC m=+35.616512337 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-registry-tls") pod "image-registry-76b686c878-267wj" (UID: "aba3d093-7afd-4f0e-8407-895e98526798") : secret "image-registry-tls" not found
Aug 05 17:12:58.878500 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.878471    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pcknb" event={"ID":"50a0dce8-3a32-4b5e-a07a-7a4492cd5166","Type":"ContainerStarted","Data":"141ba504cc16a971623e31fd4cd7965c50cedf1c5437dd9bfa1ae2a8de7d2798"}
Aug 05 17:12:58.881126 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.881099    2561 generic.go:356] "Generic (PLEG): container finished" podID="66f685fa-db6d-474f-b836-d0d9768e2d09" containerID="ddf77b4e96a17a80902cbbe02d3453813a0b1c20d851174ec29cdceaf88901bf" exitCode=0
Aug 05 17:12:58.881241 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:58.881141    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7kqb" event={"ID":"66f685fa-db6d-474f-b836-d0d9768e2d09","Type":"ContainerDied","Data":"ddf77b4e96a17a80902cbbe02d3453813a0b1c20d851174ec29cdceaf88901bf"}
Aug 05 17:12:59.724315 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:59.724281    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:12:59.724315 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:59.724314    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:12:59.729711 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:59.729685    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Aug 05 17:12:59.729711 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:59.729688    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 05 17:12:59.729909 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:59.729688    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4r2js\""
Aug 05 17:12:59.730857 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:59.730839    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Aug 05 17:12:59.880798 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:59.880766    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00d8f640-e14d-4601-b45d-81ef6fbaf47d-metrics-tls\") pod \"dns-default-4bxkx\" (UID: \"00d8f640-e14d-4601-b45d-81ef6fbaf47d\") " pod="openshift-dns/dns-default-4bxkx"
Aug 05 17:12:59.880798 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:59.880809    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-registry-tls\") pod \"image-registry-76b686c878-267wj\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") " pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:12:59.881081 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:59.880966    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 05 17:12:59.881081 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:59.880979    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 05 17:12:59.881081 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:59.880994    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76b686c878-267wj: secret "image-registry-tls" not found
Aug 05 17:12:59.881081 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:59.881057    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-registry-tls podName:aba3d093-7afd-4f0e-8407-895e98526798 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:01.881035719 +0000 UTC m=+37.625670048 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-registry-tls") pod "image-registry-76b686c878-267wj" (UID: "aba3d093-7afd-4f0e-8407-895e98526798") : secret "image-registry-tls" not found
Aug 05 17:12:59.881081 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:12:59.881077    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00d8f640-e14d-4601-b45d-81ef6fbaf47d-metrics-tls podName:00d8f640-e14d-4601-b45d-81ef6fbaf47d nodeName:}" failed. No retries permitted until 2026-08-05 17:13:01.8810628 +0000 UTC m=+37.625697105 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/00d8f640-e14d-4601-b45d-81ef6fbaf47d-metrics-tls") pod "dns-default-4bxkx" (UID: "00d8f640-e14d-4601-b45d-81ef6fbaf47d") : secret "dns-default-metrics-tls" not found
Aug 05 17:12:59.886924 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:59.886892    2561 generic.go:356] "Generic (PLEG): container finished" podID="66f685fa-db6d-474f-b836-d0d9768e2d09" containerID="99b3bc25499d75a045e26d2b3408a7ffacf3636652a2ebcc0ff54bf0391bd1fd" exitCode=0
Aug 05 17:12:59.887090 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:12:59.886973    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7kqb" event={"ID":"66f685fa-db6d-474f-b836-d0d9768e2d09","Type":"ContainerDied","Data":"99b3bc25499d75a045e26d2b3408a7ffacf3636652a2ebcc0ff54bf0391bd1fd"}
Aug 05 17:13:00.893345 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:00.893309    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7kqb" event={"ID":"66f685fa-db6d-474f-b836-d0d9768e2d09","Type":"ContainerStarted","Data":"27916d50e8d07aa3062f5fac0fd391a24fd7082a2db96fe6e0787ac0f61dd47c"}
Aug 05 17:13:00.922564 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:00.922505    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-d7kqb" podStartSLOduration=7.220118195 podStartE2EDuration="36.922486737s" totalImagesPullingTime="29.702368542s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:00.920641267 +0000 UTC m=+36.665275595" watchObservedRunningTime="2026-08-05 17:13:00.922486737 +0000 UTC m=+36.667121064"
Aug 05 17:13:01.898010 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:01.897742    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pcknb" event={"ID":"50a0dce8-3a32-4b5e-a07a-7a4492cd5166","Type":"ContainerStarted","Data":"6993afc18f5d64c02ad9d1ded0f0bfa8ead3020fad2388118a86bf9c717ff4b2"}
Aug 05 17:13:01.898347 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:01.897891    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00d8f640-e14d-4601-b45d-81ef6fbaf47d-metrics-tls\") pod \"dns-default-4bxkx\" (UID: \"00d8f640-e14d-4601-b45d-81ef6fbaf47d\") " pod="openshift-dns/dns-default-4bxkx"
Aug 05 17:13:01.898347 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:01.897991    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 05 17:13:01.898347 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:01.898157    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 05 17:13:01.898347 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:01.898167    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76b686c878-267wj: secret "image-registry-tls" not found
Aug 05 17:13:01.898347 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:01.898183    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00d8f640-e14d-4601-b45d-81ef6fbaf47d-metrics-tls podName:00d8f640-e14d-4601-b45d-81ef6fbaf47d nodeName:}" failed. No retries permitted until 2026-08-05 17:13:05.898161575 +0000 UTC m=+41.642795881 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/00d8f640-e14d-4601-b45d-81ef6fbaf47d-metrics-tls") pod "dns-default-4bxkx" (UID: "00d8f640-e14d-4601-b45d-81ef6fbaf47d") : secret "dns-default-metrics-tls" not found
Aug 05 17:13:01.898347 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:01.898104    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-registry-tls\") pod \"image-registry-76b686c878-267wj\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") " pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:13:01.898347 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:01.898204    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-registry-tls podName:aba3d093-7afd-4f0e-8407-895e98526798 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:05.898193088 +0000 UTC m=+41.642827421 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-registry-tls") pod "image-registry-76b686c878-267wj" (UID: "aba3d093-7afd-4f0e-8407-895e98526798") : secret "image-registry-tls" not found
Aug 05 17:13:01.915022 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:01.914946    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pcknb" podStartSLOduration=1.41379168 podStartE2EDuration="3.914932263s" totalImagesPullingTime="2.501140583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:12:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:01.914543901 +0000 UTC m=+37.659178229" watchObservedRunningTime="2026-08-05 17:13:01.914932263 +0000 UTC m=+37.659566590"
Aug 05 17:13:05.928503 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:05.928463    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00d8f640-e14d-4601-b45d-81ef6fbaf47d-metrics-tls\") pod \"dns-default-4bxkx\" (UID: \"00d8f640-e14d-4601-b45d-81ef6fbaf47d\") " pod="openshift-dns/dns-default-4bxkx"
Aug 05 17:13:05.928503 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:05.928504    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-registry-tls\") pod \"image-registry-76b686c878-267wj\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") " pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:13:05.929100 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:05.928619    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 05 17:13:05.929100 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:05.928681    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00d8f640-e14d-4601-b45d-81ef6fbaf47d-metrics-tls podName:00d8f640-e14d-4601-b45d-81ef6fbaf47d nodeName:}" failed. No retries permitted until 2026-08-05 17:13:13.928664228 +0000 UTC m=+49.673298534 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/00d8f640-e14d-4601-b45d-81ef6fbaf47d-metrics-tls") pod "dns-default-4bxkx" (UID: "00d8f640-e14d-4601-b45d-81ef6fbaf47d") : secret "dns-default-metrics-tls" not found
Aug 05 17:13:05.929100 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:05.928625    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 05 17:13:05.929100 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:05.928719    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76b686c878-267wj: secret "image-registry-tls" not found
Aug 05 17:13:05.929100 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:05.928771    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-registry-tls podName:aba3d093-7afd-4f0e-8407-895e98526798 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:13.928757078 +0000 UTC m=+49.673391383 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-registry-tls") pod "image-registry-76b686c878-267wj" (UID: "aba3d093-7afd-4f0e-8407-895e98526798") : secret "image-registry-tls" not found
Aug 05 17:13:07.868894 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:07.868860    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hz4nf"]
Aug 05 17:13:07.900393 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:07.900353    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hz4nf"]
Aug 05 17:13:07.900557 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:07.900487    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hz4nf"
Aug 05 17:13:07.903207 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:07.903185    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Aug 05 17:13:07.903331 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:07.903263    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Aug 05 17:13:07.903461 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:07.903445    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m4zv5\""
Aug 05 17:13:07.903514 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:07.903502    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Aug 05 17:13:07.903718 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:07.903705    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Aug 05 17:13:08.042762 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:08.042728    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/67c22df9-3392-47e6-8097-30468199cade-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hz4nf\" (UID: \"67c22df9-3392-47e6-8097-30468199cade\") " pod="openshift-insights/insights-runtime-extractor-hz4nf"
Aug 05 17:13:08.042933 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:08.042777    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p8h5\" (UniqueName: \"kubernetes.io/projected/67c22df9-3392-47e6-8097-30468199cade-kube-api-access-6p8h5\") pod \"insights-runtime-extractor-hz4nf\" (UID: \"67c22df9-3392-47e6-8097-30468199cade\") " pod="openshift-insights/insights-runtime-extractor-hz4nf"
Aug 05 17:13:08.042933 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:08.042801    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/67c22df9-3392-47e6-8097-30468199cade-crio-socket\") pod \"insights-runtime-extractor-hz4nf\" (UID: \"67c22df9-3392-47e6-8097-30468199cade\") " pod="openshift-insights/insights-runtime-extractor-hz4nf"
Aug 05 17:13:08.042933 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:08.042851    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/67c22df9-3392-47e6-8097-30468199cade-data-volume\") pod \"insights-runtime-extractor-hz4nf\" (UID: \"67c22df9-3392-47e6-8097-30468199cade\") " pod="openshift-insights/insights-runtime-extractor-hz4nf"
Aug 05 17:13:08.042933 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:08.042868    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/67c22df9-3392-47e6-8097-30468199cade-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hz4nf\" (UID: \"67c22df9-3392-47e6-8097-30468199cade\") " pod="openshift-insights/insights-runtime-extractor-hz4nf"
Aug 05 17:13:08.087844 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:08.087813    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-7tdb9"]
Aug 05 17:13:08.107756 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:08.107729    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-7tdb9"]
Aug 05 17:13:08.107893 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:08.107833    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-7tdb9"
Aug 05 17:13:08.110844 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:08.110816    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Aug 05 17:13:08.110844 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:08.110838    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-b7b5s\""
Aug 05 17:13:08.111067 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:08.110868    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Aug 05 17:13:08.144003 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:08.143904    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/67c22df9-3392-47e6-8097-30468199cade-data-volume\") pod \"insights-runtime-extractor-hz4nf\" (UID: \"67c22df9-3392-47e6-8097-30468199cade\") " pod="openshift-insights/insights-runtime-extractor-hz4nf"
Aug 05 17:13:08.144003 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:08.143944    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/67c22df9-3392-47e6-8097-30468199cade-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hz4nf\" (UID: \"67c22df9-3392-47e6-8097-30468199cade\") " pod="openshift-insights/insights-runtime-extractor-hz4nf"
Aug 05 17:13:08.144222 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:08.144069    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 05 17:13:08.144222 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:08.144136    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67c22df9-3392-47e6-8097-30468199cade-insights-runtime-extractor-tls podName:67c22df9-3392-47e6-8097-30468199cade nodeName:}" failed. No retries permitted until 2026-08-05 17:13:08.64411616 +0000 UTC m=+44.388750470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/67c22df9-3392-47e6-8097-30468199cade-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hz4nf" (UID: "67c22df9-3392-47e6-8097-30468199cade") : secret "insights-runtime-extractor-tls" not found
Aug 05 17:13:08.144222 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:08.144154    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/67c22df9-3392-47e6-8097-30468199cade-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hz4nf\" (UID: \"67c22df9-3392-47e6-8097-30468199cade\") " pod="openshift-insights/insights-runtime-extractor-hz4nf"
Aug 05 17:13:08.144222 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:08.144194    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p8h5\" (UniqueName: \"kubernetes.io/projected/67c22df9-3392-47e6-8097-30468199cade-kube-api-access-6p8h5\") pod \"insights-runtime-extractor-hz4nf\" (UID: \"67c22df9-3392-47e6-8097-30468199cade\") " pod="openshift-insights/insights-runtime-extractor-hz4nf"
Aug 05 17:13:08.144424 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:08.144248    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 05 17:13:08.144424 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:08.144268    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/67c22df9-3392-47e6-8097-30468199cade-data-volume\") pod \"insights-runtime-extractor-hz4nf\" (UID: \"67c22df9-3392-47e6-8097-30468199cade\") " pod="openshift-insights/insights-runtime-extractor-hz4nf"
Aug 05 17:13:08.144424 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:08.144309    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67c22df9-3392-47e6-8097-30468199cade-kube-rbac-proxy-cm podName:67c22df9-3392-47e6-8097-30468199cade nodeName:}" failed. No retries permitted until 2026-08-05 17:13:08.644291541 +0000 UTC m=+44.388925866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/67c22df9-3392-47e6-8097-30468199cade-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hz4nf" (UID: "67c22df9-3392-47e6-8097-30468199cade") : configmap "kube-rbac-proxy" not found
Aug 05 17:13:08.144424 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:08.144326    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/67c22df9-3392-47e6-8097-30468199cade-crio-socket\") pod \"insights-runtime-extractor-hz4nf\" (UID: \"67c22df9-3392-47e6-8097-30468199cade\") " pod="openshift-insights/insights-runtime-extractor-hz4nf"
Aug 05 17:13:08.144583 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:08.144482    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/67c22df9-3392-47e6-8097-30468199cade-crio-socket\") pod \"insights-runtime-extractor-hz4nf\" (UID: \"67c22df9-3392-47e6-8097-30468199cade\") " pod="openshift-insights/insights-runtime-extractor-hz4nf"
Aug 05 17:13:08.157338 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:08.157305    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p8h5\" (UniqueName: \"kubernetes.io/projected/67c22df9-3392-47e6-8097-30468199cade-kube-api-access-6p8h5\") pod \"insights-runtime-extractor-hz4nf\" (UID: \"67c22df9-3392-47e6-8097-30468199cade\") " pod="openshift-insights/insights-runtime-extractor-hz4nf"
Aug 05 17:13:08.244901 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:08.244860    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjwj8\" (UniqueName: \"kubernetes.io/projected/83ff891c-1a98-4abd-a563-1308fd7326b8-kube-api-access-tjwj8\") pod \"migrator-68bc745d87-7tdb9\" (UID: \"83ff891c-1a98-4abd-a563-1308fd7326b8\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-7tdb9"
Aug 05 17:13:08.345962 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:08.345911    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjwj8\" (UniqueName: \"kubernetes.io/projected/83ff891c-1a98-4abd-a563-1308fd7326b8-kube-api-access-tjwj8\") pod \"migrator-68bc745d87-7tdb9\" (UID: \"83ff891c-1a98-4abd-a563-1308fd7326b8\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-7tdb9"
Aug 05 17:13:08.358375 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:08.358344    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjwj8\" (UniqueName: \"kubernetes.io/projected/83ff891c-1a98-4abd-a563-1308fd7326b8-kube-api-access-tjwj8\") pod \"migrator-68bc745d87-7tdb9\" (UID: \"83ff891c-1a98-4abd-a563-1308fd7326b8\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-7tdb9"
Aug 05 17:13:08.416820 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:08.416746    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-7tdb9"
Aug 05 17:13:08.562974 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:08.562920    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-7tdb9"]
Aug 05 17:13:08.566036 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:13:08.566008    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ff891c_1a98_4abd_a563_1308fd7326b8.slice/crio-f119c98dd02519e14a292f0adc05c0aa28557a01cc6365f844c1d79484528bc8 WatchSource:0}: Error finding container f119c98dd02519e14a292f0adc05c0aa28557a01cc6365f844c1d79484528bc8: Status 404 returned error can't find the container with id f119c98dd02519e14a292f0adc05c0aa28557a01cc6365f844c1d79484528bc8
Aug 05 17:13:08.648754 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:08.648713    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/67c22df9-3392-47e6-8097-30468199cade-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hz4nf\" (UID: \"67c22df9-3392-47e6-8097-30468199cade\") " pod="openshift-insights/insights-runtime-extractor-hz4nf"
Aug 05 17:13:08.648929 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:08.648805    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/67c22df9-3392-47e6-8097-30468199cade-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hz4nf\" (UID: \"67c22df9-3392-47e6-8097-30468199cade\") " pod="openshift-insights/insights-runtime-extractor-hz4nf"
Aug 05 17:13:08.648929 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:08.648858    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 05 17:13:08.648929 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:08.648900    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 05 17:13:08.648929 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:08.648921    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67c22df9-3392-47e6-8097-30468199cade-insights-runtime-extractor-tls podName:67c22df9-3392-47e6-8097-30468199cade nodeName:}" failed. No retries permitted until 2026-08-05 17:13:09.648905333 +0000 UTC m=+45.393539640 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/67c22df9-3392-47e6-8097-30468199cade-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hz4nf" (UID: "67c22df9-3392-47e6-8097-30468199cade") : secret "insights-runtime-extractor-tls" not found
Aug 05 17:13:08.649137 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:08.648939    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67c22df9-3392-47e6-8097-30468199cade-kube-rbac-proxy-cm podName:67c22df9-3392-47e6-8097-30468199cade nodeName:}" failed. No retries permitted until 2026-08-05 17:13:09.64892853 +0000 UTC m=+45.393562836 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/67c22df9-3392-47e6-8097-30468199cade-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hz4nf" (UID: "67c22df9-3392-47e6-8097-30468199cade") : configmap "kube-rbac-proxy" not found
Aug 05 17:13:08.914823 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:08.914785    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-7tdb9" event={"ID":"83ff891c-1a98-4abd-a563-1308fd7326b8","Type":"ContainerStarted","Data":"f119c98dd02519e14a292f0adc05c0aa28557a01cc6365f844c1d79484528bc8"}
Aug 05 17:13:09.606029 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:09.606001    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vjnk7_47a3441a-cffb-4919-95b8-6393fd63c14b/dns-node-resolver/0.log"
Aug 05 17:13:09.655578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:09.655539    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/67c22df9-3392-47e6-8097-30468199cade-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hz4nf\" (UID: \"67c22df9-3392-47e6-8097-30468199cade\") " pod="openshift-insights/insights-runtime-extractor-hz4nf"
Aug 05 17:13:09.655768 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:09.655632    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/67c22df9-3392-47e6-8097-30468199cade-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hz4nf\" (UID: \"67c22df9-3392-47e6-8097-30468199cade\") " pod="openshift-insights/insights-runtime-extractor-hz4nf"
Aug 05 17:13:09.655768 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:09.655687    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 05 17:13:09.655768 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:09.655734    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 05 17:13:09.655768 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:09.655770    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67c22df9-3392-47e6-8097-30468199cade-insights-runtime-extractor-tls podName:67c22df9-3392-47e6-8097-30468199cade nodeName:}" failed. No retries permitted until 2026-08-05 17:13:11.655748814 +0000 UTC m=+47.400383146 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/67c22df9-3392-47e6-8097-30468199cade-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hz4nf" (UID: "67c22df9-3392-47e6-8097-30468199cade") : secret "insights-runtime-extractor-tls" not found
Aug 05 17:13:09.656011 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:09.655791    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67c22df9-3392-47e6-8097-30468199cade-kube-rbac-proxy-cm podName:67c22df9-3392-47e6-8097-30468199cade nodeName:}" failed. No retries permitted until 2026-08-05 17:13:11.65578129 +0000 UTC m=+47.400415603 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/67c22df9-3392-47e6-8097-30468199cade-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hz4nf" (UID: "67c22df9-3392-47e6-8097-30468199cade") : configmap "kube-rbac-proxy" not found
Aug 05 17:13:10.605906 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:10.605873    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tg9l4_eacfb8d3-1760-45ee-a594-7656a55c9d93/node-ca/0.log"
Aug 05 17:13:10.921366 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:10.921270    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-7tdb9" event={"ID":"83ff891c-1a98-4abd-a563-1308fd7326b8","Type":"ContainerStarted","Data":"ea8b792c6b5338a51148d8ffc2faabcb97840d9ef14d4b6fb9f993b7cb4ee5ea"}
Aug 05 17:13:10.921366 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:10.921340    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-7tdb9" event={"ID":"83ff891c-1a98-4abd-a563-1308fd7326b8","Type":"ContainerStarted","Data":"b53b5c267bc337079f2e32d20f75f92147508f8e84e77f67c1aa3bbf3aa20e6b"}
Aug 05 17:13:10.939888 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:10.939832    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-7tdb9" podStartSLOduration=1.231348996 podStartE2EDuration="2.939816719s" totalImagesPullingTime="1.708467723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:13:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:10.93941036 +0000 UTC m=+46.684044698" watchObservedRunningTime="2026-08-05 17:13:10.939816719 +0000 UTC m=+46.684451085"
Aug 05 17:13:11.203105 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:11.203016    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pcknb_50a0dce8-3a32-4b5e-a07a-7a4492cd5166/serve-healthcheck-canary/0.log"
Aug 05 17:13:11.671581 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:11.671543    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/67c22df9-3392-47e6-8097-30468199cade-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hz4nf\" (UID: \"67c22df9-3392-47e6-8097-30468199cade\") " pod="openshift-insights/insights-runtime-extractor-hz4nf"
Aug 05 17:13:11.672003 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:11.671615    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/67c22df9-3392-47e6-8097-30468199cade-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hz4nf\" (UID: \"67c22df9-3392-47e6-8097-30468199cade\") " pod="openshift-insights/insights-runtime-extractor-hz4nf"
Aug 05 17:13:11.672003 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:11.671690    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 05 17:13:11.672003 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:11.671696    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 05 17:13:11.672003 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:11.671791    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67c22df9-3392-47e6-8097-30468199cade-insights-runtime-extractor-tls podName:67c22df9-3392-47e6-8097-30468199cade nodeName:}" failed. No retries permitted until 2026-08-05 17:13:15.671775131 +0000 UTC m=+51.416409441 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/67c22df9-3392-47e6-8097-30468199cade-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hz4nf" (UID: "67c22df9-3392-47e6-8097-30468199cade") : secret "insights-runtime-extractor-tls" not found
Aug 05 17:13:11.672003 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:11.671839    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67c22df9-3392-47e6-8097-30468199cade-kube-rbac-proxy-cm podName:67c22df9-3392-47e6-8097-30468199cade nodeName:}" failed. No retries permitted until 2026-08-05 17:13:15.671826553 +0000 UTC m=+51.416460859 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/67c22df9-3392-47e6-8097-30468199cade-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hz4nf" (UID: "67c22df9-3392-47e6-8097-30468199cade") : configmap "kube-rbac-proxy" not found
Aug 05 17:13:13.992444 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:13.992403    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00d8f640-e14d-4601-b45d-81ef6fbaf47d-metrics-tls\") pod \"dns-default-4bxkx\" (UID: \"00d8f640-e14d-4601-b45d-81ef6fbaf47d\") " pod="openshift-dns/dns-default-4bxkx"
Aug 05 17:13:13.992444 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:13.992442    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-registry-tls\") pod \"image-registry-76b686c878-267wj\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") " pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:13:13.992888 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:13.992571    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 05 17:13:13.992888 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:13.992633    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00d8f640-e14d-4601-b45d-81ef6fbaf47d-metrics-tls podName:00d8f640-e14d-4601-b45d-81ef6fbaf47d nodeName:}" failed. No retries permitted until 2026-08-05 17:13:29.992618647 +0000 UTC m=+65.737252956 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/00d8f640-e14d-4601-b45d-81ef6fbaf47d-metrics-tls") pod "dns-default-4bxkx" (UID: "00d8f640-e14d-4601-b45d-81ef6fbaf47d") : secret "dns-default-metrics-tls" not found
Aug 05 17:13:13.992888 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:13.992577    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 05 17:13:13.992888 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:13.992679    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76b686c878-267wj: secret "image-registry-tls" not found
Aug 05 17:13:13.992888 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:13.992730    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-registry-tls podName:aba3d093-7afd-4f0e-8407-895e98526798 nodeName:}" failed. No retries permitted until 2026-08-05 17:13:29.992716627 +0000 UTC m=+65.737350934 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-registry-tls") pod "image-registry-76b686c878-267wj" (UID: "aba3d093-7afd-4f0e-8407-895e98526798") : secret "image-registry-tls" not found
Aug 05 17:13:15.706388 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:15.706349    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/67c22df9-3392-47e6-8097-30468199cade-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hz4nf\" (UID: \"67c22df9-3392-47e6-8097-30468199cade\") " pod="openshift-insights/insights-runtime-extractor-hz4nf"
Aug 05 17:13:15.706878 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:15.706509    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 05 17:13:15.706878 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:15.706523    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/67c22df9-3392-47e6-8097-30468199cade-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hz4nf\" (UID: \"67c22df9-3392-47e6-8097-30468199cade\") " pod="openshift-insights/insights-runtime-extractor-hz4nf"
Aug 05 17:13:15.706878 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:13:15.706581    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67c22df9-3392-47e6-8097-30468199cade-insights-runtime-extractor-tls podName:67c22df9-3392-47e6-8097-30468199cade nodeName:}" failed. No retries permitted until 2026-08-05 17:13:23.706562542 +0000 UTC m=+59.451196867 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/67c22df9-3392-47e6-8097-30468199cade-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hz4nf" (UID: "67c22df9-3392-47e6-8097-30468199cade") : secret "insights-runtime-extractor-tls" not found
Aug 05 17:13:15.707062 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:15.707053    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/67c22df9-3392-47e6-8097-30468199cade-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hz4nf\" (UID: \"67c22df9-3392-47e6-8097-30468199cade\") " pod="openshift-insights/insights-runtime-extractor-hz4nf"
Aug 05 17:13:22.869077 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:22.869044    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-87v9q"
Aug 05 17:13:23.777107 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:23.777063    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/67c22df9-3392-47e6-8097-30468199cade-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hz4nf\" (UID: \"67c22df9-3392-47e6-8097-30468199cade\") " pod="openshift-insights/insights-runtime-extractor-hz4nf"
Aug 05 17:13:23.779349 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:23.779327    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/67c22df9-3392-47e6-8097-30468199cade-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hz4nf\" (UID: \"67c22df9-3392-47e6-8097-30468199cade\") " pod="openshift-insights/insights-runtime-extractor-hz4nf"
Aug 05 17:13:23.809370 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:23.809334    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hz4nf"
Aug 05 17:13:23.934747 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:23.934611    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hz4nf"]
Aug 05 17:13:23.937597 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:13:23.937560    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67c22df9_3392_47e6_8097_30468199cade.slice/crio-6f6313946f51c44ab76d8a45f6904230171f22a7fe643c9a936e07c04059a91b WatchSource:0}: Error finding container 6f6313946f51c44ab76d8a45f6904230171f22a7fe643c9a936e07c04059a91b: Status 404 returned error can't find the container with id 6f6313946f51c44ab76d8a45f6904230171f22a7fe643c9a936e07c04059a91b
Aug 05 17:13:23.954390 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:23.954365    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hz4nf" event={"ID":"67c22df9-3392-47e6-8097-30468199cade","Type":"ContainerStarted","Data":"6f6313946f51c44ab76d8a45f6904230171f22a7fe643c9a936e07c04059a91b"}
Aug 05 17:13:24.958027 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:24.957928    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hz4nf" event={"ID":"67c22df9-3392-47e6-8097-30468199cade","Type":"ContainerStarted","Data":"986f44193bdd47259f8bd6cc6813899f8a7035090e6036e5daae0478f68bc145"}
Aug 05 17:13:24.958027 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:24.958004    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hz4nf" event={"ID":"67c22df9-3392-47e6-8097-30468199cade","Type":"ContainerStarted","Data":"ea70feb2f0202e76348029aad31c1f7b7ce2de99d73957ee61debcb01c4d5472"}
Aug 05 17:13:26.965443 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:26.965409    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hz4nf" event={"ID":"67c22df9-3392-47e6-8097-30468199cade","Type":"ContainerStarted","Data":"2fb5fe97347e1869b9d34ffdc9b9dbe66bbd0efe4b03cff2faeb587e1ae57751"}
Aug 05 17:13:26.988988 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:26.988918    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hz4nf" podStartSLOduration=18.024783283 podStartE2EDuration="19.988904565s" totalImagesPullingTime="1.964121282s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:13:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:26.987552526 +0000 UTC m=+62.732186863" watchObservedRunningTime="2026-08-05 17:13:26.988904565 +0000 UTC m=+62.733538894"
Aug 05 17:13:29.522712 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:29.522672    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25132135-9fbe-49c4-9f6d-0e9db2cce6b8-original-pull-secret\") pod \"global-pull-secret-syncer-gfsf2\" (UID: \"25132135-9fbe-49c4-9f6d-0e9db2cce6b8\") " pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:13:29.522712 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:29.522716    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba7276d8-e168-437d-adcd-3ea703e97ab1-metrics-certs\") pod \"network-metrics-daemon-v6tb4\" (UID: \"ba7276d8-e168-437d-adcd-3ea703e97ab1\") " pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:13:29.527224 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:29.527204    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 05 17:13:29.528195 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:29.528174    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 05 17:13:29.535462 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:29.535428    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/25132135-9fbe-49c4-9f6d-0e9db2cce6b8-original-pull-secret\") pod \"global-pull-secret-syncer-gfsf2\" (UID: \"25132135-9fbe-49c4-9f6d-0e9db2cce6b8\") " pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:13:29.535572 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:29.535472    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba7276d8-e168-437d-adcd-3ea703e97ab1-metrics-certs\") pod \"network-metrics-daemon-v6tb4\" (UID: \"ba7276d8-e168-437d-adcd-3ea703e97ab1\") " pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:13:29.623247 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:29.623203    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrpwv\" (UniqueName: \"kubernetes.io/projected/1cbe388d-927f-4384-9abd-49d8864afe2c-kube-api-access-qrpwv\") pod \"network-check-target-bbcvm\" (UID: \"1cbe388d-927f-4384-9abd-49d8864afe2c\") " pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:13:29.626236 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:29.626217    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Aug 05 17:13:29.636818 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:29.636792    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Aug 05 17:13:29.637138 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:29.637125    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vwprk\""
Aug 05 17:13:29.644440 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:29.644426    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6tb4"
Aug 05 17:13:29.646652 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:29.646637    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrpwv\" (UniqueName: \"kubernetes.io/projected/1cbe388d-927f-4384-9abd-49d8864afe2c-kube-api-access-qrpwv\") pod \"network-check-target-bbcvm\" (UID: \"1cbe388d-927f-4384-9abd-49d8864afe2c\") " pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:13:29.736916 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:29.736889    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfsf2"
Aug 05 17:13:29.748792 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:29.748768    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4r2js\""
Aug 05 17:13:29.752398 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:29.752380    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:13:29.819620 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:29.819546    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v6tb4"]
Aug 05 17:13:29.825838 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:13:29.823028    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba7276d8_e168_437d_adcd_3ea703e97ab1.slice/crio-e29cbdb7181a5ad3f2504d0cb616f5636f1abc58a352cca4b22cb49e169335cc WatchSource:0}: Error finding container e29cbdb7181a5ad3f2504d0cb616f5636f1abc58a352cca4b22cb49e169335cc: Status 404 returned error can't find the container with id e29cbdb7181a5ad3f2504d0cb616f5636f1abc58a352cca4b22cb49e169335cc
Aug 05 17:13:29.894716 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:29.894683    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gfsf2"]
Aug 05 17:13:29.897701 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:13:29.897673    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25132135_9fbe_49c4_9f6d_0e9db2cce6b8.slice/crio-481100572d69006520b572aff87fe313b4281e4edb1ff6e14a5593950449c91a WatchSource:0}: Error finding container 481100572d69006520b572aff87fe313b4281e4edb1ff6e14a5593950449c91a: Status 404 returned error can't find the container with id 481100572d69006520b572aff87fe313b4281e4edb1ff6e14a5593950449c91a
Aug 05 17:13:29.919389 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:29.919359    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bbcvm"]
Aug 05 17:13:29.922292 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:13:29.922263    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cbe388d_927f_4384_9abd_49d8864afe2c.slice/crio-84dc2feddad3f2b2b7e16999920a7fa503937e70f703ec7c7fe67af2f3b72272 WatchSource:0}: Error finding container 84dc2feddad3f2b2b7e16999920a7fa503937e70f703ec7c7fe67af2f3b72272: Status 404 returned error can't find the container with id 84dc2feddad3f2b2b7e16999920a7fa503937e70f703ec7c7fe67af2f3b72272
Aug 05 17:13:29.973262 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:29.973226    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v6tb4" event={"ID":"ba7276d8-e168-437d-adcd-3ea703e97ab1","Type":"ContainerStarted","Data":"e29cbdb7181a5ad3f2504d0cb616f5636f1abc58a352cca4b22cb49e169335cc"}
Aug 05 17:13:29.974333 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:29.974301    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gfsf2" event={"ID":"25132135-9fbe-49c4-9f6d-0e9db2cce6b8","Type":"ContainerStarted","Data":"481100572d69006520b572aff87fe313b4281e4edb1ff6e14a5593950449c91a"}
Aug 05 17:13:29.975346 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:29.975322    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bbcvm" event={"ID":"1cbe388d-927f-4384-9abd-49d8864afe2c","Type":"ContainerStarted","Data":"84dc2feddad3f2b2b7e16999920a7fa503937e70f703ec7c7fe67af2f3b72272"}
Aug 05 17:13:30.026680 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:30.026639    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00d8f640-e14d-4601-b45d-81ef6fbaf47d-metrics-tls\") pod \"dns-default-4bxkx\" (UID: \"00d8f640-e14d-4601-b45d-81ef6fbaf47d\") " pod="openshift-dns/dns-default-4bxkx"
Aug 05 17:13:30.026680 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:30.026682    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-registry-tls\") pod \"image-registry-76b686c878-267wj\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") " pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:13:30.028944 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:30.028925    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/00d8f640-e14d-4601-b45d-81ef6fbaf47d-metrics-tls\") pod \"dns-default-4bxkx\" (UID: \"00d8f640-e14d-4601-b45d-81ef6fbaf47d\") " pod="openshift-dns/dns-default-4bxkx"
Aug 05 17:13:30.029106 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:30.029088    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-registry-tls\") pod \"image-registry-76b686c878-267wj\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") " pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:13:30.297220 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:30.297182    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9s4jc\""
Aug 05 17:13:30.303994 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:30.303968    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mrwwg\""
Aug 05 17:13:30.305276 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:30.305250    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:13:30.312384 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:30.312354    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4bxkx"
Aug 05 17:13:30.475531 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:30.475491    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76b686c878-267wj"]
Aug 05 17:13:30.480205 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:13:30.480166    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba3d093_7afd_4f0e_8407_895e98526798.slice/crio-eda8ac6fac8906f59151a9ac84507a141edf6c993ab040dee29cad7edb152c55 WatchSource:0}: Error finding container eda8ac6fac8906f59151a9ac84507a141edf6c993ab040dee29cad7edb152c55: Status 404 returned error can't find the container with id eda8ac6fac8906f59151a9ac84507a141edf6c993ab040dee29cad7edb152c55
Aug 05 17:13:30.493392 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:30.492273    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4bxkx"]
Aug 05 17:13:30.497404 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:13:30.497353    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d8f640_e14d_4601_b45d_81ef6fbaf47d.slice/crio-fec246994e31fa9d7e5bfa60edd3c35dcb438e0b0a7a5c0fb9a474db0ad8fd41 WatchSource:0}: Error finding container fec246994e31fa9d7e5bfa60edd3c35dcb438e0b0a7a5c0fb9a474db0ad8fd41: Status 404 returned error can't find the container with id fec246994e31fa9d7e5bfa60edd3c35dcb438e0b0a7a5c0fb9a474db0ad8fd41
Aug 05 17:13:30.983752 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:30.983669    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76b686c878-267wj" event={"ID":"aba3d093-7afd-4f0e-8407-895e98526798","Type":"ContainerStarted","Data":"49ca1ae00804a17210c8806190844efd0a5944fb5d9b017bb78b26c0089de81b"}
Aug 05 17:13:30.984210 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:30.983779    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76b686c878-267wj" event={"ID":"aba3d093-7afd-4f0e-8407-895e98526798","Type":"ContainerStarted","Data":"eda8ac6fac8906f59151a9ac84507a141edf6c993ab040dee29cad7edb152c55"}
Aug 05 17:13:30.984210 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:30.983833    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:13:30.985063 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:30.985035    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4bxkx" event={"ID":"00d8f640-e14d-4601-b45d-81ef6fbaf47d","Type":"ContainerStarted","Data":"fec246994e31fa9d7e5bfa60edd3c35dcb438e0b0a7a5c0fb9a474db0ad8fd41"}
Aug 05 17:13:31.013349 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:31.012712    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-76b686c878-267wj" podStartSLOduration=62.012692317 podStartE2EDuration="1m2.012692317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:31.011409945 +0000 UTC m=+66.756044274" watchObservedRunningTime="2026-08-05 17:13:31.012692317 +0000 UTC m=+66.757326646"
Aug 05 17:13:31.992897 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:31.992861    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v6tb4" event={"ID":"ba7276d8-e168-437d-adcd-3ea703e97ab1","Type":"ContainerStarted","Data":"68a735a0bfa32b45d7af6ebbd6e581c4e7858e41612482153a566c571bcd809e"}
Aug 05 17:13:31.993315 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:31.993085    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v6tb4" event={"ID":"ba7276d8-e168-437d-adcd-3ea703e97ab1","Type":"ContainerStarted","Data":"64a4a6d4d40f8112069da8755bc0bf524e5978c25933297110864e6b0a01aa01"}
Aug 05 17:13:32.015332 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:32.015214    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-v6tb4" podStartSLOduration=66.848630025 podStartE2EDuration="1m8.015191241s" totalImagesPullingTime="1.166561216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:12:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:32.013353747 +0000 UTC m=+67.757988077" watchObservedRunningTime="2026-08-05 17:13:32.015191241 +0000 UTC m=+67.759825570"
Aug 05 17:13:34.852824 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:34.852787    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76b686c878-267wj"]
Aug 05 17:13:34.883545 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:34.883505    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b77b65f74-4qpkg"]
Aug 05 17:13:34.886235 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:34.886210    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b77b65f74-4qpkg"
Aug 05 17:13:34.890057 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:34.890031    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Aug 05 17:13:34.890395 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:34.890375    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Aug 05 17:13:34.890519 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:34.890419    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Aug 05 17:13:34.891726 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:34.891708    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Aug 05 17:13:34.907870 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:34.907843    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b77b65f74-4qpkg"]
Aug 05 17:13:34.963666 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:34.963635    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/601cbfbe-36c9-4e75-805d-f0448ed0ebc2-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b77b65f74-4qpkg\" (UID: \"601cbfbe-36c9-4e75-805d-f0448ed0ebc2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b77b65f74-4qpkg"
Aug 05 17:13:34.963779 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:34.963684    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twwxm\" (UniqueName: \"kubernetes.io/projected/601cbfbe-36c9-4e75-805d-f0448ed0ebc2-kube-api-access-twwxm\") pod \"klusterlet-addon-workmgr-5b77b65f74-4qpkg\" (UID: \"601cbfbe-36c9-4e75-805d-f0448ed0ebc2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b77b65f74-4qpkg"
Aug 05 17:13:34.963779 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:34.963720    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/601cbfbe-36c9-4e75-805d-f0448ed0ebc2-tmp\") pod \"klusterlet-addon-workmgr-5b77b65f74-4qpkg\" (UID: \"601cbfbe-36c9-4e75-805d-f0448ed0ebc2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b77b65f74-4qpkg"
Aug 05 17:13:34.979464 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:34.979432    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-vxqqd"]
Aug 05 17:13:34.982307 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:34.982282    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c944f9855-dsl5q"]
Aug 05 17:13:34.982444 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:34.982423    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-vxqqd"
Aug 05 17:13:34.984480 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:34.984234    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c944f9855-dsl5q"
Aug 05 17:13:34.987658 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:34.987623    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-9sgvm\""
Aug 05 17:13:34.987767 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:34.987694    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Aug 05 17:13:34.997061 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:34.997036    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-vxqqd"]
Aug 05 17:13:35.015336 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.015270    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c944f9855-dsl5q"]
Aug 05 17:13:35.064145 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.064039    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twwxm\" (UniqueName: \"kubernetes.io/projected/601cbfbe-36c9-4e75-805d-f0448ed0ebc2-kube-api-access-twwxm\") pod \"klusterlet-addon-workmgr-5b77b65f74-4qpkg\" (UID: \"601cbfbe-36c9-4e75-805d-f0448ed0ebc2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b77b65f74-4qpkg"
Aug 05 17:13:35.064145 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.064076    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7682b61a-d92f-4b1a-9b7f-e3404237c48f-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-vxqqd\" (UID: \"7682b61a-d92f-4b1a-9b7f-e3404237c48f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-vxqqd"
Aug 05 17:13:35.064145 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.064097    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/601cbfbe-36c9-4e75-805d-f0448ed0ebc2-tmp\") pod \"klusterlet-addon-workmgr-5b77b65f74-4qpkg\" (UID: \"601cbfbe-36c9-4e75-805d-f0448ed0ebc2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b77b65f74-4qpkg"
Aug 05 17:13:35.064145 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.064114    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6880ea65-e478-448a-a6fe-edc133ef596a-trusted-ca\") pod \"image-registry-7c944f9855-dsl5q\" (UID: \"6880ea65-e478-448a-a6fe-edc133ef596a\") " pod="openshift-image-registry/image-registry-7c944f9855-dsl5q"
Aug 05 17:13:35.064145 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.064140    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6880ea65-e478-448a-a6fe-edc133ef596a-image-registry-private-configuration\") pod \"image-registry-7c944f9855-dsl5q\" (UID: \"6880ea65-e478-448a-a6fe-edc133ef596a\") " pod="openshift-image-registry/image-registry-7c944f9855-dsl5q"
Aug 05 17:13:35.064515 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.064163    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6880ea65-e478-448a-a6fe-edc133ef596a-registry-tls\") pod \"image-registry-7c944f9855-dsl5q\" (UID: \"6880ea65-e478-448a-a6fe-edc133ef596a\") " pod="openshift-image-registry/image-registry-7c944f9855-dsl5q"
Aug 05 17:13:35.064515 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.064185    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6880ea65-e478-448a-a6fe-edc133ef596a-ca-trust-extracted\") pod \"image-registry-7c944f9855-dsl5q\" (UID: \"6880ea65-e478-448a-a6fe-edc133ef596a\") " pod="openshift-image-registry/image-registry-7c944f9855-dsl5q"
Aug 05 17:13:35.064515 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.064203    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/601cbfbe-36c9-4e75-805d-f0448ed0ebc2-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b77b65f74-4qpkg\" (UID: \"601cbfbe-36c9-4e75-805d-f0448ed0ebc2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b77b65f74-4qpkg"
Aug 05 17:13:35.064515 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.064218    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6880ea65-e478-448a-a6fe-edc133ef596a-bound-sa-token\") pod \"image-registry-7c944f9855-dsl5q\" (UID: \"6880ea65-e478-448a-a6fe-edc133ef596a\") " pod="openshift-image-registry/image-registry-7c944f9855-dsl5q"
Aug 05 17:13:35.064515 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.064236    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxmxg\" (UniqueName: \"kubernetes.io/projected/6880ea65-e478-448a-a6fe-edc133ef596a-kube-api-access-xxmxg\") pod \"image-registry-7c944f9855-dsl5q\" (UID: \"6880ea65-e478-448a-a6fe-edc133ef596a\") " pod="openshift-image-registry/image-registry-7c944f9855-dsl5q"
Aug 05 17:13:35.064515 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.064256    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6880ea65-e478-448a-a6fe-edc133ef596a-registry-certificates\") pod \"image-registry-7c944f9855-dsl5q\" (UID: \"6880ea65-e478-448a-a6fe-edc133ef596a\") " pod="openshift-image-registry/image-registry-7c944f9855-dsl5q"
Aug 05 17:13:35.064515 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.064282    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6880ea65-e478-448a-a6fe-edc133ef596a-installation-pull-secrets\") pod \"image-registry-7c944f9855-dsl5q\" (UID: \"6880ea65-e478-448a-a6fe-edc133ef596a\") " pod="openshift-image-registry/image-registry-7c944f9855-dsl5q"
Aug 05 17:13:35.064933 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.064775    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/601cbfbe-36c9-4e75-805d-f0448ed0ebc2-tmp\") pod \"klusterlet-addon-workmgr-5b77b65f74-4qpkg\" (UID: \"601cbfbe-36c9-4e75-805d-f0448ed0ebc2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b77b65f74-4qpkg"
Aug 05 17:13:35.069902 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.069878    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/601cbfbe-36c9-4e75-805d-f0448ed0ebc2-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b77b65f74-4qpkg\" (UID: \"601cbfbe-36c9-4e75-805d-f0448ed0ebc2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b77b65f74-4qpkg"
Aug 05 17:13:35.076783 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.076696    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twwxm\" (UniqueName: \"kubernetes.io/projected/601cbfbe-36c9-4e75-805d-f0448ed0ebc2-kube-api-access-twwxm\") pod \"klusterlet-addon-workmgr-5b77b65f74-4qpkg\" (UID: \"601cbfbe-36c9-4e75-805d-f0448ed0ebc2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b77b65f74-4qpkg"
Aug 05 17:13:35.165486 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.165455    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7682b61a-d92f-4b1a-9b7f-e3404237c48f-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-vxqqd\" (UID: \"7682b61a-d92f-4b1a-9b7f-e3404237c48f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-vxqqd"
Aug 05 17:13:35.165671 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.165494    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6880ea65-e478-448a-a6fe-edc133ef596a-trusted-ca\") pod \"image-registry-7c944f9855-dsl5q\" (UID: \"6880ea65-e478-448a-a6fe-edc133ef596a\") " pod="openshift-image-registry/image-registry-7c944f9855-dsl5q"
Aug 05 17:13:35.165671 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.165528    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6880ea65-e478-448a-a6fe-edc133ef596a-image-registry-private-configuration\") pod \"image-registry-7c944f9855-dsl5q\" (UID: \"6880ea65-e478-448a-a6fe-edc133ef596a\") " pod="openshift-image-registry/image-registry-7c944f9855-dsl5q"
Aug 05 17:13:35.165671 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.165558    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6880ea65-e478-448a-a6fe-edc133ef596a-registry-tls\") pod \"image-registry-7c944f9855-dsl5q\" (UID: \"6880ea65-e478-448a-a6fe-edc133ef596a\") " pod="openshift-image-registry/image-registry-7c944f9855-dsl5q"
Aug 05 17:13:35.165817 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.165729    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6880ea65-e478-448a-a6fe-edc133ef596a-ca-trust-extracted\") pod \"image-registry-7c944f9855-dsl5q\" (UID: \"6880ea65-e478-448a-a6fe-edc133ef596a\") " pod="openshift-image-registry/image-registry-7c944f9855-dsl5q"
Aug 05 17:13:35.165817 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.165783    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6880ea65-e478-448a-a6fe-edc133ef596a-bound-sa-token\") pod \"image-registry-7c944f9855-dsl5q\" (UID: \"6880ea65-e478-448a-a6fe-edc133ef596a\") " pod="openshift-image-registry/image-registry-7c944f9855-dsl5q"
Aug 05 17:13:35.165817 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.165813    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxmxg\" (UniqueName: \"kubernetes.io/projected/6880ea65-e478-448a-a6fe-edc133ef596a-kube-api-access-xxmxg\") pod \"image-registry-7c944f9855-dsl5q\" (UID: \"6880ea65-e478-448a-a6fe-edc133ef596a\") " pod="openshift-image-registry/image-registry-7c944f9855-dsl5q"
Aug 05 17:13:35.165998 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.165847    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6880ea65-e478-448a-a6fe-edc133ef596a-registry-certificates\") pod \"image-registry-7c944f9855-dsl5q\" (UID: \"6880ea65-e478-448a-a6fe-edc133ef596a\") " pod="openshift-image-registry/image-registry-7c944f9855-dsl5q"
Aug 05 17:13:35.165998 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.165892    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6880ea65-e478-448a-a6fe-edc133ef596a-installation-pull-secrets\") pod \"image-registry-7c944f9855-dsl5q\" (UID: \"6880ea65-e478-448a-a6fe-edc133ef596a\") " pod="openshift-image-registry/image-registry-7c944f9855-dsl5q"
Aug 05 17:13:35.166586 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.166534    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6880ea65-e478-448a-a6fe-edc133ef596a-ca-trust-extracted\") pod \"image-registry-7c944f9855-dsl5q\" (UID: \"6880ea65-e478-448a-a6fe-edc133ef596a\") " pod="openshift-image-registry/image-registry-7c944f9855-dsl5q"
Aug 05 17:13:35.166586 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.166555    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6880ea65-e478-448a-a6fe-edc133ef596a-trusted-ca\") pod \"image-registry-7c944f9855-dsl5q\" (UID: \"6880ea65-e478-448a-a6fe-edc133ef596a\") " pod="openshift-image-registry/image-registry-7c944f9855-dsl5q"
Aug 05 17:13:35.167030 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.167005    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6880ea65-e478-448a-a6fe-edc133ef596a-registry-certificates\") pod \"image-registry-7c944f9855-dsl5q\" (UID: \"6880ea65-e478-448a-a6fe-edc133ef596a\") " pod="openshift-image-registry/image-registry-7c944f9855-dsl5q"
Aug 05 17:13:35.168263 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.168189    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6880ea65-e478-448a-a6fe-edc133ef596a-image-registry-private-configuration\") pod \"image-registry-7c944f9855-dsl5q\" (UID: \"6880ea65-e478-448a-a6fe-edc133ef596a\") " pod="openshift-image-registry/image-registry-7c944f9855-dsl5q"
Aug 05 17:13:35.168616 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.168574    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7682b61a-d92f-4b1a-9b7f-e3404237c48f-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-vxqqd\" (UID: \"7682b61a-d92f-4b1a-9b7f-e3404237c48f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-vxqqd"
Aug 05 17:13:35.169179 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.169160    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6880ea65-e478-448a-a6fe-edc133ef596a-registry-tls\") pod \"image-registry-7c944f9855-dsl5q\" (UID: \"6880ea65-e478-448a-a6fe-edc133ef596a\") " pod="openshift-image-registry/image-registry-7c944f9855-dsl5q"
Aug 05 17:13:35.169266 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.169215    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6880ea65-e478-448a-a6fe-edc133ef596a-installation-pull-secrets\") pod \"image-registry-7c944f9855-dsl5q\" (UID: \"6880ea65-e478-448a-a6fe-edc133ef596a\") " pod="openshift-image-registry/image-registry-7c944f9855-dsl5q"
Aug 05 17:13:35.174615 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.174588    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6880ea65-e478-448a-a6fe-edc133ef596a-bound-sa-token\") pod \"image-registry-7c944f9855-dsl5q\" (UID: \"6880ea65-e478-448a-a6fe-edc133ef596a\") " pod="openshift-image-registry/image-registry-7c944f9855-dsl5q"
Aug 05 17:13:35.175624 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.175597    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxmxg\" (UniqueName: \"kubernetes.io/projected/6880ea65-e478-448a-a6fe-edc133ef596a-kube-api-access-xxmxg\") pod \"image-registry-7c944f9855-dsl5q\" (UID: \"6880ea65-e478-448a-a6fe-edc133ef596a\") " pod="openshift-image-registry/image-registry-7c944f9855-dsl5q"
Aug 05 17:13:35.196496 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.196468    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b77b65f74-4qpkg"
Aug 05 17:13:35.314727 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.314690    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-vxqqd"
Aug 05 17:13:35.331450 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.331425    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b77b65f74-4qpkg"]
Aug 05 17:13:35.333329 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.333308    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c944f9855-dsl5q"
Aug 05 17:13:35.333463 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:13:35.333442    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod601cbfbe_36c9_4e75_805d_f0448ed0ebc2.slice/crio-1e5e994216ceadf886e88e738bfaaaa255287a653ef546c2706b7da7fa03ca28 WatchSource:0}: Error finding container 1e5e994216ceadf886e88e738bfaaaa255287a653ef546c2706b7da7fa03ca28: Status 404 returned error can't find the container with id 1e5e994216ceadf886e88e738bfaaaa255287a653ef546c2706b7da7fa03ca28
Aug 05 17:13:35.452594 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.452559    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-vxqqd"]
Aug 05 17:13:35.455657 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:13:35.455628    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7682b61a_d92f_4b1a_9b7f_e3404237c48f.slice/crio-d798523758b16a3ce3ce5c0a188fad811f076850e7c1a4c21fc40be03a8110e9 WatchSource:0}: Error finding container d798523758b16a3ce3ce5c0a188fad811f076850e7c1a4c21fc40be03a8110e9: Status 404 returned error can't find the container with id d798523758b16a3ce3ce5c0a188fad811f076850e7c1a4c21fc40be03a8110e9
Aug 05 17:13:35.477591 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:35.477564    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c944f9855-dsl5q"]
Aug 05 17:13:35.480357 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:13:35.480329    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6880ea65_e478_448a_a6fe_edc133ef596a.slice/crio-a8553891c4f5559a827f7986895b28aa8b2332364f8a4c673b51b2662d17a6d7 WatchSource:0}: Error finding container a8553891c4f5559a827f7986895b28aa8b2332364f8a4c673b51b2662d17a6d7: Status 404 returned error can't find the container with id a8553891c4f5559a827f7986895b28aa8b2332364f8a4c673b51b2662d17a6d7
Aug 05 17:13:36.010975 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:36.010876    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-vxqqd" event={"ID":"7682b61a-d92f-4b1a-9b7f-e3404237c48f","Type":"ContainerStarted","Data":"d798523758b16a3ce3ce5c0a188fad811f076850e7c1a4c21fc40be03a8110e9"}
Aug 05 17:13:36.012609 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:36.012577    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c944f9855-dsl5q" event={"ID":"6880ea65-e478-448a-a6fe-edc133ef596a","Type":"ContainerStarted","Data":"c11b827d8a2f75626777cf32db53af0d3539575f62f7ec3a7dde87aec0c2fa65"}
Aug 05 17:13:36.012748 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:36.012620    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c944f9855-dsl5q" event={"ID":"6880ea65-e478-448a-a6fe-edc133ef596a","Type":"ContainerStarted","Data":"a8553891c4f5559a827f7986895b28aa8b2332364f8a4c673b51b2662d17a6d7"}
Aug 05 17:13:36.012748 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:36.012661    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c944f9855-dsl5q"
Aug 05 17:13:36.013860 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:36.013830    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b77b65f74-4qpkg" event={"ID":"601cbfbe-36c9-4e75-805d-f0448ed0ebc2","Type":"ContainerStarted","Data":"1e5e994216ceadf886e88e738bfaaaa255287a653ef546c2706b7da7fa03ca28"}
Aug 05 17:13:36.015550 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:36.015516    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gfsf2" event={"ID":"25132135-9fbe-49c4-9f6d-0e9db2cce6b8","Type":"ContainerStarted","Data":"2d75b1fcb1a808d961c43f2b7c311595b3ed0db7b152192e90c0fdfabf2a1ebb"}
Aug 05 17:13:36.017678 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:36.017549    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4bxkx" event={"ID":"00d8f640-e14d-4601-b45d-81ef6fbaf47d","Type":"ContainerStarted","Data":"0306c9522ef52893822219b821dacaa92cf696f01472d4c445b4f57fb9bac9e3"}
Aug 05 17:13:36.017790 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:36.017730    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4bxkx"
Aug 05 17:13:36.017790 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:36.017755    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4bxkx" event={"ID":"00d8f640-e14d-4601-b45d-81ef6fbaf47d","Type":"ContainerStarted","Data":"36ebddbcd9ec7b80da3d5a1d3d08358b10fb1d423734174148db6de34fc445db"}
Aug 05 17:13:36.019583 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:36.019552    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bbcvm" event={"ID":"1cbe388d-927f-4384-9abd-49d8864afe2c","Type":"ContainerStarted","Data":"f7eb3bf087b034163f1e6b30b23de8203638a565ab9aa835ec9e647bd645c78e"}
Aug 05 17:13:36.019694 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:36.019679    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:13:36.038945 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:36.038884    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7c944f9855-dsl5q" podStartSLOduration=2.03886439 podStartE2EDuration="2.03886439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:13:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:36.038516174 +0000 UTC m=+71.783150503" watchObservedRunningTime="2026-08-05 17:13:36.03886439 +0000 UTC m=+71.783498717"
Aug 05 17:13:36.128526 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:36.128460    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4bxkx" podStartSLOduration=33.696265649 podStartE2EDuration="38.128436226s" totalImagesPullingTime="4.432170577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:12:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:36.06797554 +0000 UTC m=+71.812609865" watchObservedRunningTime="2026-08-05 17:13:36.128436226 +0000 UTC m=+71.873070556"
Aug 05 17:13:36.128926 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:36.128893    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bbcvm" podStartSLOduration=67.109677601 podStartE2EDuration="1m12.128882449s" totalImagesPullingTime="5.019204848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:12:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:36.127249171 +0000 UTC m=+71.871883500" watchObservedRunningTime="2026-08-05 17:13:36.128882449 +0000 UTC m=+71.873516780"
Aug 05 17:13:36.174973 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:36.174901    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gfsf2" podStartSLOduration=67.129952797 podStartE2EDuration="1m12.174878692s" totalImagesPullingTime="5.044925895s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 17:12:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:36.171348819 +0000 UTC m=+71.915983147" watchObservedRunningTime="2026-08-05 17:13:36.174878692 +0000 UTC m=+71.919513024"
Aug 05 17:13:40.036265 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:40.036221    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b77b65f74-4qpkg" event={"ID":"601cbfbe-36c9-4e75-805d-f0448ed0ebc2","Type":"ContainerStarted","Data":"0037db8a4e1e89ab0b2efdadbe4da75e7f5ef26727b3de92d782dd9761556d8f"}
Aug 05 17:13:40.036752 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:40.036371    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b77b65f74-4qpkg"
Aug 05 17:13:40.037798 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:40.037774    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-vxqqd" event={"ID":"7682b61a-d92f-4b1a-9b7f-e3404237c48f","Type":"ContainerStarted","Data":"b29f31b5d3f03198d89832f996d7893e43846c78e2dd2a4ec3c70100391e2e35"}
Aug 05 17:13:40.038004 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:40.037980    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-vxqqd"
Aug 05 17:13:40.038292 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:40.038272    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b77b65f74-4qpkg"
Aug 05 17:13:40.042507 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:40.042488    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-vxqqd"
Aug 05 17:13:40.055028 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:40.054985    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b77b65f74-4qpkg" podStartSLOduration=2.395352701 podStartE2EDuration="6.054966524s" totalImagesPullingTime="3.659613823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:13:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:40.053118778 +0000 UTC m=+75.797753105" watchObservedRunningTime="2026-08-05 17:13:40.054966524 +0000 UTC m=+75.799600845"
Aug 05 17:13:40.087748 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:40.087703    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-vxqqd" podStartSLOduration=2.589782675 podStartE2EDuration="6.0876885s" totalImagesPullingTime="3.497905825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:13:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:40.087272735 +0000 UTC m=+75.831907063" watchObservedRunningTime="2026-08-05 17:13:40.0876885 +0000 UTC m=+75.832322828"
Aug 05 17:13:40.758805 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:40.758771    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-tcc2h"]
Aug 05 17:13:40.760817 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:40.760801    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-tcc2h"
Aug 05 17:13:40.763545 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:40.763526    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Aug 05 17:13:40.763545 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:40.763535    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Aug 05 17:13:40.764805 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:40.764786    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Aug 05 17:13:40.764888 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:40.764856    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Aug 05 17:13:40.764888 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:40.764857    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Aug 05 17:13:40.764888 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:40.764857    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-lzgh8\""
Aug 05 17:13:40.773741 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:40.773721    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-tcc2h"]
Aug 05 17:13:40.811344 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:40.811308    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/800108ba-5328-4c41-ab17-6efb3927a0ff-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-tcc2h\" (UID: \"800108ba-5328-4c41-ab17-6efb3927a0ff\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-tcc2h"
Aug 05 17:13:40.811344 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:40.811347    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/800108ba-5328-4c41-ab17-6efb3927a0ff-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-tcc2h\" (UID: \"800108ba-5328-4c41-ab17-6efb3927a0ff\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-tcc2h"
Aug 05 17:13:40.811539 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:40.811428    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/800108ba-5328-4c41-ab17-6efb3927a0ff-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-tcc2h\" (UID: \"800108ba-5328-4c41-ab17-6efb3927a0ff\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-tcc2h"
Aug 05 17:13:40.811539 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:40.811449    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lztpb\" (UniqueName: \"kubernetes.io/projected/800108ba-5328-4c41-ab17-6efb3927a0ff-kube-api-access-lztpb\") pod \"prometheus-operator-758d7cb874-tcc2h\" (UID: \"800108ba-5328-4c41-ab17-6efb3927a0ff\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-tcc2h"
Aug 05 17:13:40.912803 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:40.912760    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/800108ba-5328-4c41-ab17-6efb3927a0ff-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-tcc2h\" (UID: \"800108ba-5328-4c41-ab17-6efb3927a0ff\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-tcc2h"
Aug 05 17:13:40.912803 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:40.912804    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lztpb\" (UniqueName: \"kubernetes.io/projected/800108ba-5328-4c41-ab17-6efb3927a0ff-kube-api-access-lztpb\") pod \"prometheus-operator-758d7cb874-tcc2h\" (UID: \"800108ba-5328-4c41-ab17-6efb3927a0ff\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-tcc2h"
Aug 05 17:13:40.913028 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:40.912842    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/800108ba-5328-4c41-ab17-6efb3927a0ff-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-tcc2h\" (UID: \"800108ba-5328-4c41-ab17-6efb3927a0ff\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-tcc2h"
Aug 05 17:13:40.913028 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:40.912862    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/800108ba-5328-4c41-ab17-6efb3927a0ff-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-tcc2h\" (UID: \"800108ba-5328-4c41-ab17-6efb3927a0ff\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-tcc2h"
Aug 05 17:13:40.913494 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:40.913467    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/800108ba-5328-4c41-ab17-6efb3927a0ff-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-tcc2h\" (UID: \"800108ba-5328-4c41-ab17-6efb3927a0ff\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-tcc2h"
Aug 05 17:13:40.915292 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:40.915270    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/800108ba-5328-4c41-ab17-6efb3927a0ff-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-tcc2h\" (UID: \"800108ba-5328-4c41-ab17-6efb3927a0ff\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-tcc2h"
Aug 05 17:13:40.915384 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:40.915347    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/800108ba-5328-4c41-ab17-6efb3927a0ff-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-tcc2h\" (UID: \"800108ba-5328-4c41-ab17-6efb3927a0ff\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-tcc2h"
Aug 05 17:13:40.921930 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:40.921910    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lztpb\" (UniqueName: \"kubernetes.io/projected/800108ba-5328-4c41-ab17-6efb3927a0ff-kube-api-access-lztpb\") pod \"prometheus-operator-758d7cb874-tcc2h\" (UID: \"800108ba-5328-4c41-ab17-6efb3927a0ff\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-tcc2h"
Aug 05 17:13:41.070197 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:41.070155    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-tcc2h"
Aug 05 17:13:41.199594 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:41.199560    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-tcc2h"]
Aug 05 17:13:41.202749 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:13:41.202722    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod800108ba_5328_4c41_ab17_6efb3927a0ff.slice/crio-e6acaa688695245ea9eae20def69519e41e8fd5485903744b9dbc4bc8c75157c WatchSource:0}: Error finding container e6acaa688695245ea9eae20def69519e41e8fd5485903744b9dbc4bc8c75157c: Status 404 returned error can't find the container with id e6acaa688695245ea9eae20def69519e41e8fd5485903744b9dbc4bc8c75157c
Aug 05 17:13:42.046512 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.046470    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-tcc2h" event={"ID":"800108ba-5328-4c41-ab17-6efb3927a0ff","Type":"ContainerStarted","Data":"e6acaa688695245ea9eae20def69519e41e8fd5485903744b9dbc4bc8c75157c"}
Aug 05 17:13:42.572877 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.572851    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85f886795d-9sfjh"]
Aug 05 17:13:42.575593 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.575576    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85f886795d-9sfjh"
Aug 05 17:13:42.578200 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.578174    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Aug 05 17:13:42.578308 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.578252    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Aug 05 17:13:42.579288 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.579270    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Aug 05 17:13:42.579442 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.579422    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Aug 05 17:13:42.579559 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.579545    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Aug 05 17:13:42.579702 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.579684    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Aug 05 17:13:42.580396 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.580377    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Aug 05 17:13:42.580644 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.580630    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ngt69\""
Aug 05 17:13:42.603105 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.603078    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85f886795d-9sfjh"]
Aug 05 17:13:42.627282 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.627252    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-service-ca\") pod \"console-85f886795d-9sfjh\" (UID: \"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108\") " pod="openshift-console/console-85f886795d-9sfjh"
Aug 05 17:13:42.627389 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.627296    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-console-config\") pod \"console-85f886795d-9sfjh\" (UID: \"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108\") " pod="openshift-console/console-85f886795d-9sfjh"
Aug 05 17:13:42.627389 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.627334    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-console-serving-cert\") pod \"console-85f886795d-9sfjh\" (UID: \"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108\") " pod="openshift-console/console-85f886795d-9sfjh"
Aug 05 17:13:42.627460 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.627407    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-console-oauth-config\") pod \"console-85f886795d-9sfjh\" (UID: \"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108\") " pod="openshift-console/console-85f886795d-9sfjh"
Aug 05 17:13:42.627495 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.627463    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65ff5\" (UniqueName: \"kubernetes.io/projected/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-kube-api-access-65ff5\") pod \"console-85f886795d-9sfjh\" (UID: \"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108\") " pod="openshift-console/console-85f886795d-9sfjh"
Aug 05 17:13:42.627527 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.627500    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-oauth-serving-cert\") pod \"console-85f886795d-9sfjh\" (UID: \"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108\") " pod="openshift-console/console-85f886795d-9sfjh"
Aug 05 17:13:42.728190 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.728161    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-service-ca\") pod \"console-85f886795d-9sfjh\" (UID: \"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108\") " pod="openshift-console/console-85f886795d-9sfjh"
Aug 05 17:13:42.728190 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.728191    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-console-config\") pod \"console-85f886795d-9sfjh\" (UID: \"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108\") " pod="openshift-console/console-85f886795d-9sfjh"
Aug 05 17:13:42.728462 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.728211    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-console-serving-cert\") pod \"console-85f886795d-9sfjh\" (UID: \"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108\") " pod="openshift-console/console-85f886795d-9sfjh"
Aug 05 17:13:42.728462 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.728234    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-console-oauth-config\") pod \"console-85f886795d-9sfjh\" (UID: \"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108\") " pod="openshift-console/console-85f886795d-9sfjh"
Aug 05 17:13:42.728462 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.728265    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65ff5\" (UniqueName: \"kubernetes.io/projected/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-kube-api-access-65ff5\") pod \"console-85f886795d-9sfjh\" (UID: \"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108\") " pod="openshift-console/console-85f886795d-9sfjh"
Aug 05 17:13:42.728462 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.728290    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-oauth-serving-cert\") pod \"console-85f886795d-9sfjh\" (UID: \"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108\") " pod="openshift-console/console-85f886795d-9sfjh"
Aug 05 17:13:42.729051 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.729027    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-console-config\") pod \"console-85f886795d-9sfjh\" (UID: \"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108\") " pod="openshift-console/console-85f886795d-9sfjh"
Aug 05 17:13:42.729172 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.729062    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-service-ca\") pod \"console-85f886795d-9sfjh\" (UID: \"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108\") " pod="openshift-console/console-85f886795d-9sfjh"
Aug 05 17:13:42.729172 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.729080    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-oauth-serving-cert\") pod \"console-85f886795d-9sfjh\" (UID: \"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108\") " pod="openshift-console/console-85f886795d-9sfjh"
Aug 05 17:13:42.730915 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.730891    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-console-oauth-config\") pod \"console-85f886795d-9sfjh\" (UID: \"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108\") " pod="openshift-console/console-85f886795d-9sfjh"
Aug 05 17:13:42.731022 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.730921    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-console-serving-cert\") pod \"console-85f886795d-9sfjh\" (UID: \"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108\") " pod="openshift-console/console-85f886795d-9sfjh"
Aug 05 17:13:42.742223 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.742197    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65ff5\" (UniqueName: \"kubernetes.io/projected/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-kube-api-access-65ff5\") pod \"console-85f886795d-9sfjh\" (UID: \"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108\") " pod="openshift-console/console-85f886795d-9sfjh"
Aug 05 17:13:42.885092 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:42.885040    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85f886795d-9sfjh"
Aug 05 17:13:43.013469 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:43.013430    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85f886795d-9sfjh"]
Aug 05 17:13:43.017946 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:13:43.017915    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc1f6ca_ba3a_47c7_b6b2_b08dfe51b108.slice/crio-e587a4eb422f46b53b5124d34bbff1c2695d48f50362e738954b5153a1da1f68 WatchSource:0}: Error finding container e587a4eb422f46b53b5124d34bbff1c2695d48f50362e738954b5153a1da1f68: Status 404 returned error can't find the container with id e587a4eb422f46b53b5124d34bbff1c2695d48f50362e738954b5153a1da1f68
Aug 05 17:13:43.051434 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:43.051396    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-tcc2h" event={"ID":"800108ba-5328-4c41-ab17-6efb3927a0ff","Type":"ContainerStarted","Data":"8f0bb2fbf9101d221eac0942b9bf948928a79224f2dd5be3cad15cab57b23ac2"}
Aug 05 17:13:43.051619 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:43.051465    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-tcc2h" event={"ID":"800108ba-5328-4c41-ab17-6efb3927a0ff","Type":"ContainerStarted","Data":"79607c3b2047b96b6d86651ab19094be822dd62141147b859fea063ce56de120"}
Aug 05 17:13:43.052542 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:43.052517    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85f886795d-9sfjh" event={"ID":"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108","Type":"ContainerStarted","Data":"e587a4eb422f46b53b5124d34bbff1c2695d48f50362e738954b5153a1da1f68"}
Aug 05 17:13:43.075630 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:43.075583    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-758d7cb874-tcc2h" podStartSLOduration=1.751221465 podStartE2EDuration="3.075569318s" totalImagesPullingTime="1.324347853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:43.074041628 +0000 UTC m=+78.818675956" watchObservedRunningTime="2026-08-05 17:13:43.075569318 +0000 UTC m=+78.820203645"
Aug 05 17:13:45.188684 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.188649    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-922nb"]
Aug 05 17:13:45.193885 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.193850    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-922nb"
Aug 05 17:13:45.197782 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.197565    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Aug 05 17:13:45.197782 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.197587    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Aug 05 17:13:45.197782 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.197651    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lzm78\""
Aug 05 17:13:45.198062 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.197906    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Aug 05 17:13:45.247555 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.247518    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2091dbfe-3a2a-44be-980e-d3655d607668-node-exporter-textfile\") pod \"node-exporter-922nb\" (UID: \"2091dbfe-3a2a-44be-980e-d3655d607668\") " pod="openshift-monitoring/node-exporter-922nb"
Aug 05 17:13:45.247744 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.247562    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2091dbfe-3a2a-44be-980e-d3655d607668-node-exporter-accelerators-collector-config\") pod \"node-exporter-922nb\" (UID: \"2091dbfe-3a2a-44be-980e-d3655d607668\") " pod="openshift-monitoring/node-exporter-922nb"
Aug 05 17:13:45.247744 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.247600    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2091dbfe-3a2a-44be-980e-d3655d607668-node-exporter-tls\") pod \"node-exporter-922nb\" (UID: \"2091dbfe-3a2a-44be-980e-d3655d607668\") " pod="openshift-monitoring/node-exporter-922nb"
Aug 05 17:13:45.247744 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.247677    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2091dbfe-3a2a-44be-980e-d3655d607668-metrics-client-ca\") pod \"node-exporter-922nb\" (UID: \"2091dbfe-3a2a-44be-980e-d3655d607668\") " pod="openshift-monitoring/node-exporter-922nb"
Aug 05 17:13:45.247895 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.247762    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2091dbfe-3a2a-44be-980e-d3655d607668-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-922nb\" (UID: \"2091dbfe-3a2a-44be-980e-d3655d607668\") " pod="openshift-monitoring/node-exporter-922nb"
Aug 05 17:13:45.247895 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.247817    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2091dbfe-3a2a-44be-980e-d3655d607668-node-exporter-wtmp\") pod \"node-exporter-922nb\" (UID: \"2091dbfe-3a2a-44be-980e-d3655d607668\") " pod="openshift-monitoring/node-exporter-922nb"
Aug 05 17:13:45.247895 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.247863    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2091dbfe-3a2a-44be-980e-d3655d607668-root\") pod \"node-exporter-922nb\" (UID: \"2091dbfe-3a2a-44be-980e-d3655d607668\") " pod="openshift-monitoring/node-exporter-922nb"
Aug 05 17:13:45.247895 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.247887    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2mzw\" (UniqueName: \"kubernetes.io/projected/2091dbfe-3a2a-44be-980e-d3655d607668-kube-api-access-z2mzw\") pod \"node-exporter-922nb\" (UID: \"2091dbfe-3a2a-44be-980e-d3655d607668\") " pod="openshift-monitoring/node-exporter-922nb"
Aug 05 17:13:45.248084 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.247921    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2091dbfe-3a2a-44be-980e-d3655d607668-sys\") pod \"node-exporter-922nb\" (UID: \"2091dbfe-3a2a-44be-980e-d3655d607668\") " pod="openshift-monitoring/node-exporter-922nb"
Aug 05 17:13:45.348345 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.348303    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2091dbfe-3a2a-44be-980e-d3655d607668-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-922nb\" (UID: \"2091dbfe-3a2a-44be-980e-d3655d607668\") " pod="openshift-monitoring/node-exporter-922nb"
Aug 05 17:13:45.348541 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.348363    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2091dbfe-3a2a-44be-980e-d3655d607668-node-exporter-wtmp\") pod \"node-exporter-922nb\" (UID: \"2091dbfe-3a2a-44be-980e-d3655d607668\") " pod="openshift-monitoring/node-exporter-922nb"
Aug 05 17:13:45.348541 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.348395    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2091dbfe-3a2a-44be-980e-d3655d607668-root\") pod \"node-exporter-922nb\" (UID: \"2091dbfe-3a2a-44be-980e-d3655d607668\") " pod="openshift-monitoring/node-exporter-922nb"
Aug 05 17:13:45.348541 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.348411    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2mzw\" (UniqueName: \"kubernetes.io/projected/2091dbfe-3a2a-44be-980e-d3655d607668-kube-api-access-z2mzw\") pod \"node-exporter-922nb\" (UID: \"2091dbfe-3a2a-44be-980e-d3655d607668\") " pod="openshift-monitoring/node-exporter-922nb"
Aug 05 17:13:45.348541 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.348436    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2091dbfe-3a2a-44be-980e-d3655d607668-sys\") pod \"node-exporter-922nb\" (UID: \"2091dbfe-3a2a-44be-980e-d3655d607668\") " pod="openshift-monitoring/node-exporter-922nb"
Aug 05 17:13:45.348541 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.348474    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2091dbfe-3a2a-44be-980e-d3655d607668-node-exporter-textfile\") pod \"node-exporter-922nb\" (UID: \"2091dbfe-3a2a-44be-980e-d3655d607668\") " pod="openshift-monitoring/node-exporter-922nb"
Aug 05 17:13:45.348541 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.348497    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2091dbfe-3a2a-44be-980e-d3655d607668-node-exporter-accelerators-collector-config\") pod \"node-exporter-922nb\" (UID: \"2091dbfe-3a2a-44be-980e-d3655d607668\") " pod="openshift-monitoring/node-exporter-922nb"
Aug 05 17:13:45.348541 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.348506    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2091dbfe-3a2a-44be-980e-d3655d607668-root\") pod \"node-exporter-922nb\" (UID: \"2091dbfe-3a2a-44be-980e-d3655d607668\") " pod="openshift-monitoring/node-exporter-922nb"
Aug 05 17:13:45.348885 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.348567    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2091dbfe-3a2a-44be-980e-d3655d607668-sys\") pod \"node-exporter-922nb\" (UID: \"2091dbfe-3a2a-44be-980e-d3655d607668\") " pod="openshift-monitoring/node-exporter-922nb"
Aug 05 17:13:45.348885 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.348644    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2091dbfe-3a2a-44be-980e-d3655d607668-node-exporter-wtmp\") pod \"node-exporter-922nb\" (UID: \"2091dbfe-3a2a-44be-980e-d3655d607668\") " pod="openshift-monitoring/node-exporter-922nb"
Aug 05 17:13:45.348885 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.348776    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2091dbfe-3a2a-44be-980e-d3655d607668-node-exporter-tls\") pod \"node-exporter-922nb\" (UID: \"2091dbfe-3a2a-44be-980e-d3655d607668\") " pod="openshift-monitoring/node-exporter-922nb"
Aug 05 17:13:45.348885 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.348837    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2091dbfe-3a2a-44be-980e-d3655d607668-metrics-client-ca\") pod \"node-exporter-922nb\" (UID: \"2091dbfe-3a2a-44be-980e-d3655d607668\") " pod="openshift-monitoring/node-exporter-922nb"
Aug 05 17:13:45.348885 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.348874    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2091dbfe-3a2a-44be-980e-d3655d607668-node-exporter-textfile\") pod \"node-exporter-922nb\" (UID: \"2091dbfe-3a2a-44be-980e-d3655d607668\") " pod="openshift-monitoring/node-exporter-922nb"
Aug 05 17:13:45.349265 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.349186    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2091dbfe-3a2a-44be-980e-d3655d607668-node-exporter-accelerators-collector-config\") pod \"node-exporter-922nb\" (UID: \"2091dbfe-3a2a-44be-980e-d3655d607668\") " pod="openshift-monitoring/node-exporter-922nb"
Aug 05 17:13:45.349383 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.349361    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2091dbfe-3a2a-44be-980e-d3655d607668-metrics-client-ca\") pod \"node-exporter-922nb\" (UID: \"2091dbfe-3a2a-44be-980e-d3655d607668\") " pod="openshift-monitoring/node-exporter-922nb"
Aug 05 17:13:45.351107 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.351085    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2091dbfe-3a2a-44be-980e-d3655d607668-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-922nb\" (UID: \"2091dbfe-3a2a-44be-980e-d3655d607668\") " pod="openshift-monitoring/node-exporter-922nb"
Aug 05 17:13:45.351410 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.351387    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2091dbfe-3a2a-44be-980e-d3655d607668-node-exporter-tls\") pod \"node-exporter-922nb\" (UID: \"2091dbfe-3a2a-44be-980e-d3655d607668\") " pod="openshift-monitoring/node-exporter-922nb"
Aug 05 17:13:45.362582 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.362556    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2mzw\" (UniqueName: \"kubernetes.io/projected/2091dbfe-3a2a-44be-980e-d3655d607668-kube-api-access-z2mzw\") pod \"node-exporter-922nb\" (UID: \"2091dbfe-3a2a-44be-980e-d3655d607668\") " pod="openshift-monitoring/node-exporter-922nb"
Aug 05 17:13:45.506485 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:45.506398    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-922nb"
Aug 05 17:13:45.556945 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:13:45.556916    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2091dbfe_3a2a_44be_980e_d3655d607668.slice/crio-48455f780dbd62a400cd3404f1fe7941cfabcd242bb5b9dea799ba7f6108876b WatchSource:0}: Error finding container 48455f780dbd62a400cd3404f1fe7941cfabcd242bb5b9dea799ba7f6108876b: Status 404 returned error can't find the container with id 48455f780dbd62a400cd3404f1fe7941cfabcd242bb5b9dea799ba7f6108876b
Aug 05 17:13:46.028299 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:46.028269    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4bxkx"
Aug 05 17:13:46.067684 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:46.067615    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85f886795d-9sfjh" event={"ID":"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108","Type":"ContainerStarted","Data":"64f5f0c16920ecd3501ceb1fa8578016e68d865c7af8286c0438342c9c61914b"}
Aug 05 17:13:46.069366 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:46.069335    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-922nb" event={"ID":"2091dbfe-3a2a-44be-980e-d3655d607668","Type":"ContainerStarted","Data":"48455f780dbd62a400cd3404f1fe7941cfabcd242bb5b9dea799ba7f6108876b"}
Aug 05 17:13:46.099357 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:46.099298    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85f886795d-9sfjh" podStartSLOduration=1.529020534 podStartE2EDuration="4.099275988s" totalImagesPullingTime="2.570255454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:13:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:46.09905756 +0000 UTC m=+81.843691885" watchObservedRunningTime="2026-08-05 17:13:46.099275988 +0000 UTC m=+81.843910314"
Aug 05 17:13:47.074379 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:47.074347    2561 generic.go:356] "Generic (PLEG): container finished" podID="2091dbfe-3a2a-44be-980e-d3655d607668" containerID="c623ac1ab63034d7602f53fa3793f7c6aab069d659b895d74acf278179fa47cb" exitCode=0
Aug 05 17:13:47.074834 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:47.074432    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-922nb" event={"ID":"2091dbfe-3a2a-44be-980e-d3655d607668","Type":"ContainerDied","Data":"c623ac1ab63034d7602f53fa3793f7c6aab069d659b895d74acf278179fa47cb"}
Aug 05 17:13:48.080438 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:48.080391    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-922nb" event={"ID":"2091dbfe-3a2a-44be-980e-d3655d607668","Type":"ContainerStarted","Data":"d470319f2d7fb70be985619252ec97df3b45b4dbdfccee97b4e8e78eef1abb11"}
Aug 05 17:13:48.080892 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:48.080512    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-922nb" event={"ID":"2091dbfe-3a2a-44be-980e-d3655d607668","Type":"ContainerStarted","Data":"6e55d8b93ba2f0e14f21cfee629f800b18dab023315e35d83ee608150a56e502"}
Aug 05 17:13:49.585609 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.585550    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-922nb" podStartSLOduration=3.683007106 podStartE2EDuration="4.585531311s" totalImagesPullingTime="834.978753ms" totalInitContainerRuntime="67.545452ms" isStatefulPod=true podCreationTimestamp="2026-08-05 17:13:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:48.107168213 +0000 UTC m=+83.851802541" watchObservedRunningTime="2026-08-05 17:13:49.585531311 +0000 UTC m=+85.330165637"
Aug 05 17:13:49.586801 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.586776    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-776db5ccf4-n6kq7"]
Aug 05 17:13:49.588879 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.588858    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-776db5ccf4-n6kq7"
Aug 05 17:13:49.592525 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.592490    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Aug 05 17:13:49.592525 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.592514    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Aug 05 17:13:49.592683 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.592594    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-chmwt\""
Aug 05 17:13:49.593508 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.593494    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Aug 05 17:13:49.593572 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.593522    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Aug 05 17:13:49.593616 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.593522    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8ohgb0rsevrof\""
Aug 05 17:13:49.601288 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.601267    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-776db5ccf4-n6kq7"]
Aug 05 17:13:49.687294 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.687256    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49d65\" (UniqueName: \"kubernetes.io/projected/b3428980-76ee-439e-963b-368def9cdb48-kube-api-access-49d65\") pod \"metrics-server-776db5ccf4-n6kq7\" (UID: \"b3428980-76ee-439e-963b-368def9cdb48\") " pod="openshift-monitoring/metrics-server-776db5ccf4-n6kq7"
Aug 05 17:13:49.687294 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.687299    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b3428980-76ee-439e-963b-368def9cdb48-client-ca-bundle\") pod \"metrics-server-776db5ccf4-n6kq7\" (UID: \"b3428980-76ee-439e-963b-368def9cdb48\") " pod="openshift-monitoring/metrics-server-776db5ccf4-n6kq7"
Aug 05 17:13:49.687522 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.687325    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b3428980-76ee-439e-963b-368def9cdb48-secret-metrics-server-client-certs\") pod \"metrics-server-776db5ccf4-n6kq7\" (UID: \"b3428980-76ee-439e-963b-368def9cdb48\") " pod="openshift-monitoring/metrics-server-776db5ccf4-n6kq7"
Aug 05 17:13:49.687522 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.687354    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3428980-76ee-439e-963b-368def9cdb48-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-776db5ccf4-n6kq7\" (UID: \"b3428980-76ee-439e-963b-368def9cdb48\") " pod="openshift-monitoring/metrics-server-776db5ccf4-n6kq7"
Aug 05 17:13:49.687522 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.687381    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b3428980-76ee-439e-963b-368def9cdb48-secret-metrics-server-tls\") pod \"metrics-server-776db5ccf4-n6kq7\" (UID: \"b3428980-76ee-439e-963b-368def9cdb48\") " pod="openshift-monitoring/metrics-server-776db5ccf4-n6kq7"
Aug 05 17:13:49.687522 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.687397    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b3428980-76ee-439e-963b-368def9cdb48-audit-log\") pod \"metrics-server-776db5ccf4-n6kq7\" (UID: \"b3428980-76ee-439e-963b-368def9cdb48\") " pod="openshift-monitoring/metrics-server-776db5ccf4-n6kq7"
Aug 05 17:13:49.687522 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.687478    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b3428980-76ee-439e-963b-368def9cdb48-metrics-server-audit-profiles\") pod \"metrics-server-776db5ccf4-n6kq7\" (UID: \"b3428980-76ee-439e-963b-368def9cdb48\") " pod="openshift-monitoring/metrics-server-776db5ccf4-n6kq7"
Aug 05 17:13:49.788492 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.788454    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b3428980-76ee-439e-963b-368def9cdb48-metrics-server-audit-profiles\") pod \"metrics-server-776db5ccf4-n6kq7\" (UID: \"b3428980-76ee-439e-963b-368def9cdb48\") " pod="openshift-monitoring/metrics-server-776db5ccf4-n6kq7"
Aug 05 17:13:49.788492 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.788495    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49d65\" (UniqueName: \"kubernetes.io/projected/b3428980-76ee-439e-963b-368def9cdb48-kube-api-access-49d65\") pod \"metrics-server-776db5ccf4-n6kq7\" (UID: \"b3428980-76ee-439e-963b-368def9cdb48\") " pod="openshift-monitoring/metrics-server-776db5ccf4-n6kq7"
Aug 05 17:13:49.788697 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.788629    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b3428980-76ee-439e-963b-368def9cdb48-client-ca-bundle\") pod \"metrics-server-776db5ccf4-n6kq7\" (UID: \"b3428980-76ee-439e-963b-368def9cdb48\") " pod="openshift-monitoring/metrics-server-776db5ccf4-n6kq7"
Aug 05 17:13:49.788697 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.788682    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b3428980-76ee-439e-963b-368def9cdb48-secret-metrics-server-client-certs\") pod \"metrics-server-776db5ccf4-n6kq7\" (UID: \"b3428980-76ee-439e-963b-368def9cdb48\") " pod="openshift-monitoring/metrics-server-776db5ccf4-n6kq7"
Aug 05 17:13:49.788822 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.788799    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3428980-76ee-439e-963b-368def9cdb48-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-776db5ccf4-n6kq7\" (UID: \"b3428980-76ee-439e-963b-368def9cdb48\") " pod="openshift-monitoring/metrics-server-776db5ccf4-n6kq7"
Aug 05 17:13:49.788877 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.788822    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b3428980-76ee-439e-963b-368def9cdb48-secret-metrics-server-tls\") pod \"metrics-server-776db5ccf4-n6kq7\" (UID: \"b3428980-76ee-439e-963b-368def9cdb48\") " pod="openshift-monitoring/metrics-server-776db5ccf4-n6kq7"
Aug 05 17:13:49.788877 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.788839    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b3428980-76ee-439e-963b-368def9cdb48-audit-log\") pod \"metrics-server-776db5ccf4-n6kq7\" (UID: \"b3428980-76ee-439e-963b-368def9cdb48\") " pod="openshift-monitoring/metrics-server-776db5ccf4-n6kq7"
Aug 05 17:13:49.789173 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.789153    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b3428980-76ee-439e-963b-368def9cdb48-audit-log\") pod \"metrics-server-776db5ccf4-n6kq7\" (UID: \"b3428980-76ee-439e-963b-368def9cdb48\") " pod="openshift-monitoring/metrics-server-776db5ccf4-n6kq7"
Aug 05 17:13:49.789494 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.789474    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3428980-76ee-439e-963b-368def9cdb48-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-776db5ccf4-n6kq7\" (UID: \"b3428980-76ee-439e-963b-368def9cdb48\") " pod="openshift-monitoring/metrics-server-776db5ccf4-n6kq7"
Aug 05 17:13:49.789591 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.789504    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b3428980-76ee-439e-963b-368def9cdb48-metrics-server-audit-profiles\") pod \"metrics-server-776db5ccf4-n6kq7\" (UID: \"b3428980-76ee-439e-963b-368def9cdb48\") " pod="openshift-monitoring/metrics-server-776db5ccf4-n6kq7"
Aug 05 17:13:49.791294 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.791263    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b3428980-76ee-439e-963b-368def9cdb48-secret-metrics-server-client-certs\") pod \"metrics-server-776db5ccf4-n6kq7\" (UID: \"b3428980-76ee-439e-963b-368def9cdb48\") " pod="openshift-monitoring/metrics-server-776db5ccf4-n6kq7"
Aug 05 17:13:49.791404 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.791309    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b3428980-76ee-439e-963b-368def9cdb48-client-ca-bundle\") pod \"metrics-server-776db5ccf4-n6kq7\" (UID: \"b3428980-76ee-439e-963b-368def9cdb48\") " pod="openshift-monitoring/metrics-server-776db5ccf4-n6kq7"
Aug 05 17:13:49.791676 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.791659    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b3428980-76ee-439e-963b-368def9cdb48-secret-metrics-server-tls\") pod \"metrics-server-776db5ccf4-n6kq7\" (UID: \"b3428980-76ee-439e-963b-368def9cdb48\") " pod="openshift-monitoring/metrics-server-776db5ccf4-n6kq7"
Aug 05 17:13:49.797199 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.797175    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49d65\" (UniqueName: \"kubernetes.io/projected/b3428980-76ee-439e-963b-368def9cdb48-kube-api-access-49d65\") pod \"metrics-server-776db5ccf4-n6kq7\" (UID: \"b3428980-76ee-439e-963b-368def9cdb48\") " pod="openshift-monitoring/metrics-server-776db5ccf4-n6kq7"
Aug 05 17:13:49.898567 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:49.898475    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-776db5ccf4-n6kq7"
Aug 05 17:13:50.035790 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:50.035718    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-776db5ccf4-n6kq7"]
Aug 05 17:13:50.038389 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:13:50.038355    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3428980_76ee_439e_963b_368def9cdb48.slice/crio-b0273851257ec13a75abe76575b8798bdcf4bf611a0142bd20b3c27fb2327e0e WatchSource:0}: Error finding container b0273851257ec13a75abe76575b8798bdcf4bf611a0142bd20b3c27fb2327e0e: Status 404 returned error can't find the container with id b0273851257ec13a75abe76575b8798bdcf4bf611a0142bd20b3c27fb2327e0e
Aug 05 17:13:50.091972 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:50.091915    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-776db5ccf4-n6kq7" event={"ID":"b3428980-76ee-439e-963b-368def9cdb48","Type":"ContainerStarted","Data":"b0273851257ec13a75abe76575b8798bdcf4bf611a0142bd20b3c27fb2327e0e"}
Aug 05 17:13:52.103160 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.103127    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-776db5ccf4-n6kq7" event={"ID":"b3428980-76ee-439e-963b-368def9cdb48","Type":"ContainerStarted","Data":"721a36b3b3afafdef99cfbb40f28c18bdf0d2046838d13f2c3096cb1038b2a61"}
Aug 05 17:13:52.128377 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.128327    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-776db5ccf4-n6kq7" podStartSLOduration=1.6193197000000001 podStartE2EDuration="3.128312404s" totalImagesPullingTime="1.508992704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:13:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:52.127362494 +0000 UTC m=+87.871996824" watchObservedRunningTime="2026-08-05 17:13:52.128312404 +0000 UTC m=+87.872946732"
Aug 05 17:13:52.472042 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.471937    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d667956d9-jhpsr"]
Aug 05 17:13:52.473972 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.473935    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d667956d9-jhpsr"
Aug 05 17:13:52.487505 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.487480    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Aug 05 17:13:52.499456 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.499433    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d667956d9-jhpsr"]
Aug 05 17:13:52.612282 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.612247    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-service-ca\") pod \"console-d667956d9-jhpsr\" (UID: \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\") " pod="openshift-console/console-d667956d9-jhpsr"
Aug 05 17:13:52.612282 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.612290    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-oauth-serving-cert\") pod \"console-d667956d9-jhpsr\" (UID: \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\") " pod="openshift-console/console-d667956d9-jhpsr"
Aug 05 17:13:52.612497 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.612312    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-console-serving-cert\") pod \"console-d667956d9-jhpsr\" (UID: \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\") " pod="openshift-console/console-d667956d9-jhpsr"
Aug 05 17:13:52.612497 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.612346    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj8lp\" (UniqueName: \"kubernetes.io/projected/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-kube-api-access-nj8lp\") pod \"console-d667956d9-jhpsr\" (UID: \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\") " pod="openshift-console/console-d667956d9-jhpsr"
Aug 05 17:13:52.612497 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.612374    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-console-oauth-config\") pod \"console-d667956d9-jhpsr\" (UID: \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\") " pod="openshift-console/console-d667956d9-jhpsr"
Aug 05 17:13:52.612497 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.612437    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-console-config\") pod \"console-d667956d9-jhpsr\" (UID: \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\") " pod="openshift-console/console-d667956d9-jhpsr"
Aug 05 17:13:52.612497 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.612453    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-trusted-ca-bundle\") pod \"console-d667956d9-jhpsr\" (UID: \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\") " pod="openshift-console/console-d667956d9-jhpsr"
Aug 05 17:13:52.713274 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.713240    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-service-ca\") pod \"console-d667956d9-jhpsr\" (UID: \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\") " pod="openshift-console/console-d667956d9-jhpsr"
Aug 05 17:13:52.713462 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.713283    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-oauth-serving-cert\") pod \"console-d667956d9-jhpsr\" (UID: \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\") " pod="openshift-console/console-d667956d9-jhpsr"
Aug 05 17:13:52.713462 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.713308    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-console-serving-cert\") pod \"console-d667956d9-jhpsr\" (UID: \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\") " pod="openshift-console/console-d667956d9-jhpsr"
Aug 05 17:13:52.713462 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.713330    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj8lp\" (UniqueName: \"kubernetes.io/projected/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-kube-api-access-nj8lp\") pod \"console-d667956d9-jhpsr\" (UID: \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\") " pod="openshift-console/console-d667956d9-jhpsr"
Aug 05 17:13:52.713462 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.713382    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-console-oauth-config\") pod \"console-d667956d9-jhpsr\" (UID: \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\") " pod="openshift-console/console-d667956d9-jhpsr"
Aug 05 17:13:52.713617 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.713463    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-console-config\") pod \"console-d667956d9-jhpsr\" (UID: \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\") " pod="openshift-console/console-d667956d9-jhpsr"
Aug 05 17:13:52.713758 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.713726    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-trusted-ca-bundle\") pod \"console-d667956d9-jhpsr\" (UID: \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\") " pod="openshift-console/console-d667956d9-jhpsr"
Aug 05 17:13:52.714064 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.714041    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-service-ca\") pod \"console-d667956d9-jhpsr\" (UID: \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\") " pod="openshift-console/console-d667956d9-jhpsr"
Aug 05 17:13:52.714205 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.714182    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-oauth-serving-cert\") pod \"console-d667956d9-jhpsr\" (UID: \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\") " pod="openshift-console/console-d667956d9-jhpsr"
Aug 05 17:13:52.714288 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.714236    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-console-config\") pod \"console-d667956d9-jhpsr\" (UID: \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\") " pod="openshift-console/console-d667956d9-jhpsr"
Aug 05 17:13:52.714728 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.714706    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-trusted-ca-bundle\") pod \"console-d667956d9-jhpsr\" (UID: \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\") " pod="openshift-console/console-d667956d9-jhpsr"
Aug 05 17:13:52.715930 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.715912    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-console-serving-cert\") pod \"console-d667956d9-jhpsr\" (UID: \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\") " pod="openshift-console/console-d667956d9-jhpsr"
Aug 05 17:13:52.716088 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.716069    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-console-oauth-config\") pod \"console-d667956d9-jhpsr\" (UID: \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\") " pod="openshift-console/console-d667956d9-jhpsr"
Aug 05 17:13:52.722284 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.722234    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj8lp\" (UniqueName: \"kubernetes.io/projected/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-kube-api-access-nj8lp\") pod \"console-d667956d9-jhpsr\" (UID: \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\") " pod="openshift-console/console-d667956d9-jhpsr"
Aug 05 17:13:52.783127 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.783098    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d667956d9-jhpsr"
Aug 05 17:13:52.886183 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.886152    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85f886795d-9sfjh"
Aug 05 17:13:52.886393 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.886222    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85f886795d-9sfjh"
Aug 05 17:13:52.890875 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.890849    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85f886795d-9sfjh"
Aug 05 17:13:52.927734 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:52.927701    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d667956d9-jhpsr"]
Aug 05 17:13:52.928731 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:13:52.928703    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a14595_bc1e_4306_8d81_b2fba8b9cb9e.slice/crio-73c3c098419540daddfeda00e904c2c1f565abe6af02f870f80628eb720f7aa9 WatchSource:0}: Error finding container 73c3c098419540daddfeda00e904c2c1f565abe6af02f870f80628eb720f7aa9: Status 404 returned error can't find the container with id 73c3c098419540daddfeda00e904c2c1f565abe6af02f870f80628eb720f7aa9
Aug 05 17:13:53.108348 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:53.108304    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d667956d9-jhpsr" event={"ID":"96a14595-bc1e-4306-8d81-b2fba8b9cb9e","Type":"ContainerStarted","Data":"bc93a4beb8104819e291323e906e651db186b984812e34bfd97135f5b7691f5f"}
Aug 05 17:13:53.108759 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:53.108372    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d667956d9-jhpsr" event={"ID":"96a14595-bc1e-4306-8d81-b2fba8b9cb9e","Type":"ContainerStarted","Data":"73c3c098419540daddfeda00e904c2c1f565abe6af02f870f80628eb720f7aa9"}
Aug 05 17:13:53.112162 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:53.112141    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85f886795d-9sfjh"
Aug 05 17:13:53.132264 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:53.132207    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d667956d9-jhpsr" podStartSLOduration=1.13218702 podStartE2EDuration="1.13218702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:13:53.131984926 +0000 UTC m=+88.876619253" watchObservedRunningTime="2026-08-05 17:13:53.13218702 +0000 UTC m=+88.876821348"
Aug 05 17:13:54.858110 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:54.858084    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:13:57.029072 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:57.029040    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c944f9855-dsl5q"
Aug 05 17:13:59.876907 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:13:59.876868    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-76b686c878-267wj" podUID="aba3d093-7afd-4f0e-8407-895e98526798" containerName="registry" containerID="cri-o://49ca1ae00804a17210c8806190844efd0a5944fb5d9b017bb78b26c0089de81b" gracePeriod=30
Aug 05 17:14:00.106801 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.106773    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:14:00.137867 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.137801    2561 generic.go:356] "Generic (PLEG): container finished" podID="aba3d093-7afd-4f0e-8407-895e98526798" containerID="49ca1ae00804a17210c8806190844efd0a5944fb5d9b017bb78b26c0089de81b" exitCode=0
Aug 05 17:14:00.137867 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.137839    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76b686c878-267wj"
Aug 05 17:14:00.138077 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.137877    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76b686c878-267wj" event={"ID":"aba3d093-7afd-4f0e-8407-895e98526798","Type":"ContainerDied","Data":"49ca1ae00804a17210c8806190844efd0a5944fb5d9b017bb78b26c0089de81b"}
Aug 05 17:14:00.138077 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.137921    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76b686c878-267wj" event={"ID":"aba3d093-7afd-4f0e-8407-895e98526798","Type":"ContainerDied","Data":"eda8ac6fac8906f59151a9ac84507a141edf6c993ab040dee29cad7edb152c55"}
Aug 05 17:14:00.138578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.138566    2561 scope.go:117] "RemoveContainer" containerID="49ca1ae00804a17210c8806190844efd0a5944fb5d9b017bb78b26c0089de81b"
Aug 05 17:14:00.148686 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.148667    2561 scope.go:117] "RemoveContainer" containerID="49ca1ae00804a17210c8806190844efd0a5944fb5d9b017bb78b26c0089de81b"
Aug 05 17:14:00.148977 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:14:00.148935    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49ca1ae00804a17210c8806190844efd0a5944fb5d9b017bb78b26c0089de81b\": container with ID starting with 49ca1ae00804a17210c8806190844efd0a5944fb5d9b017bb78b26c0089de81b not found: ID does not exist" containerID="49ca1ae00804a17210c8806190844efd0a5944fb5d9b017bb78b26c0089de81b"
Aug 05 17:14:00.149067 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.148985    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ca1ae00804a17210c8806190844efd0a5944fb5d9b017bb78b26c0089de81b"} err="failed to get container status \"49ca1ae00804a17210c8806190844efd0a5944fb5d9b017bb78b26c0089de81b\": rpc error: code = NotFound desc = could not find container \"49ca1ae00804a17210c8806190844efd0a5944fb5d9b017bb78b26c0089de81b\": container with ID starting with 49ca1ae00804a17210c8806190844efd0a5944fb5d9b017bb78b26c0089de81b not found: ID does not exist"
Aug 05 17:14:00.279636 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.279601    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aba3d093-7afd-4f0e-8407-895e98526798-image-registry-private-configuration\") pod \"aba3d093-7afd-4f0e-8407-895e98526798\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") "
Aug 05 17:14:00.279826 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.279643    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aba3d093-7afd-4f0e-8407-895e98526798-installation-pull-secrets\") pod \"aba3d093-7afd-4f0e-8407-895e98526798\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") "
Aug 05 17:14:00.279826 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.279685    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aba3d093-7afd-4f0e-8407-895e98526798-registry-certificates\") pod \"aba3d093-7afd-4f0e-8407-895e98526798\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") "
Aug 05 17:14:00.279826 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.279740    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-bound-sa-token\") pod \"aba3d093-7afd-4f0e-8407-895e98526798\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") "
Aug 05 17:14:00.280014 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.279939    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7qdd\" (UniqueName: \"kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-kube-api-access-b7qdd\") pod \"aba3d093-7afd-4f0e-8407-895e98526798\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") "
Aug 05 17:14:00.280070 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.280029    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-registry-tls\") pod \"aba3d093-7afd-4f0e-8407-895e98526798\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") "
Aug 05 17:14:00.280070 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.280061    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aba3d093-7afd-4f0e-8407-895e98526798-trusted-ca\") pod \"aba3d093-7afd-4f0e-8407-895e98526798\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") "
Aug 05 17:14:00.280168 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.280085    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aba3d093-7afd-4f0e-8407-895e98526798-ca-trust-extracted\") pod \"aba3d093-7afd-4f0e-8407-895e98526798\" (UID: \"aba3d093-7afd-4f0e-8407-895e98526798\") "
Aug 05 17:14:00.280168 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.280122    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aba3d093-7afd-4f0e-8407-895e98526798-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "aba3d093-7afd-4f0e-8407-895e98526798" (UID: "aba3d093-7afd-4f0e-8407-895e98526798"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:14:00.281022 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.280965    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aba3d093-7afd-4f0e-8407-895e98526798-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "aba3d093-7afd-4f0e-8407-895e98526798" (UID: "aba3d093-7afd-4f0e-8407-895e98526798"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:14:00.283913 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.281586    2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/aba3d093-7afd-4f0e-8407-895e98526798-registry-certificates\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:14:00.283913 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.281614    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aba3d093-7afd-4f0e-8407-895e98526798-trusted-ca\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:14:00.283913 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.283899    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aba3d093-7afd-4f0e-8407-895e98526798-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "aba3d093-7afd-4f0e-8407-895e98526798" (UID: "aba3d093-7afd-4f0e-8407-895e98526798"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:14:00.284222 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.284193    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "aba3d093-7afd-4f0e-8407-895e98526798" (UID: "aba3d093-7afd-4f0e-8407-895e98526798"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:14:00.284313 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.284289    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aba3d093-7afd-4f0e-8407-895e98526798-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "aba3d093-7afd-4f0e-8407-895e98526798" (UID: "aba3d093-7afd-4f0e-8407-895e98526798"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:14:00.287300 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.285237    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "aba3d093-7afd-4f0e-8407-895e98526798" (UID: "aba3d093-7afd-4f0e-8407-895e98526798"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:14:00.287300 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.286053    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-kube-api-access-b7qdd" (OuterVolumeSpecName: "kube-api-access-b7qdd") pod "aba3d093-7afd-4f0e-8407-895e98526798" (UID: "aba3d093-7afd-4f0e-8407-895e98526798"). InnerVolumeSpecName "kube-api-access-b7qdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:14:00.293174 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.293149    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba3d093-7afd-4f0e-8407-895e98526798-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "aba3d093-7afd-4f0e-8407-895e98526798" (UID: "aba3d093-7afd-4f0e-8407-895e98526798"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:14:00.382693 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.382656    2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-bound-sa-token\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:14:00.382693 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.382686    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7qdd\" (UniqueName: \"kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-kube-api-access-b7qdd\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:14:00.382693 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.382699    2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/aba3d093-7afd-4f0e-8407-895e98526798-registry-tls\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:14:00.382693 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.382707    2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/aba3d093-7afd-4f0e-8407-895e98526798-ca-trust-extracted\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:14:00.382983 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.382717    2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/aba3d093-7afd-4f0e-8407-895e98526798-image-registry-private-configuration\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:14:00.382983 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.382727    2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/aba3d093-7afd-4f0e-8407-895e98526798-installation-pull-secrets\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:14:00.463066 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.463036    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76b686c878-267wj"]
Aug 05 17:14:00.469212 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.469187    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-76b686c878-267wj"]
Aug 05 17:14:00.728571 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:00.728498    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aba3d093-7afd-4f0e-8407-895e98526798" path="/var/lib/kubelet/pods/aba3d093-7afd-4f0e-8407-895e98526798/volumes"
Aug 05 17:14:02.783672 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:02.783632    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d667956d9-jhpsr"
Aug 05 17:14:02.784107 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:02.783702    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d667956d9-jhpsr"
Aug 05 17:14:02.788652 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:02.788629    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d667956d9-jhpsr"
Aug 05 17:14:03.154988 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:03.154894    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d667956d9-jhpsr"
Aug 05 17:14:03.207251 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:03.207208    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85f886795d-9sfjh"]
Aug 05 17:14:07.027906 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:07.027876    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bbcvm"
Aug 05 17:14:09.898873 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:09.898839    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-776db5ccf4-n6kq7"
Aug 05 17:14:09.899395 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:09.898912    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-776db5ccf4-n6kq7"
Aug 05 17:14:28.231558 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:28.231515    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-85f886795d-9sfjh" podUID="ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108" containerName="console" containerID="cri-o://64f5f0c16920ecd3501ceb1fa8578016e68d865c7af8286c0438342c9c61914b" gracePeriod=15
Aug 05 17:14:28.466811 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:28.466787    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85f886795d-9sfjh"
Aug 05 17:14:28.493323 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:28.493259    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-console-oauth-config\") pod \"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108\" (UID: \"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108\") "
Aug 05 17:14:28.493323 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:28.493306    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65ff5\" (UniqueName: \"kubernetes.io/projected/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-kube-api-access-65ff5\") pod \"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108\" (UID: \"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108\") "
Aug 05 17:14:28.493509 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:28.493344    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-console-config\") pod \"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108\" (UID: \"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108\") "
Aug 05 17:14:28.493509 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:28.493359    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-oauth-serving-cert\") pod \"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108\" (UID: \"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108\") "
Aug 05 17:14:28.493509 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:28.493381    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-service-ca\") pod \"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108\" (UID: \"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108\") "
Aug 05 17:14:28.493509 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:28.493416    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-console-serving-cert\") pod \"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108\" (UID: \"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108\") "
Aug 05 17:14:28.493778 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:28.493754    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108" (UID: "ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:14:28.493832 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:28.493770    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-console-config" (OuterVolumeSpecName: "console-config") pod "ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108" (UID: "ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:14:28.493892 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:28.493839    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-service-ca" (OuterVolumeSpecName: "service-ca") pod "ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108" (UID: "ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:14:28.495604 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:28.495578    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108" (UID: "ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:14:28.495706 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:28.495595    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-kube-api-access-65ff5" (OuterVolumeSpecName: "kube-api-access-65ff5") pod "ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108" (UID: "ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108"). InnerVolumeSpecName "kube-api-access-65ff5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:14:28.495749 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:28.495734    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108" (UID: "ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:14:28.594918 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:28.594882    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-console-oauth-config\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:14:28.594918 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:28.594911    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65ff5\" (UniqueName: \"kubernetes.io/projected/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-kube-api-access-65ff5\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:14:28.594918 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:28.594922    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-console-config\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:14:28.594918 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:28.594930    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-oauth-serving-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:14:28.595198 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:28.594939    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-service-ca\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:14:28.595198 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:28.594947    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108-console-serving-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:14:29.243168 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:29.243133    2561 generic.go:356] "Generic (PLEG): container finished" podID="ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108" containerID="64f5f0c16920ecd3501ceb1fa8578016e68d865c7af8286c0438342c9c61914b" exitCode=0
Aug 05 17:14:29.243599 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:29.243188    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85f886795d-9sfjh" event={"ID":"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108","Type":"ContainerDied","Data":"64f5f0c16920ecd3501ceb1fa8578016e68d865c7af8286c0438342c9c61914b"}
Aug 05 17:14:29.243599 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:29.243189    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85f886795d-9sfjh"
Aug 05 17:14:29.243599 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:29.243249    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85f886795d-9sfjh" event={"ID":"ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108","Type":"ContainerDied","Data":"e587a4eb422f46b53b5124d34bbff1c2695d48f50362e738954b5153a1da1f68"}
Aug 05 17:14:29.243599 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:29.243275    2561 scope.go:117] "RemoveContainer" containerID="64f5f0c16920ecd3501ceb1fa8578016e68d865c7af8286c0438342c9c61914b"
Aug 05 17:14:29.251211 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:29.251183    2561 scope.go:117] "RemoveContainer" containerID="64f5f0c16920ecd3501ceb1fa8578016e68d865c7af8286c0438342c9c61914b"
Aug 05 17:14:29.251495 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:14:29.251475    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64f5f0c16920ecd3501ceb1fa8578016e68d865c7af8286c0438342c9c61914b\": container with ID starting with 64f5f0c16920ecd3501ceb1fa8578016e68d865c7af8286c0438342c9c61914b not found: ID does not exist" containerID="64f5f0c16920ecd3501ceb1fa8578016e68d865c7af8286c0438342c9c61914b"
Aug 05 17:14:29.251543 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:29.251510    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f5f0c16920ecd3501ceb1fa8578016e68d865c7af8286c0438342c9c61914b"} err="failed to get container status \"64f5f0c16920ecd3501ceb1fa8578016e68d865c7af8286c0438342c9c61914b\": rpc error: code = NotFound desc = could not find container \"64f5f0c16920ecd3501ceb1fa8578016e68d865c7af8286c0438342c9c61914b\": container with ID starting with 64f5f0c16920ecd3501ceb1fa8578016e68d865c7af8286c0438342c9c61914b not found: ID does not exist"
Aug 05 17:14:29.264415 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:29.264391    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85f886795d-9sfjh"]
Aug 05 17:14:29.270919 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:29.270899    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85f886795d-9sfjh"]
Aug 05 17:14:29.904395 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:29.904364    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-776db5ccf4-n6kq7"
Aug 05 17:14:29.908126 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:29.908105    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-776db5ccf4-n6kq7"
Aug 05 17:14:30.728538 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:14:30.728505    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108" path="/var/lib/kubelet/pods/ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108/volumes"
Aug 05 17:15:12.699679 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:12.699649    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-859c7548f6-29p8k"]
Aug 05 17:15:12.700134 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:12.699906    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108" containerName="console"
Aug 05 17:15:12.700134 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:12.699917    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108" containerName="console"
Aug 05 17:15:12.700134 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:12.699930    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aba3d093-7afd-4f0e-8407-895e98526798" containerName="registry"
Aug 05 17:15:12.700134 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:12.699936    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="aba3d093-7afd-4f0e-8407-895e98526798" containerName="registry"
Aug 05 17:15:12.700134 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:12.699986    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebc1f6ca-ba3a-47c7-b6b2-b08dfe51b108" containerName="console"
Aug 05 17:15:12.700134 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:12.699994    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="aba3d093-7afd-4f0e-8407-895e98526798" containerName="registry"
Aug 05 17:15:12.701715 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:12.701698    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-859c7548f6-29p8k"
Aug 05 17:15:12.721558 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:12.721528    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-859c7548f6-29p8k"]
Aug 05 17:15:12.825838 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:12.825802    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-console-serving-cert\") pod \"console-859c7548f6-29p8k\" (UID: \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\") " pod="openshift-console/console-859c7548f6-29p8k"
Aug 05 17:15:12.825838 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:12.825840    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-console-oauth-config\") pod \"console-859c7548f6-29p8k\" (UID: \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\") " pod="openshift-console/console-859c7548f6-29p8k"
Aug 05 17:15:12.826111 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:12.825877    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-service-ca\") pod \"console-859c7548f6-29p8k\" (UID: \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\") " pod="openshift-console/console-859c7548f6-29p8k"
Aug 05 17:15:12.826111 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:12.826001    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-trusted-ca-bundle\") pod \"console-859c7548f6-29p8k\" (UID: \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\") " pod="openshift-console/console-859c7548f6-29p8k"
Aug 05 17:15:12.826111 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:12.826033    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-console-config\") pod \"console-859c7548f6-29p8k\" (UID: \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\") " pod="openshift-console/console-859c7548f6-29p8k"
Aug 05 17:15:12.826111 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:12.826064    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlvhn\" (UniqueName: \"kubernetes.io/projected/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-kube-api-access-vlvhn\") pod \"console-859c7548f6-29p8k\" (UID: \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\") " pod="openshift-console/console-859c7548f6-29p8k"
Aug 05 17:15:12.826111 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:12.826101    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-oauth-serving-cert\") pod \"console-859c7548f6-29p8k\" (UID: \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\") " pod="openshift-console/console-859c7548f6-29p8k"
Aug 05 17:15:12.926511 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:12.926477    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-service-ca\") pod \"console-859c7548f6-29p8k\" (UID: \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\") " pod="openshift-console/console-859c7548f6-29p8k"
Aug 05 17:15:12.926511 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:12.926518    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-trusted-ca-bundle\") pod \"console-859c7548f6-29p8k\" (UID: \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\") " pod="openshift-console/console-859c7548f6-29p8k"
Aug 05 17:15:12.926753 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:12.926534    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-console-config\") pod \"console-859c7548f6-29p8k\" (UID: \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\") " pod="openshift-console/console-859c7548f6-29p8k"
Aug 05 17:15:12.926753 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:12.926648    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlvhn\" (UniqueName: \"kubernetes.io/projected/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-kube-api-access-vlvhn\") pod \"console-859c7548f6-29p8k\" (UID: \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\") " pod="openshift-console/console-859c7548f6-29p8k"
Aug 05 17:15:12.926753 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:12.926689    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-oauth-serving-cert\") pod \"console-859c7548f6-29p8k\" (UID: \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\") " pod="openshift-console/console-859c7548f6-29p8k"
Aug 05 17:15:12.926901 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:12.926792    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-console-serving-cert\") pod \"console-859c7548f6-29p8k\" (UID: \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\") " pod="openshift-console/console-859c7548f6-29p8k"
Aug 05 17:15:12.926901 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:12.926836    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-console-oauth-config\") pod \"console-859c7548f6-29p8k\" (UID: \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\") " pod="openshift-console/console-859c7548f6-29p8k"
Aug 05 17:15:12.927371 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:12.927339    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-console-config\") pod \"console-859c7548f6-29p8k\" (UID: \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\") " pod="openshift-console/console-859c7548f6-29p8k"
Aug 05 17:15:12.927506 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:12.927417    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-service-ca\") pod \"console-859c7548f6-29p8k\" (UID: \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\") " pod="openshift-console/console-859c7548f6-29p8k"
Aug 05 17:15:12.927506 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:12.927458    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-oauth-serving-cert\") pod \"console-859c7548f6-29p8k\" (UID: \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\") " pod="openshift-console/console-859c7548f6-29p8k"
Aug 05 17:15:12.927506 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:12.927487    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-trusted-ca-bundle\") pod \"console-859c7548f6-29p8k\" (UID: \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\") " pod="openshift-console/console-859c7548f6-29p8k"
Aug 05 17:15:12.929223 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:12.929197    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-console-serving-cert\") pod \"console-859c7548f6-29p8k\" (UID: \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\") " pod="openshift-console/console-859c7548f6-29p8k"
Aug 05 17:15:12.929337 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:12.929317    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-console-oauth-config\") pod \"console-859c7548f6-29p8k\" (UID: \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\") " pod="openshift-console/console-859c7548f6-29p8k"
Aug 05 17:15:12.939328 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:12.939311    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlvhn\" (UniqueName: \"kubernetes.io/projected/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-kube-api-access-vlvhn\") pod \"console-859c7548f6-29p8k\" (UID: \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\") " pod="openshift-console/console-859c7548f6-29p8k"
Aug 05 17:15:13.011019 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:13.010900    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-859c7548f6-29p8k"
Aug 05 17:15:13.173771 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:13.173746    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-859c7548f6-29p8k"]
Aug 05 17:15:13.176158 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:15:13.176135    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c2ad75d_9f5b_49a6_aeb2_5ea020543960.slice/crio-774e2c6642b13c3c4d56e18986da8178284cef5306a3d402267133e32d41f94c WatchSource:0}: Error finding container 774e2c6642b13c3c4d56e18986da8178284cef5306a3d402267133e32d41f94c: Status 404 returned error can't find the container with id 774e2c6642b13c3c4d56e18986da8178284cef5306a3d402267133e32d41f94c
Aug 05 17:15:13.396263 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:13.396226    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-859c7548f6-29p8k" event={"ID":"7c2ad75d-9f5b-49a6-aeb2-5ea020543960","Type":"ContainerStarted","Data":"5182943b2594facbe14fa80d6876e867483f24543a2d8a4a0d3e24e1a5e11b68"}
Aug 05 17:15:13.396433 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:13.396283    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-859c7548f6-29p8k" event={"ID":"7c2ad75d-9f5b-49a6-aeb2-5ea020543960","Type":"ContainerStarted","Data":"774e2c6642b13c3c4d56e18986da8178284cef5306a3d402267133e32d41f94c"}
Aug 05 17:15:13.416041 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:13.415982    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-859c7548f6-29p8k" podStartSLOduration=1.415943123 podStartE2EDuration="1.415943123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:15:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:15:13.415586529 +0000 UTC m=+169.160220895" watchObservedRunningTime="2026-08-05 17:15:13.415943123 +0000 UTC m=+169.160577450"
Aug 05 17:15:23.012055 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:23.011894    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-859c7548f6-29p8k"
Aug 05 17:15:23.012520 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:23.012072    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-859c7548f6-29p8k"
Aug 05 17:15:23.016603 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:23.016579    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-859c7548f6-29p8k"
Aug 05 17:15:23.441539 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:23.441513    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-859c7548f6-29p8k"
Aug 05 17:15:23.500266 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:23.500230    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d667956d9-jhpsr"]
Aug 05 17:15:48.522403 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:48.522339    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d667956d9-jhpsr" podUID="96a14595-bc1e-4306-8d81-b2fba8b9cb9e" containerName="console" containerID="cri-o://bc93a4beb8104819e291323e906e651db186b984812e34bfd97135f5b7691f5f" gracePeriod=15
Aug 05 17:15:48.758301 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:48.758277    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d667956d9-jhpsr"
Aug 05 17:15:48.785692 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:48.785668    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-trusted-ca-bundle\") pod \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\" (UID: \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\") "
Aug 05 17:15:48.785879 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:48.785701    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nj8lp\" (UniqueName: \"kubernetes.io/projected/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-kube-api-access-nj8lp\") pod \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\" (UID: \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\") "
Aug 05 17:15:48.785879 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:48.785735    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-oauth-serving-cert\") pod \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\" (UID: \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\") "
Aug 05 17:15:48.785879 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:48.785790    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-service-ca\") pod \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\" (UID: \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\") "
Aug 05 17:15:48.785879 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:48.785821    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-console-config\") pod \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\" (UID: \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\") "
Aug 05 17:15:48.785879 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:48.785861    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-console-oauth-config\") pod \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\" (UID: \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\") "
Aug 05 17:15:48.786876 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:48.785942    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-console-serving-cert\") pod \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\" (UID: \"96a14595-bc1e-4306-8d81-b2fba8b9cb9e\") "
Aug 05 17:15:48.786876 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:48.786127    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "96a14595-bc1e-4306-8d81-b2fba8b9cb9e" (UID: "96a14595-bc1e-4306-8d81-b2fba8b9cb9e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:15:48.786876 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:48.786357    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-service-ca" (OuterVolumeSpecName: "service-ca") pod "96a14595-bc1e-4306-8d81-b2fba8b9cb9e" (UID: "96a14595-bc1e-4306-8d81-b2fba8b9cb9e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:15:48.786876 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:48.786397    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-console-config" (OuterVolumeSpecName: "console-config") pod "96a14595-bc1e-4306-8d81-b2fba8b9cb9e" (UID: "96a14595-bc1e-4306-8d81-b2fba8b9cb9e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:15:48.786876 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:48.786436    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "96a14595-bc1e-4306-8d81-b2fba8b9cb9e" (UID: "96a14595-bc1e-4306-8d81-b2fba8b9cb9e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:15:48.788256 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:48.788226    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "96a14595-bc1e-4306-8d81-b2fba8b9cb9e" (UID: "96a14595-bc1e-4306-8d81-b2fba8b9cb9e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:15:48.788354 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:48.788279    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "96a14595-bc1e-4306-8d81-b2fba8b9cb9e" (UID: "96a14595-bc1e-4306-8d81-b2fba8b9cb9e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:15:48.788469 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:48.788452    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-kube-api-access-nj8lp" (OuterVolumeSpecName: "kube-api-access-nj8lp") pod "96a14595-bc1e-4306-8d81-b2fba8b9cb9e" (UID: "96a14595-bc1e-4306-8d81-b2fba8b9cb9e"). InnerVolumeSpecName "kube-api-access-nj8lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:15:48.886868 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:48.886831    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-console-serving-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:15:48.886868 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:48.886859    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-trusted-ca-bundle\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:15:48.886868 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:48.886869    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj8lp\" (UniqueName: \"kubernetes.io/projected/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-kube-api-access-nj8lp\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:15:48.886868 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:48.886878    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-oauth-serving-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:15:48.887160 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:48.886891    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-service-ca\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:15:48.887160 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:48.886899    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-console-config\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:15:48.887160 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:48.886907    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/96a14595-bc1e-4306-8d81-b2fba8b9cb9e-console-oauth-config\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:15:49.528346 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:49.528314    2561 generic.go:356] "Generic (PLEG): container finished" podID="96a14595-bc1e-4306-8d81-b2fba8b9cb9e" containerID="bc93a4beb8104819e291323e906e651db186b984812e34bfd97135f5b7691f5f" exitCode=0
Aug 05 17:15:49.528798 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:49.528366    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d667956d9-jhpsr"
Aug 05 17:15:49.528798 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:49.528398    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d667956d9-jhpsr" event={"ID":"96a14595-bc1e-4306-8d81-b2fba8b9cb9e","Type":"ContainerDied","Data":"bc93a4beb8104819e291323e906e651db186b984812e34bfd97135f5b7691f5f"}
Aug 05 17:15:49.528798 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:49.528458    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d667956d9-jhpsr" event={"ID":"96a14595-bc1e-4306-8d81-b2fba8b9cb9e","Type":"ContainerDied","Data":"73c3c098419540daddfeda00e904c2c1f565abe6af02f870f80628eb720f7aa9"}
Aug 05 17:15:49.528798 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:49.528489    2561 scope.go:117] "RemoveContainer" containerID="bc93a4beb8104819e291323e906e651db186b984812e34bfd97135f5b7691f5f"
Aug 05 17:15:49.536710 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:49.536693    2561 scope.go:117] "RemoveContainer" containerID="bc93a4beb8104819e291323e906e651db186b984812e34bfd97135f5b7691f5f"
Aug 05 17:15:49.536988 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:15:49.536943    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc93a4beb8104819e291323e906e651db186b984812e34bfd97135f5b7691f5f\": container with ID starting with bc93a4beb8104819e291323e906e651db186b984812e34bfd97135f5b7691f5f not found: ID does not exist" containerID="bc93a4beb8104819e291323e906e651db186b984812e34bfd97135f5b7691f5f"
Aug 05 17:15:49.537089 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:49.536994    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc93a4beb8104819e291323e906e651db186b984812e34bfd97135f5b7691f5f"} err="failed to get container status \"bc93a4beb8104819e291323e906e651db186b984812e34bfd97135f5b7691f5f\": rpc error: code = NotFound desc = could not find container \"bc93a4beb8104819e291323e906e651db186b984812e34bfd97135f5b7691f5f\": container with ID starting with bc93a4beb8104819e291323e906e651db186b984812e34bfd97135f5b7691f5f not found: ID does not exist"
Aug 05 17:15:49.549879 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:49.549842    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d667956d9-jhpsr"]
Aug 05 17:15:49.558232 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:49.558211    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d667956d9-jhpsr"]
Aug 05 17:15:50.728249 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:15:50.728209    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96a14595-bc1e-4306-8d81-b2fba8b9cb9e" path="/var/lib/kubelet/pods/96a14595-bc1e-4306-8d81-b2fba8b9cb9e/volumes"
Aug 05 17:16:29.479811 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:29.479781    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-749868cf98-wszss"]
Aug 05 17:16:29.480212 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:29.480053    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96a14595-bc1e-4306-8d81-b2fba8b9cb9e" containerName="console"
Aug 05 17:16:29.480212 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:29.480065    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="96a14595-bc1e-4306-8d81-b2fba8b9cb9e" containerName="console"
Aug 05 17:16:29.480212 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:29.480122    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="96a14595-bc1e-4306-8d81-b2fba8b9cb9e" containerName="console"
Aug 05 17:16:29.482717 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:29.482700    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-749868cf98-wszss"
Aug 05 17:16:29.502916 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:29.502883    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-749868cf98-wszss"]
Aug 05 17:16:29.585576 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:29.585541    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzwll\" (UniqueName: \"kubernetes.io/projected/2d7a2358-52a8-4e99-a7cd-d28325db3299-kube-api-access-lzwll\") pod \"console-749868cf98-wszss\" (UID: \"2d7a2358-52a8-4e99-a7cd-d28325db3299\") " pod="openshift-console/console-749868cf98-wszss"
Aug 05 17:16:29.585770 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:29.585591    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d7a2358-52a8-4e99-a7cd-d28325db3299-console-config\") pod \"console-749868cf98-wszss\" (UID: \"2d7a2358-52a8-4e99-a7cd-d28325db3299\") " pod="openshift-console/console-749868cf98-wszss"
Aug 05 17:16:29.585770 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:29.585671    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d7a2358-52a8-4e99-a7cd-d28325db3299-console-serving-cert\") pod \"console-749868cf98-wszss\" (UID: \"2d7a2358-52a8-4e99-a7cd-d28325db3299\") " pod="openshift-console/console-749868cf98-wszss"
Aug 05 17:16:29.585770 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:29.585696    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d7a2358-52a8-4e99-a7cd-d28325db3299-oauth-serving-cert\") pod \"console-749868cf98-wszss\" (UID: \"2d7a2358-52a8-4e99-a7cd-d28325db3299\") " pod="openshift-console/console-749868cf98-wszss"
Aug 05 17:16:29.585770 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:29.585726    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d7a2358-52a8-4e99-a7cd-d28325db3299-console-oauth-config\") pod \"console-749868cf98-wszss\" (UID: \"2d7a2358-52a8-4e99-a7cd-d28325db3299\") " pod="openshift-console/console-749868cf98-wszss"
Aug 05 17:16:29.585770 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:29.585748    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d7a2358-52a8-4e99-a7cd-d28325db3299-trusted-ca-bundle\") pod \"console-749868cf98-wszss\" (UID: \"2d7a2358-52a8-4e99-a7cd-d28325db3299\") " pod="openshift-console/console-749868cf98-wszss"
Aug 05 17:16:29.585770 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:29.585764    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7a2358-52a8-4e99-a7cd-d28325db3299-service-ca\") pod \"console-749868cf98-wszss\" (UID: \"2d7a2358-52a8-4e99-a7cd-d28325db3299\") " pod="openshift-console/console-749868cf98-wszss"
Aug 05 17:16:29.686899 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:29.686861    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d7a2358-52a8-4e99-a7cd-d28325db3299-trusted-ca-bundle\") pod \"console-749868cf98-wszss\" (UID: \"2d7a2358-52a8-4e99-a7cd-d28325db3299\") " pod="openshift-console/console-749868cf98-wszss"
Aug 05 17:16:29.686899 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:29.686903    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7a2358-52a8-4e99-a7cd-d28325db3299-service-ca\") pod \"console-749868cf98-wszss\" (UID: \"2d7a2358-52a8-4e99-a7cd-d28325db3299\") " pod="openshift-console/console-749868cf98-wszss"
Aug 05 17:16:29.687159 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:29.686939    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzwll\" (UniqueName: \"kubernetes.io/projected/2d7a2358-52a8-4e99-a7cd-d28325db3299-kube-api-access-lzwll\") pod \"console-749868cf98-wszss\" (UID: \"2d7a2358-52a8-4e99-a7cd-d28325db3299\") " pod="openshift-console/console-749868cf98-wszss"
Aug 05 17:16:29.687159 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:29.687007    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d7a2358-52a8-4e99-a7cd-d28325db3299-console-config\") pod \"console-749868cf98-wszss\" (UID: \"2d7a2358-52a8-4e99-a7cd-d28325db3299\") " pod="openshift-console/console-749868cf98-wszss"
Aug 05 17:16:29.687159 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:29.687140    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d7a2358-52a8-4e99-a7cd-d28325db3299-console-serving-cert\") pod \"console-749868cf98-wszss\" (UID: \"2d7a2358-52a8-4e99-a7cd-d28325db3299\") " pod="openshift-console/console-749868cf98-wszss"
Aug 05 17:16:29.687312 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:29.687173    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d7a2358-52a8-4e99-a7cd-d28325db3299-oauth-serving-cert\") pod \"console-749868cf98-wszss\" (UID: \"2d7a2358-52a8-4e99-a7cd-d28325db3299\") " pod="openshift-console/console-749868cf98-wszss"
Aug 05 17:16:29.687312 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:29.687216    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d7a2358-52a8-4e99-a7cd-d28325db3299-console-oauth-config\") pod \"console-749868cf98-wszss\" (UID: \"2d7a2358-52a8-4e99-a7cd-d28325db3299\") " pod="openshift-console/console-749868cf98-wszss"
Aug 05 17:16:29.687750 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:29.687727    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7a2358-52a8-4e99-a7cd-d28325db3299-service-ca\") pod \"console-749868cf98-wszss\" (UID: \"2d7a2358-52a8-4e99-a7cd-d28325db3299\") " pod="openshift-console/console-749868cf98-wszss"
Aug 05 17:16:29.687851 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:29.687796    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d7a2358-52a8-4e99-a7cd-d28325db3299-console-config\") pod \"console-749868cf98-wszss\" (UID: \"2d7a2358-52a8-4e99-a7cd-d28325db3299\") " pod="openshift-console/console-749868cf98-wszss"
Aug 05 17:16:29.687851 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:29.687817    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d7a2358-52a8-4e99-a7cd-d28325db3299-oauth-serving-cert\") pod \"console-749868cf98-wszss\" (UID: \"2d7a2358-52a8-4e99-a7cd-d28325db3299\") " pod="openshift-console/console-749868cf98-wszss"
Aug 05 17:16:29.687966 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:29.687868    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d7a2358-52a8-4e99-a7cd-d28325db3299-trusted-ca-bundle\") pod \"console-749868cf98-wszss\" (UID: \"2d7a2358-52a8-4e99-a7cd-d28325db3299\") " pod="openshift-console/console-749868cf98-wszss"
Aug 05 17:16:29.689571 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:29.689547    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d7a2358-52a8-4e99-a7cd-d28325db3299-console-serving-cert\") pod \"console-749868cf98-wszss\" (UID: \"2d7a2358-52a8-4e99-a7cd-d28325db3299\") " pod="openshift-console/console-749868cf98-wszss"
Aug 05 17:16:29.689680 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:29.689662    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d7a2358-52a8-4e99-a7cd-d28325db3299-console-oauth-config\") pod \"console-749868cf98-wszss\" (UID: \"2d7a2358-52a8-4e99-a7cd-d28325db3299\") " pod="openshift-console/console-749868cf98-wszss"
Aug 05 17:16:29.695287 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:29.695263    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzwll\" (UniqueName: \"kubernetes.io/projected/2d7a2358-52a8-4e99-a7cd-d28325db3299-kube-api-access-lzwll\") pod \"console-749868cf98-wszss\" (UID: \"2d7a2358-52a8-4e99-a7cd-d28325db3299\") " pod="openshift-console/console-749868cf98-wszss"
Aug 05 17:16:29.791743 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:29.791714    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-749868cf98-wszss"
Aug 05 17:16:29.925212 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:29.925172    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-749868cf98-wszss"]
Aug 05 17:16:29.928164 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:16:29.928137    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7a2358_52a8_4e99_a7cd_d28325db3299.slice/crio-1c4fa94c39f28cc4e59ca0e81283efe948c0f43f7744e01fcafb0d9d26af8aa3 WatchSource:0}: Error finding container 1c4fa94c39f28cc4e59ca0e81283efe948c0f43f7744e01fcafb0d9d26af8aa3: Status 404 returned error can't find the container with id 1c4fa94c39f28cc4e59ca0e81283efe948c0f43f7744e01fcafb0d9d26af8aa3
Aug 05 17:16:30.675779 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:30.675744    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-749868cf98-wszss" event={"ID":"2d7a2358-52a8-4e99-a7cd-d28325db3299","Type":"ContainerStarted","Data":"fb4641fdfa88cb1060777c4a3e249fe239b4d504afd78de4ddf4aad3eb60cf43"}
Aug 05 17:16:30.676253 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:30.675819    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-749868cf98-wszss" event={"ID":"2d7a2358-52a8-4e99-a7cd-d28325db3299","Type":"ContainerStarted","Data":"1c4fa94c39f28cc4e59ca0e81283efe948c0f43f7744e01fcafb0d9d26af8aa3"}
Aug 05 17:16:30.701517 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:30.701459    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-749868cf98-wszss" podStartSLOduration=1.701444542 podStartE2EDuration="1.701444542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:16:30.700870783 +0000 UTC m=+246.445505109" watchObservedRunningTime="2026-08-05 17:16:30.701444542 +0000 UTC m=+246.446078868"
Aug 05 17:16:39.792645 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:39.792586    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-749868cf98-wszss"
Aug 05 17:16:39.793160 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:39.792703    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-749868cf98-wszss"
Aug 05 17:16:39.798904 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:39.798878    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-749868cf98-wszss"
Aug 05 17:16:40.713512 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:40.713484    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-749868cf98-wszss"
Aug 05 17:16:40.775346 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:16:40.775310    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-859c7548f6-29p8k"]
Aug 05 17:17:05.794438 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:05.794399    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-859c7548f6-29p8k" podUID="7c2ad75d-9f5b-49a6-aeb2-5ea020543960" containerName="console" containerID="cri-o://5182943b2594facbe14fa80d6876e867483f24543a2d8a4a0d3e24e1a5e11b68" gracePeriod=15
Aug 05 17:17:06.039775 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.039745    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-859c7548f6-29p8k"
Aug 05 17:17:06.068568 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.068475    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-console-oauth-config\") pod \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\" (UID: \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\") "
Aug 05 17:17:06.068568 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.068561    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-trusted-ca-bundle\") pod \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\" (UID: \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\") "
Aug 05 17:17:06.068727 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.068587    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-console-config\") pod \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\" (UID: \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\") "
Aug 05 17:17:06.068727 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.068612    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-console-serving-cert\") pod \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\" (UID: \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\") "
Aug 05 17:17:06.068727 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.068653    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-service-ca\") pod \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\" (UID: \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\") "
Aug 05 17:17:06.068727 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.068684    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlvhn\" (UniqueName: \"kubernetes.io/projected/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-kube-api-access-vlvhn\") pod \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\" (UID: \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\") "
Aug 05 17:17:06.068887 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.068737    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-oauth-serving-cert\") pod \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\" (UID: \"7c2ad75d-9f5b-49a6-aeb2-5ea020543960\") "
Aug 05 17:17:06.070229 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.069255    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-service-ca" (OuterVolumeSpecName: "service-ca") pod "7c2ad75d-9f5b-49a6-aeb2-5ea020543960" (UID: "7c2ad75d-9f5b-49a6-aeb2-5ea020543960"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:17:06.070229 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.069720    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-console-config" (OuterVolumeSpecName: "console-config") pod "7c2ad75d-9f5b-49a6-aeb2-5ea020543960" (UID: "7c2ad75d-9f5b-49a6-aeb2-5ea020543960"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:17:06.070229 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.069813    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7c2ad75d-9f5b-49a6-aeb2-5ea020543960" (UID: "7c2ad75d-9f5b-49a6-aeb2-5ea020543960"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:17:06.070485 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.070277    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7c2ad75d-9f5b-49a6-aeb2-5ea020543960" (UID: "7c2ad75d-9f5b-49a6-aeb2-5ea020543960"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:17:06.071288 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.071254    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7c2ad75d-9f5b-49a6-aeb2-5ea020543960" (UID: "7c2ad75d-9f5b-49a6-aeb2-5ea020543960"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:17:06.071400 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.071352    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7c2ad75d-9f5b-49a6-aeb2-5ea020543960" (UID: "7c2ad75d-9f5b-49a6-aeb2-5ea020543960"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:17:06.071715 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.071696    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-kube-api-access-vlvhn" (OuterVolumeSpecName: "kube-api-access-vlvhn") pod "7c2ad75d-9f5b-49a6-aeb2-5ea020543960" (UID: "7c2ad75d-9f5b-49a6-aeb2-5ea020543960"). InnerVolumeSpecName "kube-api-access-vlvhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:17:06.169424 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.169386    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-trusted-ca-bundle\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:17:06.169424 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.169420    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-console-config\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:17:06.169424 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.169432    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-console-serving-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:17:06.169698 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.169445    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-service-ca\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:17:06.169698 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.169457    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlvhn\" (UniqueName: \"kubernetes.io/projected/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-kube-api-access-vlvhn\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:17:06.169698 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.169471    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-oauth-serving-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:17:06.169698 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.169481    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c2ad75d-9f5b-49a6-aeb2-5ea020543960-console-oauth-config\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:17:06.799134 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.799099    2561 generic.go:356] "Generic (PLEG): container finished" podID="7c2ad75d-9f5b-49a6-aeb2-5ea020543960" containerID="5182943b2594facbe14fa80d6876e867483f24543a2d8a4a0d3e24e1a5e11b68" exitCode=0
Aug 05 17:17:06.799588 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.799153    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-859c7548f6-29p8k"
Aug 05 17:17:06.799588 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.799175    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-859c7548f6-29p8k" event={"ID":"7c2ad75d-9f5b-49a6-aeb2-5ea020543960","Type":"ContainerDied","Data":"5182943b2594facbe14fa80d6876e867483f24543a2d8a4a0d3e24e1a5e11b68"}
Aug 05 17:17:06.799588 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.799230    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-859c7548f6-29p8k" event={"ID":"7c2ad75d-9f5b-49a6-aeb2-5ea020543960","Type":"ContainerDied","Data":"774e2c6642b13c3c4d56e18986da8178284cef5306a3d402267133e32d41f94c"}
Aug 05 17:17:06.799588 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.799259    2561 scope.go:117] "RemoveContainer" containerID="5182943b2594facbe14fa80d6876e867483f24543a2d8a4a0d3e24e1a5e11b68"
Aug 05 17:17:06.807397 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.807376    2561 scope.go:117] "RemoveContainer" containerID="5182943b2594facbe14fa80d6876e867483f24543a2d8a4a0d3e24e1a5e11b68"
Aug 05 17:17:06.807677 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:17:06.807658    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5182943b2594facbe14fa80d6876e867483f24543a2d8a4a0d3e24e1a5e11b68\": container with ID starting with 5182943b2594facbe14fa80d6876e867483f24543a2d8a4a0d3e24e1a5e11b68 not found: ID does not exist" containerID="5182943b2594facbe14fa80d6876e867483f24543a2d8a4a0d3e24e1a5e11b68"
Aug 05 17:17:06.807734 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.807688    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5182943b2594facbe14fa80d6876e867483f24543a2d8a4a0d3e24e1a5e11b68"} err="failed to get container status \"5182943b2594facbe14fa80d6876e867483f24543a2d8a4a0d3e24e1a5e11b68\": rpc error: code = NotFound desc = could not find container \"5182943b2594facbe14fa80d6876e867483f24543a2d8a4a0d3e24e1a5e11b68\": container with ID starting with 5182943b2594facbe14fa80d6876e867483f24543a2d8a4a0d3e24e1a5e11b68 not found: ID does not exist"
Aug 05 17:17:06.816828 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.816802    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-859c7548f6-29p8k"]
Aug 05 17:17:06.821213 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:06.821185    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-859c7548f6-29p8k"]
Aug 05 17:17:08.729085 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:08.729049    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c2ad75d-9f5b-49a6-aeb2-5ea020543960" path="/var/lib/kubelet/pods/7c2ad75d-9f5b-49a6-aeb2-5ea020543960/volumes"
Aug 05 17:17:24.614009 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:17:24.613974    2561 kubelet.go:1648] "Image garbage collection succeeded"
Aug 05 17:18:51.075933 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:18:51.075898    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5d2rtq"]
Aug 05 17:18:51.076432 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:18:51.076161    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c2ad75d-9f5b-49a6-aeb2-5ea020543960" containerName="console"
Aug 05 17:18:51.076432 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:18:51.076172    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c2ad75d-9f5b-49a6-aeb2-5ea020543960" containerName="console"
Aug 05 17:18:51.076432 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:18:51.076221    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c2ad75d-9f5b-49a6-aeb2-5ea020543960" containerName="console"
Aug 05 17:18:51.081720 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:18:51.081687    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5d2rtq"
Aug 05 17:18:51.084753 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:18:51.084732    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Aug 05 17:18:51.085721 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:18:51.085697    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Aug 05 17:18:51.085846 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:18:51.085730    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-npp6p\""
Aug 05 17:18:51.095722 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:18:51.095701    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5d2rtq"]
Aug 05 17:18:51.131078 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:18:51.131036    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3acd82fb-cbc5-414e-8d56-72f1ec2c22e4-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5d2rtq\" (UID: \"3acd82fb-cbc5-414e-8d56-72f1ec2c22e4\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5d2rtq"
Aug 05 17:18:51.131078 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:18:51.131081    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvtph\" (UniqueName: \"kubernetes.io/projected/3acd82fb-cbc5-414e-8d56-72f1ec2c22e4-kube-api-access-pvtph\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5d2rtq\" (UID: \"3acd82fb-cbc5-414e-8d56-72f1ec2c22e4\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5d2rtq"
Aug 05 17:18:51.131288 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:18:51.131157    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3acd82fb-cbc5-414e-8d56-72f1ec2c22e4-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5d2rtq\" (UID: \"3acd82fb-cbc5-414e-8d56-72f1ec2c22e4\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5d2rtq"
Aug 05 17:18:51.232292 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:18:51.232261    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3acd82fb-cbc5-414e-8d56-72f1ec2c22e4-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5d2rtq\" (UID: \"3acd82fb-cbc5-414e-8d56-72f1ec2c22e4\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5d2rtq"
Aug 05 17:18:51.232489 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:18:51.232298    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvtph\" (UniqueName: \"kubernetes.io/projected/3acd82fb-cbc5-414e-8d56-72f1ec2c22e4-kube-api-access-pvtph\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5d2rtq\" (UID: \"3acd82fb-cbc5-414e-8d56-72f1ec2c22e4\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5d2rtq"
Aug 05 17:18:51.232489 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:18:51.232343    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3acd82fb-cbc5-414e-8d56-72f1ec2c22e4-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5d2rtq\" (UID: \"3acd82fb-cbc5-414e-8d56-72f1ec2c22e4\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5d2rtq"
Aug 05 17:18:51.232712 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:18:51.232690    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3acd82fb-cbc5-414e-8d56-72f1ec2c22e4-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5d2rtq\" (UID: \"3acd82fb-cbc5-414e-8d56-72f1ec2c22e4\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5d2rtq"
Aug 05 17:18:51.232785 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:18:51.232702    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3acd82fb-cbc5-414e-8d56-72f1ec2c22e4-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5d2rtq\" (UID: \"3acd82fb-cbc5-414e-8d56-72f1ec2c22e4\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5d2rtq"
Aug 05 17:18:51.242071 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:18:51.242045    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvtph\" (UniqueName: \"kubernetes.io/projected/3acd82fb-cbc5-414e-8d56-72f1ec2c22e4-kube-api-access-pvtph\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5d2rtq\" (UID: \"3acd82fb-cbc5-414e-8d56-72f1ec2c22e4\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5d2rtq"
Aug 05 17:18:51.390823 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:18:51.390733    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5d2rtq"
Aug 05 17:18:51.521407 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:18:51.521222    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5d2rtq"]
Aug 05 17:18:51.524198 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:18:51.524169    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3acd82fb_cbc5_414e_8d56_72f1ec2c22e4.slice/crio-f4bbedd231abadd1a7e34adacd2b49c38cb812980bba85e355ff5f79bf37944d WatchSource:0}: Error finding container f4bbedd231abadd1a7e34adacd2b49c38cb812980bba85e355ff5f79bf37944d: Status 404 returned error can't find the container with id f4bbedd231abadd1a7e34adacd2b49c38cb812980bba85e355ff5f79bf37944d
Aug 05 17:18:51.526330 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:18:51.526313    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 05 17:18:52.174451 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:18:52.174411    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5d2rtq" event={"ID":"3acd82fb-cbc5-414e-8d56-72f1ec2c22e4","Type":"ContainerStarted","Data":"f4bbedd231abadd1a7e34adacd2b49c38cb812980bba85e355ff5f79bf37944d"}
Aug 05 17:18:58.199943 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:18:58.199904    2561 generic.go:356] "Generic (PLEG): container finished" podID="3acd82fb-cbc5-414e-8d56-72f1ec2c22e4" containerID="1138fd710104e90bda6a0f9618dcddcf0f51099c905b38816af8de230591872f" exitCode=0
Aug 05 17:18:58.200444 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:18:58.199996    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5d2rtq" event={"ID":"3acd82fb-cbc5-414e-8d56-72f1ec2c22e4","Type":"ContainerDied","Data":"1138fd710104e90bda6a0f9618dcddcf0f51099c905b38816af8de230591872f"}
Aug 05 17:19:01.214858 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:01.214823    2561 generic.go:356] "Generic (PLEG): container finished" podID="3acd82fb-cbc5-414e-8d56-72f1ec2c22e4" containerID="9db01a42c41c0ea7e1ce0ddcd2c709a7c4e9600b1b2e1a11d558de0c91aeb3c0" exitCode=0
Aug 05 17:19:01.215255 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:01.214904    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5d2rtq" event={"ID":"3acd82fb-cbc5-414e-8d56-72f1ec2c22e4","Type":"ContainerDied","Data":"9db01a42c41c0ea7e1ce0ddcd2c709a7c4e9600b1b2e1a11d558de0c91aeb3c0"}
Aug 05 17:19:08.243776 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:08.243683    2561 generic.go:356] "Generic (PLEG): container finished" podID="3acd82fb-cbc5-414e-8d56-72f1ec2c22e4" containerID="73af1607af99c850781e49fba13f068473a396f02a5c1b805acfd721d5fd20a9" exitCode=0
Aug 05 17:19:08.243776 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:08.243759    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5d2rtq" event={"ID":"3acd82fb-cbc5-414e-8d56-72f1ec2c22e4","Type":"ContainerDied","Data":"73af1607af99c850781e49fba13f068473a396f02a5c1b805acfd721d5fd20a9"}
Aug 05 17:19:09.369004 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:09.368980    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5d2rtq"
Aug 05 17:19:09.481828 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:09.481789    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvtph\" (UniqueName: \"kubernetes.io/projected/3acd82fb-cbc5-414e-8d56-72f1ec2c22e4-kube-api-access-pvtph\") pod \"3acd82fb-cbc5-414e-8d56-72f1ec2c22e4\" (UID: \"3acd82fb-cbc5-414e-8d56-72f1ec2c22e4\") "
Aug 05 17:19:09.482058 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:09.481854    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3acd82fb-cbc5-414e-8d56-72f1ec2c22e4-util\") pod \"3acd82fb-cbc5-414e-8d56-72f1ec2c22e4\" (UID: \"3acd82fb-cbc5-414e-8d56-72f1ec2c22e4\") "
Aug 05 17:19:09.482058 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:09.481892    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3acd82fb-cbc5-414e-8d56-72f1ec2c22e4-bundle\") pod \"3acd82fb-cbc5-414e-8d56-72f1ec2c22e4\" (UID: \"3acd82fb-cbc5-414e-8d56-72f1ec2c22e4\") "
Aug 05 17:19:09.482544 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:09.482522    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3acd82fb-cbc5-414e-8d56-72f1ec2c22e4-bundle" (OuterVolumeSpecName: "bundle") pod "3acd82fb-cbc5-414e-8d56-72f1ec2c22e4" (UID: "3acd82fb-cbc5-414e-8d56-72f1ec2c22e4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:19:09.484051 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:09.484026    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3acd82fb-cbc5-414e-8d56-72f1ec2c22e4-kube-api-access-pvtph" (OuterVolumeSpecName: "kube-api-access-pvtph") pod "3acd82fb-cbc5-414e-8d56-72f1ec2c22e4" (UID: "3acd82fb-cbc5-414e-8d56-72f1ec2c22e4"). InnerVolumeSpecName "kube-api-access-pvtph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:19:09.486286 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:09.486261    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3acd82fb-cbc5-414e-8d56-72f1ec2c22e4-util" (OuterVolumeSpecName: "util") pod "3acd82fb-cbc5-414e-8d56-72f1ec2c22e4" (UID: "3acd82fb-cbc5-414e-8d56-72f1ec2c22e4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:19:09.583079 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:09.583036    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3acd82fb-cbc5-414e-8d56-72f1ec2c22e4-util\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:19:09.583079 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:09.583076    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3acd82fb-cbc5-414e-8d56-72f1ec2c22e4-bundle\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:19:09.583079 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:09.583086    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvtph\" (UniqueName: \"kubernetes.io/projected/3acd82fb-cbc5-414e-8d56-72f1ec2c22e4-kube-api-access-pvtph\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:19:10.252279 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:10.252249    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5d2rtq"
Aug 05 17:19:10.252279 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:10.252248    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5d2rtq" event={"ID":"3acd82fb-cbc5-414e-8d56-72f1ec2c22e4","Type":"ContainerDied","Data":"f4bbedd231abadd1a7e34adacd2b49c38cb812980bba85e355ff5f79bf37944d"}
Aug 05 17:19:10.252466 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:10.252308    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4bbedd231abadd1a7e34adacd2b49c38cb812980bba85e355ff5f79bf37944d"
Aug 05 17:19:14.053617 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:14.053583    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-psgw5"]
Aug 05 17:19:14.054039 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:14.053837    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3acd82fb-cbc5-414e-8d56-72f1ec2c22e4" containerName="extract"
Aug 05 17:19:14.054039 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:14.053847    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3acd82fb-cbc5-414e-8d56-72f1ec2c22e4" containerName="extract"
Aug 05 17:19:14.054039 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:14.053857    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3acd82fb-cbc5-414e-8d56-72f1ec2c22e4" containerName="util"
Aug 05 17:19:14.054039 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:14.053862    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3acd82fb-cbc5-414e-8d56-72f1ec2c22e4" containerName="util"
Aug 05 17:19:14.054039 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:14.053868    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3acd82fb-cbc5-414e-8d56-72f1ec2c22e4" containerName="pull"
Aug 05 17:19:14.054039 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:14.053874    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3acd82fb-cbc5-414e-8d56-72f1ec2c22e4" containerName="pull"
Aug 05 17:19:14.054039 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:14.053928    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3acd82fb-cbc5-414e-8d56-72f1ec2c22e4" containerName="extract"
Aug 05 17:19:14.096362 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:14.096326    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-psgw5"]
Aug 05 17:19:14.096532 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:14.096444    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-psgw5"
Aug 05 17:19:14.100000 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:14.099949    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-km5pm\""
Aug 05 17:19:14.100000 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:14.099989    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Aug 05 17:19:14.100000 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:14.100001    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Aug 05 17:19:14.100288 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:14.100276    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Aug 05 17:19:14.221829 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:14.221789    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/04d28069-668d-4473-b6d2-c2611586988e-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-psgw5\" (UID: \"04d28069-668d-4473-b6d2-c2611586988e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-psgw5"
Aug 05 17:19:14.222012 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:14.221842    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjxb8\" (UniqueName: \"kubernetes.io/projected/04d28069-668d-4473-b6d2-c2611586988e-kube-api-access-pjxb8\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-psgw5\" (UID: \"04d28069-668d-4473-b6d2-c2611586988e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-psgw5"
Aug 05 17:19:14.322472 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:14.322384    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/04d28069-668d-4473-b6d2-c2611586988e-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-psgw5\" (UID: \"04d28069-668d-4473-b6d2-c2611586988e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-psgw5"
Aug 05 17:19:14.322472 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:14.322450    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjxb8\" (UniqueName: \"kubernetes.io/projected/04d28069-668d-4473-b6d2-c2611586988e-kube-api-access-pjxb8\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-psgw5\" (UID: \"04d28069-668d-4473-b6d2-c2611586988e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-psgw5"
Aug 05 17:19:14.324826 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:14.324801    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/04d28069-668d-4473-b6d2-c2611586988e-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-psgw5\" (UID: \"04d28069-668d-4473-b6d2-c2611586988e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-psgw5"
Aug 05 17:19:14.340060 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:14.340028    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjxb8\" (UniqueName: \"kubernetes.io/projected/04d28069-668d-4473-b6d2-c2611586988e-kube-api-access-pjxb8\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-psgw5\" (UID: \"04d28069-668d-4473-b6d2-c2611586988e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-psgw5"
Aug 05 17:19:14.407025 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:14.406983    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-psgw5"
Aug 05 17:19:14.563069 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:14.563041    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-psgw5"]
Aug 05 17:19:14.565018 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:19:14.564995    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d28069_668d_4473_b6d2_c2611586988e.slice/crio-db53d958a7a59a3ae3d45d2cbbd5ee26886c0836ddc6c9c16f963b15408d0df0 WatchSource:0}: Error finding container db53d958a7a59a3ae3d45d2cbbd5ee26886c0836ddc6c9c16f963b15408d0df0: Status 404 returned error can't find the container with id db53d958a7a59a3ae3d45d2cbbd5ee26886c0836ddc6c9c16f963b15408d0df0
Aug 05 17:19:15.271088 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:15.271051    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-psgw5" event={"ID":"04d28069-668d-4473-b6d2-c2611586988e","Type":"ContainerStarted","Data":"db53d958a7a59a3ae3d45d2cbbd5ee26886c0836ddc6c9c16f963b15408d0df0"}
Aug 05 17:19:25.309817 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:25.309783    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-psgw5" event={"ID":"04d28069-668d-4473-b6d2-c2611586988e","Type":"ContainerStarted","Data":"7aadd0b50da2aab1d98ba9dbb33194ec76b2badf7bf8efddb4fc6e39310ce99f"}
Aug 05 17:19:25.310272 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:25.309894    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-psgw5"
Aug 05 17:19:25.333941 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:25.333881    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-psgw5" podStartSLOduration=0.893195333 podStartE2EDuration="11.333849141s" totalImagesPullingTime="10.440653808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:19:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:19:25.332745785 +0000 UTC m=+421.077380113" watchObservedRunningTime="2026-08-05 17:19:25.333849141 +0000 UTC m=+421.078483469"
Aug 05 17:19:25.730580 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:25.730507    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-fvfml"]
Aug 05 17:19:25.732473 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:25.732457    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-fvfml"
Aug 05 17:19:25.735142 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:25.735115    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Aug 05 17:19:25.735256 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:25.735174    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-pvzmt\""
Aug 05 17:19:25.735310 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:25.735275    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Aug 05 17:19:25.744241 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:25.744212    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-fvfml"]
Aug 05 17:19:25.813910 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:25.813869    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dhnj\" (UniqueName: \"kubernetes.io/projected/1da5f38f-597c-4235-9c7b-ca2d608efecd-kube-api-access-7dhnj\") pod \"keda-operator-77f648d694-fvfml\" (UID: \"1da5f38f-597c-4235-9c7b-ca2d608efecd\") " pod="openshift-keda/keda-operator-77f648d694-fvfml"
Aug 05 17:19:25.813910 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:25.813914    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1da5f38f-597c-4235-9c7b-ca2d608efecd-certificates\") pod \"keda-operator-77f648d694-fvfml\" (UID: \"1da5f38f-597c-4235-9c7b-ca2d608efecd\") " pod="openshift-keda/keda-operator-77f648d694-fvfml"
Aug 05 17:19:25.814151 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:25.813972    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1da5f38f-597c-4235-9c7b-ca2d608efecd-cabundle0\") pod \"keda-operator-77f648d694-fvfml\" (UID: \"1da5f38f-597c-4235-9c7b-ca2d608efecd\") " pod="openshift-keda/keda-operator-77f648d694-fvfml"
Aug 05 17:19:25.915121 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:25.915080    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dhnj\" (UniqueName: \"kubernetes.io/projected/1da5f38f-597c-4235-9c7b-ca2d608efecd-kube-api-access-7dhnj\") pod \"keda-operator-77f648d694-fvfml\" (UID: \"1da5f38f-597c-4235-9c7b-ca2d608efecd\") " pod="openshift-keda/keda-operator-77f648d694-fvfml"
Aug 05 17:19:25.915121 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:25.915125    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1da5f38f-597c-4235-9c7b-ca2d608efecd-certificates\") pod \"keda-operator-77f648d694-fvfml\" (UID: \"1da5f38f-597c-4235-9c7b-ca2d608efecd\") " pod="openshift-keda/keda-operator-77f648d694-fvfml"
Aug 05 17:19:25.915376 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:25.915170    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1da5f38f-597c-4235-9c7b-ca2d608efecd-cabundle0\") pod \"keda-operator-77f648d694-fvfml\" (UID: \"1da5f38f-597c-4235-9c7b-ca2d608efecd\") " pod="openshift-keda/keda-operator-77f648d694-fvfml"
Aug 05 17:19:25.915376 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:19:25.915318    2561 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Aug 05 17:19:25.915376 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:19:25.915346    2561 secret.go:281] references non-existent secret key: ca.crt
Aug 05 17:19:25.915376 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:19:25.915354    2561 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Aug 05 17:19:25.915589 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:19:25.915386    2561 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-fvfml: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Aug 05 17:19:25.915589 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:19:25.915470    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1da5f38f-597c-4235-9c7b-ca2d608efecd-certificates podName:1da5f38f-597c-4235-9c7b-ca2d608efecd nodeName:}" failed. No retries permitted until 2026-08-05 17:19:26.415453216 +0000 UTC m=+422.160087526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/1da5f38f-597c-4235-9c7b-ca2d608efecd-certificates") pod "keda-operator-77f648d694-fvfml" (UID: "1da5f38f-597c-4235-9c7b-ca2d608efecd") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Aug 05 17:19:25.915909 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:25.915887    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1da5f38f-597c-4235-9c7b-ca2d608efecd-cabundle0\") pod \"keda-operator-77f648d694-fvfml\" (UID: \"1da5f38f-597c-4235-9c7b-ca2d608efecd\") " pod="openshift-keda/keda-operator-77f648d694-fvfml"
Aug 05 17:19:25.930335 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:25.930164    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dhnj\" (UniqueName: \"kubernetes.io/projected/1da5f38f-597c-4235-9c7b-ca2d608efecd-kube-api-access-7dhnj\") pod \"keda-operator-77f648d694-fvfml\" (UID: \"1da5f38f-597c-4235-9c7b-ca2d608efecd\") " pod="openshift-keda/keda-operator-77f648d694-fvfml"
Aug 05 17:19:25.931881 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:25.931857    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-gvwkg"]
Aug 05 17:19:25.934616 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:25.934600    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-gvwkg"
Aug 05 17:19:25.937007 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:25.936982    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Aug 05 17:19:25.943679 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:25.943657    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-gvwkg"]
Aug 05 17:19:26.016420 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:26.016306    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d1c060d8-3ba8-4093-93f9-410cf888fd38-certificates\") pod \"keda-metrics-apiserver-55557df8c-gvwkg\" (UID: \"d1c060d8-3ba8-4093-93f9-410cf888fd38\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-gvwkg"
Aug 05 17:19:26.016420 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:26.016363    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/d1c060d8-3ba8-4093-93f9-410cf888fd38-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-gvwkg\" (UID: \"d1c060d8-3ba8-4093-93f9-410cf888fd38\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-gvwkg"
Aug 05 17:19:26.016633 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:26.016478    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x7h5\" (UniqueName: \"kubernetes.io/projected/d1c060d8-3ba8-4093-93f9-410cf888fd38-kube-api-access-2x7h5\") pod \"keda-metrics-apiserver-55557df8c-gvwkg\" (UID: \"d1c060d8-3ba8-4093-93f9-410cf888fd38\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-gvwkg"
Aug 05 17:19:26.117274 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:26.117236    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/d1c060d8-3ba8-4093-93f9-410cf888fd38-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-gvwkg\" (UID: \"d1c060d8-3ba8-4093-93f9-410cf888fd38\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-gvwkg"
Aug 05 17:19:26.117483 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:26.117306    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x7h5\" (UniqueName: \"kubernetes.io/projected/d1c060d8-3ba8-4093-93f9-410cf888fd38-kube-api-access-2x7h5\") pod \"keda-metrics-apiserver-55557df8c-gvwkg\" (UID: \"d1c060d8-3ba8-4093-93f9-410cf888fd38\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-gvwkg"
Aug 05 17:19:26.117483 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:26.117350    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d1c060d8-3ba8-4093-93f9-410cf888fd38-certificates\") pod \"keda-metrics-apiserver-55557df8c-gvwkg\" (UID: \"d1c060d8-3ba8-4093-93f9-410cf888fd38\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-gvwkg"
Aug 05 17:19:26.117483 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:19:26.117429    2561 secret.go:281] references non-existent secret key: tls.crt
Aug 05 17:19:26.117483 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:19:26.117441    2561 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Aug 05 17:19:26.117483 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:19:26.117455    2561 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Aug 05 17:19:26.117483 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:19:26.117469    2561 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-gvwkg: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Aug 05 17:19:26.117772 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:19:26.117528    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1c060d8-3ba8-4093-93f9-410cf888fd38-certificates podName:d1c060d8-3ba8-4093-93f9-410cf888fd38 nodeName:}" failed. No retries permitted until 2026-08-05 17:19:26.617515222 +0000 UTC m=+422.362149528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/d1c060d8-3ba8-4093-93f9-410cf888fd38-certificates") pod "keda-metrics-apiserver-55557df8c-gvwkg" (UID: "d1c060d8-3ba8-4093-93f9-410cf888fd38") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Aug 05 17:19:26.117772 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:26.117625    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/d1c060d8-3ba8-4093-93f9-410cf888fd38-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-gvwkg\" (UID: \"d1c060d8-3ba8-4093-93f9-410cf888fd38\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-gvwkg"
Aug 05 17:19:26.126735 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:26.126700    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x7h5\" (UniqueName: \"kubernetes.io/projected/d1c060d8-3ba8-4093-93f9-410cf888fd38-kube-api-access-2x7h5\") pod \"keda-metrics-apiserver-55557df8c-gvwkg\" (UID: \"d1c060d8-3ba8-4093-93f9-410cf888fd38\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-gvwkg"
Aug 05 17:19:26.419817 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:26.419774    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1da5f38f-597c-4235-9c7b-ca2d608efecd-certificates\") pod \"keda-operator-77f648d694-fvfml\" (UID: \"1da5f38f-597c-4235-9c7b-ca2d608efecd\") " pod="openshift-keda/keda-operator-77f648d694-fvfml"
Aug 05 17:19:26.422211 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:26.422188    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1da5f38f-597c-4235-9c7b-ca2d608efecd-certificates\") pod \"keda-operator-77f648d694-fvfml\" (UID: \"1da5f38f-597c-4235-9c7b-ca2d608efecd\") " pod="openshift-keda/keda-operator-77f648d694-fvfml"
Aug 05 17:19:26.513529 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:26.513496    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-r7m9s"]
Aug 05 17:19:26.515712 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:26.515684    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-r7m9s"
Aug 05 17:19:26.518154 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:26.518138    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Aug 05 17:19:26.525742 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:26.525710    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-r7m9s"]
Aug 05 17:19:26.621150 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:26.621117    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/077d16b9-1b2e-4bfc-9ad2-62547fbc8cf4-certificates\") pod \"keda-admission-56f9798bb4-r7m9s\" (UID: \"077d16b9-1b2e-4bfc-9ad2-62547fbc8cf4\") " pod="openshift-keda/keda-admission-56f9798bb4-r7m9s"
Aug 05 17:19:26.621331 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:26.621171    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d1c060d8-3ba8-4093-93f9-410cf888fd38-certificates\") pod \"keda-metrics-apiserver-55557df8c-gvwkg\" (UID: \"d1c060d8-3ba8-4093-93f9-410cf888fd38\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-gvwkg"
Aug 05 17:19:26.621331 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:26.621286    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7ngr\" (UniqueName: \"kubernetes.io/projected/077d16b9-1b2e-4bfc-9ad2-62547fbc8cf4-kube-api-access-p7ngr\") pod \"keda-admission-56f9798bb4-r7m9s\" (UID: \"077d16b9-1b2e-4bfc-9ad2-62547fbc8cf4\") " pod="openshift-keda/keda-admission-56f9798bb4-r7m9s"
Aug 05 17:19:26.623685 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:26.623650    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d1c060d8-3ba8-4093-93f9-410cf888fd38-certificates\") pod \"keda-metrics-apiserver-55557df8c-gvwkg\" (UID: \"d1c060d8-3ba8-4093-93f9-410cf888fd38\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-gvwkg"
Aug 05 17:19:26.642462 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:26.642429    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-fvfml"
Aug 05 17:19:26.722103 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:26.722074    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7ngr\" (UniqueName: \"kubernetes.io/projected/077d16b9-1b2e-4bfc-9ad2-62547fbc8cf4-kube-api-access-p7ngr\") pod \"keda-admission-56f9798bb4-r7m9s\" (UID: \"077d16b9-1b2e-4bfc-9ad2-62547fbc8cf4\") " pod="openshift-keda/keda-admission-56f9798bb4-r7m9s"
Aug 05 17:19:26.722252 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:26.722123    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/077d16b9-1b2e-4bfc-9ad2-62547fbc8cf4-certificates\") pod \"keda-admission-56f9798bb4-r7m9s\" (UID: \"077d16b9-1b2e-4bfc-9ad2-62547fbc8cf4\") " pod="openshift-keda/keda-admission-56f9798bb4-r7m9s"
Aug 05 17:19:26.725048 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:26.725017    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/077d16b9-1b2e-4bfc-9ad2-62547fbc8cf4-certificates\") pod \"keda-admission-56f9798bb4-r7m9s\" (UID: \"077d16b9-1b2e-4bfc-9ad2-62547fbc8cf4\") " pod="openshift-keda/keda-admission-56f9798bb4-r7m9s"
Aug 05 17:19:26.734448 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:26.734415    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7ngr\" (UniqueName: \"kubernetes.io/projected/077d16b9-1b2e-4bfc-9ad2-62547fbc8cf4-kube-api-access-p7ngr\") pod \"keda-admission-56f9798bb4-r7m9s\" (UID: \"077d16b9-1b2e-4bfc-9ad2-62547fbc8cf4\") " pod="openshift-keda/keda-admission-56f9798bb4-r7m9s"
Aug 05 17:19:26.806538 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:26.805697    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-fvfml"]
Aug 05 17:19:26.807464 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:19:26.807430    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da5f38f_597c_4235_9c7b_ca2d608efecd.slice/crio-997a1857821149284ed751a43ab2082a58938cf593609910e732b48e83feb661 WatchSource:0}: Error finding container 997a1857821149284ed751a43ab2082a58938cf593609910e732b48e83feb661: Status 404 returned error can't find the container with id 997a1857821149284ed751a43ab2082a58938cf593609910e732b48e83feb661
Aug 05 17:19:26.826130 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:26.826098    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-r7m9s"
Aug 05 17:19:26.847129 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:26.847100    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-gvwkg"
Aug 05 17:19:26.992115 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:26.992088    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-r7m9s"]
Aug 05 17:19:26.994495 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:19:26.994468    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod077d16b9_1b2e_4bfc_9ad2_62547fbc8cf4.slice/crio-5ec050978bdac056bae9332dc7b1ee32a9f74faacbacca243e64fcc42884a551 WatchSource:0}: Error finding container 5ec050978bdac056bae9332dc7b1ee32a9f74faacbacca243e64fcc42884a551: Status 404 returned error can't find the container with id 5ec050978bdac056bae9332dc7b1ee32a9f74faacbacca243e64fcc42884a551
Aug 05 17:19:27.012101 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:27.012048    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-gvwkg"]
Aug 05 17:19:27.015329 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:19:27.015299    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c060d8_3ba8_4093_93f9_410cf888fd38.slice/crio-bbf84bd926cd3954fd72d0029aebef4958f9b72e27114c2ef9449b8ea37de7d9 WatchSource:0}: Error finding container bbf84bd926cd3954fd72d0029aebef4958f9b72e27114c2ef9449b8ea37de7d9: Status 404 returned error can't find the container with id bbf84bd926cd3954fd72d0029aebef4958f9b72e27114c2ef9449b8ea37de7d9
Aug 05 17:19:27.321620 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:27.321585    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-fvfml" event={"ID":"1da5f38f-597c-4235-9c7b-ca2d608efecd","Type":"ContainerStarted","Data":"997a1857821149284ed751a43ab2082a58938cf593609910e732b48e83feb661"}
Aug 05 17:19:27.322819 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:27.322786    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-gvwkg" event={"ID":"d1c060d8-3ba8-4093-93f9-410cf888fd38","Type":"ContainerStarted","Data":"bbf84bd926cd3954fd72d0029aebef4958f9b72e27114c2ef9449b8ea37de7d9"}
Aug 05 17:19:27.323849 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:27.323830    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-r7m9s" event={"ID":"077d16b9-1b2e-4bfc-9ad2-62547fbc8cf4","Type":"ContainerStarted","Data":"5ec050978bdac056bae9332dc7b1ee32a9f74faacbacca243e64fcc42884a551"}
Aug 05 17:19:29.339510 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:29.339475    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-r7m9s" event={"ID":"077d16b9-1b2e-4bfc-9ad2-62547fbc8cf4","Type":"ContainerStarted","Data":"78dd41a22baeedb75214b749d024ad535e7bd17ad314d4602a8d89031373a2ac"}
Aug 05 17:19:29.340008 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:29.339610    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-r7m9s"
Aug 05 17:19:29.357919 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:29.357848    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-r7m9s" podStartSLOduration=1.6824933610000001 podStartE2EDuration="3.357827843s" totalImagesPullingTime="1.675334482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:19:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:19:29.356355702 +0000 UTC m=+425.100990030" watchObservedRunningTime="2026-08-05 17:19:29.357827843 +0000 UTC m=+425.102462170"
Aug 05 17:19:31.350405 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:31.350367    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-gvwkg" event={"ID":"d1c060d8-3ba8-4093-93f9-410cf888fd38","Type":"ContainerStarted","Data":"2e52fc7e166e1f9e449ca12348a1fa623c27287b202eb5e792e3f154b81f7a2e"}
Aug 05 17:19:31.350872 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:31.350425    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-gvwkg"
Aug 05 17:19:31.351802 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:31.351775    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-fvfml" event={"ID":"1da5f38f-597c-4235-9c7b-ca2d608efecd","Type":"ContainerStarted","Data":"be333d7fac7fc5a9291fea6a363a5cb2de29062e6357efe6827122f548f29e38"}
Aug 05 17:19:31.351933 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:31.351918    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-fvfml"
Aug 05 17:19:31.368547 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:31.368498    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-gvwkg" podStartSLOduration=2.464252787 podStartE2EDuration="6.368483342s" totalImagesPullingTime="3.904230555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:19:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:19:31.367498287 +0000 UTC m=+427.112132634" watchObservedRunningTime="2026-08-05 17:19:31.368483342 +0000 UTC m=+427.113117669"
Aug 05 17:19:31.385753 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:31.385707    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-fvfml" podStartSLOduration=2.268855491 podStartE2EDuration="6.385691564s" totalImagesPullingTime="4.116836073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:19:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:19:31.385299925 +0000 UTC m=+427.129934253" watchObservedRunningTime="2026-08-05 17:19:31.385691564 +0000 UTC m=+427.130325921"
Aug 05 17:19:42.360919 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:42.360880    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-gvwkg"
Aug 05 17:19:46.317175 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:46.317145    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-psgw5"
Aug 05 17:19:50.347054 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:50.347025    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-r7m9s"
Aug 05 17:19:52.358658 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:19:52.358629    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-fvfml"
Aug 05 17:20:19.148183 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:19.148151    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g9m6j"]
Aug 05 17:20:19.150297 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:19.150280    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g9m6j"
Aug 05 17:20:19.153305 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:19.153285    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Aug 05 17:20:19.154204 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:19.154187    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-npp6p\""
Aug 05 17:20:19.154265 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:19.154187    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Aug 05 17:20:19.168344 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:19.168304    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g9m6j"]
Aug 05 17:20:19.224635 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:19.224597    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/937239d9-e3b8-4f23-9c94-d7a03d7fad1a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g9m6j\" (UID: \"937239d9-e3b8-4f23-9c94-d7a03d7fad1a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g9m6j"
Aug 05 17:20:19.224825 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:19.224646    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztxvm\" (UniqueName: \"kubernetes.io/projected/937239d9-e3b8-4f23-9c94-d7a03d7fad1a-kube-api-access-ztxvm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g9m6j\" (UID: \"937239d9-e3b8-4f23-9c94-d7a03d7fad1a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g9m6j"
Aug 05 17:20:19.224825 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:19.224733    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/937239d9-e3b8-4f23-9c94-d7a03d7fad1a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g9m6j\" (UID: \"937239d9-e3b8-4f23-9c94-d7a03d7fad1a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g9m6j"
Aug 05 17:20:19.326052 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:19.326013    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztxvm\" (UniqueName: \"kubernetes.io/projected/937239d9-e3b8-4f23-9c94-d7a03d7fad1a-kube-api-access-ztxvm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g9m6j\" (UID: \"937239d9-e3b8-4f23-9c94-d7a03d7fad1a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g9m6j"
Aug 05 17:20:19.326187 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:19.326073    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/937239d9-e3b8-4f23-9c94-d7a03d7fad1a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g9m6j\" (UID: \"937239d9-e3b8-4f23-9c94-d7a03d7fad1a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g9m6j"
Aug 05 17:20:19.326187 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:19.326116    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/937239d9-e3b8-4f23-9c94-d7a03d7fad1a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g9m6j\" (UID: \"937239d9-e3b8-4f23-9c94-d7a03d7fad1a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g9m6j"
Aug 05 17:20:19.326443 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:19.326429    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/937239d9-e3b8-4f23-9c94-d7a03d7fad1a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g9m6j\" (UID: \"937239d9-e3b8-4f23-9c94-d7a03d7fad1a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g9m6j"
Aug 05 17:20:19.326482 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:19.326465    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/937239d9-e3b8-4f23-9c94-d7a03d7fad1a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g9m6j\" (UID: \"937239d9-e3b8-4f23-9c94-d7a03d7fad1a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g9m6j"
Aug 05 17:20:19.337324 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:19.337290    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztxvm\" (UniqueName: \"kubernetes.io/projected/937239d9-e3b8-4f23-9c94-d7a03d7fad1a-kube-api-access-ztxvm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g9m6j\" (UID: \"937239d9-e3b8-4f23-9c94-d7a03d7fad1a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g9m6j"
Aug 05 17:20:19.458667 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:19.458560    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g9m6j"
Aug 05 17:20:19.590966 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:19.590930    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g9m6j"]
Aug 05 17:20:19.593457 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:20:19.593433    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod937239d9_e3b8_4f23_9c94_d7a03d7fad1a.slice/crio-5ee6e108313aa5951d68e8ae82d24e37818d7a07d681c29662d47dcb9cf09762 WatchSource:0}: Error finding container 5ee6e108313aa5951d68e8ae82d24e37818d7a07d681c29662d47dcb9cf09762: Status 404 returned error can't find the container with id 5ee6e108313aa5951d68e8ae82d24e37818d7a07d681c29662d47dcb9cf09762
Aug 05 17:20:20.549814 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:20.549778    2561 generic.go:356] "Generic (PLEG): container finished" podID="937239d9-e3b8-4f23-9c94-d7a03d7fad1a" containerID="16ea717e3c4adc41501a9df4b29a5d900e29505bff7dbd327436c497e343c803" exitCode=0
Aug 05 17:20:20.549814 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:20.549820    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g9m6j" event={"ID":"937239d9-e3b8-4f23-9c94-d7a03d7fad1a","Type":"ContainerDied","Data":"16ea717e3c4adc41501a9df4b29a5d900e29505bff7dbd327436c497e343c803"}
Aug 05 17:20:20.550380 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:20.549855    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g9m6j" event={"ID":"937239d9-e3b8-4f23-9c94-d7a03d7fad1a","Type":"ContainerStarted","Data":"5ee6e108313aa5951d68e8ae82d24e37818d7a07d681c29662d47dcb9cf09762"}
Aug 05 17:20:23.564966 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:23.564928    2561 generic.go:356] "Generic (PLEG): container finished" podID="937239d9-e3b8-4f23-9c94-d7a03d7fad1a" containerID="869da9d801ac4d10d388b748f5d375fc7e1fe2b4496fd0c6d04e5002a17bb1bb" exitCode=0
Aug 05 17:20:23.565361 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:23.565013    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g9m6j" event={"ID":"937239d9-e3b8-4f23-9c94-d7a03d7fad1a","Type":"ContainerDied","Data":"869da9d801ac4d10d388b748f5d375fc7e1fe2b4496fd0c6d04e5002a17bb1bb"}
Aug 05 17:20:24.573835 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:24.573805    2561 generic.go:356] "Generic (PLEG): container finished" podID="937239d9-e3b8-4f23-9c94-d7a03d7fad1a" containerID="d0a147d610abc2f8132b3a500be584abae92f8a2289fcd64f36afb3d5570fc87" exitCode=0
Aug 05 17:20:24.574256 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:24.573868    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g9m6j" event={"ID":"937239d9-e3b8-4f23-9c94-d7a03d7fad1a","Type":"ContainerDied","Data":"d0a147d610abc2f8132b3a500be584abae92f8a2289fcd64f36afb3d5570fc87"}
Aug 05 17:20:25.700788 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:25.700763    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g9m6j"
Aug 05 17:20:25.779582 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:25.779546    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/937239d9-e3b8-4f23-9c94-d7a03d7fad1a-bundle\") pod \"937239d9-e3b8-4f23-9c94-d7a03d7fad1a\" (UID: \"937239d9-e3b8-4f23-9c94-d7a03d7fad1a\") "
Aug 05 17:20:25.779766 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:25.779611    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztxvm\" (UniqueName: \"kubernetes.io/projected/937239d9-e3b8-4f23-9c94-d7a03d7fad1a-kube-api-access-ztxvm\") pod \"937239d9-e3b8-4f23-9c94-d7a03d7fad1a\" (UID: \"937239d9-e3b8-4f23-9c94-d7a03d7fad1a\") "
Aug 05 17:20:25.779766 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:25.779631    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/937239d9-e3b8-4f23-9c94-d7a03d7fad1a-util\") pod \"937239d9-e3b8-4f23-9c94-d7a03d7fad1a\" (UID: \"937239d9-e3b8-4f23-9c94-d7a03d7fad1a\") "
Aug 05 17:20:25.780573 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:25.780547    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937239d9-e3b8-4f23-9c94-d7a03d7fad1a-bundle" (OuterVolumeSpecName: "bundle") pod "937239d9-e3b8-4f23-9c94-d7a03d7fad1a" (UID: "937239d9-e3b8-4f23-9c94-d7a03d7fad1a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:20:25.782131 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:25.782107    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937239d9-e3b8-4f23-9c94-d7a03d7fad1a-kube-api-access-ztxvm" (OuterVolumeSpecName: "kube-api-access-ztxvm") pod "937239d9-e3b8-4f23-9c94-d7a03d7fad1a" (UID: "937239d9-e3b8-4f23-9c94-d7a03d7fad1a"). InnerVolumeSpecName "kube-api-access-ztxvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:20:25.785487 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:25.785462    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937239d9-e3b8-4f23-9c94-d7a03d7fad1a-util" (OuterVolumeSpecName: "util") pod "937239d9-e3b8-4f23-9c94-d7a03d7fad1a" (UID: "937239d9-e3b8-4f23-9c94-d7a03d7fad1a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:20:25.881066 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:25.880945    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztxvm\" (UniqueName: \"kubernetes.io/projected/937239d9-e3b8-4f23-9c94-d7a03d7fad1a-kube-api-access-ztxvm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:20:25.881066 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:25.881009    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/937239d9-e3b8-4f23-9c94-d7a03d7fad1a-util\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:20:25.881066 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:25.881020    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/937239d9-e3b8-4f23-9c94-d7a03d7fad1a-bundle\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:20:26.585488 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:26.585450    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g9m6j" event={"ID":"937239d9-e3b8-4f23-9c94-d7a03d7fad1a","Type":"ContainerDied","Data":"5ee6e108313aa5951d68e8ae82d24e37818d7a07d681c29662d47dcb9cf09762"}
Aug 05 17:20:26.585488 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:26.585481    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g9m6j"
Aug 05 17:20:26.585735 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:26.585543    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ee6e108313aa5951d68e8ae82d24e37818d7a07d681c29662d47dcb9cf09762"
Aug 05 17:20:32.247718 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:32.247683    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cpfjr"]
Aug 05 17:20:32.248131 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:32.247969    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="937239d9-e3b8-4f23-9c94-d7a03d7fad1a" containerName="util"
Aug 05 17:20:32.248131 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:32.247981    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="937239d9-e3b8-4f23-9c94-d7a03d7fad1a" containerName="util"
Aug 05 17:20:32.248131 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:32.247988    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="937239d9-e3b8-4f23-9c94-d7a03d7fad1a" containerName="pull"
Aug 05 17:20:32.248131 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:32.247993    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="937239d9-e3b8-4f23-9c94-d7a03d7fad1a" containerName="pull"
Aug 05 17:20:32.248131 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:32.248001    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="937239d9-e3b8-4f23-9c94-d7a03d7fad1a" containerName="extract"
Aug 05 17:20:32.248131 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:32.248007    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="937239d9-e3b8-4f23-9c94-d7a03d7fad1a" containerName="extract"
Aug 05 17:20:32.248131 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:32.248055    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="937239d9-e3b8-4f23-9c94-d7a03d7fad1a" containerName="extract"
Aug 05 17:20:32.250178 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:32.250162    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cpfjr"
Aug 05 17:20:32.252933 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:32.252895    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-mghld\""
Aug 05 17:20:32.252933 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:32.252936    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Aug 05 17:20:32.253111 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:32.252895    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Aug 05 17:20:32.266515 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:32.264104    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cpfjr"]
Aug 05 17:20:32.332251 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:32.332212    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9844fd23-126c-4a70-b511-63f046112798-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-cpfjr\" (UID: \"9844fd23-126c-4a70-b511-63f046112798\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cpfjr"
Aug 05 17:20:32.332251 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:32.332256    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjlzj\" (UniqueName: \"kubernetes.io/projected/9844fd23-126c-4a70-b511-63f046112798-kube-api-access-tjlzj\") pod \"cert-manager-operator-controller-manager-684f79c9b9-cpfjr\" (UID: \"9844fd23-126c-4a70-b511-63f046112798\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cpfjr"
Aug 05 17:20:32.433704 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:32.433656    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9844fd23-126c-4a70-b511-63f046112798-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-cpfjr\" (UID: \"9844fd23-126c-4a70-b511-63f046112798\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cpfjr"
Aug 05 17:20:32.433704 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:32.433710    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjlzj\" (UniqueName: \"kubernetes.io/projected/9844fd23-126c-4a70-b511-63f046112798-kube-api-access-tjlzj\") pod \"cert-manager-operator-controller-manager-684f79c9b9-cpfjr\" (UID: \"9844fd23-126c-4a70-b511-63f046112798\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cpfjr"
Aug 05 17:20:32.434161 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:32.434140    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9844fd23-126c-4a70-b511-63f046112798-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-cpfjr\" (UID: \"9844fd23-126c-4a70-b511-63f046112798\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cpfjr"
Aug 05 17:20:32.444058 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:32.444038    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjlzj\" (UniqueName: \"kubernetes.io/projected/9844fd23-126c-4a70-b511-63f046112798-kube-api-access-tjlzj\") pod \"cert-manager-operator-controller-manager-684f79c9b9-cpfjr\" (UID: \"9844fd23-126c-4a70-b511-63f046112798\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cpfjr"
Aug 05 17:20:32.559309 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:32.559277    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cpfjr"
Aug 05 17:20:32.701273 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:32.701249    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cpfjr"]
Aug 05 17:20:32.704072 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:20:32.704048    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9844fd23_126c_4a70_b511_63f046112798.slice/crio-b9a9fe0b7d16c01a9a8b4dc1ee287af787f5b44222c7b95b6954ad9f78c0e747 WatchSource:0}: Error finding container b9a9fe0b7d16c01a9a8b4dc1ee287af787f5b44222c7b95b6954ad9f78c0e747: Status 404 returned error can't find the container with id b9a9fe0b7d16c01a9a8b4dc1ee287af787f5b44222c7b95b6954ad9f78c0e747
Aug 05 17:20:33.621169 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:33.621131    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cpfjr" event={"ID":"9844fd23-126c-4a70-b511-63f046112798","Type":"ContainerStarted","Data":"b9a9fe0b7d16c01a9a8b4dc1ee287af787f5b44222c7b95b6954ad9f78c0e747"}
Aug 05 17:20:35.631733 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:35.631704    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cpfjr" event={"ID":"9844fd23-126c-4a70-b511-63f046112798","Type":"ContainerStarted","Data":"bc162610df5e7be0b916d2ab931d57bf9a4c8d0c44c9d0ba2000fdbb9a615c94"}
Aug 05 17:20:35.678177 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:35.678121    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cpfjr" podStartSLOduration=0.831857395 podStartE2EDuration="3.678103039s" totalImagesPullingTime="2.846245644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:20:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:20:35.676229685 +0000 UTC m=+491.420864012" watchObservedRunningTime="2026-08-05 17:20:35.678103039 +0000 UTC m=+491.422737367"
Aug 05 17:20:40.585106 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:40.585072    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fswljb"]
Aug 05 17:20:40.587471 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:40.587455    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fswljb"
Aug 05 17:20:40.591330 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:40.591307    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Aug 05 17:20:40.591455 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:40.591312    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-npp6p\""
Aug 05 17:20:40.592295 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:40.592280    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Aug 05 17:20:40.601558 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:40.601537    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fswljb"]
Aug 05 17:20:40.703917 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:40.703878    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbd8l\" (UniqueName: \"kubernetes.io/projected/f47cd013-a6df-40bb-8017-9a885fbcb894-kube-api-access-hbd8l\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fswljb\" (UID: \"f47cd013-a6df-40bb-8017-9a885fbcb894\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fswljb"
Aug 05 17:20:40.703917 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:40.703922    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f47cd013-a6df-40bb-8017-9a885fbcb894-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fswljb\" (UID: \"f47cd013-a6df-40bb-8017-9a885fbcb894\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fswljb"
Aug 05 17:20:40.704160 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:40.703946    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f47cd013-a6df-40bb-8017-9a885fbcb894-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fswljb\" (UID: \"f47cd013-a6df-40bb-8017-9a885fbcb894\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fswljb"
Aug 05 17:20:40.805307 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:40.805269    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbd8l\" (UniqueName: \"kubernetes.io/projected/f47cd013-a6df-40bb-8017-9a885fbcb894-kube-api-access-hbd8l\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fswljb\" (UID: \"f47cd013-a6df-40bb-8017-9a885fbcb894\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fswljb"
Aug 05 17:20:40.805307 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:40.805313    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f47cd013-a6df-40bb-8017-9a885fbcb894-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fswljb\" (UID: \"f47cd013-a6df-40bb-8017-9a885fbcb894\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fswljb"
Aug 05 17:20:40.805552 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:40.805335    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f47cd013-a6df-40bb-8017-9a885fbcb894-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fswljb\" (UID: \"f47cd013-a6df-40bb-8017-9a885fbcb894\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fswljb"
Aug 05 17:20:40.805728 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:40.805705    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f47cd013-a6df-40bb-8017-9a885fbcb894-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fswljb\" (UID: \"f47cd013-a6df-40bb-8017-9a885fbcb894\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fswljb"
Aug 05 17:20:40.805796 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:40.805752    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f47cd013-a6df-40bb-8017-9a885fbcb894-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fswljb\" (UID: \"f47cd013-a6df-40bb-8017-9a885fbcb894\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fswljb"
Aug 05 17:20:40.814232 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:40.814208    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbd8l\" (UniqueName: \"kubernetes.io/projected/f47cd013-a6df-40bb-8017-9a885fbcb894-kube-api-access-hbd8l\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fswljb\" (UID: \"f47cd013-a6df-40bb-8017-9a885fbcb894\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fswljb"
Aug 05 17:20:40.896463 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:40.896375    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fswljb"
Aug 05 17:20:41.031100 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:41.031075    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fswljb"]
Aug 05 17:20:41.032625 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:20:41.032597    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf47cd013_a6df_40bb_8017_9a885fbcb894.slice/crio-7be3ed8542da625bdc9b20c7370c791304f254574f74b88d148d3e1ef45d1e0a WatchSource:0}: Error finding container 7be3ed8542da625bdc9b20c7370c791304f254574f74b88d148d3e1ef45d1e0a: Status 404 returned error can't find the container with id 7be3ed8542da625bdc9b20c7370c791304f254574f74b88d148d3e1ef45d1e0a
Aug 05 17:20:41.662012 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:41.661976    2561 generic.go:356] "Generic (PLEG): container finished" podID="f47cd013-a6df-40bb-8017-9a885fbcb894" containerID="cf42e7348519d00d5b747854e48b9cbef313a10ab670e9e1e1b3174c5388085b" exitCode=0
Aug 05 17:20:41.662455 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:41.662057    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fswljb" event={"ID":"f47cd013-a6df-40bb-8017-9a885fbcb894","Type":"ContainerDied","Data":"cf42e7348519d00d5b747854e48b9cbef313a10ab670e9e1e1b3174c5388085b"}
Aug 05 17:20:41.662455 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:41.662147    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fswljb" event={"ID":"f47cd013-a6df-40bb-8017-9a885fbcb894","Type":"ContainerStarted","Data":"7be3ed8542da625bdc9b20c7370c791304f254574f74b88d148d3e1ef45d1e0a"}
Aug 05 17:20:43.673356 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:43.673323    2561 generic.go:356] "Generic (PLEG): container finished" podID="f47cd013-a6df-40bb-8017-9a885fbcb894" containerID="567f0b4e1a755049fe3992d789465e178f567cfdeb16bd8d0c5f467b28dfa67a" exitCode=0
Aug 05 17:20:43.673751 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:43.673373    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fswljb" event={"ID":"f47cd013-a6df-40bb-8017-9a885fbcb894","Type":"ContainerDied","Data":"567f0b4e1a755049fe3992d789465e178f567cfdeb16bd8d0c5f467b28dfa67a"}
Aug 05 17:20:44.679970 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:44.679919    2561 generic.go:356] "Generic (PLEG): container finished" podID="f47cd013-a6df-40bb-8017-9a885fbcb894" containerID="cd61524dcc702f0c253306acf4cc3ada4e41f26a3816adf301708354ac2d18b6" exitCode=0
Aug 05 17:20:44.680379 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:44.679995    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fswljb" event={"ID":"f47cd013-a6df-40bb-8017-9a885fbcb894","Type":"ContainerDied","Data":"cd61524dcc702f0c253306acf4cc3ada4e41f26a3816adf301708354ac2d18b6"}
Aug 05 17:20:45.810880 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:45.810852    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fswljb"
Aug 05 17:20:45.951869 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:45.951772    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f47cd013-a6df-40bb-8017-9a885fbcb894-bundle\") pod \"f47cd013-a6df-40bb-8017-9a885fbcb894\" (UID: \"f47cd013-a6df-40bb-8017-9a885fbcb894\") "
Aug 05 17:20:45.951869 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:45.951810    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f47cd013-a6df-40bb-8017-9a885fbcb894-util\") pod \"f47cd013-a6df-40bb-8017-9a885fbcb894\" (UID: \"f47cd013-a6df-40bb-8017-9a885fbcb894\") "
Aug 05 17:20:45.952233 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:45.951888    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbd8l\" (UniqueName: \"kubernetes.io/projected/f47cd013-a6df-40bb-8017-9a885fbcb894-kube-api-access-hbd8l\") pod \"f47cd013-a6df-40bb-8017-9a885fbcb894\" (UID: \"f47cd013-a6df-40bb-8017-9a885fbcb894\") "
Aug 05 17:20:45.952339 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:45.952307    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47cd013-a6df-40bb-8017-9a885fbcb894-bundle" (OuterVolumeSpecName: "bundle") pod "f47cd013-a6df-40bb-8017-9a885fbcb894" (UID: "f47cd013-a6df-40bb-8017-9a885fbcb894"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:20:45.954289 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:45.954251    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47cd013-a6df-40bb-8017-9a885fbcb894-kube-api-access-hbd8l" (OuterVolumeSpecName: "kube-api-access-hbd8l") pod "f47cd013-a6df-40bb-8017-9a885fbcb894" (UID: "f47cd013-a6df-40bb-8017-9a885fbcb894"). InnerVolumeSpecName "kube-api-access-hbd8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:20:45.957472 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:45.957444    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f47cd013-a6df-40bb-8017-9a885fbcb894-util" (OuterVolumeSpecName: "util") pod "f47cd013-a6df-40bb-8017-9a885fbcb894" (UID: "f47cd013-a6df-40bb-8017-9a885fbcb894"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:20:46.053151 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:46.053103    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f47cd013-a6df-40bb-8017-9a885fbcb894-bundle\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:20:46.053151 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:46.053140    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f47cd013-a6df-40bb-8017-9a885fbcb894-util\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:20:46.053151 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:46.053150    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbd8l\" (UniqueName: \"kubernetes.io/projected/f47cd013-a6df-40bb-8017-9a885fbcb894-kube-api-access-hbd8l\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:20:46.689724 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:46.689694    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fswljb"
Aug 05 17:20:46.689891 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:46.689690    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fswljb" event={"ID":"f47cd013-a6df-40bb-8017-9a885fbcb894","Type":"ContainerDied","Data":"7be3ed8542da625bdc9b20c7370c791304f254574f74b88d148d3e1ef45d1e0a"}
Aug 05 17:20:46.689891 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:20:46.689852    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7be3ed8542da625bdc9b20c7370c791304f254574f74b88d148d3e1ef45d1e0a"
Aug 05 17:21:08.954988 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:08.954939    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"]
Aug 05 17:21:08.955512 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:08.955245    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f47cd013-a6df-40bb-8017-9a885fbcb894" containerName="pull"
Aug 05 17:21:08.955512 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:08.955256    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f47cd013-a6df-40bb-8017-9a885fbcb894" containerName="pull"
Aug 05 17:21:08.955512 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:08.955265    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f47cd013-a6df-40bb-8017-9a885fbcb894" containerName="extract"
Aug 05 17:21:08.955512 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:08.955271    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f47cd013-a6df-40bb-8017-9a885fbcb894" containerName="extract"
Aug 05 17:21:08.955512 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:08.955280    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f47cd013-a6df-40bb-8017-9a885fbcb894" containerName="util"
Aug 05 17:21:08.955512 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:08.955286    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f47cd013-a6df-40bb-8017-9a885fbcb894" containerName="util"
Aug 05 17:21:08.955512 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:08.955332    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f47cd013-a6df-40bb-8017-9a885fbcb894" containerName="extract"
Aug 05 17:21:08.963675 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:08.963654    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"
Aug 05 17:21:08.966225 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:08.966197    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Aug 05 17:21:08.966225 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:08.966217    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-777hc\""
Aug 05 17:21:08.966433 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:08.966415    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Aug 05 17:21:08.966520 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:08.966500    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Aug 05 17:21:08.966638 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:08.966525    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Aug 05 17:21:08.966708 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:08.966536    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Aug 05 17:21:08.966708 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:08.966651    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Aug 05 17:21:08.972454 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:08.972435    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"]
Aug 05 17:21:09.041411 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:09.041367    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmbv2\" (UniqueName: \"kubernetes.io/projected/6d624442-959d-4265-8f55-f0be6ef1af83-kube-api-access-rmbv2\") pod \"istiod-openshift-gateway-7b7995d5cd-x9nz9\" (UID: \"6d624442-959d-4265-8f55-f0be6ef1af83\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"
Aug 05 17:21:09.041411 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:09.041411    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6d624442-959d-4265-8f55-f0be6ef1af83-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-x9nz9\" (UID: \"6d624442-959d-4265-8f55-f0be6ef1af83\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"
Aug 05 17:21:09.041608 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:09.041445    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6d624442-959d-4265-8f55-f0be6ef1af83-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-x9nz9\" (UID: \"6d624442-959d-4265-8f55-f0be6ef1af83\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"
Aug 05 17:21:09.041608 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:09.041474    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d624442-959d-4265-8f55-f0be6ef1af83-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-x9nz9\" (UID: \"6d624442-959d-4265-8f55-f0be6ef1af83\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"
Aug 05 17:21:09.041608 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:09.041493    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d624442-959d-4265-8f55-f0be6ef1af83-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-x9nz9\" (UID: \"6d624442-959d-4265-8f55-f0be6ef1af83\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"
Aug 05 17:21:09.041608 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:09.041542    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d624442-959d-4265-8f55-f0be6ef1af83-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-x9nz9\" (UID: \"6d624442-959d-4265-8f55-f0be6ef1af83\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"
Aug 05 17:21:09.041608 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:09.041557    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6d624442-959d-4265-8f55-f0be6ef1af83-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-x9nz9\" (UID: \"6d624442-959d-4265-8f55-f0be6ef1af83\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"
Aug 05 17:21:09.142007 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:09.141968    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d624442-959d-4265-8f55-f0be6ef1af83-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-x9nz9\" (UID: \"6d624442-959d-4265-8f55-f0be6ef1af83\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"
Aug 05 17:21:09.142209 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:09.142022    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d624442-959d-4265-8f55-f0be6ef1af83-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-x9nz9\" (UID: \"6d624442-959d-4265-8f55-f0be6ef1af83\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"
Aug 05 17:21:09.142209 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:09.142041    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6d624442-959d-4265-8f55-f0be6ef1af83-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-x9nz9\" (UID: \"6d624442-959d-4265-8f55-f0be6ef1af83\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"
Aug 05 17:21:09.142209 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:09.142066    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmbv2\" (UniqueName: \"kubernetes.io/projected/6d624442-959d-4265-8f55-f0be6ef1af83-kube-api-access-rmbv2\") pod \"istiod-openshift-gateway-7b7995d5cd-x9nz9\" (UID: \"6d624442-959d-4265-8f55-f0be6ef1af83\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"
Aug 05 17:21:09.142209 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:09.142085    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6d624442-959d-4265-8f55-f0be6ef1af83-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-x9nz9\" (UID: \"6d624442-959d-4265-8f55-f0be6ef1af83\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"
Aug 05 17:21:09.142209 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:09.142123    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6d624442-959d-4265-8f55-f0be6ef1af83-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-x9nz9\" (UID: \"6d624442-959d-4265-8f55-f0be6ef1af83\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"
Aug 05 17:21:09.142209 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:09.142173    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d624442-959d-4265-8f55-f0be6ef1af83-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-x9nz9\" (UID: \"6d624442-959d-4265-8f55-f0be6ef1af83\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"
Aug 05 17:21:09.142803 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:09.142779    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6d624442-959d-4265-8f55-f0be6ef1af83-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-x9nz9\" (UID: \"6d624442-959d-4265-8f55-f0be6ef1af83\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"
Aug 05 17:21:09.144908 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:09.144883    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d624442-959d-4265-8f55-f0be6ef1af83-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-x9nz9\" (UID: \"6d624442-959d-4265-8f55-f0be6ef1af83\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"
Aug 05 17:21:09.145014 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:09.144977    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d624442-959d-4265-8f55-f0be6ef1af83-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-x9nz9\" (UID: \"6d624442-959d-4265-8f55-f0be6ef1af83\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"
Aug 05 17:21:09.145055 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:09.145026    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6d624442-959d-4265-8f55-f0be6ef1af83-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-x9nz9\" (UID: \"6d624442-959d-4265-8f55-f0be6ef1af83\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"
Aug 05 17:21:09.145122 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:09.145103    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6d624442-959d-4265-8f55-f0be6ef1af83-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-x9nz9\" (UID: \"6d624442-959d-4265-8f55-f0be6ef1af83\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"
Aug 05 17:21:09.156027 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:09.156004    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d624442-959d-4265-8f55-f0be6ef1af83-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-x9nz9\" (UID: \"6d624442-959d-4265-8f55-f0be6ef1af83\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"
Aug 05 17:21:09.156125 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:09.156101    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmbv2\" (UniqueName: \"kubernetes.io/projected/6d624442-959d-4265-8f55-f0be6ef1af83-kube-api-access-rmbv2\") pod \"istiod-openshift-gateway-7b7995d5cd-x9nz9\" (UID: \"6d624442-959d-4265-8f55-f0be6ef1af83\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"
Aug 05 17:21:09.274301 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:09.274203    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"
Aug 05 17:21:09.423316 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:09.423287    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"]
Aug 05 17:21:09.425609 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:21:09.425583    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d624442_959d_4265_8f55_f0be6ef1af83.slice/crio-82829ba9c36d526979f36031ec1c83b81df663dd55f9af4ff635c021426b1d54 WatchSource:0}: Error finding container 82829ba9c36d526979f36031ec1c83b81df663dd55f9af4ff635c021426b1d54: Status 404 returned error can't find the container with id 82829ba9c36d526979f36031ec1c83b81df663dd55f9af4ff635c021426b1d54
Aug 05 17:21:09.791285 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:09.791248    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9" event={"ID":"6d624442-959d-4265-8f55-f0be6ef1af83","Type":"ContainerStarted","Data":"82829ba9c36d526979f36031ec1c83b81df663dd55f9af4ff635c021426b1d54"}
Aug 05 17:21:15.730702 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:15.730664    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 17:21:15.730936 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:15.730749    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 17:21:15.819854 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:15.819814    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9" event={"ID":"6d624442-959d-4265-8f55-f0be6ef1af83","Type":"ContainerStarted","Data":"7c82abc62ab60623336f7045d7ad294326aa47413e339c7c15b28944735aba31"}
Aug 05 17:21:15.820037 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:15.819898    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"
Aug 05 17:21:16.826999 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:16.826905    2561 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-x9nz9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Aug 05 17:21:16.826999 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:16.826978    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9" podUID="6d624442-959d-4265-8f55-f0be6ef1af83" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Aug 05 17:21:19.826007 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:19.825971    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"
Aug 05 17:21:19.851237 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:19.851172    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9" podStartSLOduration=5.548289967 podStartE2EDuration="11.851158002s" totalImagesPullingTime="6.302868035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:21:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:21:15.852470701 +0000 UTC m=+531.597105030" watchObservedRunningTime="2026-08-05 17:21:19.851158002 +0000 UTC m=+535.595792331"
Aug 05 17:21:20.367637 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.367608    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"]
Aug 05 17:21:20.370698 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.370677    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:20.374061 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.374034    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-nj65l\""
Aug 05 17:21:20.391948 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.391914    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"]
Aug 05 17:21:20.434804 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.434769    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecc8132-d5f3-4c33-aca3-e7e0a30e7531-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9\" (UID: \"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:20.434804 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.434805    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecc8132-d5f3-4c33-aca3-e7e0a30e7531-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9\" (UID: \"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:20.435089 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.434827    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecc8132-d5f3-4c33-aca3-e7e0a30e7531-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9\" (UID: \"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:20.435089 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.434842    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxmdj\" (UniqueName: \"kubernetes.io/projected/2ecc8132-d5f3-4c33-aca3-e7e0a30e7531-kube-api-access-xxmdj\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9\" (UID: \"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:20.435089 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.434924    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecc8132-d5f3-4c33-aca3-e7e0a30e7531-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9\" (UID: \"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:20.435089 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.434980    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecc8132-d5f3-4c33-aca3-e7e0a30e7531-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9\" (UID: \"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:20.435089 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.435043    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecc8132-d5f3-4c33-aca3-e7e0a30e7531-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9\" (UID: \"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:20.435295 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.435092    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecc8132-d5f3-4c33-aca3-e7e0a30e7531-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9\" (UID: \"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:20.435295 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.435129    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecc8132-d5f3-4c33-aca3-e7e0a30e7531-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9\" (UID: \"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:20.535994 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.535936    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecc8132-d5f3-4c33-aca3-e7e0a30e7531-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9\" (UID: \"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:20.535994 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.535994    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecc8132-d5f3-4c33-aca3-e7e0a30e7531-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9\" (UID: \"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:20.536247 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.536040    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecc8132-d5f3-4c33-aca3-e7e0a30e7531-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9\" (UID: \"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:20.536247 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.536078    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecc8132-d5f3-4c33-aca3-e7e0a30e7531-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9\" (UID: \"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:20.536247 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.536107    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecc8132-d5f3-4c33-aca3-e7e0a30e7531-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9\" (UID: \"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:20.536247 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.536144    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecc8132-d5f3-4c33-aca3-e7e0a30e7531-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9\" (UID: \"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:20.536247 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.536176    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecc8132-d5f3-4c33-aca3-e7e0a30e7531-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9\" (UID: \"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:20.536247 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.536200    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecc8132-d5f3-4c33-aca3-e7e0a30e7531-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9\" (UID: \"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:20.536529 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.536472    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecc8132-d5f3-4c33-aca3-e7e0a30e7531-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9\" (UID: \"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:20.536585 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.536536    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecc8132-d5f3-4c33-aca3-e7e0a30e7531-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9\" (UID: \"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:20.536585 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.536553    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecc8132-d5f3-4c33-aca3-e7e0a30e7531-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9\" (UID: \"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:20.536686 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.536607    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecc8132-d5f3-4c33-aca3-e7e0a30e7531-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9\" (UID: \"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:20.536766 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.536745    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxmdj\" (UniqueName: \"kubernetes.io/projected/2ecc8132-d5f3-4c33-aca3-e7e0a30e7531-kube-api-access-xxmdj\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9\" (UID: \"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:20.537086 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.537061    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecc8132-d5f3-4c33-aca3-e7e0a30e7531-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9\" (UID: \"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:20.538355 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.538329    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecc8132-d5f3-4c33-aca3-e7e0a30e7531-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9\" (UID: \"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:20.539056 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.539035    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecc8132-d5f3-4c33-aca3-e7e0a30e7531-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9\" (UID: \"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:20.555848 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.555820    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecc8132-d5f3-4c33-aca3-e7e0a30e7531-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9\" (UID: \"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:20.556522 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.556499    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxmdj\" (UniqueName: \"kubernetes.io/projected/2ecc8132-d5f3-4c33-aca3-e7e0a30e7531-kube-api-access-xxmdj\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9\" (UID: \"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:20.683738 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.683633    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:20.823478 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.823450    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"]
Aug 05 17:21:20.826016 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:21:20.825935    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ecc8132_d5f3_4c33_aca3_e7e0a30e7531.slice/crio-eba57fa6c84213e478128344b3b42b2d517ef94472d87de7bb669e275f0f4527 WatchSource:0}: Error finding container eba57fa6c84213e478128344b3b42b2d517ef94472d87de7bb669e275f0f4527: Status 404 returned error can't find the container with id eba57fa6c84213e478128344b3b42b2d517ef94472d87de7bb669e275f0f4527
Aug 05 17:21:20.845004 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:20.844975    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9" event={"ID":"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531","Type":"ContainerStarted","Data":"eba57fa6c84213e478128344b3b42b2d517ef94472d87de7bb669e275f0f4527"}
Aug 05 17:21:23.755116 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:23.755072    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 17:21:23.755387 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:23.755169    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 17:21:23.755387 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:23.755231    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 17:21:23.861017 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:23.860979    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9" event={"ID":"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531","Type":"ContainerStarted","Data":"c2ce578300db2f2ccd034518e7839c7e043864518a00022050d52306d9d6645a"}
Aug 05 17:21:23.905734 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:23.905671    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9" podStartSLOduration=0.97893898 podStartE2EDuration="3.905654519s" totalImagesPullingTime="2.926715539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:21:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:21:23.904665304 +0000 UTC m=+539.649299648" watchObservedRunningTime="2026-08-05 17:21:23.905654519 +0000 UTC m=+539.650288846"
Aug 05 17:21:24.684148 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:24.684072    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:24.688970 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:24.688932    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:24.865318 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:24.865292    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:24.866515 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:24.866495    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:21:31.023029 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.022992    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9zwkr"]
Aug 05 17:21:31.026770 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.026748    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9zwkr"
Aug 05 17:21:31.029496 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.029479    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Aug 05 17:21:31.029701 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.029688    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Aug 05 17:21:31.030597 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.030574    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-npp6p\""
Aug 05 17:21:31.037078 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.037054    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9zwkr"]
Aug 05 17:21:31.119370 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.119331    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rtqtn"]
Aug 05 17:21:31.123051 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.123028    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rtqtn"
Aug 05 17:21:31.134063 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.134033    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rtqtn"]
Aug 05 17:21:31.134319 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.134297    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7182211b-cfe8-4912-b5d1-339569fb3a33-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9zwkr\" (UID: \"7182211b-cfe8-4912-b5d1-339569fb3a33\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9zwkr"
Aug 05 17:21:31.134379 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.134328    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfjzs\" (UniqueName: \"kubernetes.io/projected/7182211b-cfe8-4912-b5d1-339569fb3a33-kube-api-access-nfjzs\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9zwkr\" (UID: \"7182211b-cfe8-4912-b5d1-339569fb3a33\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9zwkr"
Aug 05 17:21:31.134424 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.134397    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7182211b-cfe8-4912-b5d1-339569fb3a33-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9zwkr\" (UID: \"7182211b-cfe8-4912-b5d1-339569fb3a33\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9zwkr"
Aug 05 17:21:31.220099 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.220061    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2"]
Aug 05 17:21:31.223438 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.223416    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2"
Aug 05 17:21:31.235725 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.235694    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/feda1dee-d447-4e5c-8f53-7cd9c3cded8f-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rtqtn\" (UID: \"feda1dee-d447-4e5c-8f53-7cd9c3cded8f\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rtqtn"
Aug 05 17:21:31.235863 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.235746    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7182211b-cfe8-4912-b5d1-339569fb3a33-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9zwkr\" (UID: \"7182211b-cfe8-4912-b5d1-339569fb3a33\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9zwkr"
Aug 05 17:21:31.235863 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.235811    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfw2s\" (UniqueName: \"kubernetes.io/projected/feda1dee-d447-4e5c-8f53-7cd9c3cded8f-kube-api-access-nfw2s\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rtqtn\" (UID: \"feda1dee-d447-4e5c-8f53-7cd9c3cded8f\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rtqtn"
Aug 05 17:21:31.236012 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.235891    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/feda1dee-d447-4e5c-8f53-7cd9c3cded8f-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rtqtn\" (UID: \"feda1dee-d447-4e5c-8f53-7cd9c3cded8f\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rtqtn"
Aug 05 17:21:31.236012 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.235921    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7182211b-cfe8-4912-b5d1-339569fb3a33-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9zwkr\" (UID: \"7182211b-cfe8-4912-b5d1-339569fb3a33\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9zwkr"
Aug 05 17:21:31.236012 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.235979    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfjzs\" (UniqueName: \"kubernetes.io/projected/7182211b-cfe8-4912-b5d1-339569fb3a33-kube-api-access-nfjzs\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9zwkr\" (UID: \"7182211b-cfe8-4912-b5d1-339569fb3a33\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9zwkr"
Aug 05 17:21:31.236162 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.236100    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7182211b-cfe8-4912-b5d1-339569fb3a33-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9zwkr\" (UID: \"7182211b-cfe8-4912-b5d1-339569fb3a33\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9zwkr"
Aug 05 17:21:31.236255 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.236206    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7182211b-cfe8-4912-b5d1-339569fb3a33-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9zwkr\" (UID: \"7182211b-cfe8-4912-b5d1-339569fb3a33\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9zwkr"
Aug 05 17:21:31.239340 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.239317    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2"]
Aug 05 17:21:31.253126 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.253096    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfjzs\" (UniqueName: \"kubernetes.io/projected/7182211b-cfe8-4912-b5d1-339569fb3a33-kube-api-access-nfjzs\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9zwkr\" (UID: \"7182211b-cfe8-4912-b5d1-339569fb3a33\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9zwkr"
Aug 05 17:21:31.336305 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.336271    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9zwkr"
Aug 05 17:21:31.336486 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.336465    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfw2s\" (UniqueName: \"kubernetes.io/projected/feda1dee-d447-4e5c-8f53-7cd9c3cded8f-kube-api-access-nfw2s\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rtqtn\" (UID: \"feda1dee-d447-4e5c-8f53-7cd9c3cded8f\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rtqtn"
Aug 05 17:21:31.336551 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.336513    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/feda1dee-d447-4e5c-8f53-7cd9c3cded8f-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rtqtn\" (UID: \"feda1dee-d447-4e5c-8f53-7cd9c3cded8f\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rtqtn"
Aug 05 17:21:31.336551 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.336547    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66a878c9-7899-487b-9363-a6dcadca08d9-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2\" (UID: \"66a878c9-7899-487b-9363-a6dcadca08d9\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2"
Aug 05 17:21:31.336623 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.336563    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66a878c9-7899-487b-9363-a6dcadca08d9-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2\" (UID: \"66a878c9-7899-487b-9363-a6dcadca08d9\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2"
Aug 05 17:21:31.336623 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.336592    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b68r\" (UniqueName: \"kubernetes.io/projected/66a878c9-7899-487b-9363-a6dcadca08d9-kube-api-access-8b68r\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2\" (UID: \"66a878c9-7899-487b-9363-a6dcadca08d9\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2"
Aug 05 17:21:31.336623 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.336612    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/feda1dee-d447-4e5c-8f53-7cd9c3cded8f-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rtqtn\" (UID: \"feda1dee-d447-4e5c-8f53-7cd9c3cded8f\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rtqtn"
Aug 05 17:21:31.336942 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.336924    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/feda1dee-d447-4e5c-8f53-7cd9c3cded8f-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rtqtn\" (UID: \"feda1dee-d447-4e5c-8f53-7cd9c3cded8f\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rtqtn"
Aug 05 17:21:31.337033 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.336940    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/feda1dee-d447-4e5c-8f53-7cd9c3cded8f-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rtqtn\" (UID: \"feda1dee-d447-4e5c-8f53-7cd9c3cded8f\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rtqtn"
Aug 05 17:21:31.343578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.343553    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhbblx"]
Aug 05 17:21:31.347672 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.347649    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhbblx"
Aug 05 17:21:31.349178 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.349148    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfw2s\" (UniqueName: \"kubernetes.io/projected/feda1dee-d447-4e5c-8f53-7cd9c3cded8f-kube-api-access-nfw2s\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rtqtn\" (UID: \"feda1dee-d447-4e5c-8f53-7cd9c3cded8f\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rtqtn"
Aug 05 17:21:31.357431 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.357397    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhbblx"]
Aug 05 17:21:31.433359 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.433330    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rtqtn"
Aug 05 17:21:31.437668 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.437647    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66a878c9-7899-487b-9363-a6dcadca08d9-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2\" (UID: \"66a878c9-7899-487b-9363-a6dcadca08d9\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2"
Aug 05 17:21:31.437786 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.437731    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66a878c9-7899-487b-9363-a6dcadca08d9-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2\" (UID: \"66a878c9-7899-487b-9363-a6dcadca08d9\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2"
Aug 05 17:21:31.437909 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.437827    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0f773d9-d471-405d-bcf8-5834e5b114ca-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhbblx\" (UID: \"a0f773d9-d471-405d-bcf8-5834e5b114ca\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhbblx"
Aug 05 17:21:31.438089 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.438064    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66a878c9-7899-487b-9363-a6dcadca08d9-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2\" (UID: \"66a878c9-7899-487b-9363-a6dcadca08d9\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2"
Aug 05 17:21:31.438216 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.438178    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66a878c9-7899-487b-9363-a6dcadca08d9-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2\" (UID: \"66a878c9-7899-487b-9363-a6dcadca08d9\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2"
Aug 05 17:21:31.438298 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.438194    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b68r\" (UniqueName: \"kubernetes.io/projected/66a878c9-7899-487b-9363-a6dcadca08d9-kube-api-access-8b68r\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2\" (UID: \"66a878c9-7899-487b-9363-a6dcadca08d9\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2"
Aug 05 17:21:31.438298 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.438249    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5hmd\" (UniqueName: \"kubernetes.io/projected/a0f773d9-d471-405d-bcf8-5834e5b114ca-kube-api-access-x5hmd\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhbblx\" (UID: \"a0f773d9-d471-405d-bcf8-5834e5b114ca\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhbblx"
Aug 05 17:21:31.438393 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.438296    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0f773d9-d471-405d-bcf8-5834e5b114ca-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhbblx\" (UID: \"a0f773d9-d471-405d-bcf8-5834e5b114ca\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhbblx"
Aug 05 17:21:31.449219 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.449184    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b68r\" (UniqueName: \"kubernetes.io/projected/66a878c9-7899-487b-9363-a6dcadca08d9-kube-api-access-8b68r\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2\" (UID: \"66a878c9-7899-487b-9363-a6dcadca08d9\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2"
Aug 05 17:21:31.482100 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.482069    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9zwkr"]
Aug 05 17:21:31.483588 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:21:31.483562    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7182211b_cfe8_4912_b5d1_339569fb3a33.slice/crio-c541a727ec4296abf00b80e77f32b90c7f7a7c03ea3d1bb21d43211862deaee9 WatchSource:0}: Error finding container c541a727ec4296abf00b80e77f32b90c7f7a7c03ea3d1bb21d43211862deaee9: Status 404 returned error can't find the container with id c541a727ec4296abf00b80e77f32b90c7f7a7c03ea3d1bb21d43211862deaee9
Aug 05 17:21:31.532013 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.531987    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2"
Aug 05 17:21:31.539444 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.539416    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5hmd\" (UniqueName: \"kubernetes.io/projected/a0f773d9-d471-405d-bcf8-5834e5b114ca-kube-api-access-x5hmd\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhbblx\" (UID: \"a0f773d9-d471-405d-bcf8-5834e5b114ca\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhbblx"
Aug 05 17:21:31.539544 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.539465    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0f773d9-d471-405d-bcf8-5834e5b114ca-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhbblx\" (UID: \"a0f773d9-d471-405d-bcf8-5834e5b114ca\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhbblx"
Aug 05 17:21:31.539611 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.539556    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0f773d9-d471-405d-bcf8-5834e5b114ca-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhbblx\" (UID: \"a0f773d9-d471-405d-bcf8-5834e5b114ca\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhbblx"
Aug 05 17:21:31.539992 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.539969    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0f773d9-d471-405d-bcf8-5834e5b114ca-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhbblx\" (UID: \"a0f773d9-d471-405d-bcf8-5834e5b114ca\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhbblx"
Aug 05 17:21:31.540061 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.539990    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0f773d9-d471-405d-bcf8-5834e5b114ca-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhbblx\" (UID: \"a0f773d9-d471-405d-bcf8-5834e5b114ca\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhbblx"
Aug 05 17:21:31.549657 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.549609    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5hmd\" (UniqueName: \"kubernetes.io/projected/a0f773d9-d471-405d-bcf8-5834e5b114ca-kube-api-access-x5hmd\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhbblx\" (UID: \"a0f773d9-d471-405d-bcf8-5834e5b114ca\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhbblx"
Aug 05 17:21:31.590940 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.590869    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rtqtn"]
Aug 05 17:21:31.594405 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:21:31.594355    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeda1dee_d447_4e5c_8f53_7cd9c3cded8f.slice/crio-085f69152f130192a71ba2532c232580686b3636ed47cc7de1e26d0808527dec WatchSource:0}: Error finding container 085f69152f130192a71ba2532c232580686b3636ed47cc7de1e26d0808527dec: Status 404 returned error can't find the container with id 085f69152f130192a71ba2532c232580686b3636ed47cc7de1e26d0808527dec
Aug 05 17:21:31.675707 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.675676    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhbblx"
Aug 05 17:21:31.682000 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.681973    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2"]
Aug 05 17:21:31.694603 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:21:31.694574    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a878c9_7899_487b_9363_a6dcadca08d9.slice/crio-277f4960501e2c337bedbc912e8e1c0c78743cab1cfb9765145b21e5f6db9f8f WatchSource:0}: Error finding container 277f4960501e2c337bedbc912e8e1c0c78743cab1cfb9765145b21e5f6db9f8f: Status 404 returned error can't find the container with id 277f4960501e2c337bedbc912e8e1c0c78743cab1cfb9765145b21e5f6db9f8f
Aug 05 17:21:31.842490 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.842395    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhbblx"]
Aug 05 17:21:31.876846 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:21:31.876808    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f773d9_d471_405d_bcf8_5834e5b114ca.slice/crio-b3152b9b8550dba46771984f679cf7508339aa69b99e12059705b5b1a340ebe8 WatchSource:0}: Error finding container b3152b9b8550dba46771984f679cf7508339aa69b99e12059705b5b1a340ebe8: Status 404 returned error can't find the container with id b3152b9b8550dba46771984f679cf7508339aa69b99e12059705b5b1a340ebe8
Aug 05 17:21:31.897034 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.897006    2561 generic.go:356] "Generic (PLEG): container finished" podID="66a878c9-7899-487b-9363-a6dcadca08d9" containerID="4e7b8bc30245901f7364a8e4229a4a587c62610a5bd9659adee7368843c8db09" exitCode=0
Aug 05 17:21:31.897139 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.897091    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2" event={"ID":"66a878c9-7899-487b-9363-a6dcadca08d9","Type":"ContainerDied","Data":"4e7b8bc30245901f7364a8e4229a4a587c62610a5bd9659adee7368843c8db09"}
Aug 05 17:21:31.897195 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.897139    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2" event={"ID":"66a878c9-7899-487b-9363-a6dcadca08d9","Type":"ContainerStarted","Data":"277f4960501e2c337bedbc912e8e1c0c78743cab1cfb9765145b21e5f6db9f8f"}
Aug 05 17:21:31.898654 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.898635    2561 generic.go:356] "Generic (PLEG): container finished" podID="7182211b-cfe8-4912-b5d1-339569fb3a33" containerID="ae2b3b2ecbc11f9ec1c9d0879c832ec363f009051995554a619b479c737a8b63" exitCode=0
Aug 05 17:21:31.898748 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.898731    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9zwkr" event={"ID":"7182211b-cfe8-4912-b5d1-339569fb3a33","Type":"ContainerDied","Data":"ae2b3b2ecbc11f9ec1c9d0879c832ec363f009051995554a619b479c737a8b63"}
Aug 05 17:21:31.898803 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.898775    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9zwkr" event={"ID":"7182211b-cfe8-4912-b5d1-339569fb3a33","Type":"ContainerStarted","Data":"c541a727ec4296abf00b80e77f32b90c7f7a7c03ea3d1bb21d43211862deaee9"}
Aug 05 17:21:31.899853 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.899834    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhbblx" event={"ID":"a0f773d9-d471-405d-bcf8-5834e5b114ca","Type":"ContainerStarted","Data":"b3152b9b8550dba46771984f679cf7508339aa69b99e12059705b5b1a340ebe8"}
Aug 05 17:21:31.901347 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.901328    2561 generic.go:356] "Generic (PLEG): container finished" podID="feda1dee-d447-4e5c-8f53-7cd9c3cded8f" containerID="9e2ebf0e0062392d6622eed57c2b9e427040f170fae364d153c3770be069e9e6" exitCode=0
Aug 05 17:21:31.901459 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.901368    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rtqtn" event={"ID":"feda1dee-d447-4e5c-8f53-7cd9c3cded8f","Type":"ContainerDied","Data":"9e2ebf0e0062392d6622eed57c2b9e427040f170fae364d153c3770be069e9e6"}
Aug 05 17:21:31.901459 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:31.901399    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rtqtn" event={"ID":"feda1dee-d447-4e5c-8f53-7cd9c3cded8f","Type":"ContainerStarted","Data":"085f69152f130192a71ba2532c232580686b3636ed47cc7de1e26d0808527dec"}
Aug 05 17:21:32.908696 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:32.908667    2561 generic.go:356] "Generic (PLEG): container finished" podID="a0f773d9-d471-405d-bcf8-5834e5b114ca" containerID="373e0847f23661620c9ca1ad94da512cc867a06bb105a3e493fbffb5040fc2c9" exitCode=0
Aug 05 17:21:32.909127 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:32.908752    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhbblx" event={"ID":"a0f773d9-d471-405d-bcf8-5834e5b114ca","Type":"ContainerDied","Data":"373e0847f23661620c9ca1ad94da512cc867a06bb105a3e493fbffb5040fc2c9"}
Aug 05 17:21:32.911019 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:32.910995    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2" event={"ID":"66a878c9-7899-487b-9363-a6dcadca08d9","Type":"ContainerStarted","Data":"f1724881dade3ec96b52a2665d278ae5bf329abb36e59e8c329d3a36460c0cef"}
Aug 05 17:21:32.913270 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:32.913247    2561 generic.go:356] "Generic (PLEG): container finished" podID="7182211b-cfe8-4912-b5d1-339569fb3a33" containerID="d5c1428ca31cd9b52a2a6d6896f4301132c0f42fd513fd87b98b25acf082f6b5" exitCode=0
Aug 05 17:21:32.913363 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:32.913293    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9zwkr" event={"ID":"7182211b-cfe8-4912-b5d1-339569fb3a33","Type":"ContainerDied","Data":"d5c1428ca31cd9b52a2a6d6896f4301132c0f42fd513fd87b98b25acf082f6b5"}
Aug 05 17:21:33.919702 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:33.919671    2561 generic.go:356] "Generic (PLEG): container finished" podID="66a878c9-7899-487b-9363-a6dcadca08d9" containerID="f1724881dade3ec96b52a2665d278ae5bf329abb36e59e8c329d3a36460c0cef" exitCode=0
Aug 05 17:21:33.920186 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:33.919763    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2" event={"ID":"66a878c9-7899-487b-9363-a6dcadca08d9","Type":"ContainerDied","Data":"f1724881dade3ec96b52a2665d278ae5bf329abb36e59e8c329d3a36460c0cef"}
Aug 05 17:21:33.922096 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:33.922071    2561 generic.go:356] "Generic (PLEG): container finished" podID="7182211b-cfe8-4912-b5d1-339569fb3a33" containerID="2e3208e223d7eb7156c65feb10b43caa6d8348c5e6fd89928fd63697b7ec548d" exitCode=0
Aug 05 17:21:33.922205 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:33.922119    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9zwkr" event={"ID":"7182211b-cfe8-4912-b5d1-339569fb3a33","Type":"ContainerDied","Data":"2e3208e223d7eb7156c65feb10b43caa6d8348c5e6fd89928fd63697b7ec548d"}
Aug 05 17:21:34.928624 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:34.928587    2561 generic.go:356] "Generic (PLEG): container finished" podID="66a878c9-7899-487b-9363-a6dcadca08d9" containerID="0b6c904a0ee7cff6ef66c3c941de75e3acfc7760ca0fd3cc167a29285127bb40" exitCode=0
Aug 05 17:21:34.929048 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:34.928626    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2" event={"ID":"66a878c9-7899-487b-9363-a6dcadca08d9","Type":"ContainerDied","Data":"0b6c904a0ee7cff6ef66c3c941de75e3acfc7760ca0fd3cc167a29285127bb40"}
Aug 05 17:21:35.058492 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:35.058461    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9zwkr"
Aug 05 17:21:35.171616 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:35.171578    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7182211b-cfe8-4912-b5d1-339569fb3a33-util\") pod \"7182211b-cfe8-4912-b5d1-339569fb3a33\" (UID: \"7182211b-cfe8-4912-b5d1-339569fb3a33\") "
Aug 05 17:21:35.171796 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:35.171698    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7182211b-cfe8-4912-b5d1-339569fb3a33-bundle\") pod \"7182211b-cfe8-4912-b5d1-339569fb3a33\" (UID: \"7182211b-cfe8-4912-b5d1-339569fb3a33\") "
Aug 05 17:21:35.171796 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:35.171745    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfjzs\" (UniqueName: \"kubernetes.io/projected/7182211b-cfe8-4912-b5d1-339569fb3a33-kube-api-access-nfjzs\") pod \"7182211b-cfe8-4912-b5d1-339569fb3a33\" (UID: \"7182211b-cfe8-4912-b5d1-339569fb3a33\") "
Aug 05 17:21:35.172210 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:35.172187    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7182211b-cfe8-4912-b5d1-339569fb3a33-bundle" (OuterVolumeSpecName: "bundle") pod "7182211b-cfe8-4912-b5d1-339569fb3a33" (UID: "7182211b-cfe8-4912-b5d1-339569fb3a33"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:21:35.173776 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:35.173753    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7182211b-cfe8-4912-b5d1-339569fb3a33-kube-api-access-nfjzs" (OuterVolumeSpecName: "kube-api-access-nfjzs") pod "7182211b-cfe8-4912-b5d1-339569fb3a33" (UID: "7182211b-cfe8-4912-b5d1-339569fb3a33"). InnerVolumeSpecName "kube-api-access-nfjzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:21:35.177415 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:35.177393    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7182211b-cfe8-4912-b5d1-339569fb3a33-util" (OuterVolumeSpecName: "util") pod "7182211b-cfe8-4912-b5d1-339569fb3a33" (UID: "7182211b-cfe8-4912-b5d1-339569fb3a33"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:21:35.272711 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:35.272624    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7182211b-cfe8-4912-b5d1-339569fb3a33-util\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:21:35.272711 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:35.272656    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7182211b-cfe8-4912-b5d1-339569fb3a33-bundle\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:21:35.272711 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:35.272668    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfjzs\" (UniqueName: \"kubernetes.io/projected/7182211b-cfe8-4912-b5d1-339569fb3a33-kube-api-access-nfjzs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:21:35.934972 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:35.934931    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9zwkr"
Aug 05 17:21:35.934972 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:35.934937    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c9zwkr" event={"ID":"7182211b-cfe8-4912-b5d1-339569fb3a33","Type":"ContainerDied","Data":"c541a727ec4296abf00b80e77f32b90c7f7a7c03ea3d1bb21d43211862deaee9"}
Aug 05 17:21:35.935462 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:35.935000    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c541a727ec4296abf00b80e77f32b90c7f7a7c03ea3d1bb21d43211862deaee9"
Aug 05 17:21:35.936750 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:35.936722    2561 generic.go:356] "Generic (PLEG): container finished" podID="a0f773d9-d471-405d-bcf8-5834e5b114ca" containerID="e62c8e6af174c1009001e08c5d76d907e46b1c9f3ee35ce34c68a47ac8e51ebd" exitCode=0
Aug 05 17:21:35.936876 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:35.936788    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhbblx" event={"ID":"a0f773d9-d471-405d-bcf8-5834e5b114ca","Type":"ContainerDied","Data":"e62c8e6af174c1009001e08c5d76d907e46b1c9f3ee35ce34c68a47ac8e51ebd"}
Aug 05 17:21:35.938706 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:35.938684    2561 generic.go:356] "Generic (PLEG): container finished" podID="feda1dee-d447-4e5c-8f53-7cd9c3cded8f" containerID="24f54c9a4cc97861df45c11ec0294fe4b420426c2280035d64e2f1f884033c4f" exitCode=0
Aug 05 17:21:35.938793 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:35.938767    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rtqtn" event={"ID":"feda1dee-d447-4e5c-8f53-7cd9c3cded8f","Type":"ContainerDied","Data":"24f54c9a4cc97861df45c11ec0294fe4b420426c2280035d64e2f1f884033c4f"}
Aug 05 17:21:36.087119 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:36.087086    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2"
Aug 05 17:21:36.180621 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:36.180587    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66a878c9-7899-487b-9363-a6dcadca08d9-bundle\") pod \"66a878c9-7899-487b-9363-a6dcadca08d9\" (UID: \"66a878c9-7899-487b-9363-a6dcadca08d9\") "
Aug 05 17:21:36.180809 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:36.180640    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8b68r\" (UniqueName: \"kubernetes.io/projected/66a878c9-7899-487b-9363-a6dcadca08d9-kube-api-access-8b68r\") pod \"66a878c9-7899-487b-9363-a6dcadca08d9\" (UID: \"66a878c9-7899-487b-9363-a6dcadca08d9\") "
Aug 05 17:21:36.180809 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:36.180672    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66a878c9-7899-487b-9363-a6dcadca08d9-util\") pod \"66a878c9-7899-487b-9363-a6dcadca08d9\" (UID: \"66a878c9-7899-487b-9363-a6dcadca08d9\") "
Aug 05 17:21:36.181124 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:36.181099    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a878c9-7899-487b-9363-a6dcadca08d9-bundle" (OuterVolumeSpecName: "bundle") pod "66a878c9-7899-487b-9363-a6dcadca08d9" (UID: "66a878c9-7899-487b-9363-a6dcadca08d9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:21:36.182734 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:36.182709    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66a878c9-7899-487b-9363-a6dcadca08d9-kube-api-access-8b68r" (OuterVolumeSpecName: "kube-api-access-8b68r") pod "66a878c9-7899-487b-9363-a6dcadca08d9" (UID: "66a878c9-7899-487b-9363-a6dcadca08d9"). InnerVolumeSpecName "kube-api-access-8b68r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:21:36.185879 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:36.185856    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a878c9-7899-487b-9363-a6dcadca08d9-util" (OuterVolumeSpecName: "util") pod "66a878c9-7899-487b-9363-a6dcadca08d9" (UID: "66a878c9-7899-487b-9363-a6dcadca08d9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:21:36.281818 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:36.281782    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66a878c9-7899-487b-9363-a6dcadca08d9-util\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:21:36.281818 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:36.281812    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66a878c9-7899-487b-9363-a6dcadca08d9-bundle\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:21:36.281818 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:36.281822    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b68r\" (UniqueName: \"kubernetes.io/projected/66a878c9-7899-487b-9363-a6dcadca08d9-kube-api-access-8b68r\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:21:36.945826 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:36.945788    2561 generic.go:356] "Generic (PLEG): container finished" podID="a0f773d9-d471-405d-bcf8-5834e5b114ca" containerID="6573c139f558da8abb664a0e9a6e7f0f1ca8c31e87a513cf0ab6c518ac09142c" exitCode=0
Aug 05 17:21:36.946332 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:36.945865    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhbblx" event={"ID":"a0f773d9-d471-405d-bcf8-5834e5b114ca","Type":"ContainerDied","Data":"6573c139f558da8abb664a0e9a6e7f0f1ca8c31e87a513cf0ab6c518ac09142c"}
Aug 05 17:21:36.948016 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:36.947992    2561 generic.go:356] "Generic (PLEG): container finished" podID="feda1dee-d447-4e5c-8f53-7cd9c3cded8f" containerID="b7267fd0abd8649bfbb1844d55ddc2e0e640542b0e0cda3af520c0958547522d" exitCode=0
Aug 05 17:21:36.948129 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:36.948060    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rtqtn" event={"ID":"feda1dee-d447-4e5c-8f53-7cd9c3cded8f","Type":"ContainerDied","Data":"b7267fd0abd8649bfbb1844d55ddc2e0e640542b0e0cda3af520c0958547522d"}
Aug 05 17:21:36.949816 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:36.949801    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2"
Aug 05 17:21:36.949919 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:36.949836    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0g9vj2" event={"ID":"66a878c9-7899-487b-9363-a6dcadca08d9","Type":"ContainerDied","Data":"277f4960501e2c337bedbc912e8e1c0c78743cab1cfb9765145b21e5f6db9f8f"}
Aug 05 17:21:36.949919 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:36.949869    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="277f4960501e2c337bedbc912e8e1c0c78743cab1cfb9765145b21e5f6db9f8f"
Aug 05 17:21:38.108497 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:38.108474    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rtqtn"
Aug 05 17:21:38.110871 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:38.110853    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhbblx"
Aug 05 17:21:38.197733 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:38.197695    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfw2s\" (UniqueName: \"kubernetes.io/projected/feda1dee-d447-4e5c-8f53-7cd9c3cded8f-kube-api-access-nfw2s\") pod \"feda1dee-d447-4e5c-8f53-7cd9c3cded8f\" (UID: \"feda1dee-d447-4e5c-8f53-7cd9c3cded8f\") "
Aug 05 17:21:38.197934 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:38.197772    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/feda1dee-d447-4e5c-8f53-7cd9c3cded8f-util\") pod \"feda1dee-d447-4e5c-8f53-7cd9c3cded8f\" (UID: \"feda1dee-d447-4e5c-8f53-7cd9c3cded8f\") "
Aug 05 17:21:38.197934 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:38.197797    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5hmd\" (UniqueName: \"kubernetes.io/projected/a0f773d9-d471-405d-bcf8-5834e5b114ca-kube-api-access-x5hmd\") pod \"a0f773d9-d471-405d-bcf8-5834e5b114ca\" (UID: \"a0f773d9-d471-405d-bcf8-5834e5b114ca\") "
Aug 05 17:21:38.197934 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:38.197858    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0f773d9-d471-405d-bcf8-5834e5b114ca-util\") pod \"a0f773d9-d471-405d-bcf8-5834e5b114ca\" (UID: \"a0f773d9-d471-405d-bcf8-5834e5b114ca\") "
Aug 05 17:21:38.197934 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:38.197923    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/feda1dee-d447-4e5c-8f53-7cd9c3cded8f-bundle\") pod \"feda1dee-d447-4e5c-8f53-7cd9c3cded8f\" (UID: \"feda1dee-d447-4e5c-8f53-7cd9c3cded8f\") "
Aug 05 17:21:38.198183 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:38.198011    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0f773d9-d471-405d-bcf8-5834e5b114ca-bundle\") pod \"a0f773d9-d471-405d-bcf8-5834e5b114ca\" (UID: \"a0f773d9-d471-405d-bcf8-5834e5b114ca\") "
Aug 05 17:21:38.198775 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:38.198584    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f773d9-d471-405d-bcf8-5834e5b114ca-bundle" (OuterVolumeSpecName: "bundle") pod "a0f773d9-d471-405d-bcf8-5834e5b114ca" (UID: "a0f773d9-d471-405d-bcf8-5834e5b114ca"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:21:38.198775 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:38.198722    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feda1dee-d447-4e5c-8f53-7cd9c3cded8f-bundle" (OuterVolumeSpecName: "bundle") pod "feda1dee-d447-4e5c-8f53-7cd9c3cded8f" (UID: "feda1dee-d447-4e5c-8f53-7cd9c3cded8f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:21:38.200020 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:38.199996    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f773d9-d471-405d-bcf8-5834e5b114ca-kube-api-access-x5hmd" (OuterVolumeSpecName: "kube-api-access-x5hmd") pod "a0f773d9-d471-405d-bcf8-5834e5b114ca" (UID: "a0f773d9-d471-405d-bcf8-5834e5b114ca"). InnerVolumeSpecName "kube-api-access-x5hmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:21:38.200108 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:38.200024    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feda1dee-d447-4e5c-8f53-7cd9c3cded8f-kube-api-access-nfw2s" (OuterVolumeSpecName: "kube-api-access-nfw2s") pod "feda1dee-d447-4e5c-8f53-7cd9c3cded8f" (UID: "feda1dee-d447-4e5c-8f53-7cd9c3cded8f"). InnerVolumeSpecName "kube-api-access-nfw2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:21:38.203156 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:38.203137    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feda1dee-d447-4e5c-8f53-7cd9c3cded8f-util" (OuterVolumeSpecName: "util") pod "feda1dee-d447-4e5c-8f53-7cd9c3cded8f" (UID: "feda1dee-d447-4e5c-8f53-7cd9c3cded8f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:21:38.204892 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:38.204869    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0f773d9-d471-405d-bcf8-5834e5b114ca-util" (OuterVolumeSpecName: "util") pod "a0f773d9-d471-405d-bcf8-5834e5b114ca" (UID: "a0f773d9-d471-405d-bcf8-5834e5b114ca"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:21:38.299569 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:38.299532    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0f773d9-d471-405d-bcf8-5834e5b114ca-util\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:21:38.299569 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:38.299563    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/feda1dee-d447-4e5c-8f53-7cd9c3cded8f-bundle\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:21:38.299569 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:38.299575    2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0f773d9-d471-405d-bcf8-5834e5b114ca-bundle\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:21:38.299789 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:38.299584    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfw2s\" (UniqueName: \"kubernetes.io/projected/feda1dee-d447-4e5c-8f53-7cd9c3cded8f-kube-api-access-nfw2s\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:21:38.299789 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:38.299593    2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/feda1dee-d447-4e5c-8f53-7cd9c3cded8f-util\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:21:38.299789 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:38.299602    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5hmd\" (UniqueName: \"kubernetes.io/projected/a0f773d9-d471-405d-bcf8-5834e5b114ca-kube-api-access-x5hmd\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:21:38.961465 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:38.961433    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhbblx" event={"ID":"a0f773d9-d471-405d-bcf8-5834e5b114ca","Type":"ContainerDied","Data":"b3152b9b8550dba46771984f679cf7508339aa69b99e12059705b5b1a340ebe8"}
Aug 05 17:21:38.961465 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:38.961448    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbhbblx"
Aug 05 17:21:38.961734 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:38.961476    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3152b9b8550dba46771984f679cf7508339aa69b99e12059705b5b1a340ebe8"
Aug 05 17:21:38.963344 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:38.963325    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rtqtn"
Aug 05 17:21:38.963344 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:38.963337    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053rtqtn" event={"ID":"feda1dee-d447-4e5c-8f53-7cd9c3cded8f","Type":"ContainerDied","Data":"085f69152f130192a71ba2532c232580686b3636ed47cc7de1e26d0808527dec"}
Aug 05 17:21:38.963515 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:38.963375    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="085f69152f130192a71ba2532c232580686b3636ed47cc7de1e26d0808527dec"
Aug 05 17:21:44.280258 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.280224    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-b7lz5"]
Aug 05 17:21:44.280703 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.280533    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0f773d9-d471-405d-bcf8-5834e5b114ca" containerName="extract"
Aug 05 17:21:44.280703 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.280544    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0f773d9-d471-405d-bcf8-5834e5b114ca" containerName="extract"
Aug 05 17:21:44.280703 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.280553    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feda1dee-d447-4e5c-8f53-7cd9c3cded8f" containerName="pull"
Aug 05 17:21:44.280703 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.280558    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="feda1dee-d447-4e5c-8f53-7cd9c3cded8f" containerName="pull"
Aug 05 17:21:44.280703 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.280571    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7182211b-cfe8-4912-b5d1-339569fb3a33" containerName="pull"
Aug 05 17:21:44.280703 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.280579    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7182211b-cfe8-4912-b5d1-339569fb3a33" containerName="pull"
Aug 05 17:21:44.280703 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.280585    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0f773d9-d471-405d-bcf8-5834e5b114ca" containerName="pull"
Aug 05 17:21:44.280703 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.280591    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0f773d9-d471-405d-bcf8-5834e5b114ca" containerName="pull"
Aug 05 17:21:44.280703 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.280600    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feda1dee-d447-4e5c-8f53-7cd9c3cded8f" containerName="util"
Aug 05 17:21:44.280703 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.280606    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="feda1dee-d447-4e5c-8f53-7cd9c3cded8f" containerName="util"
Aug 05 17:21:44.280703 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.280614    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0f773d9-d471-405d-bcf8-5834e5b114ca" containerName="util"
Aug 05 17:21:44.280703 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.280619    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0f773d9-d471-405d-bcf8-5834e5b114ca" containerName="util"
Aug 05 17:21:44.280703 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.280625    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7182211b-cfe8-4912-b5d1-339569fb3a33" containerName="util"
Aug 05 17:21:44.280703 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.280630    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7182211b-cfe8-4912-b5d1-339569fb3a33" containerName="util"
Aug 05 17:21:44.280703 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.280634    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66a878c9-7899-487b-9363-a6dcadca08d9" containerName="extract"
Aug 05 17:21:44.280703 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.280639    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="66a878c9-7899-487b-9363-a6dcadca08d9" containerName="extract"
Aug 05 17:21:44.280703 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.280647    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66a878c9-7899-487b-9363-a6dcadca08d9" containerName="pull"
Aug 05 17:21:44.280703 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.280656    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="66a878c9-7899-487b-9363-a6dcadca08d9" containerName="pull"
Aug 05 17:21:44.280703 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.280662    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66a878c9-7899-487b-9363-a6dcadca08d9" containerName="util"
Aug 05 17:21:44.280703 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.280667    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="66a878c9-7899-487b-9363-a6dcadca08d9" containerName="util"
Aug 05 17:21:44.280703 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.280675    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7182211b-cfe8-4912-b5d1-339569fb3a33" containerName="extract"
Aug 05 17:21:44.280703 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.280680    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7182211b-cfe8-4912-b5d1-339569fb3a33" containerName="extract"
Aug 05 17:21:44.280703 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.280685    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feda1dee-d447-4e5c-8f53-7cd9c3cded8f" containerName="extract"
Aug 05 17:21:44.280703 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.280690    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="feda1dee-d447-4e5c-8f53-7cd9c3cded8f" containerName="extract"
Aug 05 17:21:44.281411 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.280739    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7182211b-cfe8-4912-b5d1-339569fb3a33" containerName="extract"
Aug 05 17:21:44.281411 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.280748    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="66a878c9-7899-487b-9363-a6dcadca08d9" containerName="extract"
Aug 05 17:21:44.281411 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.280754    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0f773d9-d471-405d-bcf8-5834e5b114ca" containerName="extract"
Aug 05 17:21:44.281411 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.280761    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="feda1dee-d447-4e5c-8f53-7cd9c3cded8f" containerName="extract"
Aug 05 17:21:44.287723 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.287702    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-b7lz5"
Aug 05 17:21:44.291830 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.291806    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Aug 05 17:21:44.291980 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.291890    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-96db6\""
Aug 05 17:21:44.292601 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.292586    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Aug 05 17:21:44.292601 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.292589    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Aug 05 17:21:44.297216 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.297192    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-b7lz5"]
Aug 05 17:21:44.450236 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.450205    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbwxq\" (UniqueName: \"kubernetes.io/projected/6efaf77d-b2f1-405c-93e3-27533d612e34-kube-api-access-pbwxq\") pod \"dns-operator-controller-manager-ccbdf686-b7lz5\" (UID: \"6efaf77d-b2f1-405c-93e3-27533d612e34\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-b7lz5"
Aug 05 17:21:44.550846 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.550752    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbwxq\" (UniqueName: \"kubernetes.io/projected/6efaf77d-b2f1-405c-93e3-27533d612e34-kube-api-access-pbwxq\") pod \"dns-operator-controller-manager-ccbdf686-b7lz5\" (UID: \"6efaf77d-b2f1-405c-93e3-27533d612e34\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-b7lz5"
Aug 05 17:21:44.582495 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.582461    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbwxq\" (UniqueName: \"kubernetes.io/projected/6efaf77d-b2f1-405c-93e3-27533d612e34-kube-api-access-pbwxq\") pod \"dns-operator-controller-manager-ccbdf686-b7lz5\" (UID: \"6efaf77d-b2f1-405c-93e3-27533d612e34\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-b7lz5"
Aug 05 17:21:44.598307 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.598280    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-b7lz5"
Aug 05 17:21:44.747452 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.747429    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-b7lz5"]
Aug 05 17:21:44.749435 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:21:44.749410    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6efaf77d_b2f1_405c_93e3_27533d612e34.slice/crio-707df614734d7a07a8804a73d9552b8b94ad50315e822d233a71b052e7a82334 WatchSource:0}: Error finding container 707df614734d7a07a8804a73d9552b8b94ad50315e822d233a71b052e7a82334: Status 404 returned error can't find the container with id 707df614734d7a07a8804a73d9552b8b94ad50315e822d233a71b052e7a82334
Aug 05 17:21:44.992755 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:44.992664    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-b7lz5" event={"ID":"6efaf77d-b2f1-405c-93e3-27533d612e34","Type":"ContainerStarted","Data":"707df614734d7a07a8804a73d9552b8b94ad50315e822d233a71b052e7a82334"}
Aug 05 17:21:48.012105 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:48.012004    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-b7lz5" event={"ID":"6efaf77d-b2f1-405c-93e3-27533d612e34","Type":"ContainerStarted","Data":"5ebd9f5973e2faec21154419ae07c9660af320ada8ffb6e3ff420e656c3984aa"}
Aug 05 17:21:48.012465 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:48.012170    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-b7lz5"
Aug 05 17:21:48.032804 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:48.032747    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-b7lz5" podStartSLOduration=1.17654581 podStartE2EDuration="4.032728548s" totalImagesPullingTime="2.856182738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:21:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:21:48.031985526 +0000 UTC m=+563.776619854" watchObservedRunningTime="2026-08-05 17:21:48.032728548 +0000 UTC m=+563.777362874"
Aug 05 17:21:48.653712 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:48.653677    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gb5pp"]
Aug 05 17:21:48.656940 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:48.656920    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gb5pp"
Aug 05 17:21:48.659553 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:48.659534    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jnhb9\""
Aug 05 17:21:48.670129 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:48.670104    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gb5pp"]
Aug 05 17:21:48.787979 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:48.787922    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dw6j\" (UniqueName: \"kubernetes.io/projected/b9847013-4287-414a-85a2-48ba79c78131-kube-api-access-7dw6j\") pod \"kuadrant-operator-controller-manager-66866f5fcc-gb5pp\" (UID: \"b9847013-4287-414a-85a2-48ba79c78131\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gb5pp"
Aug 05 17:21:48.788144 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:48.788069    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9847013-4287-414a-85a2-48ba79c78131-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-gb5pp\" (UID: \"b9847013-4287-414a-85a2-48ba79c78131\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gb5pp"
Aug 05 17:21:48.889054 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:48.889020    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dw6j\" (UniqueName: \"kubernetes.io/projected/b9847013-4287-414a-85a2-48ba79c78131-kube-api-access-7dw6j\") pod \"kuadrant-operator-controller-manager-66866f5fcc-gb5pp\" (UID: \"b9847013-4287-414a-85a2-48ba79c78131\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gb5pp"
Aug 05 17:21:48.889247 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:48.889105    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9847013-4287-414a-85a2-48ba79c78131-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-gb5pp\" (UID: \"b9847013-4287-414a-85a2-48ba79c78131\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gb5pp"
Aug 05 17:21:48.889437 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:48.889416    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9847013-4287-414a-85a2-48ba79c78131-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-gb5pp\" (UID: \"b9847013-4287-414a-85a2-48ba79c78131\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gb5pp"
Aug 05 17:21:48.913394 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:48.913322    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dw6j\" (UniqueName: \"kubernetes.io/projected/b9847013-4287-414a-85a2-48ba79c78131-kube-api-access-7dw6j\") pod \"kuadrant-operator-controller-manager-66866f5fcc-gb5pp\" (UID: \"b9847013-4287-414a-85a2-48ba79c78131\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gb5pp"
Aug 05 17:21:48.966833 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:48.966793    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gb5pp"
Aug 05 17:21:49.108924 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:49.108899    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gb5pp"]
Aug 05 17:21:49.110965 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:21:49.110929    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9847013_4287_414a_85a2_48ba79c78131.slice/crio-13787db33e811a53cdb3c6e706196cd75e519754c9e7f2792b7235968a570230 WatchSource:0}: Error finding container 13787db33e811a53cdb3c6e706196cd75e519754c9e7f2792b7235968a570230: Status 404 returned error can't find the container with id 13787db33e811a53cdb3c6e706196cd75e519754c9e7f2792b7235968a570230
Aug 05 17:21:50.027136 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:50.027095    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gb5pp" event={"ID":"b9847013-4287-414a-85a2-48ba79c78131","Type":"ContainerStarted","Data":"13787db33e811a53cdb3c6e706196cd75e519754c9e7f2792b7235968a570230"}
Aug 05 17:21:51.225823 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:51.225792    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-nkm22"]
Aug 05 17:21:51.229269 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:51.229247    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-56fb6f6856-nkm22"
Aug 05 17:21:51.231988 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:51.231947    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-xllg9\""
Aug 05 17:21:51.260383 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:51.260351    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-nkm22"]
Aug 05 17:21:51.409677 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:51.409645    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2j5z\" (UniqueName: \"kubernetes.io/projected/9ece522f-5b6b-47a1-b254-a92f050d50af-kube-api-access-d2j5z\") pod \"authorino-operator-56fb6f6856-nkm22\" (UID: \"9ece522f-5b6b-47a1-b254-a92f050d50af\") " pod="kuadrant-system/authorino-operator-56fb6f6856-nkm22"
Aug 05 17:21:51.511067 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:51.510984    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2j5z\" (UniqueName: \"kubernetes.io/projected/9ece522f-5b6b-47a1-b254-a92f050d50af-kube-api-access-d2j5z\") pod \"authorino-operator-56fb6f6856-nkm22\" (UID: \"9ece522f-5b6b-47a1-b254-a92f050d50af\") " pod="kuadrant-system/authorino-operator-56fb6f6856-nkm22"
Aug 05 17:21:51.524704 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:51.524676    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2j5z\" (UniqueName: \"kubernetes.io/projected/9ece522f-5b6b-47a1-b254-a92f050d50af-kube-api-access-d2j5z\") pod \"authorino-operator-56fb6f6856-nkm22\" (UID: \"9ece522f-5b6b-47a1-b254-a92f050d50af\") " pod="kuadrant-system/authorino-operator-56fb6f6856-nkm22"
Aug 05 17:21:51.542515 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:51.542481    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-56fb6f6856-nkm22"
Aug 05 17:21:51.700233 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:51.700204    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-nkm22"]
Aug 05 17:21:52.831304 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:21:52.831269    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ece522f_5b6b_47a1_b254_a92f050d50af.slice/crio-ad9bee58d70b5fd1c895dd41a6cd8fb5e93c03ef6c799fcaafdd61cb45a9bcfa WatchSource:0}: Error finding container ad9bee58d70b5fd1c895dd41a6cd8fb5e93c03ef6c799fcaafdd61cb45a9bcfa: Status 404 returned error can't find the container with id ad9bee58d70b5fd1c895dd41a6cd8fb5e93c03ef6c799fcaafdd61cb45a9bcfa
Aug 05 17:21:53.044570 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:53.044536    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gb5pp" event={"ID":"b9847013-4287-414a-85a2-48ba79c78131","Type":"ContainerStarted","Data":"5c99aeb4f587cb7ff8b75f718390e7a1e16c22ac254a545415789ca2573b2b5a"}
Aug 05 17:21:53.044787 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:53.044608    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gb5pp"
Aug 05 17:21:53.045942 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:53.045901    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-56fb6f6856-nkm22" event={"ID":"9ece522f-5b6b-47a1-b254-a92f050d50af","Type":"ContainerStarted","Data":"ad9bee58d70b5fd1c895dd41a6cd8fb5e93c03ef6c799fcaafdd61cb45a9bcfa"}
Aug 05 17:21:53.074632 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:53.074573    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gb5pp" podStartSLOduration=1.2884789460000001 podStartE2EDuration="5.074553432s" totalImagesPullingTime="3.786074486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:21:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:21:53.073378533 +0000 UTC m=+568.818012860" watchObservedRunningTime="2026-08-05 17:21:53.074553432 +0000 UTC m=+568.819187763"
Aug 05 17:21:55.060033 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:55.059990    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-56fb6f6856-nkm22" event={"ID":"9ece522f-5b6b-47a1-b254-a92f050d50af","Type":"ContainerStarted","Data":"df5f0cc18a6ee15c310e8e2bdee2cfb9fd1083a807f7fe94aea919b7e02cc130"}
Aug 05 17:21:55.060451 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:55.060134    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-56fb6f6856-nkm22"
Aug 05 17:21:59.021463 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:59.021435    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-b7lz5"
Aug 05 17:21:59.047427 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:21:59.047374    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-56fb6f6856-nkm22" podStartSLOduration=6.693306389 podStartE2EDuration="8.047357997s" totalImagesPullingTime="1.354051608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:21:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:21:55.098324049 +0000 UTC m=+570.842958377" watchObservedRunningTime="2026-08-05 17:21:59.047357997 +0000 UTC m=+574.791992323"
Aug 05 17:22:04.056521 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:22:04.056493    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gb5pp"
Aug 05 17:22:06.068531 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:22:06.068501    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-56fb6f6856-nkm22"
Aug 05 17:22:35.788729 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:22:35.788689    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tfk9h"]
Aug 05 17:22:35.792584 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:22:35.792560    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-tfk9h"
Aug 05 17:22:35.795128 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:22:35.795106    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Aug 05 17:22:35.795245 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:22:35.795136    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-96dlj\""
Aug 05 17:22:35.801263 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:22:35.801240    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tfk9h"]
Aug 05 17:22:35.835637 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:22:35.835603    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tfk9h"]
Aug 05 17:22:35.880475 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:22:35.880442    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b4vd\" (UniqueName: \"kubernetes.io/projected/a84e6912-5ce1-4155-a1f1-88c0b20dd5ff-kube-api-access-8b4vd\") pod \"limitador-limitador-85d9db4d5d-tfk9h\" (UID: \"a84e6912-5ce1-4155-a1f1-88c0b20dd5ff\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-tfk9h"
Aug 05 17:22:35.880671 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:22:35.880571    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a84e6912-5ce1-4155-a1f1-88c0b20dd5ff-config-file\") pod \"limitador-limitador-85d9db4d5d-tfk9h\" (UID: \"a84e6912-5ce1-4155-a1f1-88c0b20dd5ff\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-tfk9h"
Aug 05 17:22:35.981352 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:22:35.981314    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a84e6912-5ce1-4155-a1f1-88c0b20dd5ff-config-file\") pod \"limitador-limitador-85d9db4d5d-tfk9h\" (UID: \"a84e6912-5ce1-4155-a1f1-88c0b20dd5ff\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-tfk9h"
Aug 05 17:22:35.981542 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:22:35.981371    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b4vd\" (UniqueName: \"kubernetes.io/projected/a84e6912-5ce1-4155-a1f1-88c0b20dd5ff-kube-api-access-8b4vd\") pod \"limitador-limitador-85d9db4d5d-tfk9h\" (UID: \"a84e6912-5ce1-4155-a1f1-88c0b20dd5ff\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-tfk9h"
Aug 05 17:22:35.982065 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:22:35.982044    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a84e6912-5ce1-4155-a1f1-88c0b20dd5ff-config-file\") pod \"limitador-limitador-85d9db4d5d-tfk9h\" (UID: \"a84e6912-5ce1-4155-a1f1-88c0b20dd5ff\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-tfk9h"
Aug 05 17:22:35.990662 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:22:35.990634    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b4vd\" (UniqueName: \"kubernetes.io/projected/a84e6912-5ce1-4155-a1f1-88c0b20dd5ff-kube-api-access-8b4vd\") pod \"limitador-limitador-85d9db4d5d-tfk9h\" (UID: \"a84e6912-5ce1-4155-a1f1-88c0b20dd5ff\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-tfk9h"
Aug 05 17:22:36.104124 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:22:36.104043    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-tfk9h"
Aug 05 17:22:36.251834 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:22:36.251702    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-tfk9h"]
Aug 05 17:22:36.254940 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:22:36.254913    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda84e6912_5ce1_4155_a1f1_88c0b20dd5ff.slice/crio-899907c941b6f43f8c9a2f269e7d343a4388db0c3703526b212ce48ca3210d5f WatchSource:0}: Error finding container 899907c941b6f43f8c9a2f269e7d343a4388db0c3703526b212ce48ca3210d5f: Status 404 returned error can't find the container with id 899907c941b6f43f8c9a2f269e7d343a4388db0c3703526b212ce48ca3210d5f
Aug 05 17:22:36.265723 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:22:36.265696    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-tfk9h" event={"ID":"a84e6912-5ce1-4155-a1f1-88c0b20dd5ff","Type":"ContainerStarted","Data":"899907c941b6f43f8c9a2f269e7d343a4388db0c3703526b212ce48ca3210d5f"}
Aug 05 17:22:38.278270 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:22:38.278235    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-tfk9h" event={"ID":"a84e6912-5ce1-4155-a1f1-88c0b20dd5ff","Type":"ContainerStarted","Data":"54fc4cd26d80bd695cf5e9540a7747780815bd6d08de382a03f64efe5fdfc9c8"}
Aug 05 17:22:38.278681 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:22:38.278347    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-tfk9h"
Aug 05 17:22:38.298722 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:22:38.298674    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-tfk9h" podStartSLOduration=2.208908213 podStartE2EDuration="3.298656709s" totalImagesPullingTime="1.089748496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:22:38.297434277 +0000 UTC m=+614.042068618" watchObservedRunningTime="2026-08-05 17:22:38.298656709 +0000 UTC m=+614.043291094"
Aug 05 17:22:49.285301 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:22:49.285226    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-tfk9h"
Aug 05 17:23:28.271291 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.271258    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"]
Aug 05 17:23:28.273835 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.271443    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9" podUID="6d624442-959d-4265-8f55-f0be6ef1af83" containerName="discovery" containerID="cri-o://7c82abc62ab60623336f7045d7ad294326aa47413e339c7c15b28944735aba31" gracePeriod=30
Aug 05 17:23:28.521728 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.521678    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"
Aug 05 17:23:28.537295 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.537266    2561 generic.go:356] "Generic (PLEG): container finished" podID="6d624442-959d-4265-8f55-f0be6ef1af83" containerID="7c82abc62ab60623336f7045d7ad294326aa47413e339c7c15b28944735aba31" exitCode=0
Aug 05 17:23:28.537416 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.537305    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"
Aug 05 17:23:28.537416 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.537306    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9" event={"ID":"6d624442-959d-4265-8f55-f0be6ef1af83","Type":"ContainerDied","Data":"7c82abc62ab60623336f7045d7ad294326aa47413e339c7c15b28944735aba31"}
Aug 05 17:23:28.537557 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.537485    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9" event={"ID":"6d624442-959d-4265-8f55-f0be6ef1af83","Type":"ContainerDied","Data":"82829ba9c36d526979f36031ec1c83b81df663dd55f9af4ff635c021426b1d54"}
Aug 05 17:23:28.537557 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.537525    2561 scope.go:117] "RemoveContainer" containerID="7c82abc62ab60623336f7045d7ad294326aa47413e339c7c15b28944735aba31"
Aug 05 17:23:28.546557 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.546037    2561 scope.go:117] "RemoveContainer" containerID="7c82abc62ab60623336f7045d7ad294326aa47413e339c7c15b28944735aba31"
Aug 05 17:23:28.546557 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:23:28.546370    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c82abc62ab60623336f7045d7ad294326aa47413e339c7c15b28944735aba31\": container with ID starting with 7c82abc62ab60623336f7045d7ad294326aa47413e339c7c15b28944735aba31 not found: ID does not exist" containerID="7c82abc62ab60623336f7045d7ad294326aa47413e339c7c15b28944735aba31"
Aug 05 17:23:28.546557 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.546406    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c82abc62ab60623336f7045d7ad294326aa47413e339c7c15b28944735aba31"} err="failed to get container status \"7c82abc62ab60623336f7045d7ad294326aa47413e339c7c15b28944735aba31\": rpc error: code = NotFound desc = could not find container \"7c82abc62ab60623336f7045d7ad294326aa47413e339c7c15b28944735aba31\": container with ID starting with 7c82abc62ab60623336f7045d7ad294326aa47413e339c7c15b28944735aba31 not found: ID does not exist"
Aug 05 17:23:28.623354 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.623305    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d624442-959d-4265-8f55-f0be6ef1af83-istio-token\") pod \"6d624442-959d-4265-8f55-f0be6ef1af83\" (UID: \"6d624442-959d-4265-8f55-f0be6ef1af83\") "
Aug 05 17:23:28.623354 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.623360    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmbv2\" (UniqueName: \"kubernetes.io/projected/6d624442-959d-4265-8f55-f0be6ef1af83-kube-api-access-rmbv2\") pod \"6d624442-959d-4265-8f55-f0be6ef1af83\" (UID: \"6d624442-959d-4265-8f55-f0be6ef1af83\") "
Aug 05 17:23:28.623608 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.623385    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d624442-959d-4265-8f55-f0be6ef1af83-istio-kubeconfig\") pod \"6d624442-959d-4265-8f55-f0be6ef1af83\" (UID: \"6d624442-959d-4265-8f55-f0be6ef1af83\") "
Aug 05 17:23:28.623608 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.623412    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6d624442-959d-4265-8f55-f0be6ef1af83-cacerts\") pod \"6d624442-959d-4265-8f55-f0be6ef1af83\" (UID: \"6d624442-959d-4265-8f55-f0be6ef1af83\") "
Aug 05 17:23:28.623608 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.623453    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6d624442-959d-4265-8f55-f0be6ef1af83-istio-csr-ca-configmap\") pod \"6d624442-959d-4265-8f55-f0be6ef1af83\" (UID: \"6d624442-959d-4265-8f55-f0be6ef1af83\") "
Aug 05 17:23:28.623608 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.623578    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6d624442-959d-4265-8f55-f0be6ef1af83-istio-csr-dns-cert\") pod \"6d624442-959d-4265-8f55-f0be6ef1af83\" (UID: \"6d624442-959d-4265-8f55-f0be6ef1af83\") "
Aug 05 17:23:28.623797 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.623633    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d624442-959d-4265-8f55-f0be6ef1af83-local-certs\") pod \"6d624442-959d-4265-8f55-f0be6ef1af83\" (UID: \"6d624442-959d-4265-8f55-f0be6ef1af83\") "
Aug 05 17:23:28.623877 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.623851    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d624442-959d-4265-8f55-f0be6ef1af83-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "6d624442-959d-4265-8f55-f0be6ef1af83" (UID: "6d624442-959d-4265-8f55-f0be6ef1af83"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:23:28.625801 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.625767    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d624442-959d-4265-8f55-f0be6ef1af83-kube-api-access-rmbv2" (OuterVolumeSpecName: "kube-api-access-rmbv2") pod "6d624442-959d-4265-8f55-f0be6ef1af83" (UID: "6d624442-959d-4265-8f55-f0be6ef1af83"). InnerVolumeSpecName "kube-api-access-rmbv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:23:28.625978 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.625899    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d624442-959d-4265-8f55-f0be6ef1af83-local-certs" (OuterVolumeSpecName: "local-certs") pod "6d624442-959d-4265-8f55-f0be6ef1af83" (UID: "6d624442-959d-4265-8f55-f0be6ef1af83"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:23:28.626049 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.626013    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d624442-959d-4265-8f55-f0be6ef1af83-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "6d624442-959d-4265-8f55-f0be6ef1af83" (UID: "6d624442-959d-4265-8f55-f0be6ef1af83"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:23:28.626164 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.626081    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d624442-959d-4265-8f55-f0be6ef1af83-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "6d624442-959d-4265-8f55-f0be6ef1af83" (UID: "6d624442-959d-4265-8f55-f0be6ef1af83"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:23:28.626164 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.626091    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d624442-959d-4265-8f55-f0be6ef1af83-cacerts" (OuterVolumeSpecName: "cacerts") pod "6d624442-959d-4265-8f55-f0be6ef1af83" (UID: "6d624442-959d-4265-8f55-f0be6ef1af83"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:23:28.626616 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.626590    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d624442-959d-4265-8f55-f0be6ef1af83-istio-token" (OuterVolumeSpecName: "istio-token") pod "6d624442-959d-4265-8f55-f0be6ef1af83" (UID: "6d624442-959d-4265-8f55-f0be6ef1af83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:23:28.725097 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.725065    2561 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6d624442-959d-4265-8f55-f0be6ef1af83-istio-csr-dns-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:23:28.725097 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.725092    2561 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d624442-959d-4265-8f55-f0be6ef1af83-local-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:23:28.725097 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.725101    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d624442-959d-4265-8f55-f0be6ef1af83-istio-token\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:23:28.725097 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.725110    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmbv2\" (UniqueName: \"kubernetes.io/projected/6d624442-959d-4265-8f55-f0be6ef1af83-kube-api-access-rmbv2\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:23:28.725449 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.725118    2561 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d624442-959d-4265-8f55-f0be6ef1af83-istio-kubeconfig\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:23:28.725449 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.725127    2561 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6d624442-959d-4265-8f55-f0be6ef1af83-cacerts\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:23:28.725449 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.725135    2561 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6d624442-959d-4265-8f55-f0be6ef1af83-istio-csr-ca-configmap\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:23:28.856501 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.856469    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"]
Aug 05 17:23:28.861771 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:28.861743    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x9nz9"]
Aug 05 17:23:30.729539 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:30.729505    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d624442-959d-4265-8f55-f0be6ef1af83" path="/var/lib/kubelet/pods/6d624442-959d-4265-8f55-f0be6ef1af83/volumes"
Aug 05 17:23:32.445865 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.445825    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6f49dcd9ff-th5l2"]
Aug 05 17:23:32.446279 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.446216    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d624442-959d-4265-8f55-f0be6ef1af83" containerName="discovery"
Aug 05 17:23:32.446279 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.446233    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d624442-959d-4265-8f55-f0be6ef1af83" containerName="discovery"
Aug 05 17:23:32.446374 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.446288    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d624442-959d-4265-8f55-f0be6ef1af83" containerName="discovery"
Aug 05 17:23:32.450857 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.450841    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f49dcd9ff-th5l2"
Aug 05 17:23:32.453238 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.453214    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Aug 05 17:23:32.453375 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.453250    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Aug 05 17:23:32.454442 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.454418    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-d65n9\""
Aug 05 17:23:32.454534 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.454453    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Aug 05 17:23:32.460998 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.460972    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6f49dcd9ff-th5l2"]
Aug 05 17:23:32.473873 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.473846    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-54d86dcccd-69dsf"]
Aug 05 17:23:32.477622 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.477602    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-54d86dcccd-69dsf"
Aug 05 17:23:32.480214 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.480197    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Aug 05 17:23:32.480334 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.480232    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-9dk7c\""
Aug 05 17:23:32.488074 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.488051    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-54d86dcccd-69dsf"]
Aug 05 17:23:32.495383 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.495362    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-qnq2d"]
Aug 05 17:23:32.501590 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.501570    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-qnq2d"
Aug 05 17:23:32.504070 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.504051    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-fj5b7\""
Aug 05 17:23:32.504175 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.504103    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Aug 05 17:23:32.509597 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.509575    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-qnq2d"]
Aug 05 17:23:32.555905 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.555875    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57894026-1268-467e-b264-8eb72ae47296-cert\") pod \"llmisvc-controller-manager-54d86dcccd-69dsf\" (UID: \"57894026-1268-467e-b264-8eb72ae47296\") " pod="kserve/llmisvc-controller-manager-54d86dcccd-69dsf"
Aug 05 17:23:32.555905 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.555909    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73bcbe9e-2965-4ad2-bb10-f49183c8f4c1-cert\") pod \"kserve-controller-manager-6f49dcd9ff-th5l2\" (UID: \"73bcbe9e-2965-4ad2-bb10-f49183c8f4c1\") " pod="kserve/kserve-controller-manager-6f49dcd9ff-th5l2"
Aug 05 17:23:32.556195 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.555943    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9l5j\" (UniqueName: \"kubernetes.io/projected/57894026-1268-467e-b264-8eb72ae47296-kube-api-access-z9l5j\") pod \"llmisvc-controller-manager-54d86dcccd-69dsf\" (UID: \"57894026-1268-467e-b264-8eb72ae47296\") " pod="kserve/llmisvc-controller-manager-54d86dcccd-69dsf"
Aug 05 17:23:32.556195 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.556040    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrbcd\" (UniqueName: \"kubernetes.io/projected/73bcbe9e-2965-4ad2-bb10-f49183c8f4c1-kube-api-access-lrbcd\") pod \"kserve-controller-manager-6f49dcd9ff-th5l2\" (UID: \"73bcbe9e-2965-4ad2-bb10-f49183c8f4c1\") " pod="kserve/kserve-controller-manager-6f49dcd9ff-th5l2"
Aug 05 17:23:32.556195 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.556131    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c9534844-5bc2-4f1e-950a-a833156cd036-data\") pod \"seaweedfs-64568bcd49-qnq2d\" (UID: \"c9534844-5bc2-4f1e-950a-a833156cd036\") " pod="kserve/seaweedfs-64568bcd49-qnq2d"
Aug 05 17:23:32.556195 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.556164    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hctf\" (UniqueName: \"kubernetes.io/projected/c9534844-5bc2-4f1e-950a-a833156cd036-kube-api-access-7hctf\") pod \"seaweedfs-64568bcd49-qnq2d\" (UID: \"c9534844-5bc2-4f1e-950a-a833156cd036\") " pod="kserve/seaweedfs-64568bcd49-qnq2d"
Aug 05 17:23:32.656922 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.656889    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hctf\" (UniqueName: \"kubernetes.io/projected/c9534844-5bc2-4f1e-950a-a833156cd036-kube-api-access-7hctf\") pod \"seaweedfs-64568bcd49-qnq2d\" (UID: \"c9534844-5bc2-4f1e-950a-a833156cd036\") " pod="kserve/seaweedfs-64568bcd49-qnq2d"
Aug 05 17:23:32.657147 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.656939    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57894026-1268-467e-b264-8eb72ae47296-cert\") pod \"llmisvc-controller-manager-54d86dcccd-69dsf\" (UID: \"57894026-1268-467e-b264-8eb72ae47296\") " pod="kserve/llmisvc-controller-manager-54d86dcccd-69dsf"
Aug 05 17:23:32.657147 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.657053    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73bcbe9e-2965-4ad2-bb10-f49183c8f4c1-cert\") pod \"kserve-controller-manager-6f49dcd9ff-th5l2\" (UID: \"73bcbe9e-2965-4ad2-bb10-f49183c8f4c1\") " pod="kserve/kserve-controller-manager-6f49dcd9ff-th5l2"
Aug 05 17:23:32.657147 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.657086    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9l5j\" (UniqueName: \"kubernetes.io/projected/57894026-1268-467e-b264-8eb72ae47296-kube-api-access-z9l5j\") pod \"llmisvc-controller-manager-54d86dcccd-69dsf\" (UID: \"57894026-1268-467e-b264-8eb72ae47296\") " pod="kserve/llmisvc-controller-manager-54d86dcccd-69dsf"
Aug 05 17:23:32.657147 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.657116    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrbcd\" (UniqueName: \"kubernetes.io/projected/73bcbe9e-2965-4ad2-bb10-f49183c8f4c1-kube-api-access-lrbcd\") pod \"kserve-controller-manager-6f49dcd9ff-th5l2\" (UID: \"73bcbe9e-2965-4ad2-bb10-f49183c8f4c1\") " pod="kserve/kserve-controller-manager-6f49dcd9ff-th5l2"
Aug 05 17:23:32.657345 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:23:32.657178    2561 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Aug 05 17:23:32.657345 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.657204    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c9534844-5bc2-4f1e-950a-a833156cd036-data\") pod \"seaweedfs-64568bcd49-qnq2d\" (UID: \"c9534844-5bc2-4f1e-950a-a833156cd036\") " pod="kserve/seaweedfs-64568bcd49-qnq2d"
Aug 05 17:23:32.657345 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:23:32.657255    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73bcbe9e-2965-4ad2-bb10-f49183c8f4c1-cert podName:73bcbe9e-2965-4ad2-bb10-f49183c8f4c1 nodeName:}" failed. No retries permitted until 2026-08-05 17:23:33.157233054 +0000 UTC m=+668.901867385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/73bcbe9e-2965-4ad2-bb10-f49183c8f4c1-cert") pod "kserve-controller-manager-6f49dcd9ff-th5l2" (UID: "73bcbe9e-2965-4ad2-bb10-f49183c8f4c1") : secret "kserve-webhook-server-cert" not found
Aug 05 17:23:32.657595 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.657575    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c9534844-5bc2-4f1e-950a-a833156cd036-data\") pod \"seaweedfs-64568bcd49-qnq2d\" (UID: \"c9534844-5bc2-4f1e-950a-a833156cd036\") " pod="kserve/seaweedfs-64568bcd49-qnq2d"
Aug 05 17:23:32.659722 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.659701    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57894026-1268-467e-b264-8eb72ae47296-cert\") pod \"llmisvc-controller-manager-54d86dcccd-69dsf\" (UID: \"57894026-1268-467e-b264-8eb72ae47296\") " pod="kserve/llmisvc-controller-manager-54d86dcccd-69dsf"
Aug 05 17:23:32.666023 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.665991    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hctf\" (UniqueName: \"kubernetes.io/projected/c9534844-5bc2-4f1e-950a-a833156cd036-kube-api-access-7hctf\") pod \"seaweedfs-64568bcd49-qnq2d\" (UID: \"c9534844-5bc2-4f1e-950a-a833156cd036\") " pod="kserve/seaweedfs-64568bcd49-qnq2d"
Aug 05 17:23:32.666151 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.665991    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrbcd\" (UniqueName: \"kubernetes.io/projected/73bcbe9e-2965-4ad2-bb10-f49183c8f4c1-kube-api-access-lrbcd\") pod \"kserve-controller-manager-6f49dcd9ff-th5l2\" (UID: \"73bcbe9e-2965-4ad2-bb10-f49183c8f4c1\") " pod="kserve/kserve-controller-manager-6f49dcd9ff-th5l2"
Aug 05 17:23:32.666218 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.666148    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9l5j\" (UniqueName: \"kubernetes.io/projected/57894026-1268-467e-b264-8eb72ae47296-kube-api-access-z9l5j\") pod \"llmisvc-controller-manager-54d86dcccd-69dsf\" (UID: \"57894026-1268-467e-b264-8eb72ae47296\") " pod="kserve/llmisvc-controller-manager-54d86dcccd-69dsf"
Aug 05 17:23:32.789285 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.789248    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-54d86dcccd-69dsf"
Aug 05 17:23:32.813621 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.813591    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-qnq2d"
Aug 05 17:23:32.936445 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.936400    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-54d86dcccd-69dsf"]
Aug 05 17:23:32.939288 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:23:32.939262    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod57894026_1268_467e_b264_8eb72ae47296.slice/crio-57e2d0f592355eab14c700b0807162fe80762b5fe9afeb4e9e42cc90721ca712 WatchSource:0}: Error finding container 57e2d0f592355eab14c700b0807162fe80762b5fe9afeb4e9e42cc90721ca712: Status 404 returned error can't find the container with id 57e2d0f592355eab14c700b0807162fe80762b5fe9afeb4e9e42cc90721ca712
Aug 05 17:23:32.965177 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:32.965149    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-qnq2d"]
Aug 05 17:23:32.966590 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:23:32.966566    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9534844_5bc2_4f1e_950a_a833156cd036.slice/crio-4806fa4bb73a4354ec27bf40f45a75305507ee6c0b42234b4de6e195d415081c WatchSource:0}: Error finding container 4806fa4bb73a4354ec27bf40f45a75305507ee6c0b42234b4de6e195d415081c: Status 404 returned error can't find the container with id 4806fa4bb73a4354ec27bf40f45a75305507ee6c0b42234b4de6e195d415081c
Aug 05 17:23:33.162220 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:33.162137    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73bcbe9e-2965-4ad2-bb10-f49183c8f4c1-cert\") pod \"kserve-controller-manager-6f49dcd9ff-th5l2\" (UID: \"73bcbe9e-2965-4ad2-bb10-f49183c8f4c1\") " pod="kserve/kserve-controller-manager-6f49dcd9ff-th5l2"
Aug 05 17:23:33.164483 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:33.164466    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73bcbe9e-2965-4ad2-bb10-f49183c8f4c1-cert\") pod \"kserve-controller-manager-6f49dcd9ff-th5l2\" (UID: \"73bcbe9e-2965-4ad2-bb10-f49183c8f4c1\") " pod="kserve/kserve-controller-manager-6f49dcd9ff-th5l2"
Aug 05 17:23:33.361818 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:33.361785    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f49dcd9ff-th5l2"
Aug 05 17:23:33.515810 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:33.515766    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6f49dcd9ff-th5l2"]
Aug 05 17:23:33.518281 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:23:33.518244    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73bcbe9e_2965_4ad2_bb10_f49183c8f4c1.slice/crio-16d731f997ff1cea9b0705303b79e7682b7b071e3edef79ee0085af5696e111b WatchSource:0}: Error finding container 16d731f997ff1cea9b0705303b79e7682b7b071e3edef79ee0085af5696e111b: Status 404 returned error can't find the container with id 16d731f997ff1cea9b0705303b79e7682b7b071e3edef79ee0085af5696e111b
Aug 05 17:23:33.568467 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:33.568428    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-54d86dcccd-69dsf" event={"ID":"57894026-1268-467e-b264-8eb72ae47296","Type":"ContainerStarted","Data":"57e2d0f592355eab14c700b0807162fe80762b5fe9afeb4e9e42cc90721ca712"}
Aug 05 17:23:33.570010 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:33.569976    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f49dcd9ff-th5l2" event={"ID":"73bcbe9e-2965-4ad2-bb10-f49183c8f4c1","Type":"ContainerStarted","Data":"16d731f997ff1cea9b0705303b79e7682b7b071e3edef79ee0085af5696e111b"}
Aug 05 17:23:33.571288 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:33.571254    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-qnq2d" event={"ID":"c9534844-5bc2-4f1e-950a-a833156cd036","Type":"ContainerStarted","Data":"4806fa4bb73a4354ec27bf40f45a75305507ee6c0b42234b4de6e195d415081c"}
Aug 05 17:23:38.605037 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:38.604995    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-qnq2d" event={"ID":"c9534844-5bc2-4f1e-950a-a833156cd036","Type":"ContainerStarted","Data":"268999680c6c27d84a4d7ef2e573ce01d9e39f84b1d9da6c51cc94aa6610e955"}
Aug 05 17:23:38.605474 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:38.605098    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-qnq2d"
Aug 05 17:23:38.606833 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:38.606804    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-54d86dcccd-69dsf" event={"ID":"57894026-1268-467e-b264-8eb72ae47296","Type":"ContainerStarted","Data":"7d5abb24d36bf2fdae0188284a3c240c3b5cba58e4ea9b17ecf2ba08a6406d61"}
Aug 05 17:23:38.607383 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:38.607363    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-54d86dcccd-69dsf"
Aug 05 17:23:38.608877 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:38.608849    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f49dcd9ff-th5l2" event={"ID":"73bcbe9e-2965-4ad2-bb10-f49183c8f4c1","Type":"ContainerStarted","Data":"f50172ff2e89591e26f9cf4d16a0c48f85cec4ea0727ffb91ae5ebf31cf002b0"}
Aug 05 17:23:38.609016 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:38.609002    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6f49dcd9ff-th5l2"
Aug 05 17:23:38.622731 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:38.622685    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-qnq2d" podStartSLOduration=1.542330691 podStartE2EDuration="6.622671847s" totalImagesPullingTime="5.080341156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:23:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:23:38.620572411 +0000 UTC m=+674.365206738" watchObservedRunningTime="2026-08-05 17:23:38.622671847 +0000 UTC m=+674.367306175"
Aug 05 17:23:38.637506 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:38.637463    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-54d86dcccd-69dsf" podStartSLOduration=1.5866459819999998 podStartE2EDuration="6.637447448s" totalImagesPullingTime="5.050801466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:23:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:23:38.636163612 +0000 UTC m=+674.380797939" watchObservedRunningTime="2026-08-05 17:23:38.637447448 +0000 UTC m=+674.382081754"
Aug 05 17:23:38.653735 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:38.653684    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6f49dcd9ff-th5l2" podStartSLOduration=2.183187527 podStartE2EDuration="6.653667828s" totalImagesPullingTime="4.470480301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:23:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:23:38.652459616 +0000 UTC m=+674.397093944" watchObservedRunningTime="2026-08-05 17:23:38.653667828 +0000 UTC m=+674.398302155"
Aug 05 17:23:44.617380 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:23:44.617347    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-qnq2d"
Aug 05 17:24:09.620438 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:09.620411    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6f49dcd9ff-th5l2"
Aug 05 17:24:10.622224 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:10.622191    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-54d86dcccd-69dsf"
Aug 05 17:24:19.491043 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:19.490940    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6f49dcd9ff-th5l2"]
Aug 05 17:24:19.491505 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:19.491171    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-6f49dcd9ff-th5l2" podUID="73bcbe9e-2965-4ad2-bb10-f49183c8f4c1" containerName="manager" containerID="cri-o://f50172ff2e89591e26f9cf4d16a0c48f85cec4ea0727ffb91ae5ebf31cf002b0" gracePeriod=10
Aug 05 17:24:19.515515 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:19.515489    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6f49dcd9ff-cgtr7"]
Aug 05 17:24:19.517700 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:19.517684    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f49dcd9ff-cgtr7"
Aug 05 17:24:19.529660 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:19.529635    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6f49dcd9ff-cgtr7"]
Aug 05 17:24:19.570023 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:19.569972    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjvsv\" (UniqueName: \"kubernetes.io/projected/8a3c78ca-b8ed-4973-b09e-55d7cac74626-kube-api-access-fjvsv\") pod \"kserve-controller-manager-6f49dcd9ff-cgtr7\" (UID: \"8a3c78ca-b8ed-4973-b09e-55d7cac74626\") " pod="kserve/kserve-controller-manager-6f49dcd9ff-cgtr7"
Aug 05 17:24:19.570176 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:19.570049    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a3c78ca-b8ed-4973-b09e-55d7cac74626-cert\") pod \"kserve-controller-manager-6f49dcd9ff-cgtr7\" (UID: \"8a3c78ca-b8ed-4973-b09e-55d7cac74626\") " pod="kserve/kserve-controller-manager-6f49dcd9ff-cgtr7"
Aug 05 17:24:19.614854 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:19.614809    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve/kserve-controller-manager-6f49dcd9ff-th5l2" podUID="73bcbe9e-2965-4ad2-bb10-f49183c8f4c1" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.38:8081/readyz\": dial tcp 10.133.0.38:8081: connect: connection refused"
Aug 05 17:24:19.671486 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:19.671456    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjvsv\" (UniqueName: \"kubernetes.io/projected/8a3c78ca-b8ed-4973-b09e-55d7cac74626-kube-api-access-fjvsv\") pod \"kserve-controller-manager-6f49dcd9ff-cgtr7\" (UID: \"8a3c78ca-b8ed-4973-b09e-55d7cac74626\") " pod="kserve/kserve-controller-manager-6f49dcd9ff-cgtr7"
Aug 05 17:24:19.671636 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:19.671500    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a3c78ca-b8ed-4973-b09e-55d7cac74626-cert\") pod \"kserve-controller-manager-6f49dcd9ff-cgtr7\" (UID: \"8a3c78ca-b8ed-4973-b09e-55d7cac74626\") " pod="kserve/kserve-controller-manager-6f49dcd9ff-cgtr7"
Aug 05 17:24:19.674179 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:19.674151    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a3c78ca-b8ed-4973-b09e-55d7cac74626-cert\") pod \"kserve-controller-manager-6f49dcd9ff-cgtr7\" (UID: \"8a3c78ca-b8ed-4973-b09e-55d7cac74626\") " pod="kserve/kserve-controller-manager-6f49dcd9ff-cgtr7"
Aug 05 17:24:19.680211 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:19.680185    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjvsv\" (UniqueName: \"kubernetes.io/projected/8a3c78ca-b8ed-4973-b09e-55d7cac74626-kube-api-access-fjvsv\") pod \"kserve-controller-manager-6f49dcd9ff-cgtr7\" (UID: \"8a3c78ca-b8ed-4973-b09e-55d7cac74626\") " pod="kserve/kserve-controller-manager-6f49dcd9ff-cgtr7"
Aug 05 17:24:19.736236 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:19.736214    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f49dcd9ff-th5l2"
Aug 05 17:24:19.820860 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:19.820823    2561 generic.go:356] "Generic (PLEG): container finished" podID="73bcbe9e-2965-4ad2-bb10-f49183c8f4c1" containerID="f50172ff2e89591e26f9cf4d16a0c48f85cec4ea0727ffb91ae5ebf31cf002b0" exitCode=0
Aug 05 17:24:19.820860 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:19.820865    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f49dcd9ff-th5l2"
Aug 05 17:24:19.821103 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:19.820900    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f49dcd9ff-th5l2" event={"ID":"73bcbe9e-2965-4ad2-bb10-f49183c8f4c1","Type":"ContainerDied","Data":"f50172ff2e89591e26f9cf4d16a0c48f85cec4ea0727ffb91ae5ebf31cf002b0"}
Aug 05 17:24:19.821103 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:19.820973    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f49dcd9ff-th5l2" event={"ID":"73bcbe9e-2965-4ad2-bb10-f49183c8f4c1","Type":"ContainerDied","Data":"16d731f997ff1cea9b0705303b79e7682b7b071e3edef79ee0085af5696e111b"}
Aug 05 17:24:19.821103 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:19.820997    2561 scope.go:117] "RemoveContainer" containerID="f50172ff2e89591e26f9cf4d16a0c48f85cec4ea0727ffb91ae5ebf31cf002b0"
Aug 05 17:24:19.829313 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:19.829298    2561 scope.go:117] "RemoveContainer" containerID="f50172ff2e89591e26f9cf4d16a0c48f85cec4ea0727ffb91ae5ebf31cf002b0"
Aug 05 17:24:19.829585 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:24:19.829567    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f50172ff2e89591e26f9cf4d16a0c48f85cec4ea0727ffb91ae5ebf31cf002b0\": container with ID starting with f50172ff2e89591e26f9cf4d16a0c48f85cec4ea0727ffb91ae5ebf31cf002b0 not found: ID does not exist" containerID="f50172ff2e89591e26f9cf4d16a0c48f85cec4ea0727ffb91ae5ebf31cf002b0"
Aug 05 17:24:19.829638 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:19.829594    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f50172ff2e89591e26f9cf4d16a0c48f85cec4ea0727ffb91ae5ebf31cf002b0"} err="failed to get container status \"f50172ff2e89591e26f9cf4d16a0c48f85cec4ea0727ffb91ae5ebf31cf002b0\": rpc error: code = NotFound desc = could not find container \"f50172ff2e89591e26f9cf4d16a0c48f85cec4ea0727ffb91ae5ebf31cf002b0\": container with ID starting with f50172ff2e89591e26f9cf4d16a0c48f85cec4ea0727ffb91ae5ebf31cf002b0 not found: ID does not exist"
Aug 05 17:24:19.863922 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:19.863880    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f49dcd9ff-cgtr7"
Aug 05 17:24:19.873438 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:19.873412    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrbcd\" (UniqueName: \"kubernetes.io/projected/73bcbe9e-2965-4ad2-bb10-f49183c8f4c1-kube-api-access-lrbcd\") pod \"73bcbe9e-2965-4ad2-bb10-f49183c8f4c1\" (UID: \"73bcbe9e-2965-4ad2-bb10-f49183c8f4c1\") "
Aug 05 17:24:19.873555 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:19.873474    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73bcbe9e-2965-4ad2-bb10-f49183c8f4c1-cert\") pod \"73bcbe9e-2965-4ad2-bb10-f49183c8f4c1\" (UID: \"73bcbe9e-2965-4ad2-bb10-f49183c8f4c1\") "
Aug 05 17:24:19.875614 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:19.875581    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73bcbe9e-2965-4ad2-bb10-f49183c8f4c1-cert" (OuterVolumeSpecName: "cert") pod "73bcbe9e-2965-4ad2-bb10-f49183c8f4c1" (UID: "73bcbe9e-2965-4ad2-bb10-f49183c8f4c1"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:24:19.875716 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:19.875660    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73bcbe9e-2965-4ad2-bb10-f49183c8f4c1-kube-api-access-lrbcd" (OuterVolumeSpecName: "kube-api-access-lrbcd") pod "73bcbe9e-2965-4ad2-bb10-f49183c8f4c1" (UID: "73bcbe9e-2965-4ad2-bb10-f49183c8f4c1"). InnerVolumeSpecName "kube-api-access-lrbcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:24:19.974707 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:19.974667    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrbcd\" (UniqueName: \"kubernetes.io/projected/73bcbe9e-2965-4ad2-bb10-f49183c8f4c1-kube-api-access-lrbcd\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:24:19.974707 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:19.974706    2561 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73bcbe9e-2965-4ad2-bb10-f49183c8f4c1-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:24:19.995087 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:19.995057    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6f49dcd9ff-cgtr7"]
Aug 05 17:24:19.996997 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:24:19.996970    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3c78ca_b8ed_4973_b09e_55d7cac74626.slice/crio-0dfaf65ad111322d59128ce31fb4e4ad4337c430a863b18870eaa9586312336b WatchSource:0}: Error finding container 0dfaf65ad111322d59128ce31fb4e4ad4337c430a863b18870eaa9586312336b: Status 404 returned error can't find the container with id 0dfaf65ad111322d59128ce31fb4e4ad4337c430a863b18870eaa9586312336b
Aug 05 17:24:19.998296 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:19.998280    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 05 17:24:20.144935 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:20.144907    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6f49dcd9ff-th5l2"]
Aug 05 17:24:20.152416 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:20.152390    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-6f49dcd9ff-th5l2"]
Aug 05 17:24:20.729754 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:20.729678    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73bcbe9e-2965-4ad2-bb10-f49183c8f4c1" path="/var/lib/kubelet/pods/73bcbe9e-2965-4ad2-bb10-f49183c8f4c1/volumes"
Aug 05 17:24:20.833267 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:20.833232    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f49dcd9ff-cgtr7" event={"ID":"8a3c78ca-b8ed-4973-b09e-55d7cac74626","Type":"ContainerStarted","Data":"7d71e2265aabbb758750b3e297d2c305d9b33a04e326daac7d9449005f3e3101"}
Aug 05 17:24:20.833443 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:20.833304    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f49dcd9ff-cgtr7" event={"ID":"8a3c78ca-b8ed-4973-b09e-55d7cac74626","Type":"ContainerStarted","Data":"0dfaf65ad111322d59128ce31fb4e4ad4337c430a863b18870eaa9586312336b"}
Aug 05 17:24:20.833443 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:20.833345    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6f49dcd9ff-cgtr7"
Aug 05 17:24:20.853151 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:20.853104    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6f49dcd9ff-cgtr7" podStartSLOduration=1.498753466 podStartE2EDuration="1.853088547s" totalImagesPullingTime="354.335081ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:24:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:24:20.850987744 +0000 UTC m=+716.595622073" watchObservedRunningTime="2026-08-05 17:24:20.853088547 +0000 UTC m=+716.597722873"
Aug 05 17:24:32.638977 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:32.638915    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57d57488d-wcvsn"]
Aug 05 17:24:32.639518 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:32.639450    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73bcbe9e-2965-4ad2-bb10-f49183c8f4c1" containerName="manager"
Aug 05 17:24:32.639518 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:32.639469    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="73bcbe9e-2965-4ad2-bb10-f49183c8f4c1" containerName="manager"
Aug 05 17:24:32.639637 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:32.639554    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="73bcbe9e-2965-4ad2-bb10-f49183c8f4c1" containerName="manager"
Aug 05 17:24:32.642542 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:32.642519    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d57488d-wcvsn"
Aug 05 17:24:32.659491 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:32.659448    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57d57488d-wcvsn"]
Aug 05 17:24:32.784569 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:32.784530    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eae812f8-7c5a-4017-9dbf-f04df9cf563d-console-serving-cert\") pod \"console-57d57488d-wcvsn\" (UID: \"eae812f8-7c5a-4017-9dbf-f04df9cf563d\") " pod="openshift-console/console-57d57488d-wcvsn"
Aug 05 17:24:32.784569 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:32.784571    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eae812f8-7c5a-4017-9dbf-f04df9cf563d-console-oauth-config\") pod \"console-57d57488d-wcvsn\" (UID: \"eae812f8-7c5a-4017-9dbf-f04df9cf563d\") " pod="openshift-console/console-57d57488d-wcvsn"
Aug 05 17:24:32.784791 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:32.784592    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eae812f8-7c5a-4017-9dbf-f04df9cf563d-oauth-serving-cert\") pod \"console-57d57488d-wcvsn\" (UID: \"eae812f8-7c5a-4017-9dbf-f04df9cf563d\") " pod="openshift-console/console-57d57488d-wcvsn"
Aug 05 17:24:32.784791 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:32.784643    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzw84\" (UniqueName: \"kubernetes.io/projected/eae812f8-7c5a-4017-9dbf-f04df9cf563d-kube-api-access-zzw84\") pod \"console-57d57488d-wcvsn\" (UID: \"eae812f8-7c5a-4017-9dbf-f04df9cf563d\") " pod="openshift-console/console-57d57488d-wcvsn"
Aug 05 17:24:32.784791 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:32.784696    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eae812f8-7c5a-4017-9dbf-f04df9cf563d-console-config\") pod \"console-57d57488d-wcvsn\" (UID: \"eae812f8-7c5a-4017-9dbf-f04df9cf563d\") " pod="openshift-console/console-57d57488d-wcvsn"
Aug 05 17:24:32.784791 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:32.784713    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eae812f8-7c5a-4017-9dbf-f04df9cf563d-trusted-ca-bundle\") pod \"console-57d57488d-wcvsn\" (UID: \"eae812f8-7c5a-4017-9dbf-f04df9cf563d\") " pod="openshift-console/console-57d57488d-wcvsn"
Aug 05 17:24:32.784791 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:32.784755    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eae812f8-7c5a-4017-9dbf-f04df9cf563d-service-ca\") pod \"console-57d57488d-wcvsn\" (UID: \"eae812f8-7c5a-4017-9dbf-f04df9cf563d\") " pod="openshift-console/console-57d57488d-wcvsn"
Aug 05 17:24:32.886191 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:32.886155    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eae812f8-7c5a-4017-9dbf-f04df9cf563d-oauth-serving-cert\") pod \"console-57d57488d-wcvsn\" (UID: \"eae812f8-7c5a-4017-9dbf-f04df9cf563d\") " pod="openshift-console/console-57d57488d-wcvsn"
Aug 05 17:24:32.886191 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:32.886190    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzw84\" (UniqueName: \"kubernetes.io/projected/eae812f8-7c5a-4017-9dbf-f04df9cf563d-kube-api-access-zzw84\") pod \"console-57d57488d-wcvsn\" (UID: \"eae812f8-7c5a-4017-9dbf-f04df9cf563d\") " pod="openshift-console/console-57d57488d-wcvsn"
Aug 05 17:24:32.886410 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:32.886216    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eae812f8-7c5a-4017-9dbf-f04df9cf563d-console-config\") pod \"console-57d57488d-wcvsn\" (UID: \"eae812f8-7c5a-4017-9dbf-f04df9cf563d\") " pod="openshift-console/console-57d57488d-wcvsn"
Aug 05 17:24:32.886410 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:32.886240    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eae812f8-7c5a-4017-9dbf-f04df9cf563d-trusted-ca-bundle\") pod \"console-57d57488d-wcvsn\" (UID: \"eae812f8-7c5a-4017-9dbf-f04df9cf563d\") " pod="openshift-console/console-57d57488d-wcvsn"
Aug 05 17:24:32.886410 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:32.886377    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eae812f8-7c5a-4017-9dbf-f04df9cf563d-service-ca\") pod \"console-57d57488d-wcvsn\" (UID: \"eae812f8-7c5a-4017-9dbf-f04df9cf563d\") " pod="openshift-console/console-57d57488d-wcvsn"
Aug 05 17:24:32.886555 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:32.886541    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eae812f8-7c5a-4017-9dbf-f04df9cf563d-console-serving-cert\") pod \"console-57d57488d-wcvsn\" (UID: \"eae812f8-7c5a-4017-9dbf-f04df9cf563d\") " pod="openshift-console/console-57d57488d-wcvsn"
Aug 05 17:24:32.886763 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:32.886736    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eae812f8-7c5a-4017-9dbf-f04df9cf563d-console-oauth-config\") pod \"console-57d57488d-wcvsn\" (UID: \"eae812f8-7c5a-4017-9dbf-f04df9cf563d\") " pod="openshift-console/console-57d57488d-wcvsn"
Aug 05 17:24:32.887010 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:32.886991    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eae812f8-7c5a-4017-9dbf-f04df9cf563d-oauth-serving-cert\") pod \"console-57d57488d-wcvsn\" (UID: \"eae812f8-7c5a-4017-9dbf-f04df9cf563d\") " pod="openshift-console/console-57d57488d-wcvsn"
Aug 05 17:24:32.887114 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:32.887046    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eae812f8-7c5a-4017-9dbf-f04df9cf563d-console-config\") pod \"console-57d57488d-wcvsn\" (UID: \"eae812f8-7c5a-4017-9dbf-f04df9cf563d\") " pod="openshift-console/console-57d57488d-wcvsn"
Aug 05 17:24:32.887114 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:32.887107    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eae812f8-7c5a-4017-9dbf-f04df9cf563d-service-ca\") pod \"console-57d57488d-wcvsn\" (UID: \"eae812f8-7c5a-4017-9dbf-f04df9cf563d\") " pod="openshift-console/console-57d57488d-wcvsn"
Aug 05 17:24:32.887218 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:32.887167    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eae812f8-7c5a-4017-9dbf-f04df9cf563d-trusted-ca-bundle\") pod \"console-57d57488d-wcvsn\" (UID: \"eae812f8-7c5a-4017-9dbf-f04df9cf563d\") " pod="openshift-console/console-57d57488d-wcvsn"
Aug 05 17:24:32.889026 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:32.888976    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eae812f8-7c5a-4017-9dbf-f04df9cf563d-console-oauth-config\") pod \"console-57d57488d-wcvsn\" (UID: \"eae812f8-7c5a-4017-9dbf-f04df9cf563d\") " pod="openshift-console/console-57d57488d-wcvsn"
Aug 05 17:24:32.889090 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:32.889046    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eae812f8-7c5a-4017-9dbf-f04df9cf563d-console-serving-cert\") pod \"console-57d57488d-wcvsn\" (UID: \"eae812f8-7c5a-4017-9dbf-f04df9cf563d\") " pod="openshift-console/console-57d57488d-wcvsn"
Aug 05 17:24:32.895145 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:32.895123    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzw84\" (UniqueName: \"kubernetes.io/projected/eae812f8-7c5a-4017-9dbf-f04df9cf563d-kube-api-access-zzw84\") pod \"console-57d57488d-wcvsn\" (UID: \"eae812f8-7c5a-4017-9dbf-f04df9cf563d\") " pod="openshift-console/console-57d57488d-wcvsn"
Aug 05 17:24:32.956779 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:32.956738    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d57488d-wcvsn"
Aug 05 17:24:33.090858 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:33.090833    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57d57488d-wcvsn"]
Aug 05 17:24:33.092734 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:24:33.092707    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae812f8_7c5a_4017_9dbf_f04df9cf563d.slice/crio-ffebc81e7e49189e9a5e972809a700283db2e05d2bce082514e04554c4fdd55b WatchSource:0}: Error finding container ffebc81e7e49189e9a5e972809a700283db2e05d2bce082514e04554c4fdd55b: Status 404 returned error can't find the container with id ffebc81e7e49189e9a5e972809a700283db2e05d2bce082514e04554c4fdd55b
Aug 05 17:24:33.900604 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:33.900571    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d57488d-wcvsn" event={"ID":"eae812f8-7c5a-4017-9dbf-f04df9cf563d","Type":"ContainerStarted","Data":"f406953e2643b163dc200031c5055f3847a88bd03e022dc2298f0c906cabb222"}
Aug 05 17:24:33.900991 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:33.900621    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d57488d-wcvsn" event={"ID":"eae812f8-7c5a-4017-9dbf-f04df9cf563d","Type":"ContainerStarted","Data":"ffebc81e7e49189e9a5e972809a700283db2e05d2bce082514e04554c4fdd55b"}
Aug 05 17:24:33.920441 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:33.920389    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57d57488d-wcvsn" podStartSLOduration=1.920373004 podStartE2EDuration="1.920373004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:24:33.919030461 +0000 UTC m=+729.663664789" watchObservedRunningTime="2026-08-05 17:24:33.920373004 +0000 UTC m=+729.665007330"
Aug 05 17:24:42.957650 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:42.957618    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57d57488d-wcvsn"
Aug 05 17:24:42.958298 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:42.957697    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57d57488d-wcvsn"
Aug 05 17:24:42.962702 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:42.962678    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57d57488d-wcvsn"
Aug 05 17:24:43.955327 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:43.955297    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57d57488d-wcvsn"
Aug 05 17:24:44.007289 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:44.007249    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-749868cf98-wszss"]
Aug 05 17:24:51.843371 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:51.843336    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6f49dcd9ff-cgtr7"
Aug 05 17:24:53.000136 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.000104    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-qzlbc"]
Aug 05 17:24:53.003482 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.003461    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-qzlbc"
Aug 05 17:24:53.007997 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.007946    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-tdmrs\""
Aug 05 17:24:53.008141 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.008021    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Aug 05 17:24:53.016099 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.016073    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-qzlbc"]
Aug 05 17:24:53.029323 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.029298    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-8588677cfb-dl669"]
Aug 05 17:24:53.032627 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.032608    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-8588677cfb-dl669"
Aug 05 17:24:53.036096 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.036075    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Aug 05 17:24:53.036181 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.036160    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Aug 05 17:24:53.036652 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.036637    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-67p99\""
Aug 05 17:24:53.047328 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.047306    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-8588677cfb-dl669"]
Aug 05 17:24:53.054475 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.054453    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68sdk\" (UniqueName: \"kubernetes.io/projected/c26bd63d-5f44-405d-b181-d239062c25f4-kube-api-access-68sdk\") pod \"odh-model-controller-8588677cfb-dl669\" (UID: \"c26bd63d-5f44-405d-b181-d239062c25f4\") " pod="kserve/odh-model-controller-8588677cfb-dl669"
Aug 05 17:24:53.054596 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.054487    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c26bd63d-5f44-405d-b181-d239062c25f4-metrics-certs\") pod \"odh-model-controller-8588677cfb-dl669\" (UID: \"c26bd63d-5f44-405d-b181-d239062c25f4\") " pod="kserve/odh-model-controller-8588677cfb-dl669"
Aug 05 17:24:53.054596 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.054509    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c26bd63d-5f44-405d-b181-d239062c25f4-cert\") pod \"odh-model-controller-8588677cfb-dl669\" (UID: \"c26bd63d-5f44-405d-b181-d239062c25f4\") " pod="kserve/odh-model-controller-8588677cfb-dl669"
Aug 05 17:24:53.054596 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.054574    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf98fcfc-4562-48f9-8d84-4654b6cd399a-tls-certs\") pod \"model-serving-api-7b954ffc95-qzlbc\" (UID: \"bf98fcfc-4562-48f9-8d84-4654b6cd399a\") " pod="kserve/model-serving-api-7b954ffc95-qzlbc"
Aug 05 17:24:53.054773 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.054627    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42p6z\" (UniqueName: \"kubernetes.io/projected/bf98fcfc-4562-48f9-8d84-4654b6cd399a-kube-api-access-42p6z\") pod \"model-serving-api-7b954ffc95-qzlbc\" (UID: \"bf98fcfc-4562-48f9-8d84-4654b6cd399a\") " pod="kserve/model-serving-api-7b954ffc95-qzlbc"
Aug 05 17:24:53.155797 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.155754    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c26bd63d-5f44-405d-b181-d239062c25f4-metrics-certs\") pod \"odh-model-controller-8588677cfb-dl669\" (UID: \"c26bd63d-5f44-405d-b181-d239062c25f4\") " pod="kserve/odh-model-controller-8588677cfb-dl669"
Aug 05 17:24:53.155797 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.155801    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c26bd63d-5f44-405d-b181-d239062c25f4-cert\") pod \"odh-model-controller-8588677cfb-dl669\" (UID: \"c26bd63d-5f44-405d-b181-d239062c25f4\") " pod="kserve/odh-model-controller-8588677cfb-dl669"
Aug 05 17:24:53.156104 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.155843    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf98fcfc-4562-48f9-8d84-4654b6cd399a-tls-certs\") pod \"model-serving-api-7b954ffc95-qzlbc\" (UID: \"bf98fcfc-4562-48f9-8d84-4654b6cd399a\") " pod="kserve/model-serving-api-7b954ffc95-qzlbc"
Aug 05 17:24:53.156104 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.155908    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42p6z\" (UniqueName: \"kubernetes.io/projected/bf98fcfc-4562-48f9-8d84-4654b6cd399a-kube-api-access-42p6z\") pod \"model-serving-api-7b954ffc95-qzlbc\" (UID: \"bf98fcfc-4562-48f9-8d84-4654b6cd399a\") " pod="kserve/model-serving-api-7b954ffc95-qzlbc"
Aug 05 17:24:53.156104 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:24:53.155917    2561 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Aug 05 17:24:53.156104 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:24:53.155985    2561 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Aug 05 17:24:53.156104 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.156002    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68sdk\" (UniqueName: \"kubernetes.io/projected/c26bd63d-5f44-405d-b181-d239062c25f4-kube-api-access-68sdk\") pod \"odh-model-controller-8588677cfb-dl669\" (UID: \"c26bd63d-5f44-405d-b181-d239062c25f4\") " pod="kserve/odh-model-controller-8588677cfb-dl669"
Aug 05 17:24:53.156104 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:24:53.156052    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c26bd63d-5f44-405d-b181-d239062c25f4-cert podName:c26bd63d-5f44-405d-b181-d239062c25f4 nodeName:}" failed. No retries permitted until 2026-08-05 17:24:53.656030912 +0000 UTC m=+749.400665217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c26bd63d-5f44-405d-b181-d239062c25f4-cert") pod "odh-model-controller-8588677cfb-dl669" (UID: "c26bd63d-5f44-405d-b181-d239062c25f4") : secret "odh-model-controller-webhook-cert" not found
Aug 05 17:24:53.156104 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:24:53.156069    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c26bd63d-5f44-405d-b181-d239062c25f4-metrics-certs podName:c26bd63d-5f44-405d-b181-d239062c25f4 nodeName:}" failed. No retries permitted until 2026-08-05 17:24:53.656060762 +0000 UTC m=+749.400695068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c26bd63d-5f44-405d-b181-d239062c25f4-metrics-certs") pod "odh-model-controller-8588677cfb-dl669" (UID: "c26bd63d-5f44-405d-b181-d239062c25f4") : secret "odh-model-controller-metrics-tls" not found
Aug 05 17:24:53.158461 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.158442    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf98fcfc-4562-48f9-8d84-4654b6cd399a-tls-certs\") pod \"model-serving-api-7b954ffc95-qzlbc\" (UID: \"bf98fcfc-4562-48f9-8d84-4654b6cd399a\") " pod="kserve/model-serving-api-7b954ffc95-qzlbc"
Aug 05 17:24:53.167382 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.167352    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68sdk\" (UniqueName: \"kubernetes.io/projected/c26bd63d-5f44-405d-b181-d239062c25f4-kube-api-access-68sdk\") pod \"odh-model-controller-8588677cfb-dl669\" (UID: \"c26bd63d-5f44-405d-b181-d239062c25f4\") " pod="kserve/odh-model-controller-8588677cfb-dl669"
Aug 05 17:24:53.167571 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.167552    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42p6z\" (UniqueName: \"kubernetes.io/projected/bf98fcfc-4562-48f9-8d84-4654b6cd399a-kube-api-access-42p6z\") pod \"model-serving-api-7b954ffc95-qzlbc\" (UID: \"bf98fcfc-4562-48f9-8d84-4654b6cd399a\") " pod="kserve/model-serving-api-7b954ffc95-qzlbc"
Aug 05 17:24:53.316372 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.316331    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-qzlbc"
Aug 05 17:24:53.452224 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.452195    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-qzlbc"]
Aug 05 17:24:53.455336 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:24:53.455309    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf98fcfc_4562_48f9_8d84_4654b6cd399a.slice/crio-5286e0eb1cc757e92024f87c03e5e71e4e37691374880af2e21e2f300a3f33ab WatchSource:0}: Error finding container 5286e0eb1cc757e92024f87c03e5e71e4e37691374880af2e21e2f300a3f33ab: Status 404 returned error can't find the container with id 5286e0eb1cc757e92024f87c03e5e71e4e37691374880af2e21e2f300a3f33ab
Aug 05 17:24:53.659363 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.659272    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c26bd63d-5f44-405d-b181-d239062c25f4-metrics-certs\") pod \"odh-model-controller-8588677cfb-dl669\" (UID: \"c26bd63d-5f44-405d-b181-d239062c25f4\") " pod="kserve/odh-model-controller-8588677cfb-dl669"
Aug 05 17:24:53.659363 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.659309    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c26bd63d-5f44-405d-b181-d239062c25f4-cert\") pod \"odh-model-controller-8588677cfb-dl669\" (UID: \"c26bd63d-5f44-405d-b181-d239062c25f4\") " pod="kserve/odh-model-controller-8588677cfb-dl669"
Aug 05 17:24:53.661706 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.661677    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c26bd63d-5f44-405d-b181-d239062c25f4-cert\") pod \"odh-model-controller-8588677cfb-dl669\" (UID: \"c26bd63d-5f44-405d-b181-d239062c25f4\") " pod="kserve/odh-model-controller-8588677cfb-dl669"
Aug 05 17:24:53.661706 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.661691    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c26bd63d-5f44-405d-b181-d239062c25f4-metrics-certs\") pod \"odh-model-controller-8588677cfb-dl669\" (UID: \"c26bd63d-5f44-405d-b181-d239062c25f4\") " pod="kserve/odh-model-controller-8588677cfb-dl669"
Aug 05 17:24:53.942652 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:53.942570    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-8588677cfb-dl669"
Aug 05 17:24:54.006026 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:54.005996    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-qzlbc" event={"ID":"bf98fcfc-4562-48f9-8d84-4654b6cd399a","Type":"ContainerStarted","Data":"5286e0eb1cc757e92024f87c03e5e71e4e37691374880af2e21e2f300a3f33ab"}
Aug 05 17:24:54.089327 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:54.089297    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-8588677cfb-dl669"]
Aug 05 17:24:54.092704 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:24:54.092671    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26bd63d_5f44_405d_b181_d239062c25f4.slice/crio-00417caabb7193e70509806dc293ac0a6f897387f32253c6f98cf8f3d2befb36 WatchSource:0}: Error finding container 00417caabb7193e70509806dc293ac0a6f897387f32253c6f98cf8f3d2befb36: Status 404 returned error can't find the container with id 00417caabb7193e70509806dc293ac0a6f897387f32253c6f98cf8f3d2befb36
Aug 05 17:24:55.012104 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:55.012074    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-8588677cfb-dl669" event={"ID":"c26bd63d-5f44-405d-b181-d239062c25f4","Type":"ContainerStarted","Data":"00417caabb7193e70509806dc293ac0a6f897387f32253c6f98cf8f3d2befb36"}
Aug 05 17:24:56.020530 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:56.020496    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-qzlbc" event={"ID":"bf98fcfc-4562-48f9-8d84-4654b6cd399a","Type":"ContainerStarted","Data":"2730364e252fc872b9a49d34e1e0e5cd309732ce7afc1279f1c0ba1102aeec2c"}
Aug 05 17:24:56.020988 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:56.020561    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-qzlbc"
Aug 05 17:24:56.040111 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:56.040061    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-qzlbc" podStartSLOduration=2.536875529 podStartE2EDuration="4.040044803s" totalImagesPullingTime="1.503169274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:24:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:24:56.038267817 +0000 UTC m=+751.782902147" watchObservedRunningTime="2026-08-05 17:24:56.040044803 +0000 UTC m=+751.784679129"
Aug 05 17:24:58.033774 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:58.033739    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-8588677cfb-dl669" event={"ID":"c26bd63d-5f44-405d-b181-d239062c25f4","Type":"ContainerStarted","Data":"4aa4a3fd3a8660e9c613f3dce5c027aa0260d7ed33142daa2040f678e0db952b"}
Aug 05 17:24:58.034257 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:58.033878    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-8588677cfb-dl669"
Aug 05 17:24:58.055013 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:24:58.054930    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-8588677cfb-dl669" podStartSLOduration=2.0775255440000002 podStartE2EDuration="5.054907249s" totalImagesPullingTime="2.977381705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:24:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:24:58.051991489 +0000 UTC m=+753.796625817" watchObservedRunningTime="2026-08-05 17:24:58.054907249 +0000 UTC m=+753.799541578"
Aug 05 17:25:07.029665 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:07.029638    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-qzlbc"
Aug 05 17:25:09.026923 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:09.026862    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-749868cf98-wszss" podUID="2d7a2358-52a8-4e99-a7cd-d28325db3299" containerName="console" containerID="cri-o://fb4641fdfa88cb1060777c4a3e249fe239b4d504afd78de4ddf4aad3eb60cf43" gracePeriod=15
Aug 05 17:25:09.041266 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:09.041243    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-8588677cfb-dl669"
Aug 05 17:25:09.280633 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:09.280611    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-749868cf98-wszss"
Aug 05 17:25:09.404081 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:09.404048    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d7a2358-52a8-4e99-a7cd-d28325db3299-oauth-serving-cert\") pod \"2d7a2358-52a8-4e99-a7cd-d28325db3299\" (UID: \"2d7a2358-52a8-4e99-a7cd-d28325db3299\") "
Aug 05 17:25:09.404255 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:09.404092    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d7a2358-52a8-4e99-a7cd-d28325db3299-trusted-ca-bundle\") pod \"2d7a2358-52a8-4e99-a7cd-d28325db3299\" (UID: \"2d7a2358-52a8-4e99-a7cd-d28325db3299\") "
Aug 05 17:25:09.404255 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:09.404149    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d7a2358-52a8-4e99-a7cd-d28325db3299-console-serving-cert\") pod \"2d7a2358-52a8-4e99-a7cd-d28325db3299\" (UID: \"2d7a2358-52a8-4e99-a7cd-d28325db3299\") "
Aug 05 17:25:09.404255 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:09.404186    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzwll\" (UniqueName: \"kubernetes.io/projected/2d7a2358-52a8-4e99-a7cd-d28325db3299-kube-api-access-lzwll\") pod \"2d7a2358-52a8-4e99-a7cd-d28325db3299\" (UID: \"2d7a2358-52a8-4e99-a7cd-d28325db3299\") "
Aug 05 17:25:09.404255 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:09.404205    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d7a2358-52a8-4e99-a7cd-d28325db3299-console-oauth-config\") pod \"2d7a2358-52a8-4e99-a7cd-d28325db3299\" (UID: \"2d7a2358-52a8-4e99-a7cd-d28325db3299\") "
Aug 05 17:25:09.404255 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:09.404241    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d7a2358-52a8-4e99-a7cd-d28325db3299-console-config\") pod \"2d7a2358-52a8-4e99-a7cd-d28325db3299\" (UID: \"2d7a2358-52a8-4e99-a7cd-d28325db3299\") "
Aug 05 17:25:09.404502 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:09.404267    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7a2358-52a8-4e99-a7cd-d28325db3299-service-ca\") pod \"2d7a2358-52a8-4e99-a7cd-d28325db3299\" (UID: \"2d7a2358-52a8-4e99-a7cd-d28325db3299\") "
Aug 05 17:25:09.404622 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:09.404587    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d7a2358-52a8-4e99-a7cd-d28325db3299-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2d7a2358-52a8-4e99-a7cd-d28325db3299" (UID: "2d7a2358-52a8-4e99-a7cd-d28325db3299"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:25:09.404750 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:09.404681    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d7a2358-52a8-4e99-a7cd-d28325db3299-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2d7a2358-52a8-4e99-a7cd-d28325db3299" (UID: "2d7a2358-52a8-4e99-a7cd-d28325db3299"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:25:09.404750 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:09.404680    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d7a2358-52a8-4e99-a7cd-d28325db3299-console-config" (OuterVolumeSpecName: "console-config") pod "2d7a2358-52a8-4e99-a7cd-d28325db3299" (UID: "2d7a2358-52a8-4e99-a7cd-d28325db3299"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:25:09.404850 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:09.404789    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d7a2358-52a8-4e99-a7cd-d28325db3299-service-ca" (OuterVolumeSpecName: "service-ca") pod "2d7a2358-52a8-4e99-a7cd-d28325db3299" (UID: "2d7a2358-52a8-4e99-a7cd-d28325db3299"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:25:09.406453 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:09.406421    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d7a2358-52a8-4e99-a7cd-d28325db3299-kube-api-access-lzwll" (OuterVolumeSpecName: "kube-api-access-lzwll") pod "2d7a2358-52a8-4e99-a7cd-d28325db3299" (UID: "2d7a2358-52a8-4e99-a7cd-d28325db3299"). InnerVolumeSpecName "kube-api-access-lzwll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:25:09.406634 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:09.406604    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d7a2358-52a8-4e99-a7cd-d28325db3299-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2d7a2358-52a8-4e99-a7cd-d28325db3299" (UID: "2d7a2358-52a8-4e99-a7cd-d28325db3299"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:25:09.406725 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:09.406672    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d7a2358-52a8-4e99-a7cd-d28325db3299-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2d7a2358-52a8-4e99-a7cd-d28325db3299" (UID: "2d7a2358-52a8-4e99-a7cd-d28325db3299"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:25:09.505440 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:09.505382    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d7a2358-52a8-4e99-a7cd-d28325db3299-oauth-serving-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:25:09.505440 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:09.505429    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d7a2358-52a8-4e99-a7cd-d28325db3299-trusted-ca-bundle\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:25:09.505440 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:09.505439    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d7a2358-52a8-4e99-a7cd-d28325db3299-console-serving-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:25:09.505440 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:09.505448    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzwll\" (UniqueName: \"kubernetes.io/projected/2d7a2358-52a8-4e99-a7cd-d28325db3299-kube-api-access-lzwll\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:25:09.505440 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:09.505456    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d7a2358-52a8-4e99-a7cd-d28325db3299-console-oauth-config\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:25:09.505440 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:09.505465    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d7a2358-52a8-4e99-a7cd-d28325db3299-console-config\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:25:09.505774 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:09.505473    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7a2358-52a8-4e99-a7cd-d28325db3299-service-ca\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:25:09.884049 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:09.884021    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-hxqxk"]
Aug 05 17:25:09.884376 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:09.884365    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d7a2358-52a8-4e99-a7cd-d28325db3299" containerName="console"
Aug 05 17:25:09.884416 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:09.884379    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d7a2358-52a8-4e99-a7cd-d28325db3299" containerName="console"
Aug 05 17:25:09.884449 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:09.884441    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d7a2358-52a8-4e99-a7cd-d28325db3299" containerName="console"
Aug 05 17:25:09.887810 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:09.887797    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-hxqxk"
Aug 05 17:25:09.896832 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:09.896809    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-hxqxk"]
Aug 05 17:25:10.009838 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:10.009789    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5717793-e01e-4e74-8f1b-d65fbc81682a-model-cache\") pod \"s3-init-hxqxk\" (UID: \"c5717793-e01e-4e74-8f1b-d65fbc81682a\") " pod="kserve/s3-init-hxqxk"
Aug 05 17:25:10.010090 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:10.009856    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phlg7\" (UniqueName: \"kubernetes.io/projected/c5717793-e01e-4e74-8f1b-d65fbc81682a-kube-api-access-phlg7\") pod \"s3-init-hxqxk\" (UID: \"c5717793-e01e-4e74-8f1b-d65fbc81682a\") " pod="kserve/s3-init-hxqxk"
Aug 05 17:25:10.101828 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:10.101793    2561 generic.go:356] "Generic (PLEG): container finished" podID="2d7a2358-52a8-4e99-a7cd-d28325db3299" containerID="fb4641fdfa88cb1060777c4a3e249fe239b4d504afd78de4ddf4aad3eb60cf43" exitCode=0
Aug 05 17:25:10.102302 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:10.101844    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-749868cf98-wszss"
Aug 05 17:25:10.102302 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:10.101844    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-749868cf98-wszss" event={"ID":"2d7a2358-52a8-4e99-a7cd-d28325db3299","Type":"ContainerDied","Data":"fb4641fdfa88cb1060777c4a3e249fe239b4d504afd78de4ddf4aad3eb60cf43"}
Aug 05 17:25:10.102302 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:10.101895    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-749868cf98-wszss" event={"ID":"2d7a2358-52a8-4e99-a7cd-d28325db3299","Type":"ContainerDied","Data":"1c4fa94c39f28cc4e59ca0e81283efe948c0f43f7744e01fcafb0d9d26af8aa3"}
Aug 05 17:25:10.102302 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:10.101930    2561 scope.go:117] "RemoveContainer" containerID="fb4641fdfa88cb1060777c4a3e249fe239b4d504afd78de4ddf4aad3eb60cf43"
Aug 05 17:25:10.110886 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:10.110866    2561 scope.go:117] "RemoveContainer" containerID="fb4641fdfa88cb1060777c4a3e249fe239b4d504afd78de4ddf4aad3eb60cf43"
Aug 05 17:25:10.110987 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:10.110965    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5717793-e01e-4e74-8f1b-d65fbc81682a-model-cache\") pod \"s3-init-hxqxk\" (UID: \"c5717793-e01e-4e74-8f1b-d65fbc81682a\") " pod="kserve/s3-init-hxqxk"
Aug 05 17:25:10.111039 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:10.111024    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phlg7\" (UniqueName: \"kubernetes.io/projected/c5717793-e01e-4e74-8f1b-d65fbc81682a-kube-api-access-phlg7\") pod \"s3-init-hxqxk\" (UID: \"c5717793-e01e-4e74-8f1b-d65fbc81682a\") " pod="kserve/s3-init-hxqxk"
Aug 05 17:25:10.111214 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:25:10.111186    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4641fdfa88cb1060777c4a3e249fe239b4d504afd78de4ddf4aad3eb60cf43\": container with ID starting with fb4641fdfa88cb1060777c4a3e249fe239b4d504afd78de4ddf4aad3eb60cf43 not found: ID does not exist" containerID="fb4641fdfa88cb1060777c4a3e249fe239b4d504afd78de4ddf4aad3eb60cf43"
Aug 05 17:25:10.111254 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:10.111223    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4641fdfa88cb1060777c4a3e249fe239b4d504afd78de4ddf4aad3eb60cf43"} err="failed to get container status \"fb4641fdfa88cb1060777c4a3e249fe239b4d504afd78de4ddf4aad3eb60cf43\": rpc error: code = NotFound desc = could not find container \"fb4641fdfa88cb1060777c4a3e249fe239b4d504afd78de4ddf4aad3eb60cf43\": container with ID starting with fb4641fdfa88cb1060777c4a3e249fe239b4d504afd78de4ddf4aad3eb60cf43 not found: ID does not exist"
Aug 05 17:25:10.111293 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:10.111279    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5717793-e01e-4e74-8f1b-d65fbc81682a-model-cache\") pod \"s3-init-hxqxk\" (UID: \"c5717793-e01e-4e74-8f1b-d65fbc81682a\") " pod="kserve/s3-init-hxqxk"
Aug 05 17:25:10.119234 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:10.119214    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phlg7\" (UniqueName: \"kubernetes.io/projected/c5717793-e01e-4e74-8f1b-d65fbc81682a-kube-api-access-phlg7\") pod \"s3-init-hxqxk\" (UID: \"c5717793-e01e-4e74-8f1b-d65fbc81682a\") " pod="kserve/s3-init-hxqxk"
Aug 05 17:25:10.124040 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:10.124017    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-749868cf98-wszss"]
Aug 05 17:25:10.130439 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:10.130411    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-749868cf98-wszss"]
Aug 05 17:25:10.208254 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:10.208179    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-hxqxk"
Aug 05 17:25:10.341919 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:10.341887    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-hxqxk"]
Aug 05 17:25:10.343973 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:25:10.343925    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5717793_e01e_4e74_8f1b_d65fbc81682a.slice/crio-2898fd9d79be2ac955104aa5afe7a3424e66f159acf3cbaed7c423ee30aebd44 WatchSource:0}: Error finding container 2898fd9d79be2ac955104aa5afe7a3424e66f159acf3cbaed7c423ee30aebd44: Status 404 returned error can't find the container with id 2898fd9d79be2ac955104aa5afe7a3424e66f159acf3cbaed7c423ee30aebd44
Aug 05 17:25:10.730675 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:10.730639    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d7a2358-52a8-4e99-a7cd-d28325db3299" path="/var/lib/kubelet/pods/2d7a2358-52a8-4e99-a7cd-d28325db3299/volumes"
Aug 05 17:25:11.108568 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:11.108532    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hxqxk" event={"ID":"c5717793-e01e-4e74-8f1b-d65fbc81682a","Type":"ContainerStarted","Data":"2898fd9d79be2ac955104aa5afe7a3424e66f159acf3cbaed7c423ee30aebd44"}
Aug 05 17:25:14.130573 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:14.130526    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hxqxk" event={"ID":"c5717793-e01e-4e74-8f1b-d65fbc81682a","Type":"ContainerStarted","Data":"1a37a52bba4124fdf4ee7ab12ccb46b9c2e87b5f4444107ac4bc686e564b3cd5"}
Aug 05 17:25:16.142963 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:16.142929    2561 generic.go:356] "Generic (PLEG): container finished" podID="c5717793-e01e-4e74-8f1b-d65fbc81682a" containerID="1a37a52bba4124fdf4ee7ab12ccb46b9c2e87b5f4444107ac4bc686e564b3cd5" exitCode=0
Aug 05 17:25:16.143428 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:16.143000    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hxqxk" event={"ID":"c5717793-e01e-4e74-8f1b-d65fbc81682a","Type":"ContainerDied","Data":"1a37a52bba4124fdf4ee7ab12ccb46b9c2e87b5f4444107ac4bc686e564b3cd5"}
Aug 05 17:25:21.177863 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:21.177828    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hxqxk" event={"ID":"c5717793-e01e-4e74-8f1b-d65fbc81682a","Type":"ContainerStarted","Data":"5fe58f53e6eb1680de5b64a432786052f6b88ee09b8bbe477ac970e3221ce819"}
Aug 05 17:25:21.194815 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:25:21.194765    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-hxqxk" podStartSLOduration=2.866134277 podStartE2EDuration="12.194750058s" totalImagesPullingTime="7.776459546s" totalInitContainerRuntime="1.552156235s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:25:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:25:21.194097244 +0000 UTC m=+776.938731594" watchObservedRunningTime="2026-08-05 17:25:21.194750058 +0000 UTC m=+776.939384421"
Aug 05 17:26:25.511251 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:25.511223    2561 generic.go:356] "Generic (PLEG): container finished" podID="c5717793-e01e-4e74-8f1b-d65fbc81682a" containerID="5fe58f53e6eb1680de5b64a432786052f6b88ee09b8bbe477ac970e3221ce819" exitCode=0
Aug 05 17:26:25.511581 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:25.511294    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hxqxk" event={"ID":"c5717793-e01e-4e74-8f1b-d65fbc81682a","Type":"ContainerDied","Data":"5fe58f53e6eb1680de5b64a432786052f6b88ee09b8bbe477ac970e3221ce819"}
Aug 05 17:26:26.651769 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:26.651745    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-hxqxk"
Aug 05 17:26:26.756327 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:26.756290    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5717793-e01e-4e74-8f1b-d65fbc81682a-model-cache\") pod \"c5717793-e01e-4e74-8f1b-d65fbc81682a\" (UID: \"c5717793-e01e-4e74-8f1b-d65fbc81682a\") "
Aug 05 17:26:26.756529 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:26.756417    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phlg7\" (UniqueName: \"kubernetes.io/projected/c5717793-e01e-4e74-8f1b-d65fbc81682a-kube-api-access-phlg7\") pod \"c5717793-e01e-4e74-8f1b-d65fbc81682a\" (UID: \"c5717793-e01e-4e74-8f1b-d65fbc81682a\") "
Aug 05 17:26:26.758399 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:26.758371    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5717793-e01e-4e74-8f1b-d65fbc81682a-kube-api-access-phlg7" (OuterVolumeSpecName: "kube-api-access-phlg7") pod "c5717793-e01e-4e74-8f1b-d65fbc81682a" (UID: "c5717793-e01e-4e74-8f1b-d65fbc81682a"). InnerVolumeSpecName "kube-api-access-phlg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:26:26.774463 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:26.774403    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5717793-e01e-4e74-8f1b-d65fbc81682a-model-cache" (OuterVolumeSpecName: "model-cache") pod "c5717793-e01e-4e74-8f1b-d65fbc81682a" (UID: "c5717793-e01e-4e74-8f1b-d65fbc81682a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:26:26.857808 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:26.857775    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phlg7\" (UniqueName: \"kubernetes.io/projected/c5717793-e01e-4e74-8f1b-d65fbc81682a-kube-api-access-phlg7\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:26:26.857808 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:26.857803    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5717793-e01e-4e74-8f1b-d65fbc81682a-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:26:27.522646 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:27.522622    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-hxqxk"
Aug 05 17:26:27.522846 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:27.522651    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hxqxk" event={"ID":"c5717793-e01e-4e74-8f1b-d65fbc81682a","Type":"ContainerDied","Data":"2898fd9d79be2ac955104aa5afe7a3424e66f159acf3cbaed7c423ee30aebd44"}
Aug 05 17:26:27.522846 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:27.522692    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2898fd9d79be2ac955104aa5afe7a3424e66f159acf3cbaed7c423ee30aebd44"
Aug 05 17:26:48.611901 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.611869    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"]
Aug 05 17:26:48.612337 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.612264    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5717793-e01e-4e74-8f1b-d65fbc81682a" containerName="s3-init"
Aug 05 17:26:48.612337 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.612276    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5717793-e01e-4e74-8f1b-d65fbc81682a" containerName="s3-init"
Aug 05 17:26:48.612337 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.612286    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5717793-e01e-4e74-8f1b-d65fbc81682a" containerName="download-hf-model"
Aug 05 17:26:48.612337 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.612291    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5717793-e01e-4e74-8f1b-d65fbc81682a" containerName="download-hf-model"
Aug 05 17:26:48.612462 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.612349    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5717793-e01e-4e74-8f1b-d65fbc81682a" containerName="s3-init"
Aug 05 17:26:48.616371 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.616352    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"
Aug 05 17:26:48.618933 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.618910    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Aug 05 17:26:48.618933 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.618928    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Aug 05 17:26:48.619943 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.619926    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-v5w65\""
Aug 05 17:26:48.620041 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.619992    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Aug 05 17:26:48.625700 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.625678    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"]
Aug 05 17:26:48.747767 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.747735    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae215c87-4163-45fe-8b96-34439b272c41-dshm\") pod \"fc-smoke-test-kserve-7f4687897-x5gqm\" (UID: \"ae215c87-4163-45fe-8b96-34439b272c41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"
Aug 05 17:26:48.747975 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.747774    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae215c87-4163-45fe-8b96-34439b272c41-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-x5gqm\" (UID: \"ae215c87-4163-45fe-8b96-34439b272c41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"
Aug 05 17:26:48.747975 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.747865    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae215c87-4163-45fe-8b96-34439b272c41-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-x5gqm\" (UID: \"ae215c87-4163-45fe-8b96-34439b272c41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"
Aug 05 17:26:48.747975 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.747894    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae215c87-4163-45fe-8b96-34439b272c41-home\") pod \"fc-smoke-test-kserve-7f4687897-x5gqm\" (UID: \"ae215c87-4163-45fe-8b96-34439b272c41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"
Aug 05 17:26:48.747975 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.747915    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae215c87-4163-45fe-8b96-34439b272c41-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-x5gqm\" (UID: \"ae215c87-4163-45fe-8b96-34439b272c41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"
Aug 05 17:26:48.747975 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.747935    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh4q6\" (UniqueName: \"kubernetes.io/projected/ae215c87-4163-45fe-8b96-34439b272c41-kube-api-access-rh4q6\") pod \"fc-smoke-test-kserve-7f4687897-x5gqm\" (UID: \"ae215c87-4163-45fe-8b96-34439b272c41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"
Aug 05 17:26:48.848847 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.848814    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae215c87-4163-45fe-8b96-34439b272c41-dshm\") pod \"fc-smoke-test-kserve-7f4687897-x5gqm\" (UID: \"ae215c87-4163-45fe-8b96-34439b272c41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"
Aug 05 17:26:48.848847 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.848851    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae215c87-4163-45fe-8b96-34439b272c41-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-x5gqm\" (UID: \"ae215c87-4163-45fe-8b96-34439b272c41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"
Aug 05 17:26:48.849098 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.848905    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae215c87-4163-45fe-8b96-34439b272c41-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-x5gqm\" (UID: \"ae215c87-4163-45fe-8b96-34439b272c41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"
Aug 05 17:26:48.849098 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.848930    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae215c87-4163-45fe-8b96-34439b272c41-home\") pod \"fc-smoke-test-kserve-7f4687897-x5gqm\" (UID: \"ae215c87-4163-45fe-8b96-34439b272c41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"
Aug 05 17:26:48.849098 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.848986    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae215c87-4163-45fe-8b96-34439b272c41-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-x5gqm\" (UID: \"ae215c87-4163-45fe-8b96-34439b272c41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"
Aug 05 17:26:48.849098 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.849016    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh4q6\" (UniqueName: \"kubernetes.io/projected/ae215c87-4163-45fe-8b96-34439b272c41-kube-api-access-rh4q6\") pod \"fc-smoke-test-kserve-7f4687897-x5gqm\" (UID: \"ae215c87-4163-45fe-8b96-34439b272c41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"
Aug 05 17:26:48.849380 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.849356    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae215c87-4163-45fe-8b96-34439b272c41-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-x5gqm\" (UID: \"ae215c87-4163-45fe-8b96-34439b272c41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"
Aug 05 17:26:48.849492 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.849377    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae215c87-4163-45fe-8b96-34439b272c41-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-x5gqm\" (UID: \"ae215c87-4163-45fe-8b96-34439b272c41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"
Aug 05 17:26:48.849492 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.849403    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae215c87-4163-45fe-8b96-34439b272c41-home\") pod \"fc-smoke-test-kserve-7f4687897-x5gqm\" (UID: \"ae215c87-4163-45fe-8b96-34439b272c41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"
Aug 05 17:26:48.851221 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.851202    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae215c87-4163-45fe-8b96-34439b272c41-dshm\") pod \"fc-smoke-test-kserve-7f4687897-x5gqm\" (UID: \"ae215c87-4163-45fe-8b96-34439b272c41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"
Aug 05 17:26:48.851443 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.851428    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae215c87-4163-45fe-8b96-34439b272c41-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-x5gqm\" (UID: \"ae215c87-4163-45fe-8b96-34439b272c41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"
Aug 05 17:26:48.857603 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.857580    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh4q6\" (UniqueName: \"kubernetes.io/projected/ae215c87-4163-45fe-8b96-34439b272c41-kube-api-access-rh4q6\") pod \"fc-smoke-test-kserve-7f4687897-x5gqm\" (UID: \"ae215c87-4163-45fe-8b96-34439b272c41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"
Aug 05 17:26:48.927253 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:48.927157    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"
Aug 05 17:26:49.078631 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:49.078597    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"]
Aug 05 17:26:49.086178 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:26:49.086150    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae215c87_4163_45fe_8b96_34439b272c41.slice/crio-85930c3316bcb19f7b049db6d3db82bde155b385cf4dc4908eb8d6a6a297260c WatchSource:0}: Error finding container 85930c3316bcb19f7b049db6d3db82bde155b385cf4dc4908eb8d6a6a297260c: Status 404 returned error can't find the container with id 85930c3316bcb19f7b049db6d3db82bde155b385cf4dc4908eb8d6a6a297260c
Aug 05 17:26:49.650318 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:49.650273    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm" event={"ID":"ae215c87-4163-45fe-8b96-34439b272c41","Type":"ContainerStarted","Data":"85930c3316bcb19f7b049db6d3db82bde155b385cf4dc4908eb8d6a6a297260c"}
Aug 05 17:26:51.676676 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:51.676636    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm" event={"ID":"ae215c87-4163-45fe-8b96-34439b272c41","Type":"ContainerStarted","Data":"481cc7553f736486fc4bd718426344d2ea5d5c452e802745e0dd12caff6f976c"}
Aug 05 17:26:51.695263 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:51.695213    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm" podStartSLOduration=2.059315213 podStartE2EDuration="3.695183768s" totalImagesPullingTime="1.635868555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:26:51.693335412 +0000 UTC m=+867.437969752" watchObservedRunningTime="2026-08-05 17:26:51.695183768 +0000 UTC m=+867.439818096"
Aug 05 17:26:51.927506 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:51.927423    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"
Aug 05 17:26:58.715707 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:58.715612    2561 generic.go:356] "Generic (PLEG): container finished" podID="2ecc8132-d5f3-4c33-aca3-e7e0a30e7531" containerID="c2ce578300db2f2ccd034518e7839c7e043864518a00022050d52306d9d6645a" exitCode=0
Aug 05 17:26:58.715707 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:58.715673    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9" event={"ID":"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531","Type":"ContainerDied","Data":"c2ce578300db2f2ccd034518e7839c7e043864518a00022050d52306d9d6645a"}
Aug 05 17:26:58.716302 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:58.716176    2561 scope.go:117] "RemoveContainer" containerID="c2ce578300db2f2ccd034518e7839c7e043864518a00022050d52306d9d6645a"
Aug 05 17:26:58.717240 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:58.717204    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 17:26:58.717308 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:58.717277    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 17:26:58.717344 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:58.717314    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 17:26:58.928367 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:58.928323    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"
Aug 05 17:26:58.942118 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:58.942088    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"
Aug 05 17:26:59.722561 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:59.722528    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9" event={"ID":"2ecc8132-d5f3-4c33-aca3-e7e0a30e7531","Type":"ContainerStarted","Data":"8c5a25da4ff614b4058dfe077f52e6348469f3cfc1feadf36fd4a01d34f9c9bb"}
Aug 05 17:26:59.733679 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:26:59.733646    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"
Aug 05 17:27:00.683788 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.683745    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:27:00.684093 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.684070    2561 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Aug 05 17:27:00.684443 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.684126    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9" podUID="2ecc8132-d5f3-4c33-aca3-e7e0a30e7531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Aug 05 17:27:00.695626 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.695597    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Aug 05 17:27:00.698749 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.698729    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:27:00.704718 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.703010    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-wb4jg\""
Aug 05 17:27:00.704718 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.703552    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Aug 05 17:27:00.704718 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.703699    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-q5sfd\""
Aug 05 17:27:00.704718 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.704149    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Aug 05 17:27:00.704718 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.704694    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Aug 05 17:27:00.705061 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.704915    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Aug 05 17:27:00.714244 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.712352    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Aug 05 17:27:00.720224 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.720192    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Aug 05 17:27:00.775668 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.775618    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8f16bcc-d578-4678-9744-e8133174fc60-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:27:00.776192 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.775684    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8f16bcc-d578-4678-9744-e8133174fc60-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:27:00.776192 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.775729    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8f16bcc-d578-4678-9744-e8133174fc60-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:27:00.776192 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.775818    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8f16bcc-d578-4678-9744-e8133174fc60-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:27:00.776192 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.775881    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8f16bcc-d578-4678-9744-e8133174fc60-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:27:00.776192 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.775922    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfzvm\" (UniqueName: \"kubernetes.io/projected/a8f16bcc-d578-4678-9744-e8133174fc60-kube-api-access-gfzvm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:27:00.776192 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.775976    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8f16bcc-d578-4678-9744-e8133174fc60-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:27:00.776192 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.776035    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8f16bcc-d578-4678-9744-e8133174fc60-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:27:00.876885 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.876839    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8f16bcc-d578-4678-9744-e8133174fc60-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:27:00.877110 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.876929    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8f16bcc-d578-4678-9744-e8133174fc60-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:27:00.877110 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.876994    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfzvm\" (UniqueName: \"kubernetes.io/projected/a8f16bcc-d578-4678-9744-e8133174fc60-kube-api-access-gfzvm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:27:00.877110 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.877034    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8f16bcc-d578-4678-9744-e8133174fc60-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:27:00.877110 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.877088    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8f16bcc-d578-4678-9744-e8133174fc60-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:27:00.877345 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.877125    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8f16bcc-d578-4678-9744-e8133174fc60-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:27:00.877345 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.877157    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8f16bcc-d578-4678-9744-e8133174fc60-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:27:00.877345 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.877184    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8f16bcc-d578-4678-9744-e8133174fc60-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:27:00.878627 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.878263    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8f16bcc-d578-4678-9744-e8133174fc60-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:27:00.878627 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.878279    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8f16bcc-d578-4678-9744-e8133174fc60-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:27:00.878627 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.878494    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8f16bcc-d578-4678-9744-e8133174fc60-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:27:00.879122 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.879076    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8f16bcc-d578-4678-9744-e8133174fc60-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:27:00.879462 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.879444    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8f16bcc-d578-4678-9744-e8133174fc60-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:27:00.880994 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.880390    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8f16bcc-d578-4678-9744-e8133174fc60-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:27:00.882218 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.882191    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8f16bcc-d578-4678-9744-e8133174fc60-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:27:00.886943 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:00.886922    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfzvm\" (UniqueName: \"kubernetes.io/projected/a8f16bcc-d578-4678-9744-e8133174fc60-kube-api-access-gfzvm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:27:01.020982 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:01.020875    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:27:01.216875 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:01.216847    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Aug 05 17:27:01.219056 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:27:01.219025    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f16bcc_d578_4678_9744_e8133174fc60.slice/crio-df3a18bbf20e245b8e046643f9f7b8776a9b5afceee158cfed968ae00fb23ab0 WatchSource:0}: Error finding container df3a18bbf20e245b8e046643f9f7b8776a9b5afceee158cfed968ae00fb23ab0: Status 404 returned error can't find the container with id df3a18bbf20e245b8e046643f9f7b8776a9b5afceee158cfed968ae00fb23ab0
Aug 05 17:27:01.684767 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:01.684672    2561 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Aug 05 17:27:01.684767 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:01.684736    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9" podUID="2ecc8132-d5f3-4c33-aca3-e7e0a30e7531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Aug 05 17:27:01.737257 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:01.737224    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"a8f16bcc-d578-4678-9744-e8133174fc60","Type":"ContainerStarted","Data":"6e313310abe1cdd7ea3bae7309530fc74e6536fb25db33780f5a8c6403366c6f"}
Aug 05 17:27:01.737444 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:01.737279    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"a8f16bcc-d578-4678-9744-e8133174fc60","Type":"ContainerStarted","Data":"df3a18bbf20e245b8e046643f9f7b8776a9b5afceee158cfed968ae00fb23ab0"}
Aug 05 17:27:02.685988 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:02.684251    2561 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Aug 05 17:27:02.685988 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:02.684320    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9" podUID="2ecc8132-d5f3-4c33-aca3-e7e0a30e7531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Aug 05 17:27:03.685047 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:03.685009    2561 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Aug 05 17:27:03.685255 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:03.685100    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9" podUID="2ecc8132-d5f3-4c33-aca3-e7e0a30e7531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Aug 05 17:27:04.684177 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:04.684142    2561 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Aug 05 17:27:04.684576 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:04.684197    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9" podUID="2ecc8132-d5f3-4c33-aca3-e7e0a30e7531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Aug 05 17:27:04.757404 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:04.757369    2561 generic.go:356] "Generic (PLEG): container finished" podID="a8f16bcc-d578-4678-9744-e8133174fc60" containerID="6e313310abe1cdd7ea3bae7309530fc74e6536fb25db33780f5a8c6403366c6f" exitCode=0
Aug 05 17:27:04.757599 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:04.757414    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"a8f16bcc-d578-4678-9744-e8133174fc60","Type":"ContainerDied","Data":"6e313310abe1cdd7ea3bae7309530fc74e6536fb25db33780f5a8c6403366c6f"}
Aug 05 17:27:06.597534 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:06.597497    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:27:06.598907 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:06.598878    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:27:06.601609 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:06.601584    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9"
Aug 05 17:27:30.157468 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:30.157358    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"]
Aug 05 17:27:30.169712 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:27:30.169671    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 05 17:27:30.169712 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:27:30.169712 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:27:30.169712 ip-10-0-135-151 kubenswrapper[2561]:  > containerID="481cc7553f736486fc4bd718426344d2ea5d5c452e802745e0dd12caff6f976c" cmd=["/bin/sleep","15"]
Aug 05 17:27:30.170203 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:30.169740    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm" message="" err=<
Aug 05 17:27:30.170203 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:27:30.170203 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:27:30.170203 ip-10-0-135-151 kubenswrapper[2561]:  >
Aug 05 17:27:30.170203 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:27:30.169756    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 05 17:27:30.170203 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:27:30.170203 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:27:30.170203 ip-10-0-135-151 kubenswrapper[2561]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm" podUID="ae215c87-4163-45fe-8b96-34439b272c41" containerName="main" containerID="cri-o://481cc7553f736486fc4bd718426344d2ea5d5c452e802745e0dd12caff6f976c"
Aug 05 17:27:30.170203 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:30.169806    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm" podUID="ae215c87-4163-45fe-8b96-34439b272c41" containerName="main" containerID="cri-o://481cc7553f736486fc4bd718426344d2ea5d5c452e802745e0dd12caff6f976c" gracePeriod=60
Aug 05 17:27:30.723670 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:30.723625    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm" podUID="ae215c87-4163-45fe-8b96-34439b272c41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Aug 05 17:27:31.723302 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:31.723254    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm" podUID="ae215c87-4163-45fe-8b96-34439b272c41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Aug 05 17:27:31.723779 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:31.723384    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"
Aug 05 17:27:32.723720 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:32.723676    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm" podUID="ae215c87-4163-45fe-8b96-34439b272c41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Aug 05 17:27:33.724134 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:33.724083    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm" podUID="ae215c87-4163-45fe-8b96-34439b272c41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Aug 05 17:27:34.971822 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:34.971786    2561 generic.go:356] "Generic (PLEG): container finished" podID="ae215c87-4163-45fe-8b96-34439b272c41" containerID="481cc7553f736486fc4bd718426344d2ea5d5c452e802745e0dd12caff6f976c" exitCode=0
Aug 05 17:27:34.972246 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:34.971867    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm" event={"ID":"ae215c87-4163-45fe-8b96-34439b272c41","Type":"ContainerDied","Data":"481cc7553f736486fc4bd718426344d2ea5d5c452e802745e0dd12caff6f976c"}
Aug 05 17:27:35.547183 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:35.547148    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"
Aug 05 17:27:35.643893 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:35.643859    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae215c87-4163-45fe-8b96-34439b272c41-tls-certs\") pod \"ae215c87-4163-45fe-8b96-34439b272c41\" (UID: \"ae215c87-4163-45fe-8b96-34439b272c41\") "
Aug 05 17:27:35.644110 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:35.643903    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae215c87-4163-45fe-8b96-34439b272c41-dshm\") pod \"ae215c87-4163-45fe-8b96-34439b272c41\" (UID: \"ae215c87-4163-45fe-8b96-34439b272c41\") "
Aug 05 17:27:35.644110 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:35.643920    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae215c87-4163-45fe-8b96-34439b272c41-tmp-dir\") pod \"ae215c87-4163-45fe-8b96-34439b272c41\" (UID: \"ae215c87-4163-45fe-8b96-34439b272c41\") "
Aug 05 17:27:35.644110 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:35.643936    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rh4q6\" (UniqueName: \"kubernetes.io/projected/ae215c87-4163-45fe-8b96-34439b272c41-kube-api-access-rh4q6\") pod \"ae215c87-4163-45fe-8b96-34439b272c41\" (UID: \"ae215c87-4163-45fe-8b96-34439b272c41\") "
Aug 05 17:27:35.644110 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:35.644026    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae215c87-4163-45fe-8b96-34439b272c41-home\") pod \"ae215c87-4163-45fe-8b96-34439b272c41\" (UID: \"ae215c87-4163-45fe-8b96-34439b272c41\") "
Aug 05 17:27:35.644110 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:35.644055    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae215c87-4163-45fe-8b96-34439b272c41-model-cache\") pod \"ae215c87-4163-45fe-8b96-34439b272c41\" (UID: \"ae215c87-4163-45fe-8b96-34439b272c41\") "
Aug 05 17:27:35.644392 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:35.644231    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae215c87-4163-45fe-8b96-34439b272c41-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae215c87-4163-45fe-8b96-34439b272c41" (UID: "ae215c87-4163-45fe-8b96-34439b272c41"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:27:35.644392 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:35.644280    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae215c87-4163-45fe-8b96-34439b272c41-home" (OuterVolumeSpecName: "home") pod "ae215c87-4163-45fe-8b96-34439b272c41" (UID: "ae215c87-4163-45fe-8b96-34439b272c41"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:27:35.644392 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:35.644334    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae215c87-4163-45fe-8b96-34439b272c41-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:27:35.644535 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:35.644385    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae215c87-4163-45fe-8b96-34439b272c41-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae215c87-4163-45fe-8b96-34439b272c41" (UID: "ae215c87-4163-45fe-8b96-34439b272c41"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:27:35.646240 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:35.646215    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae215c87-4163-45fe-8b96-34439b272c41-dshm" (OuterVolumeSpecName: "dshm") pod "ae215c87-4163-45fe-8b96-34439b272c41" (UID: "ae215c87-4163-45fe-8b96-34439b272c41"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:27:35.646364 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:35.646326    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae215c87-4163-45fe-8b96-34439b272c41-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae215c87-4163-45fe-8b96-34439b272c41" (UID: "ae215c87-4163-45fe-8b96-34439b272c41"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:27:35.646364 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:35.646322    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae215c87-4163-45fe-8b96-34439b272c41-kube-api-access-rh4q6" (OuterVolumeSpecName: "kube-api-access-rh4q6") pod "ae215c87-4163-45fe-8b96-34439b272c41" (UID: "ae215c87-4163-45fe-8b96-34439b272c41"). InnerVolumeSpecName "kube-api-access-rh4q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:27:35.723536 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:35.723479    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm" podUID="ae215c87-4163-45fe-8b96-34439b272c41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded"
Aug 05 17:27:35.745705 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:35.745669    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae215c87-4163-45fe-8b96-34439b272c41-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:27:35.745705 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:35.745702    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae215c87-4163-45fe-8b96-34439b272c41-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:27:35.745705 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:35.745713    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh4q6\" (UniqueName: \"kubernetes.io/projected/ae215c87-4163-45fe-8b96-34439b272c41-kube-api-access-rh4q6\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:27:35.745994 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:35.745725    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae215c87-4163-45fe-8b96-34439b272c41-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:27:35.745994 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:35.745737    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae215c87-4163-45fe-8b96-34439b272c41-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:27:35.979276 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:35.979250    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"
Aug 05 17:27:35.979276 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:35.979254    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm" event={"ID":"ae215c87-4163-45fe-8b96-34439b272c41","Type":"ContainerDied","Data":"85930c3316bcb19f7b049db6d3db82bde155b385cf4dc4908eb8d6a6a297260c"}
Aug 05 17:27:35.979703 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:35.979324    2561 scope.go:117] "RemoveContainer" containerID="481cc7553f736486fc4bd718426344d2ea5d5c452e802745e0dd12caff6f976c"
Aug 05 17:27:36.011574 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:36.011547    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"]
Aug 05 17:27:36.015014 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:36.014987    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-x5gqm"]
Aug 05 17:27:36.730225 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:36.730193    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae215c87-4163-45fe-8b96-34439b272c41" path="/var/lib/kubelet/pods/ae215c87-4163-45fe-8b96-34439b272c41/volumes"
Aug 05 17:27:36.987727 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:36.987636    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"a8f16bcc-d578-4678-9744-e8133174fc60","Type":"ContainerStarted","Data":"c257702af7b9bd0e935f2d93fa933fa85db707f1218667f1a72c3ea776a82aa1"}
Aug 05 17:27:37.013448 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:37.013386    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=3.170291932 podStartE2EDuration="37.013370068s" totalImagesPullingTime="31.179274329s" totalInitContainerRuntime="2.663803807s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:27:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:27:37.01008805 +0000 UTC m=+912.754722379" watchObservedRunningTime="2026-08-05 17:27:37.013370068 +0000 UTC m=+912.758004395"
Aug 05 17:27:37.022063 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:37.022041    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:27:41.021455 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:41.021414    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:27:41.023478 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:41.023450    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="a8f16bcc-d578-4678-9744-e8133174fc60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Aug 05 17:27:51.021496 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:27:51.021451    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="a8f16bcc-d578-4678-9744-e8133174fc60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Aug 05 17:28:00.317529 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:00.317493    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"]
Aug 05 17:28:00.318140 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:00.318120    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae215c87-4163-45fe-8b96-34439b272c41" containerName="main"
Aug 05 17:28:00.318214 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:00.318145    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae215c87-4163-45fe-8b96-34439b272c41" containerName="main"
Aug 05 17:28:00.318252 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:00.318229    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae215c87-4163-45fe-8b96-34439b272c41" containerName="main"
Aug 05 17:28:00.534198 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:00.534156    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"]
Aug 05 17:28:00.534385 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:00.534290    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"
Aug 05 17:28:00.538998 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:00.538941    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Aug 05 17:28:00.539164 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:00.539149    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Aug 05 17:28:00.539513 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:00.539490    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-z4rm5\""
Aug 05 17:28:00.539588 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:00.539570    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Aug 05 17:28:00.668867 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:00.668778    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvwfn\" (UniqueName: \"kubernetes.io/projected/72c2be53-669f-41b0-bc59-ab258185d1d4-kube-api-access-rvwfn\") pod \"fc-concurrency-test-kserve-b5d44467c-8hgjk\" (UID: \"72c2be53-669f-41b0-bc59-ab258185d1d4\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"
Aug 05 17:28:00.668867 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:00.668827    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72c2be53-669f-41b0-bc59-ab258185d1d4-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-8hgjk\" (UID: \"72c2be53-669f-41b0-bc59-ab258185d1d4\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"
Aug 05 17:28:00.669132 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:00.668882    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72c2be53-669f-41b0-bc59-ab258185d1d4-home\") pod \"fc-concurrency-test-kserve-b5d44467c-8hgjk\" (UID: \"72c2be53-669f-41b0-bc59-ab258185d1d4\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"
Aug 05 17:28:00.669132 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:00.668937    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72c2be53-669f-41b0-bc59-ab258185d1d4-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-8hgjk\" (UID: \"72c2be53-669f-41b0-bc59-ab258185d1d4\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"
Aug 05 17:28:00.669132 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:00.669019    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72c2be53-669f-41b0-bc59-ab258185d1d4-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-8hgjk\" (UID: \"72c2be53-669f-41b0-bc59-ab258185d1d4\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"
Aug 05 17:28:00.669132 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:00.669082    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72c2be53-669f-41b0-bc59-ab258185d1d4-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-8hgjk\" (UID: \"72c2be53-669f-41b0-bc59-ab258185d1d4\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"
Aug 05 17:28:00.770366 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:00.770332    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72c2be53-669f-41b0-bc59-ab258185d1d4-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-8hgjk\" (UID: \"72c2be53-669f-41b0-bc59-ab258185d1d4\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"
Aug 05 17:28:00.770568 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:00.770377    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvwfn\" (UniqueName: \"kubernetes.io/projected/72c2be53-669f-41b0-bc59-ab258185d1d4-kube-api-access-rvwfn\") pod \"fc-concurrency-test-kserve-b5d44467c-8hgjk\" (UID: \"72c2be53-669f-41b0-bc59-ab258185d1d4\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"
Aug 05 17:28:00.770568 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:00.770520    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72c2be53-669f-41b0-bc59-ab258185d1d4-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-8hgjk\" (UID: \"72c2be53-669f-41b0-bc59-ab258185d1d4\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"
Aug 05 17:28:00.770696 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:00.770569    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72c2be53-669f-41b0-bc59-ab258185d1d4-home\") pod \"fc-concurrency-test-kserve-b5d44467c-8hgjk\" (UID: \"72c2be53-669f-41b0-bc59-ab258185d1d4\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"
Aug 05 17:28:00.770696 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:00.770610    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72c2be53-669f-41b0-bc59-ab258185d1d4-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-8hgjk\" (UID: \"72c2be53-669f-41b0-bc59-ab258185d1d4\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"
Aug 05 17:28:00.770696 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:00.770678    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72c2be53-669f-41b0-bc59-ab258185d1d4-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-8hgjk\" (UID: \"72c2be53-669f-41b0-bc59-ab258185d1d4\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"
Aug 05 17:28:00.770942 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:00.770922    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72c2be53-669f-41b0-bc59-ab258185d1d4-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-8hgjk\" (UID: \"72c2be53-669f-41b0-bc59-ab258185d1d4\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"
Aug 05 17:28:00.771038 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:00.770940    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72c2be53-669f-41b0-bc59-ab258185d1d4-home\") pod \"fc-concurrency-test-kserve-b5d44467c-8hgjk\" (UID: \"72c2be53-669f-41b0-bc59-ab258185d1d4\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"
Aug 05 17:28:00.771181 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:00.771159    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72c2be53-669f-41b0-bc59-ab258185d1d4-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-8hgjk\" (UID: \"72c2be53-669f-41b0-bc59-ab258185d1d4\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"
Aug 05 17:28:00.772916 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:00.772889    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72c2be53-669f-41b0-bc59-ab258185d1d4-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-8hgjk\" (UID: \"72c2be53-669f-41b0-bc59-ab258185d1d4\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"
Aug 05 17:28:00.773177 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:00.773160    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72c2be53-669f-41b0-bc59-ab258185d1d4-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-8hgjk\" (UID: \"72c2be53-669f-41b0-bc59-ab258185d1d4\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"
Aug 05 17:28:00.785060 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:00.785031    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvwfn\" (UniqueName: \"kubernetes.io/projected/72c2be53-669f-41b0-bc59-ab258185d1d4-kube-api-access-rvwfn\") pod \"fc-concurrency-test-kserve-b5d44467c-8hgjk\" (UID: \"72c2be53-669f-41b0-bc59-ab258185d1d4\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"
Aug 05 17:28:00.845691 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:00.845650    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"
Aug 05 17:28:01.018241 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:01.018212    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"]
Aug 05 17:28:01.019916 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:28:01.019886    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c2be53_669f_41b0_bc59_ab258185d1d4.slice/crio-1ea71f98c135838f33717edc98d7d793687883b046a3d4dede13e9a7ee0f0bf3 WatchSource:0}: Error finding container 1ea71f98c135838f33717edc98d7d793687883b046a3d4dede13e9a7ee0f0bf3: Status 404 returned error can't find the container with id 1ea71f98c135838f33717edc98d7d793687883b046a3d4dede13e9a7ee0f0bf3
Aug 05 17:28:01.021304 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:01.021272    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="a8f16bcc-d578-4678-9744-e8133174fc60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Aug 05 17:28:01.128997 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:01.128945    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk" event={"ID":"72c2be53-669f-41b0-bc59-ab258185d1d4","Type":"ContainerStarted","Data":"ee9dc586cdb8a8d02ab3883bb26d0efce4c4f7924d2140756dd3f0bde19f9abd"}
Aug 05 17:28:01.129132 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:01.129020    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk" event={"ID":"72c2be53-669f-41b0-bc59-ab258185d1d4","Type":"ContainerStarted","Data":"1ea71f98c135838f33717edc98d7d793687883b046a3d4dede13e9a7ee0f0bf3"}
Aug 05 17:28:02.155627 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:02.155512    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk" podStartSLOduration=2.155497628 podStartE2EDuration="2.155497628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:28:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:28:02.153042136 +0000 UTC m=+937.897676464" watchObservedRunningTime="2026-08-05 17:28:02.155497628 +0000 UTC m=+937.900131954"
Aug 05 17:28:02.846533 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:02.846493    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"
Aug 05 17:28:10.846500 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:10.846453    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"
Aug 05 17:28:10.865576 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:10.865544    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"
Aug 05 17:28:11.021503 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:11.021453    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="a8f16bcc-d578-4678-9744-e8133174fc60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Aug 05 17:28:11.210563 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:11.210471    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"
Aug 05 17:28:21.021737 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:21.021696    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="a8f16bcc-d578-4678-9744-e8133174fc60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Aug 05 17:28:31.021835 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:31.021777    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="a8f16bcc-d578-4678-9744-e8133174fc60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Aug 05 17:28:39.163200 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:39.163163    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"]
Aug 05 17:28:39.174155 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:28:39.174117    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 05 17:28:39.174155 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:28:39.174155 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:28:39.174155 ip-10-0-135-151 kubenswrapper[2561]:  > containerID="ee9dc586cdb8a8d02ab3883bb26d0efce4c4f7924d2140756dd3f0bde19f9abd" cmd=["/bin/sleep","15"]
Aug 05 17:28:39.174431 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:39.174170    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk" message="" err=<
Aug 05 17:28:39.174431 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:28:39.174431 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:28:39.174431 ip-10-0-135-151 kubenswrapper[2561]:  >
Aug 05 17:28:39.174431 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:28:39.174182    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 05 17:28:39.174431 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:28:39.174431 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:28:39.174431 ip-10-0-135-151 kubenswrapper[2561]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk" podUID="72c2be53-669f-41b0-bc59-ab258185d1d4" containerName="main" containerID="cri-o://ee9dc586cdb8a8d02ab3883bb26d0efce4c4f7924d2140756dd3f0bde19f9abd"
Aug 05 17:28:39.174431 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:39.174238    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk" podUID="72c2be53-669f-41b0-bc59-ab258185d1d4" containerName="main" containerID="cri-o://ee9dc586cdb8a8d02ab3883bb26d0efce4c4f7924d2140756dd3f0bde19f9abd" gracePeriod=60
Aug 05 17:28:39.200083 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:39.200042    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk" podUID="72c2be53-669f-41b0-bc59-ab258185d1d4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Aug 05 17:28:39.539199 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:39.539171    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"
Aug 05 17:28:39.638693 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:39.638657    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvwfn\" (UniqueName: \"kubernetes.io/projected/72c2be53-669f-41b0-bc59-ab258185d1d4-kube-api-access-rvwfn\") pod \"72c2be53-669f-41b0-bc59-ab258185d1d4\" (UID: \"72c2be53-669f-41b0-bc59-ab258185d1d4\") "
Aug 05 17:28:39.638896 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:39.638729    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72c2be53-669f-41b0-bc59-ab258185d1d4-dshm\") pod \"72c2be53-669f-41b0-bc59-ab258185d1d4\" (UID: \"72c2be53-669f-41b0-bc59-ab258185d1d4\") "
Aug 05 17:28:39.638896 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:39.638782    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72c2be53-669f-41b0-bc59-ab258185d1d4-tmp-dir\") pod \"72c2be53-669f-41b0-bc59-ab258185d1d4\" (UID: \"72c2be53-669f-41b0-bc59-ab258185d1d4\") "
Aug 05 17:28:39.638896 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:39.638825    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72c2be53-669f-41b0-bc59-ab258185d1d4-home\") pod \"72c2be53-669f-41b0-bc59-ab258185d1d4\" (UID: \"72c2be53-669f-41b0-bc59-ab258185d1d4\") "
Aug 05 17:28:39.638896 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:39.638871    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72c2be53-669f-41b0-bc59-ab258185d1d4-model-cache\") pod \"72c2be53-669f-41b0-bc59-ab258185d1d4\" (UID: \"72c2be53-669f-41b0-bc59-ab258185d1d4\") "
Aug 05 17:28:39.638896 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:39.638891    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72c2be53-669f-41b0-bc59-ab258185d1d4-tls-certs\") pod \"72c2be53-669f-41b0-bc59-ab258185d1d4\" (UID: \"72c2be53-669f-41b0-bc59-ab258185d1d4\") "
Aug 05 17:28:39.639188 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:39.639081    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c2be53-669f-41b0-bc59-ab258185d1d4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "72c2be53-669f-41b0-bc59-ab258185d1d4" (UID: "72c2be53-669f-41b0-bc59-ab258185d1d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:28:39.639188 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:39.639096    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c2be53-669f-41b0-bc59-ab258185d1d4-home" (OuterVolumeSpecName: "home") pod "72c2be53-669f-41b0-bc59-ab258185d1d4" (UID: "72c2be53-669f-41b0-bc59-ab258185d1d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:28:39.639188 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:39.639102    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c2be53-669f-41b0-bc59-ab258185d1d4-model-cache" (OuterVolumeSpecName: "model-cache") pod "72c2be53-669f-41b0-bc59-ab258185d1d4" (UID: "72c2be53-669f-41b0-bc59-ab258185d1d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:28:39.639335 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:39.639224    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72c2be53-669f-41b0-bc59-ab258185d1d4-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:28:39.639335 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:39.639239    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72c2be53-669f-41b0-bc59-ab258185d1d4-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:28:39.639335 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:39.639247    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72c2be53-669f-41b0-bc59-ab258185d1d4-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:28:39.641198 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:39.641162    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72c2be53-669f-41b0-bc59-ab258185d1d4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "72c2be53-669f-41b0-bc59-ab258185d1d4" (UID: "72c2be53-669f-41b0-bc59-ab258185d1d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:28:39.641405 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:39.641386    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72c2be53-669f-41b0-bc59-ab258185d1d4-kube-api-access-rvwfn" (OuterVolumeSpecName: "kube-api-access-rvwfn") pod "72c2be53-669f-41b0-bc59-ab258185d1d4" (UID: "72c2be53-669f-41b0-bc59-ab258185d1d4"). InnerVolumeSpecName "kube-api-access-rvwfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:28:39.641471 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:39.641395    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c2be53-669f-41b0-bc59-ab258185d1d4-dshm" (OuterVolumeSpecName: "dshm") pod "72c2be53-669f-41b0-bc59-ab258185d1d4" (UID: "72c2be53-669f-41b0-bc59-ab258185d1d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:28:39.740534 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:39.740434    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72c2be53-669f-41b0-bc59-ab258185d1d4-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:28:39.740534 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:39.740480    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvwfn\" (UniqueName: \"kubernetes.io/projected/72c2be53-669f-41b0-bc59-ab258185d1d4-kube-api-access-rvwfn\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:28:39.740534 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:39.740497    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72c2be53-669f-41b0-bc59-ab258185d1d4-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:28:40.369263 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:40.369227    2561 generic.go:356] "Generic (PLEG): container finished" podID="72c2be53-669f-41b0-bc59-ab258185d1d4" containerID="ee9dc586cdb8a8d02ab3883bb26d0efce4c4f7924d2140756dd3f0bde19f9abd" exitCode=0
Aug 05 17:28:40.369689 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:40.369293    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"
Aug 05 17:28:40.369689 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:40.369310    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk" event={"ID":"72c2be53-669f-41b0-bc59-ab258185d1d4","Type":"ContainerDied","Data":"ee9dc586cdb8a8d02ab3883bb26d0efce4c4f7924d2140756dd3f0bde19f9abd"}
Aug 05 17:28:40.369689 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:40.369358    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk" event={"ID":"72c2be53-669f-41b0-bc59-ab258185d1d4","Type":"ContainerDied","Data":"1ea71f98c135838f33717edc98d7d793687883b046a3d4dede13e9a7ee0f0bf3"}
Aug 05 17:28:40.369689 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:40.369380    2561 scope.go:117] "RemoveContainer" containerID="ee9dc586cdb8a8d02ab3883bb26d0efce4c4f7924d2140756dd3f0bde19f9abd"
Aug 05 17:28:40.381048 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:40.381031    2561 scope.go:117] "RemoveContainer" containerID="ee9dc586cdb8a8d02ab3883bb26d0efce4c4f7924d2140756dd3f0bde19f9abd"
Aug 05 17:28:40.381336 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:28:40.381318    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee9dc586cdb8a8d02ab3883bb26d0efce4c4f7924d2140756dd3f0bde19f9abd\": container with ID starting with ee9dc586cdb8a8d02ab3883bb26d0efce4c4f7924d2140756dd3f0bde19f9abd not found: ID does not exist" containerID="ee9dc586cdb8a8d02ab3883bb26d0efce4c4f7924d2140756dd3f0bde19f9abd"
Aug 05 17:28:40.381402 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:40.381343    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee9dc586cdb8a8d02ab3883bb26d0efce4c4f7924d2140756dd3f0bde19f9abd"} err="failed to get container status \"ee9dc586cdb8a8d02ab3883bb26d0efce4c4f7924d2140756dd3f0bde19f9abd\": rpc error: code = NotFound desc = could not find container \"ee9dc586cdb8a8d02ab3883bb26d0efce4c4f7924d2140756dd3f0bde19f9abd\": container with ID starting with ee9dc586cdb8a8d02ab3883bb26d0efce4c4f7924d2140756dd3f0bde19f9abd not found: ID does not exist"
Aug 05 17:28:40.395594 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:40.395562    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"]
Aug 05 17:28:40.400068 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:40.400042    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-8hgjk"]
Aug 05 17:28:40.730559 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:40.730482    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72c2be53-669f-41b0-bc59-ab258185d1d4" path="/var/lib/kubelet/pods/72c2be53-669f-41b0-bc59-ab258185d1d4/volumes"
Aug 05 17:28:41.021765 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:41.021665    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="a8f16bcc-d578-4678-9744-e8133174fc60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Aug 05 17:28:51.021905 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:51.021809    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="a8f16bcc-d578-4678-9744-e8133174fc60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Aug 05 17:28:59.218746 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.218705    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"]
Aug 05 17:28:59.219360 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.219322    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72c2be53-669f-41b0-bc59-ab258185d1d4" containerName="main"
Aug 05 17:28:59.219360 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.219345    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="72c2be53-669f-41b0-bc59-ab258185d1d4" containerName="main"
Aug 05 17:28:59.219551 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.219473    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="72c2be53-669f-41b0-bc59-ab258185d1d4" containerName="main"
Aug 05 17:28:59.222156 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.222133    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:28:59.224702 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.224684    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-876wf\""
Aug 05 17:28:59.224878 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.224686    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Aug 05 17:28:59.224984 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.224750    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Aug 05 17:28:59.224984 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.224889    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Aug 05 17:28:59.237701 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.237672    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"]
Aug 05 17:28:59.309031 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.309000    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd2dd37-b748-4349-ac16-9e1511a26da4-workload-socket\") pod \"router-gateway-1-openshift-default-5944c46dd4-jt48h\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:28:59.309031 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.309033    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fd2dd37-b748-4349-ac16-9e1511a26da4-istio-podinfo\") pod \"router-gateway-1-openshift-default-5944c46dd4-jt48h\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:28:59.309296 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.309059    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fd2dd37-b748-4349-ac16-9e1511a26da4-workload-certs\") pod \"router-gateway-1-openshift-default-5944c46dd4-jt48h\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:28:59.309296 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.309078    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fd2dd37-b748-4349-ac16-9e1511a26da4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5944c46dd4-jt48h\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:28:59.309296 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.309172    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fd2dd37-b748-4349-ac16-9e1511a26da4-istio-envoy\") pod \"router-gateway-1-openshift-default-5944c46dd4-jt48h\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:28:59.309296 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.309220    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59lcz\" (UniqueName: \"kubernetes.io/projected/3fd2dd37-b748-4349-ac16-9e1511a26da4-kube-api-access-59lcz\") pod \"router-gateway-1-openshift-default-5944c46dd4-jt48h\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:28:59.309296 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.309261    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fd2dd37-b748-4349-ac16-9e1511a26da4-istio-data\") pod \"router-gateway-1-openshift-default-5944c46dd4-jt48h\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:28:59.309296 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.309293    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd2dd37-b748-4349-ac16-9e1511a26da4-credential-socket\") pod \"router-gateway-1-openshift-default-5944c46dd4-jt48h\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:28:59.309522 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.309332    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fd2dd37-b748-4349-ac16-9e1511a26da4-istio-token\") pod \"router-gateway-1-openshift-default-5944c46dd4-jt48h\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:28:59.410300 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.410252    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fd2dd37-b748-4349-ac16-9e1511a26da4-istio-data\") pod \"router-gateway-1-openshift-default-5944c46dd4-jt48h\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:28:59.410499 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.410321    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd2dd37-b748-4349-ac16-9e1511a26da4-credential-socket\") pod \"router-gateway-1-openshift-default-5944c46dd4-jt48h\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:28:59.410499 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.410351    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fd2dd37-b748-4349-ac16-9e1511a26da4-istio-token\") pod \"router-gateway-1-openshift-default-5944c46dd4-jt48h\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:28:59.410499 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.410399    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd2dd37-b748-4349-ac16-9e1511a26da4-workload-socket\") pod \"router-gateway-1-openshift-default-5944c46dd4-jt48h\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:28:59.410499 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.410422    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fd2dd37-b748-4349-ac16-9e1511a26da4-istio-podinfo\") pod \"router-gateway-1-openshift-default-5944c46dd4-jt48h\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:28:59.410499 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.410452    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fd2dd37-b748-4349-ac16-9e1511a26da4-workload-certs\") pod \"router-gateway-1-openshift-default-5944c46dd4-jt48h\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:28:59.410499 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.410479    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fd2dd37-b748-4349-ac16-9e1511a26da4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5944c46dd4-jt48h\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:28:59.410835 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.410546    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fd2dd37-b748-4349-ac16-9e1511a26da4-istio-envoy\") pod \"router-gateway-1-openshift-default-5944c46dd4-jt48h\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:28:59.410835 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.410586    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59lcz\" (UniqueName: \"kubernetes.io/projected/3fd2dd37-b748-4349-ac16-9e1511a26da4-kube-api-access-59lcz\") pod \"router-gateway-1-openshift-default-5944c46dd4-jt48h\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:28:59.410835 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.410727    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fd2dd37-b748-4349-ac16-9e1511a26da4-istio-data\") pod \"router-gateway-1-openshift-default-5944c46dd4-jt48h\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:28:59.411020 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.410892    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd2dd37-b748-4349-ac16-9e1511a26da4-credential-socket\") pod \"router-gateway-1-openshift-default-5944c46dd4-jt48h\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:28:59.411020 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.411009    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fd2dd37-b748-4349-ac16-9e1511a26da4-workload-certs\") pod \"router-gateway-1-openshift-default-5944c46dd4-jt48h\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:28:59.411117 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.411059    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd2dd37-b748-4349-ac16-9e1511a26da4-workload-socket\") pod \"router-gateway-1-openshift-default-5944c46dd4-jt48h\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:28:59.411678 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.411648    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fd2dd37-b748-4349-ac16-9e1511a26da4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5944c46dd4-jt48h\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:28:59.412967 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.412937    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fd2dd37-b748-4349-ac16-9e1511a26da4-istio-podinfo\") pod \"router-gateway-1-openshift-default-5944c46dd4-jt48h\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:28:59.413143 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.413120    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fd2dd37-b748-4349-ac16-9e1511a26da4-istio-envoy\") pod \"router-gateway-1-openshift-default-5944c46dd4-jt48h\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:28:59.418482 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.418460    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fd2dd37-b748-4349-ac16-9e1511a26da4-istio-token\") pod \"router-gateway-1-openshift-default-5944c46dd4-jt48h\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:28:59.418785 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.418765    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59lcz\" (UniqueName: \"kubernetes.io/projected/3fd2dd37-b748-4349-ac16-9e1511a26da4-kube-api-access-59lcz\") pod \"router-gateway-1-openshift-default-5944c46dd4-jt48h\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:28:59.533748 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.533715    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:28:59.686034 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.686008    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"]
Aug 05 17:28:59.688616 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:28:59.688589    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fd2dd37_b748_4349_ac16_9e1511a26da4.slice/crio-0b1acfe02281acabe3358debd4618fe36cf00256a718ff9cd9d4781e154c575e WatchSource:0}: Error finding container 0b1acfe02281acabe3358debd4618fe36cf00256a718ff9cd9d4781e154c575e: Status 404 returned error can't find the container with id 0b1acfe02281acabe3358debd4618fe36cf00256a718ff9cd9d4781e154c575e
Aug 05 17:28:59.691335 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.691298    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 17:28:59.691436 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.691370    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 17:28:59.691436 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:28:59.691419    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 17:29:00.499645 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:00.499601    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h" event={"ID":"3fd2dd37-b748-4349-ac16-9e1511a26da4","Type":"ContainerStarted","Data":"c60f5cf2cc25ac2b709643b238c891cb22b466cb769bcc31a4bae623eb34533e"}
Aug 05 17:29:00.500063 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:00.499656    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h" event={"ID":"3fd2dd37-b748-4349-ac16-9e1511a26da4","Type":"ContainerStarted","Data":"0b1acfe02281acabe3358debd4618fe36cf00256a718ff9cd9d4781e154c575e"}
Aug 05 17:29:00.521184 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:00.521127    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h" podStartSLOduration=1.521112368 podStartE2EDuration="1.521112368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:29:00.520693062 +0000 UTC m=+996.265327390" watchObservedRunningTime="2026-08-05 17:29:00.521112368 +0000 UTC m=+996.265746695"
Aug 05 17:29:00.534782 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:00.534749    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:29:00.540158 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:00.540130    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:29:01.021606 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:01.021566    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="a8f16bcc-d578-4678-9744-e8133174fc60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Aug 05 17:29:01.505458 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:01.505430    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:29:01.506749 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:01.506733    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:29:11.031721 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:11.031682    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:29:11.039252 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:11.039222    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:29:42.575676 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.575638    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"]
Aug 05 17:29:42.582688 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.582660    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:42.585523 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.585493    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Aug 05 17:29:42.585674 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.585657    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-m45dc\""
Aug 05 17:29:42.585921 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.585892    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Aug 05 17:29:42.586040 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.585903    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Aug 05 17:29:42.595403 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.595380    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"]
Aug 05 17:29:42.704491 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.704442    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9da257a8-cc1d-4a85-ade1-171029eddcf2-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-78668f9df6-tlgzn\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:42.704704 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.704496    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9da257a8-cc1d-4a85-ade1-171029eddcf2-istio-envoy\") pod \"router-gateway-1-openshift-default-78668f9df6-tlgzn\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:42.704704 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.704573    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9da257a8-cc1d-4a85-ade1-171029eddcf2-istio-podinfo\") pod \"router-gateway-1-openshift-default-78668f9df6-tlgzn\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:42.704704 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.704602    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da257a8-cc1d-4a85-ade1-171029eddcf2-credential-socket\") pod \"router-gateway-1-openshift-default-78668f9df6-tlgzn\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:42.704704 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.704622    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tthtv\" (UniqueName: \"kubernetes.io/projected/9da257a8-cc1d-4a85-ade1-171029eddcf2-kube-api-access-tthtv\") pod \"router-gateway-1-openshift-default-78668f9df6-tlgzn\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:42.704704 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.704649    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da257a8-cc1d-4a85-ade1-171029eddcf2-workload-socket\") pod \"router-gateway-1-openshift-default-78668f9df6-tlgzn\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:42.705004 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.704751    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9da257a8-cc1d-4a85-ade1-171029eddcf2-istio-data\") pod \"router-gateway-1-openshift-default-78668f9df6-tlgzn\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:42.705004 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.704778    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9da257a8-cc1d-4a85-ade1-171029eddcf2-workload-certs\") pod \"router-gateway-1-openshift-default-78668f9df6-tlgzn\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:42.705004 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.704870    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9da257a8-cc1d-4a85-ade1-171029eddcf2-istio-token\") pod \"router-gateway-1-openshift-default-78668f9df6-tlgzn\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:42.805771 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.805732    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9da257a8-cc1d-4a85-ade1-171029eddcf2-istio-token\") pod \"router-gateway-1-openshift-default-78668f9df6-tlgzn\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:42.806006 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.805786    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9da257a8-cc1d-4a85-ade1-171029eddcf2-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-78668f9df6-tlgzn\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:42.806006 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.805905    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9da257a8-cc1d-4a85-ade1-171029eddcf2-istio-envoy\") pod \"router-gateway-1-openshift-default-78668f9df6-tlgzn\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:42.806006 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.805980    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9da257a8-cc1d-4a85-ade1-171029eddcf2-istio-podinfo\") pod \"router-gateway-1-openshift-default-78668f9df6-tlgzn\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:42.806217 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.806010    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da257a8-cc1d-4a85-ade1-171029eddcf2-credential-socket\") pod \"router-gateway-1-openshift-default-78668f9df6-tlgzn\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:42.806217 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.806031    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tthtv\" (UniqueName: \"kubernetes.io/projected/9da257a8-cc1d-4a85-ade1-171029eddcf2-kube-api-access-tthtv\") pod \"router-gateway-1-openshift-default-78668f9df6-tlgzn\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:42.806217 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.806050    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da257a8-cc1d-4a85-ade1-171029eddcf2-workload-socket\") pod \"router-gateway-1-openshift-default-78668f9df6-tlgzn\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:42.806217 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.806109    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9da257a8-cc1d-4a85-ade1-171029eddcf2-istio-data\") pod \"router-gateway-1-openshift-default-78668f9df6-tlgzn\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:42.806217 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.806147    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9da257a8-cc1d-4a85-ade1-171029eddcf2-workload-certs\") pod \"router-gateway-1-openshift-default-78668f9df6-tlgzn\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:42.806484 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.806463    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9da257a8-cc1d-4a85-ade1-171029eddcf2-workload-certs\") pod \"router-gateway-1-openshift-default-78668f9df6-tlgzn\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:42.806541 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.806515    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da257a8-cc1d-4a85-ade1-171029eddcf2-workload-socket\") pod \"router-gateway-1-openshift-default-78668f9df6-tlgzn\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:42.806597 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.806546    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9da257a8-cc1d-4a85-ade1-171029eddcf2-istio-data\") pod \"router-gateway-1-openshift-default-78668f9df6-tlgzn\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:42.806597 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.806573    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9da257a8-cc1d-4a85-ade1-171029eddcf2-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-78668f9df6-tlgzn\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:42.806705 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.806688    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da257a8-cc1d-4a85-ade1-171029eddcf2-credential-socket\") pod \"router-gateway-1-openshift-default-78668f9df6-tlgzn\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:42.808338 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.808313    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9da257a8-cc1d-4a85-ade1-171029eddcf2-istio-envoy\") pod \"router-gateway-1-openshift-default-78668f9df6-tlgzn\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:42.808448 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.808400    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9da257a8-cc1d-4a85-ade1-171029eddcf2-istio-podinfo\") pod \"router-gateway-1-openshift-default-78668f9df6-tlgzn\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:42.814623 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.814593    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9da257a8-cc1d-4a85-ade1-171029eddcf2-istio-token\") pod \"router-gateway-1-openshift-default-78668f9df6-tlgzn\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:42.814855 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.814834    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tthtv\" (UniqueName: \"kubernetes.io/projected/9da257a8-cc1d-4a85-ade1-171029eddcf2-kube-api-access-tthtv\") pod \"router-gateway-1-openshift-default-78668f9df6-tlgzn\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:42.895196 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:42.895102    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:43.101151 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:43.101120    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"]
Aug 05 17:29:43.103171 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:29:43.103144    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da257a8_cc1d_4a85_ade1_171029eddcf2.slice/crio-347a46c6d3e80352a6e3ef1d7e85c2be49c27d060ffbe703f642c37ea727d4b3 WatchSource:0}: Error finding container 347a46c6d3e80352a6e3ef1d7e85c2be49c27d060ffbe703f642c37ea727d4b3: Status 404 returned error can't find the container with id 347a46c6d3e80352a6e3ef1d7e85c2be49c27d060ffbe703f642c37ea727d4b3
Aug 05 17:29:43.105446 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:43.105414    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 17:29:43.105509 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:43.105486    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 17:29:43.105544 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:43.105517    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 17:29:43.731354 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:43.731316    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn" event={"ID":"9da257a8-cc1d-4a85-ade1-171029eddcf2","Type":"ContainerStarted","Data":"b04b504035794f11d1dba3343a241d6ae1f3110bc9986443ae11da1a2d31ef97"}
Aug 05 17:29:43.731738 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:43.731383    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn" event={"ID":"9da257a8-cc1d-4a85-ade1-171029eddcf2","Type":"ContainerStarted","Data":"347a46c6d3e80352a6e3ef1d7e85c2be49c27d060ffbe703f642c37ea727d4b3"}
Aug 05 17:29:43.752062 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:43.751998    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn" podStartSLOduration=1.7519741789999999 podStartE2EDuration="1.751974179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:29:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:29:43.751613123 +0000 UTC m=+1039.496247450" watchObservedRunningTime="2026-08-05 17:29:43.751974179 +0000 UTC m=+1039.496608508"
Aug 05 17:29:43.895678 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:43.895608    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:44.901101 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:44.901070    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:45.741529 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:45.741488    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:45.742438 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:45.742421    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:29:47.316618 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:47.316575    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"]
Aug 05 17:29:47.317102 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:47.316841    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h" podUID="3fd2dd37-b748-4349-ac16-9e1511a26da4" containerName="istio-proxy" containerID="cri-o://c60f5cf2cc25ac2b709643b238c891cb22b466cb769bcc31a4bae623eb34533e" gracePeriod=30
Aug 05 17:29:50.084923 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.084879    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:29:50.177448 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.176255    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd2dd37-b748-4349-ac16-9e1511a26da4-workload-socket\") pod \"3fd2dd37-b748-4349-ac16-9e1511a26da4\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") "
Aug 05 17:29:50.177448 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.176390    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fd2dd37-b748-4349-ac16-9e1511a26da4-istio-podinfo\") pod \"3fd2dd37-b748-4349-ac16-9e1511a26da4\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") "
Aug 05 17:29:50.177448 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.176426    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fd2dd37-b748-4349-ac16-9e1511a26da4-istio-token\") pod \"3fd2dd37-b748-4349-ac16-9e1511a26da4\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") "
Aug 05 17:29:50.177448 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.176455    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd2dd37-b748-4349-ac16-9e1511a26da4-credential-socket\") pod \"3fd2dd37-b748-4349-ac16-9e1511a26da4\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") "
Aug 05 17:29:50.177448 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.176496    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fd2dd37-b748-4349-ac16-9e1511a26da4-istiod-ca-cert\") pod \"3fd2dd37-b748-4349-ac16-9e1511a26da4\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") "
Aug 05 17:29:50.177448 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.176564    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fd2dd37-b748-4349-ac16-9e1511a26da4-istio-data\") pod \"3fd2dd37-b748-4349-ac16-9e1511a26da4\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") "
Aug 05 17:29:50.177448 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.176597    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fd2dd37-b748-4349-ac16-9e1511a26da4-workload-certs\") pod \"3fd2dd37-b748-4349-ac16-9e1511a26da4\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") "
Aug 05 17:29:50.177448 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.176623    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fd2dd37-b748-4349-ac16-9e1511a26da4-istio-envoy\") pod \"3fd2dd37-b748-4349-ac16-9e1511a26da4\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") "
Aug 05 17:29:50.177448 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.176664    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59lcz\" (UniqueName: \"kubernetes.io/projected/3fd2dd37-b748-4349-ac16-9e1511a26da4-kube-api-access-59lcz\") pod \"3fd2dd37-b748-4349-ac16-9e1511a26da4\" (UID: \"3fd2dd37-b748-4349-ac16-9e1511a26da4\") "
Aug 05 17:29:50.178788 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.178528    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd2dd37-b748-4349-ac16-9e1511a26da4-istio-data" (OuterVolumeSpecName: "istio-data") pod "3fd2dd37-b748-4349-ac16-9e1511a26da4" (UID: "3fd2dd37-b748-4349-ac16-9e1511a26da4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:29:50.179386 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.179099    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fd2dd37-b748-4349-ac16-9e1511a26da4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3fd2dd37-b748-4349-ac16-9e1511a26da4" (UID: "3fd2dd37-b748-4349-ac16-9e1511a26da4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:29:50.179637 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.179563    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd2dd37-b748-4349-ac16-9e1511a26da4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3fd2dd37-b748-4349-ac16-9e1511a26da4" (UID: "3fd2dd37-b748-4349-ac16-9e1511a26da4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:29:50.180095 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.180062    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd2dd37-b748-4349-ac16-9e1511a26da4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3fd2dd37-b748-4349-ac16-9e1511a26da4" (UID: "3fd2dd37-b748-4349-ac16-9e1511a26da4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:29:50.180497 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.180474    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd2dd37-b748-4349-ac16-9e1511a26da4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3fd2dd37-b748-4349-ac16-9e1511a26da4" (UID: "3fd2dd37-b748-4349-ac16-9e1511a26da4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:29:50.183475 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.183437    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd2dd37-b748-4349-ac16-9e1511a26da4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3fd2dd37-b748-4349-ac16-9e1511a26da4" (UID: "3fd2dd37-b748-4349-ac16-9e1511a26da4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:29:50.183685 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.183659    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fd2dd37-b748-4349-ac16-9e1511a26da4-kube-api-access-59lcz" (OuterVolumeSpecName: "kube-api-access-59lcz") pod "3fd2dd37-b748-4349-ac16-9e1511a26da4" (UID: "3fd2dd37-b748-4349-ac16-9e1511a26da4"). InnerVolumeSpecName "kube-api-access-59lcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:29:50.185381 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.185352    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fd2dd37-b748-4349-ac16-9e1511a26da4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3fd2dd37-b748-4349-ac16-9e1511a26da4" (UID: "3fd2dd37-b748-4349-ac16-9e1511a26da4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Aug 05 17:29:50.185655 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.185631    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fd2dd37-b748-4349-ac16-9e1511a26da4-istio-token" (OuterVolumeSpecName: "istio-token") pod "3fd2dd37-b748-4349-ac16-9e1511a26da4" (UID: "3fd2dd37-b748-4349-ac16-9e1511a26da4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:29:50.278074 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.278022    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd2dd37-b748-4349-ac16-9e1511a26da4-credential-socket\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:29:50.278309 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.278087    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fd2dd37-b748-4349-ac16-9e1511a26da4-istiod-ca-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:29:50.278309 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.278107    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fd2dd37-b748-4349-ac16-9e1511a26da4-istio-data\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:29:50.278309 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.278122    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fd2dd37-b748-4349-ac16-9e1511a26da4-workload-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:29:50.278309 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.278136    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fd2dd37-b748-4349-ac16-9e1511a26da4-istio-envoy\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:29:50.278309 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.278155    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59lcz\" (UniqueName: \"kubernetes.io/projected/3fd2dd37-b748-4349-ac16-9e1511a26da4-kube-api-access-59lcz\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:29:50.278309 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.278171    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fd2dd37-b748-4349-ac16-9e1511a26da4-workload-socket\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:29:50.278309 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.278190    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fd2dd37-b748-4349-ac16-9e1511a26da4-istio-podinfo\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:29:50.278309 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.278206    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fd2dd37-b748-4349-ac16-9e1511a26da4-istio-token\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:29:50.775005 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.774968    2561 generic.go:356] "Generic (PLEG): container finished" podID="3fd2dd37-b748-4349-ac16-9e1511a26da4" containerID="c60f5cf2cc25ac2b709643b238c891cb22b466cb769bcc31a4bae623eb34533e" exitCode=0
Aug 05 17:29:50.775161 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.775026    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"
Aug 05 17:29:50.775161 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.775030    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h" event={"ID":"3fd2dd37-b748-4349-ac16-9e1511a26da4","Type":"ContainerDied","Data":"c60f5cf2cc25ac2b709643b238c891cb22b466cb769bcc31a4bae623eb34533e"}
Aug 05 17:29:50.775300 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.775173    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h" event={"ID":"3fd2dd37-b748-4349-ac16-9e1511a26da4","Type":"ContainerDied","Data":"0b1acfe02281acabe3358debd4618fe36cf00256a718ff9cd9d4781e154c575e"}
Aug 05 17:29:50.775300 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.775195    2561 scope.go:117] "RemoveContainer" containerID="c60f5cf2cc25ac2b709643b238c891cb22b466cb769bcc31a4bae623eb34533e"
Aug 05 17:29:50.784732 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.784714    2561 scope.go:117] "RemoveContainer" containerID="c60f5cf2cc25ac2b709643b238c891cb22b466cb769bcc31a4bae623eb34533e"
Aug 05 17:29:50.785009 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:29:50.784989    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c60f5cf2cc25ac2b709643b238c891cb22b466cb769bcc31a4bae623eb34533e\": container with ID starting with c60f5cf2cc25ac2b709643b238c891cb22b466cb769bcc31a4bae623eb34533e not found: ID does not exist" containerID="c60f5cf2cc25ac2b709643b238c891cb22b466cb769bcc31a4bae623eb34533e"
Aug 05 17:29:50.785074 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.785017    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c60f5cf2cc25ac2b709643b238c891cb22b466cb769bcc31a4bae623eb34533e"} err="failed to get container status \"c60f5cf2cc25ac2b709643b238c891cb22b466cb769bcc31a4bae623eb34533e\": rpc error: code = NotFound desc = could not find container \"c60f5cf2cc25ac2b709643b238c891cb22b466cb769bcc31a4bae623eb34533e\": container with ID starting with c60f5cf2cc25ac2b709643b238c891cb22b466cb769bcc31a4bae623eb34533e not found: ID does not exist"
Aug 05 17:29:50.795398 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.795372    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"]
Aug 05 17:29:50.799161 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:50.799138    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5944c46dd4-jt48h"]
Aug 05 17:29:52.730872 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:29:52.730839    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fd2dd37-b748-4349-ac16-9e1511a26da4" path="/var/lib/kubelet/pods/3fd2dd37-b748-4349-ac16-9e1511a26da4/volumes"
Aug 05 17:30:07.614908 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:07.614869    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Aug 05 17:30:12.450158 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.450119    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"]
Aug 05 17:30:12.450583 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.450453    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn" podUID="9da257a8-cc1d-4a85-ade1-171029eddcf2" containerName="istio-proxy" containerID="cri-o://b04b504035794f11d1dba3343a241d6ae1f3110bc9986443ae11da1a2d31ef97" gracePeriod=30
Aug 05 17:30:12.818872 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.818834    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"]
Aug 05 17:30:12.819429 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.819411    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fd2dd37-b748-4349-ac16-9e1511a26da4" containerName="istio-proxy"
Aug 05 17:30:12.819517 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.819431    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fd2dd37-b748-4349-ac16-9e1511a26da4" containerName="istio-proxy"
Aug 05 17:30:12.819566 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.819542    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fd2dd37-b748-4349-ac16-9e1511a26da4" containerName="istio-proxy"
Aug 05 17:30:12.824173 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.824146    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:30:12.827028 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.826826    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Aug 05 17:30:12.827028 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.826855    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Aug 05 17:30:12.827028 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.826868    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-58dwq\""
Aug 05 17:30:12.827028 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.826884    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Aug 05 17:30:12.827028 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.826902    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Aug 05 17:30:12.827321 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.827254    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Aug 05 17:30:12.836031 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.836009    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"]
Aug 05 17:30:12.884286 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.884243    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-tls-certs\") pod \"auth-enabled-test-kserve-6c5944c47b-5jxp8\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:30:12.884286 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.884289    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz29t\" (UniqueName: \"kubernetes.io/projected/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-kube-api-access-dz29t\") pod \"auth-enabled-test-kserve-6c5944c47b-5jxp8\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:30:12.884539 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.884323    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-cabundle-cert\") pod \"auth-enabled-test-kserve-6c5944c47b-5jxp8\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:30:12.884539 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.884353    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-home\") pod \"auth-enabled-test-kserve-6c5944c47b-5jxp8\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:30:12.884539 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.884374    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-tmp-dir\") pod \"auth-enabled-test-kserve-6c5944c47b-5jxp8\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:30:12.884539 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.884401    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-kserve-provision-location\") pod \"auth-enabled-test-kserve-6c5944c47b-5jxp8\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:30:12.884539 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.884427    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-dshm\") pod \"auth-enabled-test-kserve-6c5944c47b-5jxp8\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:30:12.884539 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.884471    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-model-cache\") pod \"auth-enabled-test-kserve-6c5944c47b-5jxp8\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:30:12.985321 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.985285    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz29t\" (UniqueName: \"kubernetes.io/projected/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-kube-api-access-dz29t\") pod \"auth-enabled-test-kserve-6c5944c47b-5jxp8\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:30:12.985321 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.985332    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-cabundle-cert\") pod \"auth-enabled-test-kserve-6c5944c47b-5jxp8\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:30:12.985577 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.985360    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-home\") pod \"auth-enabled-test-kserve-6c5944c47b-5jxp8\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:30:12.985577 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.985393    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-tmp-dir\") pod \"auth-enabled-test-kserve-6c5944c47b-5jxp8\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:30:12.985577 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.985443    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-kserve-provision-location\") pod \"auth-enabled-test-kserve-6c5944c47b-5jxp8\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:30:12.985577 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.985570    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-dshm\") pod \"auth-enabled-test-kserve-6c5944c47b-5jxp8\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:30:12.985810 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.985594    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-model-cache\") pod \"auth-enabled-test-kserve-6c5944c47b-5jxp8\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:30:12.985810 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.985668    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-tls-certs\") pod \"auth-enabled-test-kserve-6c5944c47b-5jxp8\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:30:12.985967 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.985930    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-kserve-provision-location\") pod \"auth-enabled-test-kserve-6c5944c47b-5jxp8\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:30:12.986028 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.986010    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-model-cache\") pod \"auth-enabled-test-kserve-6c5944c47b-5jxp8\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:30:12.986208 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.986173    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-cabundle-cert\") pod \"auth-enabled-test-kserve-6c5944c47b-5jxp8\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:30:12.986333 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.986252    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-tmp-dir\") pod \"auth-enabled-test-kserve-6c5944c47b-5jxp8\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:30:12.986333 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.986299    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-home\") pod \"auth-enabled-test-kserve-6c5944c47b-5jxp8\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:30:12.988444 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.988423    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-tls-certs\") pod \"auth-enabled-test-kserve-6c5944c47b-5jxp8\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:30:12.988541 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:12.988423    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-dshm\") pod \"auth-enabled-test-kserve-6c5944c47b-5jxp8\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:30:13.003546 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:13.003513    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz29t\" (UniqueName: \"kubernetes.io/projected/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-kube-api-access-dz29t\") pod \"auth-enabled-test-kserve-6c5944c47b-5jxp8\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:30:13.137310 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:13.137209    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:30:13.294659 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:13.294630    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"]
Aug 05 17:30:13.294998 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:30:13.294970    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e8d6602_6f6f_48a2_99b5_4d6905f42ea6.slice/crio-7360c1cb9bad64267eba7cf0ab965bbac7e7c465c41d95b0d509aa8a22f03da9 WatchSource:0}: Error finding container 7360c1cb9bad64267eba7cf0ab965bbac7e7c465c41d95b0d509aa8a22f03da9: Status 404 returned error can't find the container with id 7360c1cb9bad64267eba7cf0ab965bbac7e7c465c41d95b0d509aa8a22f03da9
Aug 05 17:30:13.918100 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:13.918060    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8" event={"ID":"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6","Type":"ContainerStarted","Data":"23fe3867b892c778f54446d0074fc1a84b6f4b2591766f5fb73122f1b4bb9025"}
Aug 05 17:30:13.918523 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:13.918123    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8" event={"ID":"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6","Type":"ContainerStarted","Data":"7360c1cb9bad64267eba7cf0ab965bbac7e7c465c41d95b0d509aa8a22f03da9"}
Aug 05 17:30:15.249027 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.248988    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:30:15.407966 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.407916    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9da257a8-cc1d-4a85-ade1-171029eddcf2-istio-envoy\") pod \"9da257a8-cc1d-4a85-ade1-171029eddcf2\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") "
Aug 05 17:30:15.408143 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.407975    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9da257a8-cc1d-4a85-ade1-171029eddcf2-istio-podinfo\") pod \"9da257a8-cc1d-4a85-ade1-171029eddcf2\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") "
Aug 05 17:30:15.408143 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.407997    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9da257a8-cc1d-4a85-ade1-171029eddcf2-istiod-ca-cert\") pod \"9da257a8-cc1d-4a85-ade1-171029eddcf2\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") "
Aug 05 17:30:15.408143 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.408017    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da257a8-cc1d-4a85-ade1-171029eddcf2-credential-socket\") pod \"9da257a8-cc1d-4a85-ade1-171029eddcf2\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") "
Aug 05 17:30:15.408143 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.408061    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tthtv\" (UniqueName: \"kubernetes.io/projected/9da257a8-cc1d-4a85-ade1-171029eddcf2-kube-api-access-tthtv\") pod \"9da257a8-cc1d-4a85-ade1-171029eddcf2\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") "
Aug 05 17:30:15.408353 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.408154    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9da257a8-cc1d-4a85-ade1-171029eddcf2-workload-certs\") pod \"9da257a8-cc1d-4a85-ade1-171029eddcf2\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") "
Aug 05 17:30:15.408353 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.408205    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9da257a8-cc1d-4a85-ade1-171029eddcf2-istio-token\") pod \"9da257a8-cc1d-4a85-ade1-171029eddcf2\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") "
Aug 05 17:30:15.408353 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.408262    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9da257a8-cc1d-4a85-ade1-171029eddcf2-istio-data\") pod \"9da257a8-cc1d-4a85-ade1-171029eddcf2\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") "
Aug 05 17:30:15.408353 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.408314    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da257a8-cc1d-4a85-ade1-171029eddcf2-workload-socket\") pod \"9da257a8-cc1d-4a85-ade1-171029eddcf2\" (UID: \"9da257a8-cc1d-4a85-ade1-171029eddcf2\") "
Aug 05 17:30:15.408353 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.408311    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da257a8-cc1d-4a85-ade1-171029eddcf2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9da257a8-cc1d-4a85-ade1-171029eddcf2" (UID: "9da257a8-cc1d-4a85-ade1-171029eddcf2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:30:15.408614 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.408399    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9da257a8-cc1d-4a85-ade1-171029eddcf2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9da257a8-cc1d-4a85-ade1-171029eddcf2" (UID: "9da257a8-cc1d-4a85-ade1-171029eddcf2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:30:15.408614 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.408400    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da257a8-cc1d-4a85-ade1-171029eddcf2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9da257a8-cc1d-4a85-ade1-171029eddcf2" (UID: "9da257a8-cc1d-4a85-ade1-171029eddcf2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:30:15.408614 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.408554    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da257a8-cc1d-4a85-ade1-171029eddcf2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9da257a8-cc1d-4a85-ade1-171029eddcf2" (UID: "9da257a8-cc1d-4a85-ade1-171029eddcf2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:30:15.408614 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.408570    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da257a8-cc1d-4a85-ade1-171029eddcf2-istio-data" (OuterVolumeSpecName: "istio-data") pod "9da257a8-cc1d-4a85-ade1-171029eddcf2" (UID: "9da257a8-cc1d-4a85-ade1-171029eddcf2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:30:15.408796 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.408645    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9da257a8-cc1d-4a85-ade1-171029eddcf2-workload-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:30:15.408796 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.408664    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9da257a8-cc1d-4a85-ade1-171029eddcf2-istio-data\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:30:15.408796 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.408677    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da257a8-cc1d-4a85-ade1-171029eddcf2-workload-socket\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:30:15.408796 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.408691    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9da257a8-cc1d-4a85-ade1-171029eddcf2-istiod-ca-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:30:15.408796 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.408705    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da257a8-cc1d-4a85-ade1-171029eddcf2-credential-socket\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:30:15.410392 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.410366    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da257a8-cc1d-4a85-ade1-171029eddcf2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9da257a8-cc1d-4a85-ade1-171029eddcf2" (UID: "9da257a8-cc1d-4a85-ade1-171029eddcf2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:30:15.410527 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.410415    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9da257a8-cc1d-4a85-ade1-171029eddcf2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9da257a8-cc1d-4a85-ade1-171029eddcf2" (UID: "9da257a8-cc1d-4a85-ade1-171029eddcf2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Aug 05 17:30:15.410527 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.410490    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9da257a8-cc1d-4a85-ade1-171029eddcf2-kube-api-access-tthtv" (OuterVolumeSpecName: "kube-api-access-tthtv") pod "9da257a8-cc1d-4a85-ade1-171029eddcf2" (UID: "9da257a8-cc1d-4a85-ade1-171029eddcf2"). InnerVolumeSpecName "kube-api-access-tthtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:30:15.410527 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.410506    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9da257a8-cc1d-4a85-ade1-171029eddcf2-istio-token" (OuterVolumeSpecName: "istio-token") pod "9da257a8-cc1d-4a85-ade1-171029eddcf2" (UID: "9da257a8-cc1d-4a85-ade1-171029eddcf2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:30:15.509237 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.509197    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9da257a8-cc1d-4a85-ade1-171029eddcf2-istio-podinfo\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:30:15.509237 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.509231    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tthtv\" (UniqueName: \"kubernetes.io/projected/9da257a8-cc1d-4a85-ade1-171029eddcf2-kube-api-access-tthtv\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:30:15.509237 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.509244    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9da257a8-cc1d-4a85-ade1-171029eddcf2-istio-token\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:30:15.509480 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.509256    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9da257a8-cc1d-4a85-ade1-171029eddcf2-istio-envoy\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:30:15.930356 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.930323    2561 generic.go:356] "Generic (PLEG): container finished" podID="2e8d6602-6f6f-48a2-99b5-4d6905f42ea6" containerID="23fe3867b892c778f54446d0074fc1a84b6f4b2591766f5fb73122f1b4bb9025" exitCode=0
Aug 05 17:30:15.930543 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.930395    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8" event={"ID":"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6","Type":"ContainerDied","Data":"23fe3867b892c778f54446d0074fc1a84b6f4b2591766f5fb73122f1b4bb9025"}
Aug 05 17:30:15.931689 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.931665    2561 generic.go:356] "Generic (PLEG): container finished" podID="9da257a8-cc1d-4a85-ade1-171029eddcf2" containerID="b04b504035794f11d1dba3343a241d6ae1f3110bc9986443ae11da1a2d31ef97" exitCode=0
Aug 05 17:30:15.931789 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.931709    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"
Aug 05 17:30:15.931789 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.931738    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn" event={"ID":"9da257a8-cc1d-4a85-ade1-171029eddcf2","Type":"ContainerDied","Data":"b04b504035794f11d1dba3343a241d6ae1f3110bc9986443ae11da1a2d31ef97"}
Aug 05 17:30:15.931789 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.931782    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn" event={"ID":"9da257a8-cc1d-4a85-ade1-171029eddcf2","Type":"ContainerDied","Data":"347a46c6d3e80352a6e3ef1d7e85c2be49c27d060ffbe703f642c37ea727d4b3"}
Aug 05 17:30:15.931946 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.931805    2561 scope.go:117] "RemoveContainer" containerID="b04b504035794f11d1dba3343a241d6ae1f3110bc9986443ae11da1a2d31ef97"
Aug 05 17:30:15.944150 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.944123    2561 scope.go:117] "RemoveContainer" containerID="b04b504035794f11d1dba3343a241d6ae1f3110bc9986443ae11da1a2d31ef97"
Aug 05 17:30:15.944408 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:30:15.944390    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b04b504035794f11d1dba3343a241d6ae1f3110bc9986443ae11da1a2d31ef97\": container with ID starting with b04b504035794f11d1dba3343a241d6ae1f3110bc9986443ae11da1a2d31ef97 not found: ID does not exist" containerID="b04b504035794f11d1dba3343a241d6ae1f3110bc9986443ae11da1a2d31ef97"
Aug 05 17:30:15.944468 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.944416    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b04b504035794f11d1dba3343a241d6ae1f3110bc9986443ae11da1a2d31ef97"} err="failed to get container status \"b04b504035794f11d1dba3343a241d6ae1f3110bc9986443ae11da1a2d31ef97\": rpc error: code = NotFound desc = could not find container \"b04b504035794f11d1dba3343a241d6ae1f3110bc9986443ae11da1a2d31ef97\": container with ID starting with b04b504035794f11d1dba3343a241d6ae1f3110bc9986443ae11da1a2d31ef97 not found: ID does not exist"
Aug 05 17:30:15.968065 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.968033    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"]
Aug 05 17:30:15.972097 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:15.972069    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-78668f9df6-tlgzn"]
Aug 05 17:30:16.730334 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:16.730258    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9da257a8-cc1d-4a85-ade1-171029eddcf2" path="/var/lib/kubelet/pods/9da257a8-cc1d-4a85-ade1-171029eddcf2/volumes"
Aug 05 17:30:16.939735 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:16.939697    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8" event={"ID":"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6","Type":"ContainerStarted","Data":"fafc6fe4fc9aa3a5e3abc96a7507bf4b1b8744d656dd2985db06ae33b40f9ed9"}
Aug 05 17:30:16.965431 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:16.965368    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8" podStartSLOduration=2.520899359 podStartE2EDuration="4.965350897s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.444451538s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:30:16.962213624 +0000 UTC m=+1072.706847951" watchObservedRunningTime="2026-08-05 17:30:16.965350897 +0000 UTC m=+1072.709985223"
Aug 05 17:30:17.137554 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:17.137503    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:30:22.639857 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:22.639817    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="a8f16bcc-d578-4678-9744-e8133174fc60" containerName="main" containerID="cri-o://c257702af7b9bd0e935f2d93fa933fa85db707f1218667f1a72c3ea776a82aa1" gracePeriod=45
Aug 05 17:30:23.033071 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.033028    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="a8f16bcc-d578-4678-9744-e8133174fc60" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Aug 05 17:30:23.137666 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.137632    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:30:23.139498 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.139464    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8" podUID="2e8d6602-6f6f-48a2-99b5-4d6905f42ea6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Aug 05 17:30:23.392792 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.392766    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:30:23.467214 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.467181    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8f16bcc-d578-4678-9744-e8133174fc60-kserve-provision-location\") pod \"a8f16bcc-d578-4678-9744-e8133174fc60\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") "
Aug 05 17:30:23.467214 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.467214    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8f16bcc-d578-4678-9744-e8133174fc60-dshm\") pod \"a8f16bcc-d578-4678-9744-e8133174fc60\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") "
Aug 05 17:30:23.467474 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.467241    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8f16bcc-d578-4678-9744-e8133174fc60-cabundle-cert\") pod \"a8f16bcc-d578-4678-9744-e8133174fc60\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") "
Aug 05 17:30:23.467474 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.467266    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8f16bcc-d578-4678-9744-e8133174fc60-tmp-dir\") pod \"a8f16bcc-d578-4678-9744-e8133174fc60\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") "
Aug 05 17:30:23.467474 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.467328    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfzvm\" (UniqueName: \"kubernetes.io/projected/a8f16bcc-d578-4678-9744-e8133174fc60-kube-api-access-gfzvm\") pod \"a8f16bcc-d578-4678-9744-e8133174fc60\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") "
Aug 05 17:30:23.467474 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.467354    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8f16bcc-d578-4678-9744-e8133174fc60-home\") pod \"a8f16bcc-d578-4678-9744-e8133174fc60\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") "
Aug 05 17:30:23.467474 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.467377    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8f16bcc-d578-4678-9744-e8133174fc60-tls-certs\") pod \"a8f16bcc-d578-4678-9744-e8133174fc60\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") "
Aug 05 17:30:23.467474 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.467415    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8f16bcc-d578-4678-9744-e8133174fc60-model-cache\") pod \"a8f16bcc-d578-4678-9744-e8133174fc60\" (UID: \"a8f16bcc-d578-4678-9744-e8133174fc60\") "
Aug 05 17:30:23.467791 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.467557    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8f16bcc-d578-4678-9744-e8133174fc60-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a8f16bcc-d578-4678-9744-e8133174fc60" (UID: "a8f16bcc-d578-4678-9744-e8133174fc60"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:30:23.468205 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.467911    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f16bcc-d578-4678-9744-e8133174fc60-model-cache" (OuterVolumeSpecName: "model-cache") pod "a8f16bcc-d578-4678-9744-e8133174fc60" (UID: "a8f16bcc-d578-4678-9744-e8133174fc60"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:30:23.468205 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.468078    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8f16bcc-d578-4678-9744-e8133174fc60-cabundle-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:30:23.468205 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.468095    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8f16bcc-d578-4678-9744-e8133174fc60-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:30:23.468205 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.468155    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f16bcc-d578-4678-9744-e8133174fc60-home" (OuterVolumeSpecName: "home") pod "a8f16bcc-d578-4678-9744-e8133174fc60" (UID: "a8f16bcc-d578-4678-9744-e8133174fc60"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:30:23.469724 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.469688    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f16bcc-d578-4678-9744-e8133174fc60-kube-api-access-gfzvm" (OuterVolumeSpecName: "kube-api-access-gfzvm") pod "a8f16bcc-d578-4678-9744-e8133174fc60" (UID: "a8f16bcc-d578-4678-9744-e8133174fc60"). InnerVolumeSpecName "kube-api-access-gfzvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:30:23.469857 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.469811    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f16bcc-d578-4678-9744-e8133174fc60-dshm" (OuterVolumeSpecName: "dshm") pod "a8f16bcc-d578-4678-9744-e8133174fc60" (UID: "a8f16bcc-d578-4678-9744-e8133174fc60"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:30:23.469984 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.469940    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8f16bcc-d578-4678-9744-e8133174fc60-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a8f16bcc-d578-4678-9744-e8133174fc60" (UID: "a8f16bcc-d578-4678-9744-e8133174fc60"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:30:23.482920 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.482876    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f16bcc-d578-4678-9744-e8133174fc60-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a8f16bcc-d578-4678-9744-e8133174fc60" (UID: "a8f16bcc-d578-4678-9744-e8133174fc60"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:30:23.487091 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.487068    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f16bcc-d578-4678-9744-e8133174fc60-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a8f16bcc-d578-4678-9744-e8133174fc60" (UID: "a8f16bcc-d578-4678-9744-e8133174fc60"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:30:23.568659 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.568567    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8f16bcc-d578-4678-9744-e8133174fc60-kserve-provision-location\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:30:23.568659 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.568597    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8f16bcc-d578-4678-9744-e8133174fc60-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:30:23.568659 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.568608    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8f16bcc-d578-4678-9744-e8133174fc60-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:30:23.568659 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.568618    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfzvm\" (UniqueName: \"kubernetes.io/projected/a8f16bcc-d578-4678-9744-e8133174fc60-kube-api-access-gfzvm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:30:23.568659 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.568626    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8f16bcc-d578-4678-9744-e8133174fc60-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:30:23.568659 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.568634    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8f16bcc-d578-4678-9744-e8133174fc60-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:30:23.982873 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.982783    2561 generic.go:356] "Generic (PLEG): container finished" podID="a8f16bcc-d578-4678-9744-e8133174fc60" containerID="c257702af7b9bd0e935f2d93fa933fa85db707f1218667f1a72c3ea776a82aa1" exitCode=0
Aug 05 17:30:23.982873 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.982834    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Aug 05 17:30:23.983394 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.982868    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"a8f16bcc-d578-4678-9744-e8133174fc60","Type":"ContainerDied","Data":"c257702af7b9bd0e935f2d93fa933fa85db707f1218667f1a72c3ea776a82aa1"}
Aug 05 17:30:23.983394 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.982931    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"a8f16bcc-d578-4678-9744-e8133174fc60","Type":"ContainerDied","Data":"df3a18bbf20e245b8e046643f9f7b8776a9b5afceee158cfed968ae00fb23ab0"}
Aug 05 17:30:23.983394 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.982970    2561 scope.go:117] "RemoveContainer" containerID="c257702af7b9bd0e935f2d93fa933fa85db707f1218667f1a72c3ea776a82aa1"
Aug 05 17:30:23.993261 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:23.993240    2561 scope.go:117] "RemoveContainer" containerID="6e313310abe1cdd7ea3bae7309530fc74e6536fb25db33780f5a8c6403366c6f"
Aug 05 17:30:24.001522 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:24.001501    2561 scope.go:117] "RemoveContainer" containerID="c257702af7b9bd0e935f2d93fa933fa85db707f1218667f1a72c3ea776a82aa1"
Aug 05 17:30:24.001822 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:30:24.001800    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c257702af7b9bd0e935f2d93fa933fa85db707f1218667f1a72c3ea776a82aa1\": container with ID starting with c257702af7b9bd0e935f2d93fa933fa85db707f1218667f1a72c3ea776a82aa1 not found: ID does not exist" containerID="c257702af7b9bd0e935f2d93fa933fa85db707f1218667f1a72c3ea776a82aa1"
Aug 05 17:30:24.001929 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:24.001828    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c257702af7b9bd0e935f2d93fa933fa85db707f1218667f1a72c3ea776a82aa1"} err="failed to get container status \"c257702af7b9bd0e935f2d93fa933fa85db707f1218667f1a72c3ea776a82aa1\": rpc error: code = NotFound desc = could not find container \"c257702af7b9bd0e935f2d93fa933fa85db707f1218667f1a72c3ea776a82aa1\": container with ID starting with c257702af7b9bd0e935f2d93fa933fa85db707f1218667f1a72c3ea776a82aa1 not found: ID does not exist"
Aug 05 17:30:24.001929 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:24.001848    2561 scope.go:117] "RemoveContainer" containerID="6e313310abe1cdd7ea3bae7309530fc74e6536fb25db33780f5a8c6403366c6f"
Aug 05 17:30:24.002151 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:30:24.002127    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e313310abe1cdd7ea3bae7309530fc74e6536fb25db33780f5a8c6403366c6f\": container with ID starting with 6e313310abe1cdd7ea3bae7309530fc74e6536fb25db33780f5a8c6403366c6f not found: ID does not exist" containerID="6e313310abe1cdd7ea3bae7309530fc74e6536fb25db33780f5a8c6403366c6f"
Aug 05 17:30:24.002198 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:24.002160    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e313310abe1cdd7ea3bae7309530fc74e6536fb25db33780f5a8c6403366c6f"} err="failed to get container status \"6e313310abe1cdd7ea3bae7309530fc74e6536fb25db33780f5a8c6403366c6f\": rpc error: code = NotFound desc = could not find container \"6e313310abe1cdd7ea3bae7309530fc74e6536fb25db33780f5a8c6403366c6f\": container with ID starting with 6e313310abe1cdd7ea3bae7309530fc74e6536fb25db33780f5a8c6403366c6f not found: ID does not exist"
Aug 05 17:30:24.008084 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:24.008059    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Aug 05 17:30:24.012759 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:24.012736    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Aug 05 17:30:24.737134 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:24.737101    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8f16bcc-d578-4678-9744-e8133174fc60" path="/var/lib/kubelet/pods/a8f16bcc-d578-4678-9744-e8133174fc60/volumes"
Aug 05 17:30:33.138510 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:33.138464    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8" podUID="2e8d6602-6f6f-48a2-99b5-4d6905f42ea6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Aug 05 17:30:43.138561 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:43.138512    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8" podUID="2e8d6602-6f6f-48a2-99b5-4d6905f42ea6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Aug 05 17:30:53.138634 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:30:53.138592    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8" podUID="2e8d6602-6f6f-48a2-99b5-4d6905f42ea6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Aug 05 17:31:03.138556 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:03.138509    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8" podUID="2e8d6602-6f6f-48a2-99b5-4d6905f42ea6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Aug 05 17:31:13.138037 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:13.137985    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8" podUID="2e8d6602-6f6f-48a2-99b5-4d6905f42ea6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Aug 05 17:31:17.240368 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.240331    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"]
Aug 05 17:31:17.240880 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.240860    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9da257a8-cc1d-4a85-ade1-171029eddcf2" containerName="istio-proxy"
Aug 05 17:31:17.240982 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.240883    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9da257a8-cc1d-4a85-ade1-171029eddcf2" containerName="istio-proxy"
Aug 05 17:31:17.240982 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.240896    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8f16bcc-d578-4678-9744-e8133174fc60" containerName="storage-initializer"
Aug 05 17:31:17.240982 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.240906    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8f16bcc-d578-4678-9744-e8133174fc60" containerName="storage-initializer"
Aug 05 17:31:17.240982 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.240918    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8f16bcc-d578-4678-9744-e8133174fc60" containerName="main"
Aug 05 17:31:17.240982 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.240926    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8f16bcc-d578-4678-9744-e8133174fc60" containerName="main"
Aug 05 17:31:17.241228 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.241041    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8f16bcc-d578-4678-9744-e8133174fc60" containerName="main"
Aug 05 17:31:17.241228 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.241056    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9da257a8-cc1d-4a85-ade1-171029eddcf2" containerName="istio-proxy"
Aug 05 17:31:17.244346 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.244324    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"
Aug 05 17:31:17.251342 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.251319    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Aug 05 17:31:17.251465 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.251318    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Aug 05 17:31:17.251465 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.251394    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Aug 05 17:31:17.251465 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.251442    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-2v9qm\""
Aug 05 17:31:17.257343 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.257321    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"]
Aug 05 17:31:17.357556 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.357516    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c86f7b5-9abd-4d45-901d-856848f8bbd6-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-wbwld\" (UID: \"1c86f7b5-9abd-4d45-901d-856848f8bbd6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"
Aug 05 17:31:17.357556 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.357557    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c86f7b5-9abd-4d45-901d-856848f8bbd6-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-wbwld\" (UID: \"1c86f7b5-9abd-4d45-901d-856848f8bbd6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"
Aug 05 17:31:17.357825 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.357585    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c86f7b5-9abd-4d45-901d-856848f8bbd6-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-wbwld\" (UID: \"1c86f7b5-9abd-4d45-901d-856848f8bbd6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"
Aug 05 17:31:17.357825 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.357680    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c86f7b5-9abd-4d45-901d-856848f8bbd6-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-wbwld\" (UID: \"1c86f7b5-9abd-4d45-901d-856848f8bbd6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"
Aug 05 17:31:17.357825 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.357742    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c86f7b5-9abd-4d45-901d-856848f8bbd6-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-wbwld\" (UID: \"1c86f7b5-9abd-4d45-901d-856848f8bbd6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"
Aug 05 17:31:17.357825 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.357771    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtc7f\" (UniqueName: \"kubernetes.io/projected/1c86f7b5-9abd-4d45-901d-856848f8bbd6-kube-api-access-vtc7f\") pod \"scheduler-inline-config-test-kserve-8655bc757-wbwld\" (UID: \"1c86f7b5-9abd-4d45-901d-856848f8bbd6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"
Aug 05 17:31:17.458502 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.458460    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c86f7b5-9abd-4d45-901d-856848f8bbd6-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-wbwld\" (UID: \"1c86f7b5-9abd-4d45-901d-856848f8bbd6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"
Aug 05 17:31:17.458502 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.458504    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c86f7b5-9abd-4d45-901d-856848f8bbd6-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-wbwld\" (UID: \"1c86f7b5-9abd-4d45-901d-856848f8bbd6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"
Aug 05 17:31:17.458788 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.458544    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c86f7b5-9abd-4d45-901d-856848f8bbd6-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-wbwld\" (UID: \"1c86f7b5-9abd-4d45-901d-856848f8bbd6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"
Aug 05 17:31:17.458788 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.458592    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c86f7b5-9abd-4d45-901d-856848f8bbd6-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-wbwld\" (UID: \"1c86f7b5-9abd-4d45-901d-856848f8bbd6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"
Aug 05 17:31:17.458788 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.458641    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c86f7b5-9abd-4d45-901d-856848f8bbd6-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-wbwld\" (UID: \"1c86f7b5-9abd-4d45-901d-856848f8bbd6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"
Aug 05 17:31:17.458788 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.458663    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtc7f\" (UniqueName: \"kubernetes.io/projected/1c86f7b5-9abd-4d45-901d-856848f8bbd6-kube-api-access-vtc7f\") pod \"scheduler-inline-config-test-kserve-8655bc757-wbwld\" (UID: \"1c86f7b5-9abd-4d45-901d-856848f8bbd6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"
Aug 05 17:31:17.459063 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.458989    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c86f7b5-9abd-4d45-901d-856848f8bbd6-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-wbwld\" (UID: \"1c86f7b5-9abd-4d45-901d-856848f8bbd6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"
Aug 05 17:31:17.459124 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.459094    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c86f7b5-9abd-4d45-901d-856848f8bbd6-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-wbwld\" (UID: \"1c86f7b5-9abd-4d45-901d-856848f8bbd6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"
Aug 05 17:31:17.459296 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.459265    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c86f7b5-9abd-4d45-901d-856848f8bbd6-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-wbwld\" (UID: \"1c86f7b5-9abd-4d45-901d-856848f8bbd6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"
Aug 05 17:31:17.461044 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.461021    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c86f7b5-9abd-4d45-901d-856848f8bbd6-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-wbwld\" (UID: \"1c86f7b5-9abd-4d45-901d-856848f8bbd6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"
Aug 05 17:31:17.461426 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.461384    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c86f7b5-9abd-4d45-901d-856848f8bbd6-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-wbwld\" (UID: \"1c86f7b5-9abd-4d45-901d-856848f8bbd6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"
Aug 05 17:31:17.469832 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.469780    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtc7f\" (UniqueName: \"kubernetes.io/projected/1c86f7b5-9abd-4d45-901d-856848f8bbd6-kube-api-access-vtc7f\") pod \"scheduler-inline-config-test-kserve-8655bc757-wbwld\" (UID: \"1c86f7b5-9abd-4d45-901d-856848f8bbd6\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"
Aug 05 17:31:17.554452 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.554420    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"
Aug 05 17:31:17.699418 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:17.699273    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"]
Aug 05 17:31:17.702270 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:31:17.702226    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c86f7b5_9abd_4d45_901d_856848f8bbd6.slice/crio-2bde790b929f000aa430a81229d180899a3318446a508c15d5c7150b5172585d WatchSource:0}: Error finding container 2bde790b929f000aa430a81229d180899a3318446a508c15d5c7150b5172585d: Status 404 returned error can't find the container with id 2bde790b929f000aa430a81229d180899a3318446a508c15d5c7150b5172585d
Aug 05 17:31:18.288750 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:18.288712    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld" event={"ID":"1c86f7b5-9abd-4d45-901d-856848f8bbd6","Type":"ContainerStarted","Data":"8c4817c77be16bbec489419d72a75236ab92b291d813a05e9e29adcc02d30d18"}
Aug 05 17:31:18.289278 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:18.288773    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld" event={"ID":"1c86f7b5-9abd-4d45-901d-856848f8bbd6","Type":"ContainerStarted","Data":"2bde790b929f000aa430a81229d180899a3318446a508c15d5c7150b5172585d"}
Aug 05 17:31:18.307718 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:18.307652    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld" podStartSLOduration=1.307632917 podStartE2EDuration="1.307632917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:31:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:31:18.307084604 +0000 UTC m=+1134.051718935" watchObservedRunningTime="2026-08-05 17:31:18.307632917 +0000 UTC m=+1134.052267244"
Aug 05 17:31:18.554992 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:18.554874    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"
Aug 05 17:31:23.138589 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:23.138541    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8" podUID="2e8d6602-6f6f-48a2-99b5-4d6905f42ea6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Aug 05 17:31:27.555033 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:27.554993    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"
Aug 05 17:31:27.567731 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:27.567704    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"
Aug 05 17:31:28.369397 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:28.369362    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"
Aug 05 17:31:33.138498 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:33.138452    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8" podUID="2e8d6602-6f6f-48a2-99b5-4d6905f42ea6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Aug 05 17:31:43.138679 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:43.138631    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8" podUID="2e8d6602-6f6f-48a2-99b5-4d6905f42ea6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Aug 05 17:31:50.544305 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:50.544221    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"]
Aug 05 17:31:50.553764 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:31:50.553712    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 05 17:31:50.553764 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:31:50.553764 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:31:50.553764 ip-10-0-135-151 kubenswrapper[2561]:  > containerID="8c4817c77be16bbec489419d72a75236ab92b291d813a05e9e29adcc02d30d18" cmd=["/bin/sleep","15"]
Aug 05 17:31:50.554107 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:50.553793    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld" message="" err=<
Aug 05 17:31:50.554107 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:31:50.554107 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:31:50.554107 ip-10-0-135-151 kubenswrapper[2561]:  >
Aug 05 17:31:50.554107 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:31:50.553809    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 05 17:31:50.554107 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:31:50.554107 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:31:50.554107 ip-10-0-135-151 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld" podUID="1c86f7b5-9abd-4d45-901d-856848f8bbd6" containerName="main" containerID="cri-o://8c4817c77be16bbec489419d72a75236ab92b291d813a05e9e29adcc02d30d18"
Aug 05 17:31:50.554107 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:50.553871    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld" podUID="1c86f7b5-9abd-4d45-901d-856848f8bbd6" containerName="main" containerID="cri-o://8c4817c77be16bbec489419d72a75236ab92b291d813a05e9e29adcc02d30d18" gracePeriod=60
Aug 05 17:31:50.920070 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:50.920034    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"
Aug 05 17:31:50.984096 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:50.984062    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtc7f\" (UniqueName: \"kubernetes.io/projected/1c86f7b5-9abd-4d45-901d-856848f8bbd6-kube-api-access-vtc7f\") pod \"1c86f7b5-9abd-4d45-901d-856848f8bbd6\" (UID: \"1c86f7b5-9abd-4d45-901d-856848f8bbd6\") "
Aug 05 17:31:50.984277 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:50.984160    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c86f7b5-9abd-4d45-901d-856848f8bbd6-home\") pod \"1c86f7b5-9abd-4d45-901d-856848f8bbd6\" (UID: \"1c86f7b5-9abd-4d45-901d-856848f8bbd6\") "
Aug 05 17:31:50.984277 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:50.984185    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c86f7b5-9abd-4d45-901d-856848f8bbd6-model-cache\") pod \"1c86f7b5-9abd-4d45-901d-856848f8bbd6\" (UID: \"1c86f7b5-9abd-4d45-901d-856848f8bbd6\") "
Aug 05 17:31:50.984277 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:50.984232    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c86f7b5-9abd-4d45-901d-856848f8bbd6-dshm\") pod \"1c86f7b5-9abd-4d45-901d-856848f8bbd6\" (UID: \"1c86f7b5-9abd-4d45-901d-856848f8bbd6\") "
Aug 05 17:31:50.984277 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:50.984255    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c86f7b5-9abd-4d45-901d-856848f8bbd6-tmp-dir\") pod \"1c86f7b5-9abd-4d45-901d-856848f8bbd6\" (UID: \"1c86f7b5-9abd-4d45-901d-856848f8bbd6\") "
Aug 05 17:31:50.984508 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:50.984289    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c86f7b5-9abd-4d45-901d-856848f8bbd6-tls-certs\") pod \"1c86f7b5-9abd-4d45-901d-856848f8bbd6\" (UID: \"1c86f7b5-9abd-4d45-901d-856848f8bbd6\") "
Aug 05 17:31:50.984508 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:50.984407    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c86f7b5-9abd-4d45-901d-856848f8bbd6-home" (OuterVolumeSpecName: "home") pod "1c86f7b5-9abd-4d45-901d-856848f8bbd6" (UID: "1c86f7b5-9abd-4d45-901d-856848f8bbd6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:31:50.984508 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:50.984445    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c86f7b5-9abd-4d45-901d-856848f8bbd6-model-cache" (OuterVolumeSpecName: "model-cache") pod "1c86f7b5-9abd-4d45-901d-856848f8bbd6" (UID: "1c86f7b5-9abd-4d45-901d-856848f8bbd6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:31:50.984508 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:50.984501    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c86f7b5-9abd-4d45-901d-856848f8bbd6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1c86f7b5-9abd-4d45-901d-856848f8bbd6" (UID: "1c86f7b5-9abd-4d45-901d-856848f8bbd6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:31:50.984709 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:50.984616    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c86f7b5-9abd-4d45-901d-856848f8bbd6-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:31:50.984709 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:50.984630    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c86f7b5-9abd-4d45-901d-856848f8bbd6-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:31:50.984709 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:50.984643    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c86f7b5-9abd-4d45-901d-856848f8bbd6-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:31:50.986418 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:50.986382    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c86f7b5-9abd-4d45-901d-856848f8bbd6-dshm" (OuterVolumeSpecName: "dshm") pod "1c86f7b5-9abd-4d45-901d-856848f8bbd6" (UID: "1c86f7b5-9abd-4d45-901d-856848f8bbd6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:31:50.986624 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:50.986607    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c86f7b5-9abd-4d45-901d-856848f8bbd6-kube-api-access-vtc7f" (OuterVolumeSpecName: "kube-api-access-vtc7f") pod "1c86f7b5-9abd-4d45-901d-856848f8bbd6" (UID: "1c86f7b5-9abd-4d45-901d-856848f8bbd6"). InnerVolumeSpecName "kube-api-access-vtc7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:31:50.986684 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:50.986670    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c86f7b5-9abd-4d45-901d-856848f8bbd6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c86f7b5-9abd-4d45-901d-856848f8bbd6" (UID: "1c86f7b5-9abd-4d45-901d-856848f8bbd6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:31:51.085538 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:51.085456    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c86f7b5-9abd-4d45-901d-856848f8bbd6-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:31:51.085538 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:51.085493    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c86f7b5-9abd-4d45-901d-856848f8bbd6-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:31:51.085538 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:51.085511    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtc7f\" (UniqueName: \"kubernetes.io/projected/1c86f7b5-9abd-4d45-901d-856848f8bbd6-kube-api-access-vtc7f\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:31:51.488048 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:51.487943    2561 generic.go:356] "Generic (PLEG): container finished" podID="1c86f7b5-9abd-4d45-901d-856848f8bbd6" containerID="8c4817c77be16bbec489419d72a75236ab92b291d813a05e9e29adcc02d30d18" exitCode=0
Aug 05 17:31:51.488048 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:51.487981    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld" event={"ID":"1c86f7b5-9abd-4d45-901d-856848f8bbd6","Type":"ContainerDied","Data":"8c4817c77be16bbec489419d72a75236ab92b291d813a05e9e29adcc02d30d18"}
Aug 05 17:31:51.488048 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:51.488009    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"
Aug 05 17:31:51.488048 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:51.488043    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld" event={"ID":"1c86f7b5-9abd-4d45-901d-856848f8bbd6","Type":"ContainerDied","Data":"2bde790b929f000aa430a81229d180899a3318446a508c15d5c7150b5172585d"}
Aug 05 17:31:51.488348 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:51.488070    2561 scope.go:117] "RemoveContainer" containerID="8c4817c77be16bbec489419d72a75236ab92b291d813a05e9e29adcc02d30d18"
Aug 05 17:31:51.499151 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:51.499131    2561 scope.go:117] "RemoveContainer" containerID="8c4817c77be16bbec489419d72a75236ab92b291d813a05e9e29adcc02d30d18"
Aug 05 17:31:51.499422 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:31:51.499395    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c4817c77be16bbec489419d72a75236ab92b291d813a05e9e29adcc02d30d18\": container with ID starting with 8c4817c77be16bbec489419d72a75236ab92b291d813a05e9e29adcc02d30d18 not found: ID does not exist" containerID="8c4817c77be16bbec489419d72a75236ab92b291d813a05e9e29adcc02d30d18"
Aug 05 17:31:51.499532 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:51.499430    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c4817c77be16bbec489419d72a75236ab92b291d813a05e9e29adcc02d30d18"} err="failed to get container status \"8c4817c77be16bbec489419d72a75236ab92b291d813a05e9e29adcc02d30d18\": rpc error: code = NotFound desc = could not find container \"8c4817c77be16bbec489419d72a75236ab92b291d813a05e9e29adcc02d30d18\": container with ID starting with 8c4817c77be16bbec489419d72a75236ab92b291d813a05e9e29adcc02d30d18 not found: ID does not exist"
Aug 05 17:31:51.510716 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:51.510689    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"]
Aug 05 17:31:51.516523 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:51.516497    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-wbwld"]
Aug 05 17:31:52.730828 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:52.730789    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c86f7b5-9abd-4d45-901d-856848f8bbd6" path="/var/lib/kubelet/pods/1c86f7b5-9abd-4d45-901d-856848f8bbd6/volumes"
Aug 05 17:31:53.137991 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:31:53.137920    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8" podUID="2e8d6602-6f6f-48a2-99b5-4d6905f42ea6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Aug 05 17:32:03.138222 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:03.138176    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8" podUID="2e8d6602-6f6f-48a2-99b5-4d6905f42ea6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Aug 05 17:32:13.148376 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:13.148338    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:32:13.156704 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:13.156672    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:32:18.119948 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.119914    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"]
Aug 05 17:32:18.120351 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.120327    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c86f7b5-9abd-4d45-901d-856848f8bbd6" containerName="main"
Aug 05 17:32:18.120351 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.120341    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c86f7b5-9abd-4d45-901d-856848f8bbd6" containerName="main"
Aug 05 17:32:18.120427 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.120401    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c86f7b5-9abd-4d45-901d-856848f8bbd6" containerName="main"
Aug 05 17:32:18.125014 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.124995    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"
Aug 05 17:32:18.127831 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.127807    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-7xwp5\""
Aug 05 17:32:18.128899 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.128872    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Aug 05 17:32:18.129023 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.128905    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Aug 05 17:32:18.129023 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.128907    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Aug 05 17:32:18.135298 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.135277    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"]
Aug 05 17:32:18.223568 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.223521    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572\" (UID: \"9d3b145c-9623-4a20-966a-b1a9fca4cfd2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"
Aug 05 17:32:18.223568 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.223570    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572\" (UID: \"9d3b145c-9623-4a20-966a-b1a9fca4cfd2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"
Aug 05 17:32:18.223893 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.223688    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572\" (UID: \"9d3b145c-9623-4a20-966a-b1a9fca4cfd2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"
Aug 05 17:32:18.223893 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.223735    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scsz2\" (UniqueName: \"kubernetes.io/projected/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-kube-api-access-scsz2\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572\" (UID: \"9d3b145c-9623-4a20-966a-b1a9fca4cfd2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"
Aug 05 17:32:18.223893 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.223805    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572\" (UID: \"9d3b145c-9623-4a20-966a-b1a9fca4cfd2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"
Aug 05 17:32:18.223893 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.223834    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572\" (UID: \"9d3b145c-9623-4a20-966a-b1a9fca4cfd2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"
Aug 05 17:32:18.324979 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.324917    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572\" (UID: \"9d3b145c-9623-4a20-966a-b1a9fca4cfd2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"
Aug 05 17:32:18.325195 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.324985    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572\" (UID: \"9d3b145c-9623-4a20-966a-b1a9fca4cfd2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"
Aug 05 17:32:18.325195 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.325072    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572\" (UID: \"9d3b145c-9623-4a20-966a-b1a9fca4cfd2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"
Aug 05 17:32:18.325195 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.325114    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scsz2\" (UniqueName: \"kubernetes.io/projected/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-kube-api-access-scsz2\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572\" (UID: \"9d3b145c-9623-4a20-966a-b1a9fca4cfd2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"
Aug 05 17:32:18.325195 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.325150    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572\" (UID: \"9d3b145c-9623-4a20-966a-b1a9fca4cfd2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"
Aug 05 17:32:18.325409 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.325263    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572\" (UID: \"9d3b145c-9623-4a20-966a-b1a9fca4cfd2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"
Aug 05 17:32:18.325409 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.325377    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572\" (UID: \"9d3b145c-9623-4a20-966a-b1a9fca4cfd2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"
Aug 05 17:32:18.325409 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.325395    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572\" (UID: \"9d3b145c-9623-4a20-966a-b1a9fca4cfd2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"
Aug 05 17:32:18.325617 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.325584    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572\" (UID: \"9d3b145c-9623-4a20-966a-b1a9fca4cfd2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"
Aug 05 17:32:18.327306 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.327285    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572\" (UID: \"9d3b145c-9623-4a20-966a-b1a9fca4cfd2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"
Aug 05 17:32:18.327552 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.327535    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572\" (UID: \"9d3b145c-9623-4a20-966a-b1a9fca4cfd2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"
Aug 05 17:32:18.335771 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.335751    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scsz2\" (UniqueName: \"kubernetes.io/projected/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-kube-api-access-scsz2\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572\" (UID: \"9d3b145c-9623-4a20-966a-b1a9fca4cfd2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"
Aug 05 17:32:18.436364 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.436269    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"
Aug 05 17:32:18.586897 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.586874    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"]
Aug 05 17:32:18.588971 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:32:18.588921    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d3b145c_9623_4a20_966a_b1a9fca4cfd2.slice/crio-c4de75136333a306fb43f3889789bb642b690105c3f06aef08667c1ffdb01d56 WatchSource:0}: Error finding container c4de75136333a306fb43f3889789bb642b690105c3f06aef08667c1ffdb01d56: Status 404 returned error can't find the container with id c4de75136333a306fb43f3889789bb642b690105c3f06aef08667c1ffdb01d56
Aug 05 17:32:18.656054 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:18.656010    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572" event={"ID":"9d3b145c-9623-4a20-966a-b1a9fca4cfd2","Type":"ContainerStarted","Data":"c4de75136333a306fb43f3889789bb642b690105c3f06aef08667c1ffdb01d56"}
Aug 05 17:32:19.663110 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:19.663073    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572" event={"ID":"9d3b145c-9623-4a20-966a-b1a9fca4cfd2","Type":"ContainerStarted","Data":"951609d607cf76df5caf2e790b24d8604a747c936c0c86708e1613045f161587"}
Aug 05 17:32:19.681920 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:19.681865    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572" podStartSLOduration=1.681848239 podStartE2EDuration="1.681848239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:32:19.679926635 +0000 UTC m=+1195.424560990" watchObservedRunningTime="2026-08-05 17:32:19.681848239 +0000 UTC m=+1195.426482609"
Aug 05 17:32:20.436611 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:20.436567    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"
Aug 05 17:32:28.436858 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:28.436818    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"
Aug 05 17:32:28.450481 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:28.450451    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"
Aug 05 17:32:28.731245 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:28.731164    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"
Aug 05 17:32:36.635493 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:36.635455    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"]
Aug 05 17:32:51.664174 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:51.664124    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8" podUID="2e8d6602-6f6f-48a2-99b5-4d6905f42ea6" containerName="main" containerID="cri-o://fafc6fe4fc9aa3a5e3abc96a7507bf4b1b8744d656dd2985db06ae33b40f9ed9" gracePeriod=45
Aug 05 17:32:51.834385 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:51.834341    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"]
Aug 05 17:32:51.847041 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:32:51.847002    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 05 17:32:51.847041 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:32:51.847041 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:32:51.847041 ip-10-0-135-151 kubenswrapper[2561]:  > containerID="951609d607cf76df5caf2e790b24d8604a747c936c0c86708e1613045f161587" cmd=["/bin/sleep","15"]
Aug 05 17:32:51.849268 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:51.849238    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572" message="" err=<
Aug 05 17:32:51.849268 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:32:51.849268 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:32:51.849268 ip-10-0-135-151 kubenswrapper[2561]:  >
Aug 05 17:32:51.849623 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:32:51.849589    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 05 17:32:51.849623 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:32:51.849623 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:32:51.849623 ip-10-0-135-151 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572" podUID="9d3b145c-9623-4a20-966a-b1a9fca4cfd2" containerName="main" containerID="cri-o://951609d607cf76df5caf2e790b24d8604a747c936c0c86708e1613045f161587"
Aug 05 17:32:51.849925 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:51.849658    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572" podUID="9d3b145c-9623-4a20-966a-b1a9fca4cfd2" containerName="main" containerID="cri-o://951609d607cf76df5caf2e790b24d8604a747c936c0c86708e1613045f161587" gracePeriod=60
Aug 05 17:32:52.149945 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.149652    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8" podUID="2e8d6602-6f6f-48a2-99b5-4d6905f42ea6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Aug 05 17:32:52.236821 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.236790    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"
Aug 05 17:32:52.354632 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.354605    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-dshm\") pod \"9d3b145c-9623-4a20-966a-b1a9fca4cfd2\" (UID: \"9d3b145c-9623-4a20-966a-b1a9fca4cfd2\") "
Aug 05 17:32:52.355137 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.354937    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-model-cache\") pod \"9d3b145c-9623-4a20-966a-b1a9fca4cfd2\" (UID: \"9d3b145c-9623-4a20-966a-b1a9fca4cfd2\") "
Aug 05 17:32:52.355253 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.355205    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-tmp-dir\") pod \"9d3b145c-9623-4a20-966a-b1a9fca4cfd2\" (UID: \"9d3b145c-9623-4a20-966a-b1a9fca4cfd2\") "
Aug 05 17:32:52.355482 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.355280    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-home\") pod \"9d3b145c-9623-4a20-966a-b1a9fca4cfd2\" (UID: \"9d3b145c-9623-4a20-966a-b1a9fca4cfd2\") "
Aug 05 17:32:52.355482 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.355344    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scsz2\" (UniqueName: \"kubernetes.io/projected/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-kube-api-access-scsz2\") pod \"9d3b145c-9623-4a20-966a-b1a9fca4cfd2\" (UID: \"9d3b145c-9623-4a20-966a-b1a9fca4cfd2\") "
Aug 05 17:32:52.355482 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.355381    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-tls-certs\") pod \"9d3b145c-9623-4a20-966a-b1a9fca4cfd2\" (UID: \"9d3b145c-9623-4a20-966a-b1a9fca4cfd2\") "
Aug 05 17:32:52.355482 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.355397    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-model-cache" (OuterVolumeSpecName: "model-cache") pod "9d3b145c-9623-4a20-966a-b1a9fca4cfd2" (UID: "9d3b145c-9623-4a20-966a-b1a9fca4cfd2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:32:52.355797 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.355482    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9d3b145c-9623-4a20-966a-b1a9fca4cfd2" (UID: "9d3b145c-9623-4a20-966a-b1a9fca4cfd2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:32:52.355797 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.355543    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-home" (OuterVolumeSpecName: "home") pod "9d3b145c-9623-4a20-966a-b1a9fca4cfd2" (UID: "9d3b145c-9623-4a20-966a-b1a9fca4cfd2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:32:52.355797 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.355715    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:32:52.355797 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.355730    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:32:52.355797 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.355743    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:32:52.360214 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.359595    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9d3b145c-9623-4a20-966a-b1a9fca4cfd2" (UID: "9d3b145c-9623-4a20-966a-b1a9fca4cfd2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:32:52.361299 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.361255    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-kube-api-access-scsz2" (OuterVolumeSpecName: "kube-api-access-scsz2") pod "9d3b145c-9623-4a20-966a-b1a9fca4cfd2" (UID: "9d3b145c-9623-4a20-966a-b1a9fca4cfd2"). InnerVolumeSpecName "kube-api-access-scsz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:32:52.361475 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.361442    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-dshm" (OuterVolumeSpecName: "dshm") pod "9d3b145c-9623-4a20-966a-b1a9fca4cfd2" (UID: "9d3b145c-9623-4a20-966a-b1a9fca4cfd2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:32:52.422629 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.422597    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:32:52.456960 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.456917    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scsz2\" (UniqueName: \"kubernetes.io/projected/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-kube-api-access-scsz2\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:32:52.456960 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.456947    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:32:52.457167 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.456973    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d3b145c-9623-4a20-966a-b1a9fca4cfd2-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:32:52.558277 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.558245    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-model-cache\") pod \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") "
Aug 05 17:32:52.558470 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.558300    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-dshm\") pod \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") "
Aug 05 17:32:52.558470 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.558323    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-tls-certs\") pod \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") "
Aug 05 17:32:52.558470 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.558340    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-tmp-dir\") pod \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") "
Aug 05 17:32:52.558470 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.558397    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-cabundle-cert\") pod \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") "
Aug 05 17:32:52.558470 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.558415    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-home\") pod \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") "
Aug 05 17:32:52.558755 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.558558    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-model-cache" (OuterVolumeSpecName: "model-cache") pod "2e8d6602-6f6f-48a2-99b5-4d6905f42ea6" (UID: "2e8d6602-6f6f-48a2-99b5-4d6905f42ea6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:32:52.558755 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.558619    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-kserve-provision-location\") pod \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") "
Aug 05 17:32:52.558755 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.558659    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dz29t\" (UniqueName: \"kubernetes.io/projected/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-kube-api-access-dz29t\") pod \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\" (UID: \"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6\") "
Aug 05 17:32:52.558945 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.558914    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2e8d6602-6f6f-48a2-99b5-4d6905f42ea6" (UID: "2e8d6602-6f6f-48a2-99b5-4d6905f42ea6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:32:52.559044 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.559011    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:32:52.559108 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.559046    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-home" (OuterVolumeSpecName: "home") pod "2e8d6602-6f6f-48a2-99b5-4d6905f42ea6" (UID: "2e8d6602-6f6f-48a2-99b5-4d6905f42ea6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:32:52.561306 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.561255    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e8d6602-6f6f-48a2-99b5-4d6905f42ea6" (UID: "2e8d6602-6f6f-48a2-99b5-4d6905f42ea6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:32:52.561306 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.561262    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-kube-api-access-dz29t" (OuterVolumeSpecName: "kube-api-access-dz29t") pod "2e8d6602-6f6f-48a2-99b5-4d6905f42ea6" (UID: "2e8d6602-6f6f-48a2-99b5-4d6905f42ea6"). InnerVolumeSpecName "kube-api-access-dz29t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:32:52.561306 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.561299    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-dshm" (OuterVolumeSpecName: "dshm") pod "2e8d6602-6f6f-48a2-99b5-4d6905f42ea6" (UID: "2e8d6602-6f6f-48a2-99b5-4d6905f42ea6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:32:52.572417 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.572358    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2e8d6602-6f6f-48a2-99b5-4d6905f42ea6" (UID: "2e8d6602-6f6f-48a2-99b5-4d6905f42ea6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:32:52.594041 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.593996    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2e8d6602-6f6f-48a2-99b5-4d6905f42ea6" (UID: "2e8d6602-6f6f-48a2-99b5-4d6905f42ea6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:32:52.659829 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.659794    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-kserve-provision-location\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:32:52.659829 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.659830    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz29t\" (UniqueName: \"kubernetes.io/projected/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-kube-api-access-dz29t\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:32:52.660115 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.659855    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:32:52.660115 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.659869    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:32:52.660115 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.659882    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:32:52.660115 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.659894    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-cabundle-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:32:52.660115 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.659905    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:32:52.855519 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.855409    2561 generic.go:356] "Generic (PLEG): container finished" podID="9d3b145c-9623-4a20-966a-b1a9fca4cfd2" containerID="951609d607cf76df5caf2e790b24d8604a747c936c0c86708e1613045f161587" exitCode=0
Aug 05 17:32:52.855519 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.855463    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"
Aug 05 17:32:52.855519 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.855475    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572" event={"ID":"9d3b145c-9623-4a20-966a-b1a9fca4cfd2","Type":"ContainerDied","Data":"951609d607cf76df5caf2e790b24d8604a747c936c0c86708e1613045f161587"}
Aug 05 17:32:52.856165 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.855540    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572" event={"ID":"9d3b145c-9623-4a20-966a-b1a9fca4cfd2","Type":"ContainerDied","Data":"c4de75136333a306fb43f3889789bb642b690105c3f06aef08667c1ffdb01d56"}
Aug 05 17:32:52.856165 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.855568    2561 scope.go:117] "RemoveContainer" containerID="951609d607cf76df5caf2e790b24d8604a747c936c0c86708e1613045f161587"
Aug 05 17:32:52.857696 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.857657    2561 generic.go:356] "Generic (PLEG): container finished" podID="2e8d6602-6f6f-48a2-99b5-4d6905f42ea6" containerID="fafc6fe4fc9aa3a5e3abc96a7507bf4b1b8744d656dd2985db06ae33b40f9ed9" exitCode=0
Aug 05 17:32:52.857812 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.857741    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8" event={"ID":"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6","Type":"ContainerDied","Data":"fafc6fe4fc9aa3a5e3abc96a7507bf4b1b8744d656dd2985db06ae33b40f9ed9"}
Aug 05 17:32:52.857812 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.857758    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"
Aug 05 17:32:52.857974 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.857815    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8" event={"ID":"2e8d6602-6f6f-48a2-99b5-4d6905f42ea6","Type":"ContainerDied","Data":"7360c1cb9bad64267eba7cf0ab965bbac7e7c465c41d95b0d509aa8a22f03da9"}
Aug 05 17:32:52.868850 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.868823    2561 scope.go:117] "RemoveContainer" containerID="951609d607cf76df5caf2e790b24d8604a747c936c0c86708e1613045f161587"
Aug 05 17:32:52.869240 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:32:52.869209    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"951609d607cf76df5caf2e790b24d8604a747c936c0c86708e1613045f161587\": container with ID starting with 951609d607cf76df5caf2e790b24d8604a747c936c0c86708e1613045f161587 not found: ID does not exist" containerID="951609d607cf76df5caf2e790b24d8604a747c936c0c86708e1613045f161587"
Aug 05 17:32:52.869342 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.869248    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"951609d607cf76df5caf2e790b24d8604a747c936c0c86708e1613045f161587"} err="failed to get container status \"951609d607cf76df5caf2e790b24d8604a747c936c0c86708e1613045f161587\": rpc error: code = NotFound desc = could not find container \"951609d607cf76df5caf2e790b24d8604a747c936c0c86708e1613045f161587\": container with ID starting with 951609d607cf76df5caf2e790b24d8604a747c936c0c86708e1613045f161587 not found: ID does not exist"
Aug 05 17:32:52.869342 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.869274    2561 scope.go:117] "RemoveContainer" containerID="fafc6fe4fc9aa3a5e3abc96a7507bf4b1b8744d656dd2985db06ae33b40f9ed9"
Aug 05 17:32:52.879181 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.879159    2561 scope.go:117] "RemoveContainer" containerID="23fe3867b892c778f54446d0074fc1a84b6f4b2591766f5fb73122f1b4bb9025"
Aug 05 17:32:52.888276 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.888242    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"]
Aug 05 17:32:52.894667 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.894629    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-6c5944c47b-5jxp8"]
Aug 05 17:32:52.895655 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.895632    2561 scope.go:117] "RemoveContainer" containerID="fafc6fe4fc9aa3a5e3abc96a7507bf4b1b8744d656dd2985db06ae33b40f9ed9"
Aug 05 17:32:52.896278 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:32:52.896237    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fafc6fe4fc9aa3a5e3abc96a7507bf4b1b8744d656dd2985db06ae33b40f9ed9\": container with ID starting with fafc6fe4fc9aa3a5e3abc96a7507bf4b1b8744d656dd2985db06ae33b40f9ed9 not found: ID does not exist" containerID="fafc6fe4fc9aa3a5e3abc96a7507bf4b1b8744d656dd2985db06ae33b40f9ed9"
Aug 05 17:32:52.896404 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.896279    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fafc6fe4fc9aa3a5e3abc96a7507bf4b1b8744d656dd2985db06ae33b40f9ed9"} err="failed to get container status \"fafc6fe4fc9aa3a5e3abc96a7507bf4b1b8744d656dd2985db06ae33b40f9ed9\": rpc error: code = NotFound desc = could not find container \"fafc6fe4fc9aa3a5e3abc96a7507bf4b1b8744d656dd2985db06ae33b40f9ed9\": container with ID starting with fafc6fe4fc9aa3a5e3abc96a7507bf4b1b8744d656dd2985db06ae33b40f9ed9 not found: ID does not exist"
Aug 05 17:32:52.896404 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.896308    2561 scope.go:117] "RemoveContainer" containerID="23fe3867b892c778f54446d0074fc1a84b6f4b2591766f5fb73122f1b4bb9025"
Aug 05 17:32:52.896633 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:32:52.896613    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23fe3867b892c778f54446d0074fc1a84b6f4b2591766f5fb73122f1b4bb9025\": container with ID starting with 23fe3867b892c778f54446d0074fc1a84b6f4b2591766f5fb73122f1b4bb9025 not found: ID does not exist" containerID="23fe3867b892c778f54446d0074fc1a84b6f4b2591766f5fb73122f1b4bb9025"
Aug 05 17:32:52.896694 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.896641    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23fe3867b892c778f54446d0074fc1a84b6f4b2591766f5fb73122f1b4bb9025"} err="failed to get container status \"23fe3867b892c778f54446d0074fc1a84b6f4b2591766f5fb73122f1b4bb9025\": rpc error: code = NotFound desc = could not find container \"23fe3867b892c778f54446d0074fc1a84b6f4b2591766f5fb73122f1b4bb9025\": container with ID starting with 23fe3867b892c778f54446d0074fc1a84b6f4b2591766f5fb73122f1b4bb9025 not found: ID does not exist"
Aug 05 17:32:52.898787 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.898759    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"]
Aug 05 17:32:52.900998 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:52.900975    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ffx572"]
Aug 05 17:32:54.730170 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:54.730138    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e8d6602-6f6f-48a2-99b5-4d6905f42ea6" path="/var/lib/kubelet/pods/2e8d6602-6f6f-48a2-99b5-4d6905f42ea6/volumes"
Aug 05 17:32:54.730594 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:32:54.730580    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d3b145c-9623-4a20-966a-b1a9fca4cfd2" path="/var/lib/kubelet/pods/9d3b145c-9623-4a20-966a-b1a9fca4cfd2/volumes"
Aug 05 17:33:06.130845 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.130810    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"]
Aug 05 17:33:06.131264 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.131209    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e8d6602-6f6f-48a2-99b5-4d6905f42ea6" containerName="storage-initializer"
Aug 05 17:33:06.131264 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.131221    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e8d6602-6f6f-48a2-99b5-4d6905f42ea6" containerName="storage-initializer"
Aug 05 17:33:06.131264 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.131231    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e8d6602-6f6f-48a2-99b5-4d6905f42ea6" containerName="main"
Aug 05 17:33:06.131264 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.131236    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e8d6602-6f6f-48a2-99b5-4d6905f42ea6" containerName="main"
Aug 05 17:33:06.131264 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.131245    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d3b145c-9623-4a20-966a-b1a9fca4cfd2" containerName="main"
Aug 05 17:33:06.131264 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.131251    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d3b145c-9623-4a20-966a-b1a9fca4cfd2" containerName="main"
Aug 05 17:33:06.131463 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.131307    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d3b145c-9623-4a20-966a-b1a9fca4cfd2" containerName="main"
Aug 05 17:33:06.131463 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.131319    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e8d6602-6f6f-48a2-99b5-4d6905f42ea6" containerName="main"
Aug 05 17:33:06.135842 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.135822    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:33:06.138646 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.138613    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Aug 05 17:33:06.138816 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.138703    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-djpld\""
Aug 05 17:33:06.138897 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.138849    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Aug 05 17:33:06.138897 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.138852    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Aug 05 17:33:06.139043 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.138939    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Aug 05 17:33:06.139106 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.139067    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Aug 05 17:33:06.146712 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.146682    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"]
Aug 05 17:33:06.282695 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.282653    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ndc9\" (UniqueName: \"kubernetes.io/projected/697ee405-32b1-471c-b9dd-ff4a75c1decf-kube-api-access-6ndc9\") pod \"auth-invalid-token-test-kserve-f6d648757-r4cpb\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:33:06.282695 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.282702    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/697ee405-32b1-471c-b9dd-ff4a75c1decf-home\") pod \"auth-invalid-token-test-kserve-f6d648757-r4cpb\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:33:06.283017 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.282745    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/697ee405-32b1-471c-b9dd-ff4a75c1decf-tmp-dir\") pod \"auth-invalid-token-test-kserve-f6d648757-r4cpb\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:33:06.283017 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.282776    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/697ee405-32b1-471c-b9dd-ff4a75c1decf-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-f6d648757-r4cpb\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:33:06.283017 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.282845    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/697ee405-32b1-471c-b9dd-ff4a75c1decf-dshm\") pod \"auth-invalid-token-test-kserve-f6d648757-r4cpb\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:33:06.283017 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.282936    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/697ee405-32b1-471c-b9dd-ff4a75c1decf-tls-certs\") pod \"auth-invalid-token-test-kserve-f6d648757-r4cpb\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:33:06.283017 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.283008    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/697ee405-32b1-471c-b9dd-ff4a75c1decf-cabundle-cert\") pod \"auth-invalid-token-test-kserve-f6d648757-r4cpb\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:33:06.283236 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.283039    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/697ee405-32b1-471c-b9dd-ff4a75c1decf-model-cache\") pod \"auth-invalid-token-test-kserve-f6d648757-r4cpb\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:33:06.383880 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.383785    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/697ee405-32b1-471c-b9dd-ff4a75c1decf-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-f6d648757-r4cpb\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:33:06.383880 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.383833    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/697ee405-32b1-471c-b9dd-ff4a75c1decf-dshm\") pod \"auth-invalid-token-test-kserve-f6d648757-r4cpb\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:33:06.383880 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.383874    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/697ee405-32b1-471c-b9dd-ff4a75c1decf-tls-certs\") pod \"auth-invalid-token-test-kserve-f6d648757-r4cpb\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:33:06.384213 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.383894    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/697ee405-32b1-471c-b9dd-ff4a75c1decf-cabundle-cert\") pod \"auth-invalid-token-test-kserve-f6d648757-r4cpb\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:33:06.384213 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.383925    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/697ee405-32b1-471c-b9dd-ff4a75c1decf-model-cache\") pod \"auth-invalid-token-test-kserve-f6d648757-r4cpb\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:33:06.384213 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.384011    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ndc9\" (UniqueName: \"kubernetes.io/projected/697ee405-32b1-471c-b9dd-ff4a75c1decf-kube-api-access-6ndc9\") pod \"auth-invalid-token-test-kserve-f6d648757-r4cpb\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:33:06.384213 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.384046    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/697ee405-32b1-471c-b9dd-ff4a75c1decf-home\") pod \"auth-invalid-token-test-kserve-f6d648757-r4cpb\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:33:06.384213 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.384086    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/697ee405-32b1-471c-b9dd-ff4a75c1decf-tmp-dir\") pod \"auth-invalid-token-test-kserve-f6d648757-r4cpb\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:33:06.384449 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.384276    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/697ee405-32b1-471c-b9dd-ff4a75c1decf-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-f6d648757-r4cpb\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:33:06.384449 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.384381    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/697ee405-32b1-471c-b9dd-ff4a75c1decf-model-cache\") pod \"auth-invalid-token-test-kserve-f6d648757-r4cpb\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:33:06.384449 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.384381    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/697ee405-32b1-471c-b9dd-ff4a75c1decf-tmp-dir\") pod \"auth-invalid-token-test-kserve-f6d648757-r4cpb\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:33:06.384568 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.384502    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/697ee405-32b1-471c-b9dd-ff4a75c1decf-home\") pod \"auth-invalid-token-test-kserve-f6d648757-r4cpb\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:33:06.384637 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.384618    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/697ee405-32b1-471c-b9dd-ff4a75c1decf-cabundle-cert\") pod \"auth-invalid-token-test-kserve-f6d648757-r4cpb\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:33:06.386337 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.386314    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/697ee405-32b1-471c-b9dd-ff4a75c1decf-dshm\") pod \"auth-invalid-token-test-kserve-f6d648757-r4cpb\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:33:06.386741 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.386721    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/697ee405-32b1-471c-b9dd-ff4a75c1decf-tls-certs\") pod \"auth-invalid-token-test-kserve-f6d648757-r4cpb\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:33:06.393233 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.393204    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ndc9\" (UniqueName: \"kubernetes.io/projected/697ee405-32b1-471c-b9dd-ff4a75c1decf-kube-api-access-6ndc9\") pod \"auth-invalid-token-test-kserve-f6d648757-r4cpb\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:33:06.447367 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.447330    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:33:06.601690 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.601660    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"]
Aug 05 17:33:06.604113 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:33:06.604081    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod697ee405_32b1_471c_b9dd_ff4a75c1decf.slice/crio-1c8f1d5b1fa74fd080df014dfbc0536a39bc03a34f9e5968ad9e800f5c844f8a WatchSource:0}: Error finding container 1c8f1d5b1fa74fd080df014dfbc0536a39bc03a34f9e5968ad9e800f5c844f8a: Status 404 returned error can't find the container with id 1c8f1d5b1fa74fd080df014dfbc0536a39bc03a34f9e5968ad9e800f5c844f8a
Aug 05 17:33:06.939485 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.939375    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb" event={"ID":"697ee405-32b1-471c-b9dd-ff4a75c1decf","Type":"ContainerStarted","Data":"09c68d1e9bf2e2e1eccc883c0447272b3be27b1b870642b5e420cf2b3b7f2fa8"}
Aug 05 17:33:06.939485 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:06.939442    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb" event={"ID":"697ee405-32b1-471c-b9dd-ff4a75c1decf","Type":"ContainerStarted","Data":"1c8f1d5b1fa74fd080df014dfbc0536a39bc03a34f9e5968ad9e800f5c844f8a"}
Aug 05 17:33:09.961025 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:09.960987    2561 generic.go:356] "Generic (PLEG): container finished" podID="697ee405-32b1-471c-b9dd-ff4a75c1decf" containerID="09c68d1e9bf2e2e1eccc883c0447272b3be27b1b870642b5e420cf2b3b7f2fa8" exitCode=0
Aug 05 17:33:09.961400 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:09.961050    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb" event={"ID":"697ee405-32b1-471c-b9dd-ff4a75c1decf","Type":"ContainerDied","Data":"09c68d1e9bf2e2e1eccc883c0447272b3be27b1b870642b5e420cf2b3b7f2fa8"}
Aug 05 17:33:10.969836 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:10.969805    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb" event={"ID":"697ee405-32b1-471c-b9dd-ff4a75c1decf","Type":"ContainerStarted","Data":"96590d7f45f7c9a2f801ba4590d7e1286a60acc3ec2ee509c17f844b0723f171"}
Aug 05 17:33:10.994534 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:10.994472    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb" podStartSLOduration=2.55577912 podStartE2EDuration="4.994455485s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.438676365s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:33:10.991612804 +0000 UTC m=+1246.736247156" watchObservedRunningTime="2026-08-05 17:33:10.994455485 +0000 UTC m=+1246.739089813"
Aug 05 17:33:11.447928 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:11.447880    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:33:16.448113 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:16.448010    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:33:16.449852 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:16.449820    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb" podUID="697ee405-32b1-471c-b9dd-ff4a75c1decf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Aug 05 17:33:22.609089 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:22.609051    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"]
Aug 05 17:33:22.611774 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:22.611753    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"
Aug 05 17:33:22.615502 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:22.615480    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Aug 05 17:33:22.615623 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:22.615551    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-6bdnj\""
Aug 05 17:33:22.615623 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:22.615560    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Aug 05 17:33:22.615758 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:22.615723    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Aug 05 17:33:22.622912 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:22.622888    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"]
Aug 05 17:33:22.743762 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:22.743725    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stqhz\" (UniqueName: \"kubernetes.io/projected/1e8f6945-67a4-4506-8143-1ae310046ac0-kube-api-access-stqhz\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm\" (UID: \"1e8f6945-67a4-4506-8143-1ae310046ac0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"
Aug 05 17:33:22.743762 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:22.743763    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e8f6945-67a4-4506-8143-1ae310046ac0-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm\" (UID: \"1e8f6945-67a4-4506-8143-1ae310046ac0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"
Aug 05 17:33:22.744047 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:22.743787    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e8f6945-67a4-4506-8143-1ae310046ac0-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm\" (UID: \"1e8f6945-67a4-4506-8143-1ae310046ac0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"
Aug 05 17:33:22.744047 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:22.743884    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e8f6945-67a4-4506-8143-1ae310046ac0-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm\" (UID: \"1e8f6945-67a4-4506-8143-1ae310046ac0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"
Aug 05 17:33:22.744047 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:22.744032    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e8f6945-67a4-4506-8143-1ae310046ac0-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm\" (UID: \"1e8f6945-67a4-4506-8143-1ae310046ac0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"
Aug 05 17:33:22.744169 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:22.744056    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e8f6945-67a4-4506-8143-1ae310046ac0-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm\" (UID: \"1e8f6945-67a4-4506-8143-1ae310046ac0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"
Aug 05 17:33:22.845030 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:22.844989    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e8f6945-67a4-4506-8143-1ae310046ac0-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm\" (UID: \"1e8f6945-67a4-4506-8143-1ae310046ac0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"
Aug 05 17:33:22.845334 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:22.845046    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e8f6945-67a4-4506-8143-1ae310046ac0-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm\" (UID: \"1e8f6945-67a4-4506-8143-1ae310046ac0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"
Aug 05 17:33:22.845334 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:22.845098    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stqhz\" (UniqueName: \"kubernetes.io/projected/1e8f6945-67a4-4506-8143-1ae310046ac0-kube-api-access-stqhz\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm\" (UID: \"1e8f6945-67a4-4506-8143-1ae310046ac0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"
Aug 05 17:33:22.845334 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:22.845130    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e8f6945-67a4-4506-8143-1ae310046ac0-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm\" (UID: \"1e8f6945-67a4-4506-8143-1ae310046ac0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"
Aug 05 17:33:22.845334 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:22.845158    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e8f6945-67a4-4506-8143-1ae310046ac0-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm\" (UID: \"1e8f6945-67a4-4506-8143-1ae310046ac0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"
Aug 05 17:33:22.845334 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:22.845200    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e8f6945-67a4-4506-8143-1ae310046ac0-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm\" (UID: \"1e8f6945-67a4-4506-8143-1ae310046ac0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"
Aug 05 17:33:22.845579 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:22.845404    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e8f6945-67a4-4506-8143-1ae310046ac0-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm\" (UID: \"1e8f6945-67a4-4506-8143-1ae310046ac0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"
Aug 05 17:33:22.845579 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:22.845487    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e8f6945-67a4-4506-8143-1ae310046ac0-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm\" (UID: \"1e8f6945-67a4-4506-8143-1ae310046ac0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"
Aug 05 17:33:22.845713 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:22.845687    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e8f6945-67a4-4506-8143-1ae310046ac0-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm\" (UID: \"1e8f6945-67a4-4506-8143-1ae310046ac0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"
Aug 05 17:33:22.847581 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:22.847548    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e8f6945-67a4-4506-8143-1ae310046ac0-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm\" (UID: \"1e8f6945-67a4-4506-8143-1ae310046ac0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"
Aug 05 17:33:22.847923 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:22.847904    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e8f6945-67a4-4506-8143-1ae310046ac0-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm\" (UID: \"1e8f6945-67a4-4506-8143-1ae310046ac0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"
Aug 05 17:33:22.853758 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:22.853730    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stqhz\" (UniqueName: \"kubernetes.io/projected/1e8f6945-67a4-4506-8143-1ae310046ac0-kube-api-access-stqhz\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm\" (UID: \"1e8f6945-67a4-4506-8143-1ae310046ac0\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"
Aug 05 17:33:22.925692 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:22.925606    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"
Aug 05 17:33:23.072272 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:23.072245    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"]
Aug 05 17:33:23.075034 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:33:23.075007    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8f6945_67a4_4506_8143_1ae310046ac0.slice/crio-e86b2ab9e3b8bebfd2780eb3ddae97a61a12be081834400343a7b72bc6d048c7 WatchSource:0}: Error finding container e86b2ab9e3b8bebfd2780eb3ddae97a61a12be081834400343a7b72bc6d048c7: Status 404 returned error can't find the container with id e86b2ab9e3b8bebfd2780eb3ddae97a61a12be081834400343a7b72bc6d048c7
Aug 05 17:33:24.051214 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:24.051179    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm" event={"ID":"1e8f6945-67a4-4506-8143-1ae310046ac0","Type":"ContainerStarted","Data":"7e9387d379fca76156753e21c45871524d02e730c9dc13745bc3209ae6ff1a99"}
Aug 05 17:33:24.051613 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:24.051242    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm" event={"ID":"1e8f6945-67a4-4506-8143-1ae310046ac0","Type":"ContainerStarted","Data":"e86b2ab9e3b8bebfd2780eb3ddae97a61a12be081834400343a7b72bc6d048c7"}
Aug 05 17:33:24.069450 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:24.069389    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm" podStartSLOduration=2.06937131 podStartE2EDuration="2.06937131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:33:24.068810072 +0000 UTC m=+1259.813444401" watchObservedRunningTime="2026-08-05 17:33:24.06937131 +0000 UTC m=+1259.814005639"
Aug 05 17:33:24.925929 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:24.925887    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"
Aug 05 17:33:26.448728 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:26.448684    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb" podUID="697ee405-32b1-471c-b9dd-ff4a75c1decf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Aug 05 17:33:32.926097 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:32.926048    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"
Aug 05 17:33:32.939421 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:32.939394    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"
Aug 05 17:33:33.120054 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:33.120020    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"
Aug 05 17:33:36.448613 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:36.448569    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb" podUID="697ee405-32b1-471c-b9dd-ff4a75c1decf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Aug 05 17:33:46.448415 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:46.448360    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb" podUID="697ee405-32b1-471c-b9dd-ff4a75c1decf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Aug 05 17:33:56.448468 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:56.448414    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb" podUID="697ee405-32b1-471c-b9dd-ff4a75c1decf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Aug 05 17:33:56.523740 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:56.523691    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"]
Aug 05 17:33:56.534302 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:33:56.534266    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 05 17:33:56.534302 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:33:56.534302 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:33:56.534302 ip-10-0-135-151 kubenswrapper[2561]:  > containerID="7e9387d379fca76156753e21c45871524d02e730c9dc13745bc3209ae6ff1a99" cmd=["/bin/sleep","15"]
Aug 05 17:33:56.534636 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:56.534316    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm" message="" err=<
Aug 05 17:33:56.534636 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:33:56.534636 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:33:56.534636 ip-10-0-135-151 kubenswrapper[2561]:  >
Aug 05 17:33:56.534636 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:33:56.534330    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 05 17:33:56.534636 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:33:56.534636 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:33:56.534636 ip-10-0-135-151 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm" podUID="1e8f6945-67a4-4506-8143-1ae310046ac0" containerName="main" containerID="cri-o://7e9387d379fca76156753e21c45871524d02e730c9dc13745bc3209ae6ff1a99"
Aug 05 17:33:56.534636 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:56.534368    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm" podUID="1e8f6945-67a4-4506-8143-1ae310046ac0" containerName="main" containerID="cri-o://7e9387d379fca76156753e21c45871524d02e730c9dc13745bc3209ae6ff1a99" gracePeriod=60
Aug 05 17:33:56.884597 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:56.884572    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"
Aug 05 17:33:56.959432 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:56.959398    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e8f6945-67a4-4506-8143-1ae310046ac0-home\") pod \"1e8f6945-67a4-4506-8143-1ae310046ac0\" (UID: \"1e8f6945-67a4-4506-8143-1ae310046ac0\") "
Aug 05 17:33:56.959623 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:56.959495    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e8f6945-67a4-4506-8143-1ae310046ac0-tmp-dir\") pod \"1e8f6945-67a4-4506-8143-1ae310046ac0\" (UID: \"1e8f6945-67a4-4506-8143-1ae310046ac0\") "
Aug 05 17:33:56.959623 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:56.959538    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e8f6945-67a4-4506-8143-1ae310046ac0-model-cache\") pod \"1e8f6945-67a4-4506-8143-1ae310046ac0\" (UID: \"1e8f6945-67a4-4506-8143-1ae310046ac0\") "
Aug 05 17:33:56.959623 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:56.959567    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e8f6945-67a4-4506-8143-1ae310046ac0-tls-certs\") pod \"1e8f6945-67a4-4506-8143-1ae310046ac0\" (UID: \"1e8f6945-67a4-4506-8143-1ae310046ac0\") "
Aug 05 17:33:56.959623 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:56.959598    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stqhz\" (UniqueName: \"kubernetes.io/projected/1e8f6945-67a4-4506-8143-1ae310046ac0-kube-api-access-stqhz\") pod \"1e8f6945-67a4-4506-8143-1ae310046ac0\" (UID: \"1e8f6945-67a4-4506-8143-1ae310046ac0\") "
Aug 05 17:33:56.959892 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:56.959636    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e8f6945-67a4-4506-8143-1ae310046ac0-dshm\") pod \"1e8f6945-67a4-4506-8143-1ae310046ac0\" (UID: \"1e8f6945-67a4-4506-8143-1ae310046ac0\") "
Aug 05 17:33:56.959892 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:56.959713    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e8f6945-67a4-4506-8143-1ae310046ac0-home" (OuterVolumeSpecName: "home") pod "1e8f6945-67a4-4506-8143-1ae310046ac0" (UID: "1e8f6945-67a4-4506-8143-1ae310046ac0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:33:56.959892 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:56.959751    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e8f6945-67a4-4506-8143-1ae310046ac0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1e8f6945-67a4-4506-8143-1ae310046ac0" (UID: "1e8f6945-67a4-4506-8143-1ae310046ac0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:33:56.959892 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:56.959772    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e8f6945-67a4-4506-8143-1ae310046ac0-model-cache" (OuterVolumeSpecName: "model-cache") pod "1e8f6945-67a4-4506-8143-1ae310046ac0" (UID: "1e8f6945-67a4-4506-8143-1ae310046ac0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:33:56.960077 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:56.959919    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e8f6945-67a4-4506-8143-1ae310046ac0-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:33:56.960077 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:56.959937    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e8f6945-67a4-4506-8143-1ae310046ac0-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:33:56.960077 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:56.959981    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e8f6945-67a4-4506-8143-1ae310046ac0-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:33:56.961878 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:56.961857    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e8f6945-67a4-4506-8143-1ae310046ac0-dshm" (OuterVolumeSpecName: "dshm") pod "1e8f6945-67a4-4506-8143-1ae310046ac0" (UID: "1e8f6945-67a4-4506-8143-1ae310046ac0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:33:56.962192 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:56.962166    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e8f6945-67a4-4506-8143-1ae310046ac0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1e8f6945-67a4-4506-8143-1ae310046ac0" (UID: "1e8f6945-67a4-4506-8143-1ae310046ac0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:33:56.962280 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:56.962171    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e8f6945-67a4-4506-8143-1ae310046ac0-kube-api-access-stqhz" (OuterVolumeSpecName: "kube-api-access-stqhz") pod "1e8f6945-67a4-4506-8143-1ae310046ac0" (UID: "1e8f6945-67a4-4506-8143-1ae310046ac0"). InnerVolumeSpecName "kube-api-access-stqhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:33:57.060900 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:57.060864    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e8f6945-67a4-4506-8143-1ae310046ac0-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:33:57.060900 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:57.060895    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stqhz\" (UniqueName: \"kubernetes.io/projected/1e8f6945-67a4-4506-8143-1ae310046ac0-kube-api-access-stqhz\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:33:57.060900 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:57.060905    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e8f6945-67a4-4506-8143-1ae310046ac0-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:33:57.252631 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:57.252597    2561 generic.go:356] "Generic (PLEG): container finished" podID="1e8f6945-67a4-4506-8143-1ae310046ac0" containerID="7e9387d379fca76156753e21c45871524d02e730c9dc13745bc3209ae6ff1a99" exitCode=0
Aug 05 17:33:57.252796 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:57.252661    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"
Aug 05 17:33:57.252796 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:57.252683    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm" event={"ID":"1e8f6945-67a4-4506-8143-1ae310046ac0","Type":"ContainerDied","Data":"7e9387d379fca76156753e21c45871524d02e730c9dc13745bc3209ae6ff1a99"}
Aug 05 17:33:57.252796 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:57.252737    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm" event={"ID":"1e8f6945-67a4-4506-8143-1ae310046ac0","Type":"ContainerDied","Data":"e86b2ab9e3b8bebfd2780eb3ddae97a61a12be081834400343a7b72bc6d048c7"}
Aug 05 17:33:57.252796 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:57.252762    2561 scope.go:117] "RemoveContainer" containerID="7e9387d379fca76156753e21c45871524d02e730c9dc13745bc3209ae6ff1a99"
Aug 05 17:33:57.262675 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:57.262659    2561 scope.go:117] "RemoveContainer" containerID="7e9387d379fca76156753e21c45871524d02e730c9dc13745bc3209ae6ff1a99"
Aug 05 17:33:57.262992 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:33:57.262934    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e9387d379fca76156753e21c45871524d02e730c9dc13745bc3209ae6ff1a99\": container with ID starting with 7e9387d379fca76156753e21c45871524d02e730c9dc13745bc3209ae6ff1a99 not found: ID does not exist" containerID="7e9387d379fca76156753e21c45871524d02e730c9dc13745bc3209ae6ff1a99"
Aug 05 17:33:57.263105 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:57.263001    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e9387d379fca76156753e21c45871524d02e730c9dc13745bc3209ae6ff1a99"} err="failed to get container status \"7e9387d379fca76156753e21c45871524d02e730c9dc13745bc3209ae6ff1a99\": rpc error: code = NotFound desc = could not find container \"7e9387d379fca76156753e21c45871524d02e730c9dc13745bc3209ae6ff1a99\": container with ID starting with 7e9387d379fca76156753e21c45871524d02e730c9dc13745bc3209ae6ff1a99 not found: ID does not exist"
Aug 05 17:33:57.278855 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:57.278829    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"]
Aug 05 17:33:57.285180 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:57.285159    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-jvsbm"]
Aug 05 17:33:58.730476 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:33:58.730440    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e8f6945-67a4-4506-8143-1ae310046ac0" path="/var/lib/kubelet/pods/1e8f6945-67a4-4506-8143-1ae310046ac0/volumes"
Aug 05 17:34:06.447859 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:06.447810    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb" podUID="697ee405-32b1-471c-b9dd-ff4a75c1decf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Aug 05 17:34:16.447846 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:16.447801    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb" podUID="697ee405-32b1-471c-b9dd-ff4a75c1decf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Aug 05 17:34:20.809506 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:20.809472    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"]
Aug 05 17:34:20.809919 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:20.809852    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e8f6945-67a4-4506-8143-1ae310046ac0" containerName="main"
Aug 05 17:34:20.809919 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:20.809864    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e8f6945-67a4-4506-8143-1ae310046ac0" containerName="main"
Aug 05 17:34:20.810023 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:20.809937    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e8f6945-67a4-4506-8143-1ae310046ac0" containerName="main"
Aug 05 17:34:20.814645 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:20.814626    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"
Aug 05 17:34:20.818224 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:20.818196    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Aug 05 17:34:20.818381 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:20.818196    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-ffggt\""
Aug 05 17:34:20.818381 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:20.818266    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Aug 05 17:34:20.818381 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:20.818281    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Aug 05 17:34:20.822041 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:20.822016    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"]
Aug 05 17:34:20.883352 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:20.883310    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdmtq\" (UniqueName: \"kubernetes.io/projected/9025febd-fb06-4775-b9bb-75949388802f-kube-api-access-pdmtq\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv\" (UID: \"9025febd-fb06-4775-b9bb-75949388802f\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"
Aug 05 17:34:20.883541 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:20.883376    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9025febd-fb06-4775-b9bb-75949388802f-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv\" (UID: \"9025febd-fb06-4775-b9bb-75949388802f\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"
Aug 05 17:34:20.883541 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:20.883415    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9025febd-fb06-4775-b9bb-75949388802f-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv\" (UID: \"9025febd-fb06-4775-b9bb-75949388802f\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"
Aug 05 17:34:20.883541 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:20.883445    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9025febd-fb06-4775-b9bb-75949388802f-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv\" (UID: \"9025febd-fb06-4775-b9bb-75949388802f\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"
Aug 05 17:34:20.883541 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:20.883482    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9025febd-fb06-4775-b9bb-75949388802f-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv\" (UID: \"9025febd-fb06-4775-b9bb-75949388802f\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"
Aug 05 17:34:20.883707 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:20.883589    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9025febd-fb06-4775-b9bb-75949388802f-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv\" (UID: \"9025febd-fb06-4775-b9bb-75949388802f\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"
Aug 05 17:34:20.984374 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:20.984345    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdmtq\" (UniqueName: \"kubernetes.io/projected/9025febd-fb06-4775-b9bb-75949388802f-kube-api-access-pdmtq\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv\" (UID: \"9025febd-fb06-4775-b9bb-75949388802f\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"
Aug 05 17:34:20.984584 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:20.984397    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9025febd-fb06-4775-b9bb-75949388802f-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv\" (UID: \"9025febd-fb06-4775-b9bb-75949388802f\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"
Aug 05 17:34:20.984584 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:20.984431    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9025febd-fb06-4775-b9bb-75949388802f-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv\" (UID: \"9025febd-fb06-4775-b9bb-75949388802f\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"
Aug 05 17:34:20.984584 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:20.984455    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9025febd-fb06-4775-b9bb-75949388802f-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv\" (UID: \"9025febd-fb06-4775-b9bb-75949388802f\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"
Aug 05 17:34:20.984584 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:20.984481    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9025febd-fb06-4775-b9bb-75949388802f-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv\" (UID: \"9025febd-fb06-4775-b9bb-75949388802f\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"
Aug 05 17:34:20.984584 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:20.984555    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9025febd-fb06-4775-b9bb-75949388802f-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv\" (UID: \"9025febd-fb06-4775-b9bb-75949388802f\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"
Aug 05 17:34:20.984902 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:20.984852    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9025febd-fb06-4775-b9bb-75949388802f-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv\" (UID: \"9025febd-fb06-4775-b9bb-75949388802f\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"
Aug 05 17:34:20.985000 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:20.984976    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9025febd-fb06-4775-b9bb-75949388802f-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv\" (UID: \"9025febd-fb06-4775-b9bb-75949388802f\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"
Aug 05 17:34:20.985133 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:20.985112    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9025febd-fb06-4775-b9bb-75949388802f-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv\" (UID: \"9025febd-fb06-4775-b9bb-75949388802f\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"
Aug 05 17:34:20.987488 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:20.987450    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9025febd-fb06-4775-b9bb-75949388802f-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv\" (UID: \"9025febd-fb06-4775-b9bb-75949388802f\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"
Aug 05 17:34:20.987671 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:20.987649    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9025febd-fb06-4775-b9bb-75949388802f-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv\" (UID: \"9025febd-fb06-4775-b9bb-75949388802f\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"
Aug 05 17:34:20.995394 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:20.995363    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdmtq\" (UniqueName: \"kubernetes.io/projected/9025febd-fb06-4775-b9bb-75949388802f-kube-api-access-pdmtq\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv\" (UID: \"9025febd-fb06-4775-b9bb-75949388802f\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"
Aug 05 17:34:21.013438 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:21.013349    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5"]
Aug 05 17:34:21.017031 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:21.017014    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5"
Aug 05 17:34:21.029797 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:21.029769    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-6sr77\""
Aug 05 17:34:21.032493 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:21.032464    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5"]
Aug 05 17:34:21.085309 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:21.085166    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzbnx\" (UniqueName: \"kubernetes.io/projected/76079243-c5ad-44d5-90bc-bba6ef62a1d3-kube-api-access-nzbnx\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5\" (UID: \"76079243-c5ad-44d5-90bc-bba6ef62a1d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5"
Aug 05 17:34:21.085498 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:21.085337    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76079243-c5ad-44d5-90bc-bba6ef62a1d3-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5\" (UID: \"76079243-c5ad-44d5-90bc-bba6ef62a1d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5"
Aug 05 17:34:21.127914 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:21.127875    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"
Aug 05 17:34:21.186538 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:21.186469    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76079243-c5ad-44d5-90bc-bba6ef62a1d3-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5\" (UID: \"76079243-c5ad-44d5-90bc-bba6ef62a1d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5"
Aug 05 17:34:21.186730 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:21.186618    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzbnx\" (UniqueName: \"kubernetes.io/projected/76079243-c5ad-44d5-90bc-bba6ef62a1d3-kube-api-access-nzbnx\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5\" (UID: \"76079243-c5ad-44d5-90bc-bba6ef62a1d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5"
Aug 05 17:34:21.190655 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:21.190624    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76079243-c5ad-44d5-90bc-bba6ef62a1d3-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5\" (UID: \"76079243-c5ad-44d5-90bc-bba6ef62a1d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5"
Aug 05 17:34:21.198698 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:21.198217    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzbnx\" (UniqueName: \"kubernetes.io/projected/76079243-c5ad-44d5-90bc-bba6ef62a1d3-kube-api-access-nzbnx\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5\" (UID: \"76079243-c5ad-44d5-90bc-bba6ef62a1d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5"
Aug 05 17:34:21.286163 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:21.286129    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"]
Aug 05 17:34:21.288412 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:34:21.288381    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9025febd_fb06_4775_b9bb_75949388802f.slice/crio-3137abeb00ce52898de5b9f7313fcdd7f8746f93c2f609512fb3f73b1c37babe WatchSource:0}: Error finding container 3137abeb00ce52898de5b9f7313fcdd7f8746f93c2f609512fb3f73b1c37babe: Status 404 returned error can't find the container with id 3137abeb00ce52898de5b9f7313fcdd7f8746f93c2f609512fb3f73b1c37babe
Aug 05 17:34:21.336033 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:21.335939    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5"
Aug 05 17:34:21.411765 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:21.411730    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv" event={"ID":"9025febd-fb06-4775-b9bb-75949388802f","Type":"ContainerStarted","Data":"43ab8f72a16588d24af3ba674871b3e2fb606f271dbe80646e169b60877c4679"}
Aug 05 17:34:21.411934 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:21.411785    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv" event={"ID":"9025febd-fb06-4775-b9bb-75949388802f","Type":"ContainerStarted","Data":"3137abeb00ce52898de5b9f7313fcdd7f8746f93c2f609512fb3f73b1c37babe"}
Aug 05 17:34:21.433440 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:21.433391    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv" podStartSLOduration=1.4333731219999999 podStartE2EDuration="1.433373122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:34:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:34:21.430989894 +0000 UTC m=+1317.175624225" watchObservedRunningTime="2026-08-05 17:34:21.433373122 +0000 UTC m=+1317.178007450"
Aug 05 17:34:21.489303 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:21.489273    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5"]
Aug 05 17:34:21.491045 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:34:21.491018    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76079243_c5ad_44d5_90bc_bba6ef62a1d3.slice/crio-eca4dc944388b344f1362d00829a8f2d05ac91e2b22ecbcfc7222869d0b4405e WatchSource:0}: Error finding container eca4dc944388b344f1362d00829a8f2d05ac91e2b22ecbcfc7222869d0b4405e: Status 404 returned error can't find the container with id eca4dc944388b344f1362d00829a8f2d05ac91e2b22ecbcfc7222869d0b4405e
Aug 05 17:34:21.493375 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:21.493356    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 05 17:34:22.128512 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:22.128470    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"
Aug 05 17:34:22.423461 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:22.423367    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5" event={"ID":"76079243-c5ad-44d5-90bc-bba6ef62a1d3","Type":"ContainerStarted","Data":"eca4dc944388b344f1362d00829a8f2d05ac91e2b22ecbcfc7222869d0b4405e"}
Aug 05 17:34:24.436756 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:24.436662    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5" event={"ID":"76079243-c5ad-44d5-90bc-bba6ef62a1d3","Type":"ContainerStarted","Data":"1865a7b7c3258dca5d41d213e3a6766ded37ea1442fd7b03ec0bd64e03a183f6"}
Aug 05 17:34:24.437238 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:24.436758    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5"
Aug 05 17:34:24.455718 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:24.455655    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5" podStartSLOduration=1.76740607 podStartE2EDuration="4.455636898s" totalImagesPullingTime="2.688230828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:34:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:34:24.453007643 +0000 UTC m=+1320.197641973" watchObservedRunningTime="2026-08-05 17:34:24.455636898 +0000 UTC m=+1320.200271227"
Aug 05 17:34:26.448497 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:26.448445    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb" podUID="697ee405-32b1-471c-b9dd-ff4a75c1decf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Aug 05 17:34:31.128817 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:31.128776    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"
Aug 05 17:34:31.141399 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:31.141374    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"
Aug 05 17:34:31.489723 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:31.489619    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"
Aug 05 17:34:36.448552 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:36.448504    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb" podUID="697ee405-32b1-471c-b9dd-ff4a75c1decf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Aug 05 17:34:46.448071 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:46.447940    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb" podUID="697ee405-32b1-471c-b9dd-ff4a75c1decf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Aug 05 17:34:54.924745 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:54.924708    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5"]
Aug 05 17:34:54.927325 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:54.927274    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5" podUID="76079243-c5ad-44d5-90bc-bba6ef62a1d3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 05 17:34:54.933655 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:54.933630    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"]
Aug 05 17:34:54.943256 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:34:54.943213    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 05 17:34:54.943256 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:34:54.943256 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:34:54.943256 ip-10-0-135-151 kubenswrapper[2561]:  > containerID="43ab8f72a16588d24af3ba674871b3e2fb606f271dbe80646e169b60877c4679" cmd=["/bin/sleep","15"]
Aug 05 17:34:54.943571 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:54.943265    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv" message="" err=<
Aug 05 17:34:54.943571 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:34:54.943571 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:34:54.943571 ip-10-0-135-151 kubenswrapper[2561]:  >
Aug 05 17:34:54.943571 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:34:54.943278    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 05 17:34:54.943571 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:34:54.943571 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:34:54.943571 ip-10-0-135-151 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv" podUID="9025febd-fb06-4775-b9bb-75949388802f" containerName="main" containerID="cri-o://43ab8f72a16588d24af3ba674871b3e2fb606f271dbe80646e169b60877c4679"
Aug 05 17:34:54.943571 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:54.943313    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv" podUID="9025febd-fb06-4775-b9bb-75949388802f" containerName="main" containerID="cri-o://43ab8f72a16588d24af3ba674871b3e2fb606f271dbe80646e169b60877c4679" gracePeriod=60
Aug 05 17:34:55.284406 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:55.284376    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"
Aug 05 17:34:55.417046 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:55.417015    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9025febd-fb06-4775-b9bb-75949388802f-tls-certs\") pod \"9025febd-fb06-4775-b9bb-75949388802f\" (UID: \"9025febd-fb06-4775-b9bb-75949388802f\") "
Aug 05 17:34:55.417046 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:55.417054    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9025febd-fb06-4775-b9bb-75949388802f-model-cache\") pod \"9025febd-fb06-4775-b9bb-75949388802f\" (UID: \"9025febd-fb06-4775-b9bb-75949388802f\") "
Aug 05 17:34:55.417266 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:55.417108    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdmtq\" (UniqueName: \"kubernetes.io/projected/9025febd-fb06-4775-b9bb-75949388802f-kube-api-access-pdmtq\") pod \"9025febd-fb06-4775-b9bb-75949388802f\" (UID: \"9025febd-fb06-4775-b9bb-75949388802f\") "
Aug 05 17:34:55.417266 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:55.417151    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9025febd-fb06-4775-b9bb-75949388802f-home\") pod \"9025febd-fb06-4775-b9bb-75949388802f\" (UID: \"9025febd-fb06-4775-b9bb-75949388802f\") "
Aug 05 17:34:55.417266 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:55.417174    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9025febd-fb06-4775-b9bb-75949388802f-tmp-dir\") pod \"9025febd-fb06-4775-b9bb-75949388802f\" (UID: \"9025febd-fb06-4775-b9bb-75949388802f\") "
Aug 05 17:34:55.417420 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:55.417347    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9025febd-fb06-4775-b9bb-75949388802f-model-cache" (OuterVolumeSpecName: "model-cache") pod "9025febd-fb06-4775-b9bb-75949388802f" (UID: "9025febd-fb06-4775-b9bb-75949388802f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:34:55.417478 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:55.417428    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9025febd-fb06-4775-b9bb-75949388802f-home" (OuterVolumeSpecName: "home") pod "9025febd-fb06-4775-b9bb-75949388802f" (UID: "9025febd-fb06-4775-b9bb-75949388802f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:34:55.417478 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:55.417462    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9025febd-fb06-4775-b9bb-75949388802f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9025febd-fb06-4775-b9bb-75949388802f" (UID: "9025febd-fb06-4775-b9bb-75949388802f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:34:55.417574 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:55.417476    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9025febd-fb06-4775-b9bb-75949388802f-dshm\") pod \"9025febd-fb06-4775-b9bb-75949388802f\" (UID: \"9025febd-fb06-4775-b9bb-75949388802f\") "
Aug 05 17:34:55.417781 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:55.417757    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9025febd-fb06-4775-b9bb-75949388802f-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:34:55.417781 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:55.417778    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9025febd-fb06-4775-b9bb-75949388802f-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:34:55.417781 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:55.417786    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9025febd-fb06-4775-b9bb-75949388802f-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:34:55.419642 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:55.419594    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9025febd-fb06-4775-b9bb-75949388802f-dshm" (OuterVolumeSpecName: "dshm") pod "9025febd-fb06-4775-b9bb-75949388802f" (UID: "9025febd-fb06-4775-b9bb-75949388802f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:34:55.419734 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:55.419719    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9025febd-fb06-4775-b9bb-75949388802f-kube-api-access-pdmtq" (OuterVolumeSpecName: "kube-api-access-pdmtq") pod "9025febd-fb06-4775-b9bb-75949388802f" (UID: "9025febd-fb06-4775-b9bb-75949388802f"). InnerVolumeSpecName "kube-api-access-pdmtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:34:55.419774 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:55.419730    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9025febd-fb06-4775-b9bb-75949388802f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9025febd-fb06-4775-b9bb-75949388802f" (UID: "9025febd-fb06-4775-b9bb-75949388802f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:34:55.518903 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:55.518791    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9025febd-fb06-4775-b9bb-75949388802f-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:34:55.518903 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:55.518838    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9025febd-fb06-4775-b9bb-75949388802f-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:34:55.518903 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:55.518857    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdmtq\" (UniqueName: \"kubernetes.io/projected/9025febd-fb06-4775-b9bb-75949388802f-kube-api-access-pdmtq\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:34:55.613889 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:55.613851    2561 generic.go:356] "Generic (PLEG): container finished" podID="9025febd-fb06-4775-b9bb-75949388802f" containerID="43ab8f72a16588d24af3ba674871b3e2fb606f271dbe80646e169b60877c4679" exitCode=0
Aug 05 17:34:55.614084 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:55.613900    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"
Aug 05 17:34:55.614084 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:55.613938    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv" event={"ID":"9025febd-fb06-4775-b9bb-75949388802f","Type":"ContainerDied","Data":"43ab8f72a16588d24af3ba674871b3e2fb606f271dbe80646e169b60877c4679"}
Aug 05 17:34:55.614084 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:55.614016    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv" event={"ID":"9025febd-fb06-4775-b9bb-75949388802f","Type":"ContainerDied","Data":"3137abeb00ce52898de5b9f7313fcdd7f8746f93c2f609512fb3f73b1c37babe"}
Aug 05 17:34:55.614084 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:55.614069    2561 scope.go:117] "RemoveContainer" containerID="43ab8f72a16588d24af3ba674871b3e2fb606f271dbe80646e169b60877c4679"
Aug 05 17:34:55.627558 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:55.627539    2561 scope.go:117] "RemoveContainer" containerID="43ab8f72a16588d24af3ba674871b3e2fb606f271dbe80646e169b60877c4679"
Aug 05 17:34:55.627839 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:34:55.627822    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43ab8f72a16588d24af3ba674871b3e2fb606f271dbe80646e169b60877c4679\": container with ID starting with 43ab8f72a16588d24af3ba674871b3e2fb606f271dbe80646e169b60877c4679 not found: ID does not exist" containerID="43ab8f72a16588d24af3ba674871b3e2fb606f271dbe80646e169b60877c4679"
Aug 05 17:34:55.627905 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:55.627846    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43ab8f72a16588d24af3ba674871b3e2fb606f271dbe80646e169b60877c4679"} err="failed to get container status \"43ab8f72a16588d24af3ba674871b3e2fb606f271dbe80646e169b60877c4679\": rpc error: code = NotFound desc = could not find container \"43ab8f72a16588d24af3ba674871b3e2fb606f271dbe80646e169b60877c4679\": container with ID starting with 43ab8f72a16588d24af3ba674871b3e2fb606f271dbe80646e169b60877c4679 not found: ID does not exist"
Aug 05 17:34:55.641358 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:55.641331    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"]
Aug 05 17:34:55.646442 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:55.646407    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-k5jkv"]
Aug 05 17:34:56.448468 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:56.448425    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb" podUID="697ee405-32b1-471c-b9dd-ff4a75c1decf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Aug 05 17:34:56.730876 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:34:56.730785    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9025febd-fb06-4775-b9bb-75949388802f" path="/var/lib/kubelet/pods/9025febd-fb06-4775-b9bb-75949388802f/volumes"
Aug 05 17:35:04.926929 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:04.926885    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5" podUID="76079243-c5ad-44d5-90bc-bba6ef62a1d3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 05 17:35:06.457869 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:06.457837    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:35:06.465398 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:06.465362    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:35:07.968005 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:07.967968    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"]
Aug 05 17:35:09.947736 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:09.947690    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5" podUID="76079243-c5ad-44d5-90bc-bba6ef62a1d3" containerName="main" containerID="cri-o://1865a7b7c3258dca5d41d213e3a6766ded37ea1442fd7b03ec0bd64e03a183f6" gracePeriod=45
Aug 05 17:35:10.206740 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:10.206682    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5"
Aug 05 17:35:10.360146 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:10.360115    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzbnx\" (UniqueName: \"kubernetes.io/projected/76079243-c5ad-44d5-90bc-bba6ef62a1d3-kube-api-access-nzbnx\") pod \"76079243-c5ad-44d5-90bc-bba6ef62a1d3\" (UID: \"76079243-c5ad-44d5-90bc-bba6ef62a1d3\") "
Aug 05 17:35:10.360345 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:10.360225    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76079243-c5ad-44d5-90bc-bba6ef62a1d3-tls-certs\") pod \"76079243-c5ad-44d5-90bc-bba6ef62a1d3\" (UID: \"76079243-c5ad-44d5-90bc-bba6ef62a1d3\") "
Aug 05 17:35:10.362202 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:10.362182    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76079243-c5ad-44d5-90bc-bba6ef62a1d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76079243-c5ad-44d5-90bc-bba6ef62a1d3" (UID: "76079243-c5ad-44d5-90bc-bba6ef62a1d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:35:10.362276 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:10.362184    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76079243-c5ad-44d5-90bc-bba6ef62a1d3-kube-api-access-nzbnx" (OuterVolumeSpecName: "kube-api-access-nzbnx") pod "76079243-c5ad-44d5-90bc-bba6ef62a1d3" (UID: "76079243-c5ad-44d5-90bc-bba6ef62a1d3"). InnerVolumeSpecName "kube-api-access-nzbnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:35:10.461698 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:10.461610    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76079243-c5ad-44d5-90bc-bba6ef62a1d3-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:35:10.461698 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:10.461644    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzbnx\" (UniqueName: \"kubernetes.io/projected/76079243-c5ad-44d5-90bc-bba6ef62a1d3-kube-api-access-nzbnx\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:35:10.702778 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:10.702742    2561 generic.go:356] "Generic (PLEG): container finished" podID="76079243-c5ad-44d5-90bc-bba6ef62a1d3" containerID="1865a7b7c3258dca5d41d213e3a6766ded37ea1442fd7b03ec0bd64e03a183f6" exitCode=0
Aug 05 17:35:10.702965 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:10.702801    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5"
Aug 05 17:35:10.702965 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:10.702820    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5" event={"ID":"76079243-c5ad-44d5-90bc-bba6ef62a1d3","Type":"ContainerDied","Data":"1865a7b7c3258dca5d41d213e3a6766ded37ea1442fd7b03ec0bd64e03a183f6"}
Aug 05 17:35:10.702965 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:10.702874    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5" event={"ID":"76079243-c5ad-44d5-90bc-bba6ef62a1d3","Type":"ContainerDied","Data":"eca4dc944388b344f1362d00829a8f2d05ac91e2b22ecbcfc7222869d0b4405e"}
Aug 05 17:35:10.702965 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:10.702887    2561 scope.go:117] "RemoveContainer" containerID="1865a7b7c3258dca5d41d213e3a6766ded37ea1442fd7b03ec0bd64e03a183f6"
Aug 05 17:35:10.718600 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:10.718581    2561 scope.go:117] "RemoveContainer" containerID="1865a7b7c3258dca5d41d213e3a6766ded37ea1442fd7b03ec0bd64e03a183f6"
Aug 05 17:35:10.718863 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:35:10.718843    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1865a7b7c3258dca5d41d213e3a6766ded37ea1442fd7b03ec0bd64e03a183f6\": container with ID starting with 1865a7b7c3258dca5d41d213e3a6766ded37ea1442fd7b03ec0bd64e03a183f6 not found: ID does not exist" containerID="1865a7b7c3258dca5d41d213e3a6766ded37ea1442fd7b03ec0bd64e03a183f6"
Aug 05 17:35:10.718923 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:10.718870    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1865a7b7c3258dca5d41d213e3a6766ded37ea1442fd7b03ec0bd64e03a183f6"} err="failed to get container status \"1865a7b7c3258dca5d41d213e3a6766ded37ea1442fd7b03ec0bd64e03a183f6\": rpc error: code = NotFound desc = could not find container \"1865a7b7c3258dca5d41d213e3a6766ded37ea1442fd7b03ec0bd64e03a183f6\": container with ID starting with 1865a7b7c3258dca5d41d213e3a6766ded37ea1442fd7b03ec0bd64e03a183f6 not found: ID does not exist"
Aug 05 17:35:10.731361 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:10.731338    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5"]
Aug 05 17:35:10.735111 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:10.735090    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfnl6w5"]
Aug 05 17:35:12.729824 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:12.729792    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76079243-c5ad-44d5-90bc-bba6ef62a1d3" path="/var/lib/kubelet/pods/76079243-c5ad-44d5-90bc-bba6ef62a1d3/volumes"
Aug 05 17:35:20.804978 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.804932    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"]
Aug 05 17:35:20.807319 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.805300    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76079243-c5ad-44d5-90bc-bba6ef62a1d3" containerName="main"
Aug 05 17:35:20.807319 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.805311    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="76079243-c5ad-44d5-90bc-bba6ef62a1d3" containerName="main"
Aug 05 17:35:20.807319 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.805334    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9025febd-fb06-4775-b9bb-75949388802f" containerName="main"
Aug 05 17:35:20.807319 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.805339    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9025febd-fb06-4775-b9bb-75949388802f" containerName="main"
Aug 05 17:35:20.807319 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.805391    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="76079243-c5ad-44d5-90bc-bba6ef62a1d3" containerName="main"
Aug 05 17:35:20.807319 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.805401    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9025febd-fb06-4775-b9bb-75949388802f" containerName="main"
Aug 05 17:35:20.808205 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.808189    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"
Aug 05 17:35:20.810814 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.810790    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Aug 05 17:35:20.810941 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.810860    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-j4crz\""
Aug 05 17:35:20.811922 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.811899    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Aug 05 17:35:20.812064 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.811905    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Aug 05 17:35:20.823359 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.823331    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"]
Aug 05 17:35:20.848890 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.848866    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-72vnm\" (UID: \"3d68ef4c-1e91-441b-804a-1f3ecddeba6e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"
Aug 05 17:35:20.849059 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.848897    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-72vnm\" (UID: \"3d68ef4c-1e91-441b-804a-1f3ecddeba6e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"
Aug 05 17:35:20.849059 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.848926    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-72vnm\" (UID: \"3d68ef4c-1e91-441b-804a-1f3ecddeba6e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"
Aug 05 17:35:20.849059 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.848946    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfnkv\" (UniqueName: \"kubernetes.io/projected/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-kube-api-access-jfnkv\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-72vnm\" (UID: \"3d68ef4c-1e91-441b-804a-1f3ecddeba6e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"
Aug 05 17:35:20.849059 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.849042    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-72vnm\" (UID: \"3d68ef4c-1e91-441b-804a-1f3ecddeba6e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"
Aug 05 17:35:20.849269 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.849085    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-72vnm\" (UID: \"3d68ef4c-1e91-441b-804a-1f3ecddeba6e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"
Aug 05 17:35:20.950000 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.949941    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-72vnm\" (UID: \"3d68ef4c-1e91-441b-804a-1f3ecddeba6e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"
Aug 05 17:35:20.950203 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.950007    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-72vnm\" (UID: \"3d68ef4c-1e91-441b-804a-1f3ecddeba6e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"
Aug 05 17:35:20.950203 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.950055    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-72vnm\" (UID: \"3d68ef4c-1e91-441b-804a-1f3ecddeba6e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"
Aug 05 17:35:20.950203 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.950073    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfnkv\" (UniqueName: \"kubernetes.io/projected/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-kube-api-access-jfnkv\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-72vnm\" (UID: \"3d68ef4c-1e91-441b-804a-1f3ecddeba6e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"
Aug 05 17:35:20.950203 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.950123    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-72vnm\" (UID: \"3d68ef4c-1e91-441b-804a-1f3ecddeba6e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"
Aug 05 17:35:20.950203 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.950162    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-72vnm\" (UID: \"3d68ef4c-1e91-441b-804a-1f3ecddeba6e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"
Aug 05 17:35:20.950470 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.950426    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-72vnm\" (UID: \"3d68ef4c-1e91-441b-804a-1f3ecddeba6e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"
Aug 05 17:35:20.950470 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.950440    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-72vnm\" (UID: \"3d68ef4c-1e91-441b-804a-1f3ecddeba6e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"
Aug 05 17:35:20.950546 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.950487    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-72vnm\" (UID: \"3d68ef4c-1e91-441b-804a-1f3ecddeba6e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"
Aug 05 17:35:20.952349 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.952328    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-72vnm\" (UID: \"3d68ef4c-1e91-441b-804a-1f3ecddeba6e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"
Aug 05 17:35:20.952658 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.952640    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-72vnm\" (UID: \"3d68ef4c-1e91-441b-804a-1f3ecddeba6e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"
Aug 05 17:35:20.958446 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:20.958428    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfnkv\" (UniqueName: \"kubernetes.io/projected/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-kube-api-access-jfnkv\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-72vnm\" (UID: \"3d68ef4c-1e91-441b-804a-1f3ecddeba6e\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"
Aug 05 17:35:21.118307 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:21.118214    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"
Aug 05 17:35:21.266262 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:21.266172    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"]
Aug 05 17:35:21.268434 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:35:21.268406    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d68ef4c_1e91_441b_804a_1f3ecddeba6e.slice/crio-f838d0b4d40af3c78ee66e993b7c27161712f39b379da3894f1185ec9eb70028 WatchSource:0}: Error finding container f838d0b4d40af3c78ee66e993b7c27161712f39b379da3894f1185ec9eb70028: Status 404 returned error can't find the container with id f838d0b4d40af3c78ee66e993b7c27161712f39b379da3894f1185ec9eb70028
Aug 05 17:35:21.762497 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:21.762462    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm" event={"ID":"3d68ef4c-1e91-441b-804a-1f3ecddeba6e","Type":"ContainerStarted","Data":"0d629359d23ce8182ccaa9ed862c7324742401856472f2adb8361927aeebf458"}
Aug 05 17:35:21.762705 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:21.762517    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm" event={"ID":"3d68ef4c-1e91-441b-804a-1f3ecddeba6e","Type":"ContainerStarted","Data":"f838d0b4d40af3c78ee66e993b7c27161712f39b379da3894f1185ec9eb70028"}
Aug 05 17:35:21.792578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:21.792521    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm" podStartSLOduration=1.792502554 podStartE2EDuration="1.792502554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:35:21.791483128 +0000 UTC m=+1377.536117454" watchObservedRunningTime="2026-08-05 17:35:21.792502554 +0000 UTC m=+1377.537136880"
Aug 05 17:35:22.118374 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:22.118337    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"
Aug 05 17:35:22.993645 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:22.993580    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb" podUID="697ee405-32b1-471c-b9dd-ff4a75c1decf" containerName="main" containerID="cri-o://96590d7f45f7c9a2f801ba4590d7e1286a60acc3ec2ee509c17f844b0723f171" gracePeriod=45
Aug 05 17:35:23.458760 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.458713    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb" podUID="697ee405-32b1-471c-b9dd-ff4a75c1decf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Aug 05 17:35:23.647754 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.647724    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:35:23.677244 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.677212    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/697ee405-32b1-471c-b9dd-ff4a75c1decf-tls-certs\") pod \"697ee405-32b1-471c-b9dd-ff4a75c1decf\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") "
Aug 05 17:35:23.677411 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.677256    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/697ee405-32b1-471c-b9dd-ff4a75c1decf-kserve-provision-location\") pod \"697ee405-32b1-471c-b9dd-ff4a75c1decf\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") "
Aug 05 17:35:23.677411 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.677287    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/697ee405-32b1-471c-b9dd-ff4a75c1decf-tmp-dir\") pod \"697ee405-32b1-471c-b9dd-ff4a75c1decf\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") "
Aug 05 17:35:23.677411 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.677315    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/697ee405-32b1-471c-b9dd-ff4a75c1decf-home\") pod \"697ee405-32b1-471c-b9dd-ff4a75c1decf\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") "
Aug 05 17:35:23.677411 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.677363    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ndc9\" (UniqueName: \"kubernetes.io/projected/697ee405-32b1-471c-b9dd-ff4a75c1decf-kube-api-access-6ndc9\") pod \"697ee405-32b1-471c-b9dd-ff4a75c1decf\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") "
Aug 05 17:35:23.677661 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.677437    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/697ee405-32b1-471c-b9dd-ff4a75c1decf-cabundle-cert\") pod \"697ee405-32b1-471c-b9dd-ff4a75c1decf\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") "
Aug 05 17:35:23.677661 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.677522    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/697ee405-32b1-471c-b9dd-ff4a75c1decf-dshm\") pod \"697ee405-32b1-471c-b9dd-ff4a75c1decf\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") "
Aug 05 17:35:23.677661 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.677549    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/697ee405-32b1-471c-b9dd-ff4a75c1decf-model-cache\") pod \"697ee405-32b1-471c-b9dd-ff4a75c1decf\" (UID: \"697ee405-32b1-471c-b9dd-ff4a75c1decf\") "
Aug 05 17:35:23.678229 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.678049    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/697ee405-32b1-471c-b9dd-ff4a75c1decf-model-cache" (OuterVolumeSpecName: "model-cache") pod "697ee405-32b1-471c-b9dd-ff4a75c1decf" (UID: "697ee405-32b1-471c-b9dd-ff4a75c1decf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:35:23.678860 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.678440    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/697ee405-32b1-471c-b9dd-ff4a75c1decf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "697ee405-32b1-471c-b9dd-ff4a75c1decf" (UID: "697ee405-32b1-471c-b9dd-ff4a75c1decf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:35:23.678860 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.678829    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/697ee405-32b1-471c-b9dd-ff4a75c1decf-home" (OuterVolumeSpecName: "home") pod "697ee405-32b1-471c-b9dd-ff4a75c1decf" (UID: "697ee405-32b1-471c-b9dd-ff4a75c1decf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:35:23.680409 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.680369    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/697ee405-32b1-471c-b9dd-ff4a75c1decf-dshm" (OuterVolumeSpecName: "dshm") pod "697ee405-32b1-471c-b9dd-ff4a75c1decf" (UID: "697ee405-32b1-471c-b9dd-ff4a75c1decf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:35:23.680963 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.680920    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/697ee405-32b1-471c-b9dd-ff4a75c1decf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "697ee405-32b1-471c-b9dd-ff4a75c1decf" (UID: "697ee405-32b1-471c-b9dd-ff4a75c1decf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:35:23.681322 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.681292    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/697ee405-32b1-471c-b9dd-ff4a75c1decf-kube-api-access-6ndc9" (OuterVolumeSpecName: "kube-api-access-6ndc9") pod "697ee405-32b1-471c-b9dd-ff4a75c1decf" (UID: "697ee405-32b1-471c-b9dd-ff4a75c1decf"). InnerVolumeSpecName "kube-api-access-6ndc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:35:23.696153 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.696115    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/697ee405-32b1-471c-b9dd-ff4a75c1decf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "697ee405-32b1-471c-b9dd-ff4a75c1decf" (UID: "697ee405-32b1-471c-b9dd-ff4a75c1decf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:35:23.703526 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.703495    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/697ee405-32b1-471c-b9dd-ff4a75c1decf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "697ee405-32b1-471c-b9dd-ff4a75c1decf" (UID: "697ee405-32b1-471c-b9dd-ff4a75c1decf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:35:23.775490 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.775402    2561 generic.go:356] "Generic (PLEG): container finished" podID="697ee405-32b1-471c-b9dd-ff4a75c1decf" containerID="96590d7f45f7c9a2f801ba4590d7e1286a60acc3ec2ee509c17f844b0723f171" exitCode=0
Aug 05 17:35:23.775490 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.775464    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"
Aug 05 17:35:23.775701 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.775483    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb" event={"ID":"697ee405-32b1-471c-b9dd-ff4a75c1decf","Type":"ContainerDied","Data":"96590d7f45f7c9a2f801ba4590d7e1286a60acc3ec2ee509c17f844b0723f171"}
Aug 05 17:35:23.775701 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.775552    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb" event={"ID":"697ee405-32b1-471c-b9dd-ff4a75c1decf","Type":"ContainerDied","Data":"1c8f1d5b1fa74fd080df014dfbc0536a39bc03a34f9e5968ad9e800f5c844f8a"}
Aug 05 17:35:23.775701 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.775585    2561 scope.go:117] "RemoveContainer" containerID="96590d7f45f7c9a2f801ba4590d7e1286a60acc3ec2ee509c17f844b0723f171"
Aug 05 17:35:23.778353 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.778327    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/697ee405-32b1-471c-b9dd-ff4a75c1decf-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:35:23.778353 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.778355    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ndc9\" (UniqueName: \"kubernetes.io/projected/697ee405-32b1-471c-b9dd-ff4a75c1decf-kube-api-access-6ndc9\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:35:23.778539 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.778365    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/697ee405-32b1-471c-b9dd-ff4a75c1decf-cabundle-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:35:23.778539 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.778374    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/697ee405-32b1-471c-b9dd-ff4a75c1decf-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:35:23.778539 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.778381    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/697ee405-32b1-471c-b9dd-ff4a75c1decf-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:35:23.778539 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.778388    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/697ee405-32b1-471c-b9dd-ff4a75c1decf-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:35:23.778539 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.778396    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/697ee405-32b1-471c-b9dd-ff4a75c1decf-kserve-provision-location\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:35:23.778539 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.778407    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/697ee405-32b1-471c-b9dd-ff4a75c1decf-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:35:23.785449 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.785425    2561 scope.go:117] "RemoveContainer" containerID="09c68d1e9bf2e2e1eccc883c0447272b3be27b1b870642b5e420cf2b3b7f2fa8"
Aug 05 17:35:23.793766 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.793751    2561 scope.go:117] "RemoveContainer" containerID="96590d7f45f7c9a2f801ba4590d7e1286a60acc3ec2ee509c17f844b0723f171"
Aug 05 17:35:23.797487 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:35:23.797431    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96590d7f45f7c9a2f801ba4590d7e1286a60acc3ec2ee509c17f844b0723f171\": container with ID starting with 96590d7f45f7c9a2f801ba4590d7e1286a60acc3ec2ee509c17f844b0723f171 not found: ID does not exist" containerID="96590d7f45f7c9a2f801ba4590d7e1286a60acc3ec2ee509c17f844b0723f171"
Aug 05 17:35:23.797487 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.797472    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96590d7f45f7c9a2f801ba4590d7e1286a60acc3ec2ee509c17f844b0723f171"} err="failed to get container status \"96590d7f45f7c9a2f801ba4590d7e1286a60acc3ec2ee509c17f844b0723f171\": rpc error: code = NotFound desc = could not find container \"96590d7f45f7c9a2f801ba4590d7e1286a60acc3ec2ee509c17f844b0723f171\": container with ID starting with 96590d7f45f7c9a2f801ba4590d7e1286a60acc3ec2ee509c17f844b0723f171 not found: ID does not exist"
Aug 05 17:35:23.797675 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.797501    2561 scope.go:117] "RemoveContainer" containerID="09c68d1e9bf2e2e1eccc883c0447272b3be27b1b870642b5e420cf2b3b7f2fa8"
Aug 05 17:35:23.798111 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:35:23.798074    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09c68d1e9bf2e2e1eccc883c0447272b3be27b1b870642b5e420cf2b3b7f2fa8\": container with ID starting with 09c68d1e9bf2e2e1eccc883c0447272b3be27b1b870642b5e420cf2b3b7f2fa8 not found: ID does not exist" containerID="09c68d1e9bf2e2e1eccc883c0447272b3be27b1b870642b5e420cf2b3b7f2fa8"
Aug 05 17:35:23.798232 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.798118    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09c68d1e9bf2e2e1eccc883c0447272b3be27b1b870642b5e420cf2b3b7f2fa8"} err="failed to get container status \"09c68d1e9bf2e2e1eccc883c0447272b3be27b1b870642b5e420cf2b3b7f2fa8\": rpc error: code = NotFound desc = could not find container \"09c68d1e9bf2e2e1eccc883c0447272b3be27b1b870642b5e420cf2b3b7f2fa8\": container with ID starting with 09c68d1e9bf2e2e1eccc883c0447272b3be27b1b870642b5e420cf2b3b7f2fa8 not found: ID does not exist"
Aug 05 17:35:23.800264 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.800238    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"]
Aug 05 17:35:23.804707 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:23.804689    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-f6d648757-r4cpb"]
Aug 05 17:35:24.729972 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:24.729936    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="697ee405-32b1-471c-b9dd-ff4a75c1decf" path="/var/lib/kubelet/pods/697ee405-32b1-471c-b9dd-ff4a75c1decf/volumes"
Aug 05 17:35:31.118617 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:31.118578    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"
Aug 05 17:35:31.131237 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:31.131212    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"
Aug 05 17:35:31.827248 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:35:31.827214    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"
Aug 05 17:36:23.333714 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.333633    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"]
Aug 05 17:36:23.334183 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.334045    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="697ee405-32b1-471c-b9dd-ff4a75c1decf" containerName="storage-initializer"
Aug 05 17:36:23.334183 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.334059    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="697ee405-32b1-471c-b9dd-ff4a75c1decf" containerName="storage-initializer"
Aug 05 17:36:23.334183 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.334076    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="697ee405-32b1-471c-b9dd-ff4a75c1decf" containerName="main"
Aug 05 17:36:23.334183 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.334081    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="697ee405-32b1-471c-b9dd-ff4a75c1decf" containerName="main"
Aug 05 17:36:23.334183 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.334149    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="697ee405-32b1-471c-b9dd-ff4a75c1decf" containerName="main"
Aug 05 17:36:23.337212 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.337196    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:36:23.339786 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.339761    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Aug 05 17:36:23.340323 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.340301    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Aug 05 17:36:23.340423 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.340357    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Aug 05 17:36:23.340423 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.340379    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Aug 05 17:36:23.340423 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.340387    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-fg72r\""
Aug 05 17:36:23.340613 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.340592    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Aug 05 17:36:23.349463 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.349440    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"]
Aug 05 17:36:23.389668 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.389639    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07b14313-4de3-4353-92da-e347fe4286bf-dshm\") pod \"auth-disabled-test-kserve-546d97dc78-4zjd2\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:36:23.389839 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.389676    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07b14313-4de3-4353-92da-e347fe4286bf-tls-certs\") pod \"auth-disabled-test-kserve-546d97dc78-4zjd2\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:36:23.389839 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.389704    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07b14313-4de3-4353-92da-e347fe4286bf-home\") pod \"auth-disabled-test-kserve-546d97dc78-4zjd2\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:36:23.389839 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.389782    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07b14313-4de3-4353-92da-e347fe4286bf-tmp-dir\") pod \"auth-disabled-test-kserve-546d97dc78-4zjd2\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:36:23.389839 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.389812    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07b14313-4de3-4353-92da-e347fe4286bf-model-cache\") pod \"auth-disabled-test-kserve-546d97dc78-4zjd2\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:36:23.390104 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.389847    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07b14313-4de3-4353-92da-e347fe4286bf-kserve-provision-location\") pod \"auth-disabled-test-kserve-546d97dc78-4zjd2\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:36:23.390104 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.389890    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vbb2\" (UniqueName: \"kubernetes.io/projected/07b14313-4de3-4353-92da-e347fe4286bf-kube-api-access-8vbb2\") pod \"auth-disabled-test-kserve-546d97dc78-4zjd2\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:36:23.390104 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.390026    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07b14313-4de3-4353-92da-e347fe4286bf-cabundle-cert\") pod \"auth-disabled-test-kserve-546d97dc78-4zjd2\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:36:23.491164 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.491132    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07b14313-4de3-4353-92da-e347fe4286bf-kserve-provision-location\") pod \"auth-disabled-test-kserve-546d97dc78-4zjd2\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:36:23.491164 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.491169    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vbb2\" (UniqueName: \"kubernetes.io/projected/07b14313-4de3-4353-92da-e347fe4286bf-kube-api-access-8vbb2\") pod \"auth-disabled-test-kserve-546d97dc78-4zjd2\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:36:23.491418 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.491250    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07b14313-4de3-4353-92da-e347fe4286bf-cabundle-cert\") pod \"auth-disabled-test-kserve-546d97dc78-4zjd2\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:36:23.491418 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.491288    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07b14313-4de3-4353-92da-e347fe4286bf-dshm\") pod \"auth-disabled-test-kserve-546d97dc78-4zjd2\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:36:23.491418 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.491310    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07b14313-4de3-4353-92da-e347fe4286bf-tls-certs\") pod \"auth-disabled-test-kserve-546d97dc78-4zjd2\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:36:23.491418 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.491336    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07b14313-4de3-4353-92da-e347fe4286bf-home\") pod \"auth-disabled-test-kserve-546d97dc78-4zjd2\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:36:23.491418 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.491364    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07b14313-4de3-4353-92da-e347fe4286bf-tmp-dir\") pod \"auth-disabled-test-kserve-546d97dc78-4zjd2\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:36:23.491418 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.491391    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07b14313-4de3-4353-92da-e347fe4286bf-model-cache\") pod \"auth-disabled-test-kserve-546d97dc78-4zjd2\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:36:23.491741 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.491619    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07b14313-4de3-4353-92da-e347fe4286bf-kserve-provision-location\") pod \"auth-disabled-test-kserve-546d97dc78-4zjd2\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:36:23.491741 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.491693    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07b14313-4de3-4353-92da-e347fe4286bf-model-cache\") pod \"auth-disabled-test-kserve-546d97dc78-4zjd2\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:36:23.491852 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.491744    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07b14313-4de3-4353-92da-e347fe4286bf-home\") pod \"auth-disabled-test-kserve-546d97dc78-4zjd2\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:36:23.491977 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.491938    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07b14313-4de3-4353-92da-e347fe4286bf-tmp-dir\") pod \"auth-disabled-test-kserve-546d97dc78-4zjd2\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:36:23.492064 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.492028    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07b14313-4de3-4353-92da-e347fe4286bf-cabundle-cert\") pod \"auth-disabled-test-kserve-546d97dc78-4zjd2\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:36:23.493506 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.493485    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07b14313-4de3-4353-92da-e347fe4286bf-dshm\") pod \"auth-disabled-test-kserve-546d97dc78-4zjd2\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:36:23.493855 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.493838    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07b14313-4de3-4353-92da-e347fe4286bf-tls-certs\") pod \"auth-disabled-test-kserve-546d97dc78-4zjd2\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:36:23.500553 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.500528    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vbb2\" (UniqueName: \"kubernetes.io/projected/07b14313-4de3-4353-92da-e347fe4286bf-kube-api-access-8vbb2\") pod \"auth-disabled-test-kserve-546d97dc78-4zjd2\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:36:23.648190 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.648093    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:36:23.793001 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:23.792975    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"]
Aug 05 17:36:23.794443 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:36:23.794406    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b14313_4de3_4353_92da_e347fe4286bf.slice/crio-f45fc1f34fa1c8e28a646e12e5f59a6bdd6903451c7a04c1ba6a0ae718e1c621 WatchSource:0}: Error finding container f45fc1f34fa1c8e28a646e12e5f59a6bdd6903451c7a04c1ba6a0ae718e1c621: Status 404 returned error can't find the container with id f45fc1f34fa1c8e28a646e12e5f59a6bdd6903451c7a04c1ba6a0ae718e1c621
Aug 05 17:36:24.089440 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:24.089404    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2" event={"ID":"07b14313-4de3-4353-92da-e347fe4286bf","Type":"ContainerStarted","Data":"25a09fa80e35d672a4febf6b88b086cf5258894edf947d8b619d114bbf59db30"}
Aug 05 17:36:24.089638 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:24.089457    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2" event={"ID":"07b14313-4de3-4353-92da-e347fe4286bf","Type":"ContainerStarted","Data":"f45fc1f34fa1c8e28a646e12e5f59a6bdd6903451c7a04c1ba6a0ae718e1c621"}
Aug 05 17:36:27.107627 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:27.107592    2561 generic.go:356] "Generic (PLEG): container finished" podID="07b14313-4de3-4353-92da-e347fe4286bf" containerID="25a09fa80e35d672a4febf6b88b086cf5258894edf947d8b619d114bbf59db30" exitCode=0
Aug 05 17:36:27.108096 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:27.107669    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2" event={"ID":"07b14313-4de3-4353-92da-e347fe4286bf","Type":"ContainerDied","Data":"25a09fa80e35d672a4febf6b88b086cf5258894edf947d8b619d114bbf59db30"}
Aug 05 17:36:28.114857 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:28.114818    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2" event={"ID":"07b14313-4de3-4353-92da-e347fe4286bf","Type":"ContainerStarted","Data":"2860bbae8c41f5bd4916e85a3ebd420df7c539a2f9b2b28cd343cc4eff254045"}
Aug 05 17:36:28.140126 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:28.140065    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2" podStartSLOduration=2.750071365 podStartE2EDuration="5.140045249s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.389973884s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:36:28.139144136 +0000 UTC m=+1443.883778486" watchObservedRunningTime="2026-08-05 17:36:28.140045249 +0000 UTC m=+1443.884679575"
Aug 05 17:36:28.648644 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:28.648604    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:36:33.648707 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:33.648662    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:36:33.650603 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:33.650567    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2" podUID="07b14313-4de3-4353-92da-e347fe4286bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Aug 05 17:36:39.080011 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:39.079969    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"]
Aug 05 17:36:39.091344 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:36:39.091289    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 05 17:36:39.091344 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:36:39.091344 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:36:39.091344 ip-10-0-135-151 kubenswrapper[2561]:  > containerID="0d629359d23ce8182ccaa9ed862c7324742401856472f2adb8361927aeebf458" cmd=["/bin/sleep","15"]
Aug 05 17:36:39.091670 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:39.091364    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm" message="" err=<
Aug 05 17:36:39.091670 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:36:39.091670 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:36:39.091670 ip-10-0-135-151 kubenswrapper[2561]:  >
Aug 05 17:36:39.091670 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:36:39.091380    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 05 17:36:39.091670 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:36:39.091670 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:36:39.091670 ip-10-0-135-151 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm" podUID="3d68ef4c-1e91-441b-804a-1f3ecddeba6e" containerName="main" containerID="cri-o://0d629359d23ce8182ccaa9ed862c7324742401856472f2adb8361927aeebf458"
Aug 05 17:36:39.091670 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:39.091428    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm" podUID="3d68ef4c-1e91-441b-804a-1f3ecddeba6e" containerName="main" containerID="cri-o://0d629359d23ce8182ccaa9ed862c7324742401856472f2adb8361927aeebf458" gracePeriod=60
Aug 05 17:36:39.446114 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:39.446083    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"
Aug 05 17:36:39.542618 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:39.542581    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-dshm\") pod \"3d68ef4c-1e91-441b-804a-1f3ecddeba6e\" (UID: \"3d68ef4c-1e91-441b-804a-1f3ecddeba6e\") "
Aug 05 17:36:39.542814 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:39.542640    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfnkv\" (UniqueName: \"kubernetes.io/projected/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-kube-api-access-jfnkv\") pod \"3d68ef4c-1e91-441b-804a-1f3ecddeba6e\" (UID: \"3d68ef4c-1e91-441b-804a-1f3ecddeba6e\") "
Aug 05 17:36:39.542814 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:39.542675    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-tmp-dir\") pod \"3d68ef4c-1e91-441b-804a-1f3ecddeba6e\" (UID: \"3d68ef4c-1e91-441b-804a-1f3ecddeba6e\") "
Aug 05 17:36:39.542814 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:39.542691    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-tls-certs\") pod \"3d68ef4c-1e91-441b-804a-1f3ecddeba6e\" (UID: \"3d68ef4c-1e91-441b-804a-1f3ecddeba6e\") "
Aug 05 17:36:39.542814 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:39.542740    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-home\") pod \"3d68ef4c-1e91-441b-804a-1f3ecddeba6e\" (UID: \"3d68ef4c-1e91-441b-804a-1f3ecddeba6e\") "
Aug 05 17:36:39.542814 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:39.542759    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-model-cache\") pod \"3d68ef4c-1e91-441b-804a-1f3ecddeba6e\" (UID: \"3d68ef4c-1e91-441b-804a-1f3ecddeba6e\") "
Aug 05 17:36:39.543136 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:39.542909    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d68ef4c-1e91-441b-804a-1f3ecddeba6e" (UID: "3d68ef4c-1e91-441b-804a-1f3ecddeba6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:36:39.543136 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:39.543043    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-home" (OuterVolumeSpecName: "home") pod "3d68ef4c-1e91-441b-804a-1f3ecddeba6e" (UID: "3d68ef4c-1e91-441b-804a-1f3ecddeba6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:36:39.543136 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:39.543119    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d68ef4c-1e91-441b-804a-1f3ecddeba6e" (UID: "3d68ef4c-1e91-441b-804a-1f3ecddeba6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:36:39.544932 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:39.544896    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d68ef4c-1e91-441b-804a-1f3ecddeba6e" (UID: "3d68ef4c-1e91-441b-804a-1f3ecddeba6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:36:39.545652 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:39.545579    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-dshm" (OuterVolumeSpecName: "dshm") pod "3d68ef4c-1e91-441b-804a-1f3ecddeba6e" (UID: "3d68ef4c-1e91-441b-804a-1f3ecddeba6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:36:39.545652 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:39.545615    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-kube-api-access-jfnkv" (OuterVolumeSpecName: "kube-api-access-jfnkv") pod "3d68ef4c-1e91-441b-804a-1f3ecddeba6e" (UID: "3d68ef4c-1e91-441b-804a-1f3ecddeba6e"). InnerVolumeSpecName "kube-api-access-jfnkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:36:39.643725 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:39.643625    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:36:39.643725 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:39.643666    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:36:39.643725 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:39.643680    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:36:39.643725 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:39.643691    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfnkv\" (UniqueName: \"kubernetes.io/projected/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-kube-api-access-jfnkv\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:36:39.643725 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:39.643701    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:36:39.643725 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:39.643718    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d68ef4c-1e91-441b-804a-1f3ecddeba6e-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:36:40.191094 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:40.191059    2561 generic.go:356] "Generic (PLEG): container finished" podID="3d68ef4c-1e91-441b-804a-1f3ecddeba6e" containerID="0d629359d23ce8182ccaa9ed862c7324742401856472f2adb8361927aeebf458" exitCode=0
Aug 05 17:36:40.191528 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:40.191125    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"
Aug 05 17:36:40.191528 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:40.191157    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm" event={"ID":"3d68ef4c-1e91-441b-804a-1f3ecddeba6e","Type":"ContainerDied","Data":"0d629359d23ce8182ccaa9ed862c7324742401856472f2adb8361927aeebf458"}
Aug 05 17:36:40.191528 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:40.191210    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm" event={"ID":"3d68ef4c-1e91-441b-804a-1f3ecddeba6e","Type":"ContainerDied","Data":"f838d0b4d40af3c78ee66e993b7c27161712f39b379da3894f1185ec9eb70028"}
Aug 05 17:36:40.191528 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:40.191232    2561 scope.go:117] "RemoveContainer" containerID="0d629359d23ce8182ccaa9ed862c7324742401856472f2adb8361927aeebf458"
Aug 05 17:36:40.201144 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:40.201119    2561 scope.go:117] "RemoveContainer" containerID="0d629359d23ce8182ccaa9ed862c7324742401856472f2adb8361927aeebf458"
Aug 05 17:36:40.201597 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:36:40.201523    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d629359d23ce8182ccaa9ed862c7324742401856472f2adb8361927aeebf458\": container with ID starting with 0d629359d23ce8182ccaa9ed862c7324742401856472f2adb8361927aeebf458 not found: ID does not exist" containerID="0d629359d23ce8182ccaa9ed862c7324742401856472f2adb8361927aeebf458"
Aug 05 17:36:40.201597 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:40.201567    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d629359d23ce8182ccaa9ed862c7324742401856472f2adb8361927aeebf458"} err="failed to get container status \"0d629359d23ce8182ccaa9ed862c7324742401856472f2adb8361927aeebf458\": rpc error: code = NotFound desc = could not find container \"0d629359d23ce8182ccaa9ed862c7324742401856472f2adb8361927aeebf458\": container with ID starting with 0d629359d23ce8182ccaa9ed862c7324742401856472f2adb8361927aeebf458 not found: ID does not exist"
Aug 05 17:36:40.217613 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:40.217571    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"]
Aug 05 17:36:40.220071 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:40.220038    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-72vnm"]
Aug 05 17:36:40.731328 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:40.731288    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d68ef4c-1e91-441b-804a-1f3ecddeba6e" path="/var/lib/kubelet/pods/3d68ef4c-1e91-441b-804a-1f3ecddeba6e/volumes"
Aug 05 17:36:43.649074 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:43.649023    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2" podUID="07b14313-4de3-4353-92da-e347fe4286bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Aug 05 17:36:53.649295 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:36:53.649249    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2" podUID="07b14313-4de3-4353-92da-e347fe4286bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Aug 05 17:37:03.649472 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:03.649423    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2" podUID="07b14313-4de3-4353-92da-e347fe4286bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Aug 05 17:37:04.928381 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:04.928336    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"]
Aug 05 17:37:04.929011 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:04.928703    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d68ef4c-1e91-441b-804a-1f3ecddeba6e" containerName="main"
Aug 05 17:37:04.929011 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:04.928714    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d68ef4c-1e91-441b-804a-1f3ecddeba6e" containerName="main"
Aug 05 17:37:04.929011 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:04.928770    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d68ef4c-1e91-441b-804a-1f3ecddeba6e" containerName="main"
Aug 05 17:37:04.932031 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:04.932005    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"
Aug 05 17:37:04.934638 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:04.934613    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-gb8zf\""
Aug 05 17:37:04.934885 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:04.934863    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Aug 05 17:37:04.934979 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:04.934927    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Aug 05 17:37:04.935799 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:04.935777    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-5wzw2\""
Aug 05 17:37:04.936120 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:04.935779    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Aug 05 17:37:04.947375 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:04.947342    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"]
Aug 05 17:37:04.976306 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:04.976273    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01a90c65-6320-429d-aca1-867753c6eb4e-home\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs\" (UID: \"01a90c65-6320-429d-aca1-867753c6eb4e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"
Aug 05 17:37:04.976494 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:04.976327    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmlll\" (UniqueName: \"kubernetes.io/projected/01a90c65-6320-429d-aca1-867753c6eb4e-kube-api-access-lmlll\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs\" (UID: \"01a90c65-6320-429d-aca1-867753c6eb4e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"
Aug 05 17:37:04.976494 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:04.976349    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01a90c65-6320-429d-aca1-867753c6eb4e-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs\" (UID: \"01a90c65-6320-429d-aca1-867753c6eb4e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"
Aug 05 17:37:04.976494 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:04.976369    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01a90c65-6320-429d-aca1-867753c6eb4e-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs\" (UID: \"01a90c65-6320-429d-aca1-867753c6eb4e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"
Aug 05 17:37:04.976494 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:04.976443    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01a90c65-6320-429d-aca1-867753c6eb4e-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs\" (UID: \"01a90c65-6320-429d-aca1-867753c6eb4e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"
Aug 05 17:37:04.976494 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:04.976484    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01a90c65-6320-429d-aca1-867753c6eb4e-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs\" (UID: \"01a90c65-6320-429d-aca1-867753c6eb4e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"
Aug 05 17:37:05.076933 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:05.076902    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01a90c65-6320-429d-aca1-867753c6eb4e-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs\" (UID: \"01a90c65-6320-429d-aca1-867753c6eb4e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"
Aug 05 17:37:05.077154 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:05.076946    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01a90c65-6320-429d-aca1-867753c6eb4e-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs\" (UID: \"01a90c65-6320-429d-aca1-867753c6eb4e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"
Aug 05 17:37:05.077154 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:05.077034    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01a90c65-6320-429d-aca1-867753c6eb4e-home\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs\" (UID: \"01a90c65-6320-429d-aca1-867753c6eb4e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"
Aug 05 17:37:05.077154 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:05.077112    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmlll\" (UniqueName: \"kubernetes.io/projected/01a90c65-6320-429d-aca1-867753c6eb4e-kube-api-access-lmlll\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs\" (UID: \"01a90c65-6320-429d-aca1-867753c6eb4e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"
Aug 05 17:37:05.077154 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:05.077143    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01a90c65-6320-429d-aca1-867753c6eb4e-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs\" (UID: \"01a90c65-6320-429d-aca1-867753c6eb4e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"
Aug 05 17:37:05.077378 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:05.077175    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01a90c65-6320-429d-aca1-867753c6eb4e-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs\" (UID: \"01a90c65-6320-429d-aca1-867753c6eb4e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"
Aug 05 17:37:05.077378 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:05.077326    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01a90c65-6320-429d-aca1-867753c6eb4e-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs\" (UID: \"01a90c65-6320-429d-aca1-867753c6eb4e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"
Aug 05 17:37:05.077485 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:05.077397    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01a90c65-6320-429d-aca1-867753c6eb4e-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs\" (UID: \"01a90c65-6320-429d-aca1-867753c6eb4e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"
Aug 05 17:37:05.077592 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:05.077571    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01a90c65-6320-429d-aca1-867753c6eb4e-home\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs\" (UID: \"01a90c65-6320-429d-aca1-867753c6eb4e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"
Aug 05 17:37:05.079851 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:05.079823    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01a90c65-6320-429d-aca1-867753c6eb4e-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs\" (UID: \"01a90c65-6320-429d-aca1-867753c6eb4e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"
Aug 05 17:37:05.080204 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:05.080176    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01a90c65-6320-429d-aca1-867753c6eb4e-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs\" (UID: \"01a90c65-6320-429d-aca1-867753c6eb4e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"
Aug 05 17:37:05.086333 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:05.086307    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmlll\" (UniqueName: \"kubernetes.io/projected/01a90c65-6320-429d-aca1-867753c6eb4e-kube-api-access-lmlll\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs\" (UID: \"01a90c65-6320-429d-aca1-867753c6eb4e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"
Aug 05 17:37:05.152775 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:05.152676    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg"]
Aug 05 17:37:05.156853 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:05.156828    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg"
Aug 05 17:37:05.159727 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:05.159703    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-87nvx\""
Aug 05 17:37:05.170731 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:05.170699    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg"]
Aug 05 17:37:05.177903 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:05.177873    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2gq2\" (UniqueName: \"kubernetes.io/projected/ce3a8738-eb1a-4936-89c5-ddbc045cca9e-kube-api-access-h2gq2\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg\" (UID: \"ce3a8738-eb1a-4936-89c5-ddbc045cca9e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg"
Aug 05 17:37:05.178074 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:05.177914    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce3a8738-eb1a-4936-89c5-ddbc045cca9e-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg\" (UID: \"ce3a8738-eb1a-4936-89c5-ddbc045cca9e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg"
Aug 05 17:37:05.248599 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:05.248512    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"
Aug 05 17:37:05.279108 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:05.279075    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2gq2\" (UniqueName: \"kubernetes.io/projected/ce3a8738-eb1a-4936-89c5-ddbc045cca9e-kube-api-access-h2gq2\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg\" (UID: \"ce3a8738-eb1a-4936-89c5-ddbc045cca9e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg"
Aug 05 17:37:05.279277 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:05.279115    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce3a8738-eb1a-4936-89c5-ddbc045cca9e-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg\" (UID: \"ce3a8738-eb1a-4936-89c5-ddbc045cca9e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg"
Aug 05 17:37:05.281846 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:05.281810    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce3a8738-eb1a-4936-89c5-ddbc045cca9e-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg\" (UID: \"ce3a8738-eb1a-4936-89c5-ddbc045cca9e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg"
Aug 05 17:37:05.287680 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:05.287645    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2gq2\" (UniqueName: \"kubernetes.io/projected/ce3a8738-eb1a-4936-89c5-ddbc045cca9e-kube-api-access-h2gq2\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg\" (UID: \"ce3a8738-eb1a-4936-89c5-ddbc045cca9e\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg"
Aug 05 17:37:05.406993 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:05.406938    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"]
Aug 05 17:37:05.408721 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:37:05.408693    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a90c65_6320_429d_aca1_867753c6eb4e.slice/crio-1071b4b282b459d19e5710e5ad841d04befa50333707683979745cca4441497b WatchSource:0}: Error finding container 1071b4b282b459d19e5710e5ad841d04befa50333707683979745cca4441497b: Status 404 returned error can't find the container with id 1071b4b282b459d19e5710e5ad841d04befa50333707683979745cca4441497b
Aug 05 17:37:05.468512 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:05.468475    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg"
Aug 05 17:37:05.621229 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:05.621204    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg"]
Aug 05 17:37:05.623443 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:37:05.623419    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3a8738_eb1a_4936_89c5_ddbc045cca9e.slice/crio-b169432bd83f6320072a9365fd8c6f695541de543b4cbd67403fac6d5ee3c3e0 WatchSource:0}: Error finding container b169432bd83f6320072a9365fd8c6f695541de543b4cbd67403fac6d5ee3c3e0: Status 404 returned error can't find the container with id b169432bd83f6320072a9365fd8c6f695541de543b4cbd67403fac6d5ee3c3e0
Aug 05 17:37:06.340845 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:06.340809    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs" event={"ID":"01a90c65-6320-429d-aca1-867753c6eb4e","Type":"ContainerStarted","Data":"1071b4b282b459d19e5710e5ad841d04befa50333707683979745cca4441497b"}
Aug 05 17:37:06.342648 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:06.342614    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg" event={"ID":"ce3a8738-eb1a-4936-89c5-ddbc045cca9e","Type":"ContainerStarted","Data":"6c48d7a50393c22c78b91b9bd9f8841522b431ac617fe768c037a3074bbab43f"}
Aug 05 17:37:06.342799 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:06.342666    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg" event={"ID":"ce3a8738-eb1a-4936-89c5-ddbc045cca9e","Type":"ContainerStarted","Data":"b169432bd83f6320072a9365fd8c6f695541de543b4cbd67403fac6d5ee3c3e0"}
Aug 05 17:37:06.342866 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:06.342827    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg"
Aug 05 17:37:06.363989 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:06.363918    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg" podStartSLOduration=1.363899806 podStartE2EDuration="1.363899806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:37:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:37:06.362119754 +0000 UTC m=+1482.106754081" watchObservedRunningTime="2026-08-05 17:37:06.363899806 +0000 UTC m=+1482.108534132"
Aug 05 17:37:07.351541 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:07.351500    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs" event={"ID":"01a90c65-6320-429d-aca1-867753c6eb4e","Type":"ContainerStarted","Data":"5d7de0929646f65ed3cb80ab507ad745bbf1386d4d9fb4bb6d442fdf6c46649c"}
Aug 05 17:37:07.352023 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:07.351724    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"
Aug 05 17:37:08.360399 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:08.360357    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs" event={"ID":"01a90c65-6320-429d-aca1-867753c6eb4e","Type":"ContainerStarted","Data":"19f2501a60728e545617e8f4a1d0ae79855022487ed8385921af2b67d1f320a7"}
Aug 05 17:37:08.388050 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:08.387982    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs" podStartSLOduration=2.817502382 podStartE2EDuration="4.387945581s" totalImagesPullingTime="1.570443199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:37:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:37:08.384608487 +0000 UTC m=+1484.129242838" watchObservedRunningTime="2026-08-05 17:37:08.387945581 +0000 UTC m=+1484.132579912"
Aug 05 17:37:09.249519 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:09.249472    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"
Aug 05 17:37:13.648771 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:13.648712    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2" podUID="07b14313-4de3-4353-92da-e347fe4286bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Aug 05 17:37:15.249265 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:15.249223    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"
Aug 05 17:37:15.261943 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:15.261919    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"
Aug 05 17:37:15.413776 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:15.413743    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"
Aug 05 17:37:23.648596 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:23.648555    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2" podUID="07b14313-4de3-4353-92da-e347fe4286bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Aug 05 17:37:26.422582 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:26.422554    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"
Aug 05 17:37:33.648975 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:33.648907    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2" podUID="07b14313-4de3-4353-92da-e347fe4286bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Aug 05 17:37:37.353903 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:37.353868    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg"
Aug 05 17:37:43.649026 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:43.648978    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2" podUID="07b14313-4de3-4353-92da-e347fe4286bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Aug 05 17:37:53.649027 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:37:53.648913    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2" podUID="07b14313-4de3-4353-92da-e347fe4286bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Aug 05 17:38:03.649567 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:03.649519    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2" podUID="07b14313-4de3-4353-92da-e347fe4286bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Aug 05 17:38:13.648965 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:13.648917    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2" podUID="07b14313-4de3-4353-92da-e347fe4286bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Aug 05 17:38:23.659287 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:23.659252    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:38:23.667484 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:23.667460    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:38:28.655826 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:28.655794    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"]
Aug 05 17:38:39.071851 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.071808    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg"]
Aug 05 17:38:39.076050 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.076018    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"]
Aug 05 17:38:39.085480 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:38:39.085443    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 05 17:38:39.085480 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:38:39.085480 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:38:39.085480 ip-10-0-135-151 kubenswrapper[2561]:  > containerID="19f2501a60728e545617e8f4a1d0ae79855022487ed8385921af2b67d1f320a7" cmd=["/bin/sleep","15"]
Aug 05 17:38:39.085789 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.085496    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs" message="" err=<
Aug 05 17:38:39.085789 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:38:39.085789 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:38:39.085789 ip-10-0-135-151 kubenswrapper[2561]:  >
Aug 05 17:38:39.085789 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:38:39.085511    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 05 17:38:39.085789 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:38:39.085789 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:38:39.085789 ip-10-0-135-151 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs" podUID="01a90c65-6320-429d-aca1-867753c6eb4e" containerName="main" containerID="cri-o://19f2501a60728e545617e8f4a1d0ae79855022487ed8385921af2b67d1f320a7"
Aug 05 17:38:39.085789 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.085550    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs" podUID="01a90c65-6320-429d-aca1-867753c6eb4e" containerName="main" containerID="cri-o://19f2501a60728e545617e8f4a1d0ae79855022487ed8385921af2b67d1f320a7" gracePeriod=60
Aug 05 17:38:39.302736 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.302700    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs" podUID="01a90c65-6320-429d-aca1-867753c6eb4e" containerName="llm-d-routing-sidecar" containerID="cri-o://5d7de0929646f65ed3cb80ab507ad745bbf1386d4d9fb4bb6d442fdf6c46649c" gracePeriod=60
Aug 05 17:38:39.403548 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.403441    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs" podUID="01a90c65-6320-429d-aca1-867753c6eb4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8001/health\": dial tcp 10.133.0.60:8001: connect: connection refused"
Aug 05 17:38:39.561540 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.561518    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"
Aug 05 17:38:39.655684 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.655604    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01a90c65-6320-429d-aca1-867753c6eb4e-dshm\") pod \"01a90c65-6320-429d-aca1-867753c6eb4e\" (UID: \"01a90c65-6320-429d-aca1-867753c6eb4e\") "
Aug 05 17:38:39.655684 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.655676    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01a90c65-6320-429d-aca1-867753c6eb4e-tls-certs\") pod \"01a90c65-6320-429d-aca1-867753c6eb4e\" (UID: \"01a90c65-6320-429d-aca1-867753c6eb4e\") "
Aug 05 17:38:39.655896 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.655698    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01a90c65-6320-429d-aca1-867753c6eb4e-model-cache\") pod \"01a90c65-6320-429d-aca1-867753c6eb4e\" (UID: \"01a90c65-6320-429d-aca1-867753c6eb4e\") "
Aug 05 17:38:39.655896 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.655723    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmlll\" (UniqueName: \"kubernetes.io/projected/01a90c65-6320-429d-aca1-867753c6eb4e-kube-api-access-lmlll\") pod \"01a90c65-6320-429d-aca1-867753c6eb4e\" (UID: \"01a90c65-6320-429d-aca1-867753c6eb4e\") "
Aug 05 17:38:39.655896 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.655751    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01a90c65-6320-429d-aca1-867753c6eb4e-tmp-dir\") pod \"01a90c65-6320-429d-aca1-867753c6eb4e\" (UID: \"01a90c65-6320-429d-aca1-867753c6eb4e\") "
Aug 05 17:38:39.655896 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.655790    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01a90c65-6320-429d-aca1-867753c6eb4e-home\") pod \"01a90c65-6320-429d-aca1-867753c6eb4e\" (UID: \"01a90c65-6320-429d-aca1-867753c6eb4e\") "
Aug 05 17:38:39.656128 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.656000    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a90c65-6320-429d-aca1-867753c6eb4e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "01a90c65-6320-429d-aca1-867753c6eb4e" (UID: "01a90c65-6320-429d-aca1-867753c6eb4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:38:39.656128 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.656031    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a90c65-6320-429d-aca1-867753c6eb4e-model-cache" (OuterVolumeSpecName: "model-cache") pod "01a90c65-6320-429d-aca1-867753c6eb4e" (UID: "01a90c65-6320-429d-aca1-867753c6eb4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:38:39.656128 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.656087    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01a90c65-6320-429d-aca1-867753c6eb4e-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:38:39.656128 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.656099    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a90c65-6320-429d-aca1-867753c6eb4e-home" (OuterVolumeSpecName: "home") pod "01a90c65-6320-429d-aca1-867753c6eb4e" (UID: "01a90c65-6320-429d-aca1-867753c6eb4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:38:39.657991 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.657939    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a90c65-6320-429d-aca1-867753c6eb4e-dshm" (OuterVolumeSpecName: "dshm") pod "01a90c65-6320-429d-aca1-867753c6eb4e" (UID: "01a90c65-6320-429d-aca1-867753c6eb4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:38:39.657991 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.657977    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01a90c65-6320-429d-aca1-867753c6eb4e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "01a90c65-6320-429d-aca1-867753c6eb4e" (UID: "01a90c65-6320-429d-aca1-867753c6eb4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:38:39.657991 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.657980    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a90c65-6320-429d-aca1-867753c6eb4e-kube-api-access-lmlll" (OuterVolumeSpecName: "kube-api-access-lmlll") pod "01a90c65-6320-429d-aca1-867753c6eb4e" (UID: "01a90c65-6320-429d-aca1-867753c6eb4e"). InnerVolumeSpecName "kube-api-access-lmlll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:38:39.757458 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.757426    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01a90c65-6320-429d-aca1-867753c6eb4e-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:38:39.757458 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.757452    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01a90c65-6320-429d-aca1-867753c6eb4e-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:38:39.757458 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.757461    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01a90c65-6320-429d-aca1-867753c6eb4e-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:38:39.757458 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.757469    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmlll\" (UniqueName: \"kubernetes.io/projected/01a90c65-6320-429d-aca1-867753c6eb4e-kube-api-access-lmlll\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:38:39.757793 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.757476    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01a90c65-6320-429d-aca1-867753c6eb4e-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:38:39.894122 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.894091    2561 generic.go:356] "Generic (PLEG): container finished" podID="01a90c65-6320-429d-aca1-867753c6eb4e" containerID="19f2501a60728e545617e8f4a1d0ae79855022487ed8385921af2b67d1f320a7" exitCode=0
Aug 05 17:38:39.894122 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.894115    2561 generic.go:356] "Generic (PLEG): container finished" podID="01a90c65-6320-429d-aca1-867753c6eb4e" containerID="5d7de0929646f65ed3cb80ab507ad745bbf1386d4d9fb4bb6d442fdf6c46649c" exitCode=0
Aug 05 17:38:39.894379 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.894158    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"
Aug 05 17:38:39.894379 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.894175    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs" event={"ID":"01a90c65-6320-429d-aca1-867753c6eb4e","Type":"ContainerDied","Data":"19f2501a60728e545617e8f4a1d0ae79855022487ed8385921af2b67d1f320a7"}
Aug 05 17:38:39.894379 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.894239    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs" event={"ID":"01a90c65-6320-429d-aca1-867753c6eb4e","Type":"ContainerDied","Data":"5d7de0929646f65ed3cb80ab507ad745bbf1386d4d9fb4bb6d442fdf6c46649c"}
Aug 05 17:38:39.894379 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.894259    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs" event={"ID":"01a90c65-6320-429d-aca1-867753c6eb4e","Type":"ContainerDied","Data":"1071b4b282b459d19e5710e5ad841d04befa50333707683979745cca4441497b"}
Aug 05 17:38:39.894379 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.894281    2561 scope.go:117] "RemoveContainer" containerID="19f2501a60728e545617e8f4a1d0ae79855022487ed8385921af2b67d1f320a7"
Aug 05 17:38:39.904082 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.904066    2561 scope.go:117] "RemoveContainer" containerID="5d7de0929646f65ed3cb80ab507ad745bbf1386d4d9fb4bb6d442fdf6c46649c"
Aug 05 17:38:39.912249 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.912232    2561 scope.go:117] "RemoveContainer" containerID="19f2501a60728e545617e8f4a1d0ae79855022487ed8385921af2b67d1f320a7"
Aug 05 17:38:39.912496 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:38:39.912479    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19f2501a60728e545617e8f4a1d0ae79855022487ed8385921af2b67d1f320a7\": container with ID starting with 19f2501a60728e545617e8f4a1d0ae79855022487ed8385921af2b67d1f320a7 not found: ID does not exist" containerID="19f2501a60728e545617e8f4a1d0ae79855022487ed8385921af2b67d1f320a7"
Aug 05 17:38:39.912552 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.912505    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f2501a60728e545617e8f4a1d0ae79855022487ed8385921af2b67d1f320a7"} err="failed to get container status \"19f2501a60728e545617e8f4a1d0ae79855022487ed8385921af2b67d1f320a7\": rpc error: code = NotFound desc = could not find container \"19f2501a60728e545617e8f4a1d0ae79855022487ed8385921af2b67d1f320a7\": container with ID starting with 19f2501a60728e545617e8f4a1d0ae79855022487ed8385921af2b67d1f320a7 not found: ID does not exist"
Aug 05 17:38:39.912552 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.912521    2561 scope.go:117] "RemoveContainer" containerID="5d7de0929646f65ed3cb80ab507ad745bbf1386d4d9fb4bb6d442fdf6c46649c"
Aug 05 17:38:39.912728 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:38:39.912712    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d7de0929646f65ed3cb80ab507ad745bbf1386d4d9fb4bb6d442fdf6c46649c\": container with ID starting with 5d7de0929646f65ed3cb80ab507ad745bbf1386d4d9fb4bb6d442fdf6c46649c not found: ID does not exist" containerID="5d7de0929646f65ed3cb80ab507ad745bbf1386d4d9fb4bb6d442fdf6c46649c"
Aug 05 17:38:39.912768 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.912732    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7de0929646f65ed3cb80ab507ad745bbf1386d4d9fb4bb6d442fdf6c46649c"} err="failed to get container status \"5d7de0929646f65ed3cb80ab507ad745bbf1386d4d9fb4bb6d442fdf6c46649c\": rpc error: code = NotFound desc = could not find container \"5d7de0929646f65ed3cb80ab507ad745bbf1386d4d9fb4bb6d442fdf6c46649c\": container with ID starting with 5d7de0929646f65ed3cb80ab507ad745bbf1386d4d9fb4bb6d442fdf6c46649c not found: ID does not exist"
Aug 05 17:38:39.912768 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.912745    2561 scope.go:117] "RemoveContainer" containerID="19f2501a60728e545617e8f4a1d0ae79855022487ed8385921af2b67d1f320a7"
Aug 05 17:38:39.912917 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.912902    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f2501a60728e545617e8f4a1d0ae79855022487ed8385921af2b67d1f320a7"} err="failed to get container status \"19f2501a60728e545617e8f4a1d0ae79855022487ed8385921af2b67d1f320a7\": rpc error: code = NotFound desc = could not find container \"19f2501a60728e545617e8f4a1d0ae79855022487ed8385921af2b67d1f320a7\": container with ID starting with 19f2501a60728e545617e8f4a1d0ae79855022487ed8385921af2b67d1f320a7 not found: ID does not exist"
Aug 05 17:38:39.912971 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.912917    2561 scope.go:117] "RemoveContainer" containerID="5d7de0929646f65ed3cb80ab507ad745bbf1386d4d9fb4bb6d442fdf6c46649c"
Aug 05 17:38:39.913171 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.913152    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7de0929646f65ed3cb80ab507ad745bbf1386d4d9fb4bb6d442fdf6c46649c"} err="failed to get container status \"5d7de0929646f65ed3cb80ab507ad745bbf1386d4d9fb4bb6d442fdf6c46649c\": rpc error: code = NotFound desc = could not find container \"5d7de0929646f65ed3cb80ab507ad745bbf1386d4d9fb4bb6d442fdf6c46649c\": container with ID starting with 5d7de0929646f65ed3cb80ab507ad745bbf1386d4d9fb4bb6d442fdf6c46649c not found: ID does not exist"
Aug 05 17:38:39.917764 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.917742    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"]
Aug 05 17:38:39.924190 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:39.924171    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-fqcbs"]
Aug 05 17:38:40.736836 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:40.736802    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01a90c65-6320-429d-aca1-867753c6eb4e" path="/var/lib/kubelet/pods/01a90c65-6320-429d-aca1-867753c6eb4e/volumes"
Aug 05 17:38:43.678121 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:43.678080    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2" podUID="07b14313-4de3-4353-92da-e347fe4286bf" containerName="main" containerID="cri-o://2860bbae8c41f5bd4916e85a3ebd420df7c539a2f9b2b28cd343cc4eff254045" gracePeriod=45
Aug 05 17:38:44.431373 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.431348    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:38:44.496127 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.496042    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07b14313-4de3-4353-92da-e347fe4286bf-tmp-dir\") pod \"07b14313-4de3-4353-92da-e347fe4286bf\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") "
Aug 05 17:38:44.496127 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.496076    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vbb2\" (UniqueName: \"kubernetes.io/projected/07b14313-4de3-4353-92da-e347fe4286bf-kube-api-access-8vbb2\") pod \"07b14313-4de3-4353-92da-e347fe4286bf\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") "
Aug 05 17:38:44.496342 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.496141    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07b14313-4de3-4353-92da-e347fe4286bf-kserve-provision-location\") pod \"07b14313-4de3-4353-92da-e347fe4286bf\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") "
Aug 05 17:38:44.496342 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.496198    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07b14313-4de3-4353-92da-e347fe4286bf-model-cache\") pod \"07b14313-4de3-4353-92da-e347fe4286bf\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") "
Aug 05 17:38:44.496342 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.496245    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07b14313-4de3-4353-92da-e347fe4286bf-cabundle-cert\") pod \"07b14313-4de3-4353-92da-e347fe4286bf\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") "
Aug 05 17:38:44.496342 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.496274    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07b14313-4de3-4353-92da-e347fe4286bf-tls-certs\") pod \"07b14313-4de3-4353-92da-e347fe4286bf\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") "
Aug 05 17:38:44.496342 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.496331    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07b14313-4de3-4353-92da-e347fe4286bf-dshm\") pod \"07b14313-4de3-4353-92da-e347fe4286bf\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") "
Aug 05 17:38:44.496607 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.496352    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07b14313-4de3-4353-92da-e347fe4286bf-home\") pod \"07b14313-4de3-4353-92da-e347fe4286bf\" (UID: \"07b14313-4de3-4353-92da-e347fe4286bf\") "
Aug 05 17:38:44.496607 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.496497    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b14313-4de3-4353-92da-e347fe4286bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "07b14313-4de3-4353-92da-e347fe4286bf" (UID: "07b14313-4de3-4353-92da-e347fe4286bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:38:44.496607 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.496582    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b14313-4de3-4353-92da-e347fe4286bf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "07b14313-4de3-4353-92da-e347fe4286bf" (UID: "07b14313-4de3-4353-92da-e347fe4286bf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:38:44.496872 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.496826    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07b14313-4de3-4353-92da-e347fe4286bf-cabundle-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:38:44.496872 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.496851    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07b14313-4de3-4353-92da-e347fe4286bf-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:38:44.497175 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.497152    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b14313-4de3-4353-92da-e347fe4286bf-home" (OuterVolumeSpecName: "home") pod "07b14313-4de3-4353-92da-e347fe4286bf" (UID: "07b14313-4de3-4353-92da-e347fe4286bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:38:44.498557 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.498528    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b14313-4de3-4353-92da-e347fe4286bf-kube-api-access-8vbb2" (OuterVolumeSpecName: "kube-api-access-8vbb2") pod "07b14313-4de3-4353-92da-e347fe4286bf" (UID: "07b14313-4de3-4353-92da-e347fe4286bf"). InnerVolumeSpecName "kube-api-access-8vbb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:38:44.498859 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.498836    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b14313-4de3-4353-92da-e347fe4286bf-dshm" (OuterVolumeSpecName: "dshm") pod "07b14313-4de3-4353-92da-e347fe4286bf" (UID: "07b14313-4de3-4353-92da-e347fe4286bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:38:44.498930 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.498908    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07b14313-4de3-4353-92da-e347fe4286bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "07b14313-4de3-4353-92da-e347fe4286bf" (UID: "07b14313-4de3-4353-92da-e347fe4286bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:38:44.508992 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.508970    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b14313-4de3-4353-92da-e347fe4286bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "07b14313-4de3-4353-92da-e347fe4286bf" (UID: "07b14313-4de3-4353-92da-e347fe4286bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:38:44.520298 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.520272    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b14313-4de3-4353-92da-e347fe4286bf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "07b14313-4de3-4353-92da-e347fe4286bf" (UID: "07b14313-4de3-4353-92da-e347fe4286bf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:38:44.597462 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.597431    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07b14313-4de3-4353-92da-e347fe4286bf-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:38:44.597462 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.597455    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07b14313-4de3-4353-92da-e347fe4286bf-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:38:44.597462 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.597464    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07b14313-4de3-4353-92da-e347fe4286bf-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:38:44.597656 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.597474    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vbb2\" (UniqueName: \"kubernetes.io/projected/07b14313-4de3-4353-92da-e347fe4286bf-kube-api-access-8vbb2\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:38:44.597656 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.597482    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07b14313-4de3-4353-92da-e347fe4286bf-kserve-provision-location\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:38:44.597656 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.597491    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07b14313-4de3-4353-92da-e347fe4286bf-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:38:44.922942 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.922906    2561 generic.go:356] "Generic (PLEG): container finished" podID="07b14313-4de3-4353-92da-e347fe4286bf" containerID="2860bbae8c41f5bd4916e85a3ebd420df7c539a2f9b2b28cd343cc4eff254045" exitCode=0
Aug 05 17:38:44.923462 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.922974    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"
Aug 05 17:38:44.923462 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.922992    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2" event={"ID":"07b14313-4de3-4353-92da-e347fe4286bf","Type":"ContainerDied","Data":"2860bbae8c41f5bd4916e85a3ebd420df7c539a2f9b2b28cd343cc4eff254045"}
Aug 05 17:38:44.923462 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.923057    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2" event={"ID":"07b14313-4de3-4353-92da-e347fe4286bf","Type":"ContainerDied","Data":"f45fc1f34fa1c8e28a646e12e5f59a6bdd6903451c7a04c1ba6a0ae718e1c621"}
Aug 05 17:38:44.923462 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.923090    2561 scope.go:117] "RemoveContainer" containerID="2860bbae8c41f5bd4916e85a3ebd420df7c539a2f9b2b28cd343cc4eff254045"
Aug 05 17:38:44.931685 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.931670    2561 scope.go:117] "RemoveContainer" containerID="25a09fa80e35d672a4febf6b88b086cf5258894edf947d8b619d114bbf59db30"
Aug 05 17:38:44.939052 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.939034    2561 scope.go:117] "RemoveContainer" containerID="2860bbae8c41f5bd4916e85a3ebd420df7c539a2f9b2b28cd343cc4eff254045"
Aug 05 17:38:44.939326 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:38:44.939309    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2860bbae8c41f5bd4916e85a3ebd420df7c539a2f9b2b28cd343cc4eff254045\": container with ID starting with 2860bbae8c41f5bd4916e85a3ebd420df7c539a2f9b2b28cd343cc4eff254045 not found: ID does not exist" containerID="2860bbae8c41f5bd4916e85a3ebd420df7c539a2f9b2b28cd343cc4eff254045"
Aug 05 17:38:44.939374 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.939336    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2860bbae8c41f5bd4916e85a3ebd420df7c539a2f9b2b28cd343cc4eff254045"} err="failed to get container status \"2860bbae8c41f5bd4916e85a3ebd420df7c539a2f9b2b28cd343cc4eff254045\": rpc error: code = NotFound desc = could not find container \"2860bbae8c41f5bd4916e85a3ebd420df7c539a2f9b2b28cd343cc4eff254045\": container with ID starting with 2860bbae8c41f5bd4916e85a3ebd420df7c539a2f9b2b28cd343cc4eff254045 not found: ID does not exist"
Aug 05 17:38:44.939374 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.939353    2561 scope.go:117] "RemoveContainer" containerID="25a09fa80e35d672a4febf6b88b086cf5258894edf947d8b619d114bbf59db30"
Aug 05 17:38:44.939596 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:38:44.939576    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25a09fa80e35d672a4febf6b88b086cf5258894edf947d8b619d114bbf59db30\": container with ID starting with 25a09fa80e35d672a4febf6b88b086cf5258894edf947d8b619d114bbf59db30 not found: ID does not exist" containerID="25a09fa80e35d672a4febf6b88b086cf5258894edf947d8b619d114bbf59db30"
Aug 05 17:38:44.939645 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.939601    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25a09fa80e35d672a4febf6b88b086cf5258894edf947d8b619d114bbf59db30"} err="failed to get container status \"25a09fa80e35d672a4febf6b88b086cf5258894edf947d8b619d114bbf59db30\": rpc error: code = NotFound desc = could not find container \"25a09fa80e35d672a4febf6b88b086cf5258894edf947d8b619d114bbf59db30\": container with ID starting with 25a09fa80e35d672a4febf6b88b086cf5258894edf947d8b619d114bbf59db30 not found: ID does not exist"
Aug 05 17:38:44.943477 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.943459    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"]
Aug 05 17:38:44.947568 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:44.947545    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-546d97dc78-4zjd2"]
Aug 05 17:38:46.729631 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:46.729599    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07b14313-4de3-4353-92da-e347fe4286bf" path="/var/lib/kubelet/pods/07b14313-4de3-4353-92da-e347fe4286bf/volumes"
Aug 05 17:38:47.353659 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:47.353620    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg" podUID="ce3a8738-eb1a-4936-89c5-ddbc045cca9e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 05 17:38:54.094174 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:54.094131    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg" podUID="ce3a8738-eb1a-4936-89c5-ddbc045cca9e" containerName="main" containerID="cri-o://6c48d7a50393c22c78b91b9bd9f8841522b431ac617fe768c037a3074bbab43f" gracePeriod=45
Aug 05 17:38:54.340708 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:54.340684    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg"
Aug 05 17:38:54.382291 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:54.382204    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2gq2\" (UniqueName: \"kubernetes.io/projected/ce3a8738-eb1a-4936-89c5-ddbc045cca9e-kube-api-access-h2gq2\") pod \"ce3a8738-eb1a-4936-89c5-ddbc045cca9e\" (UID: \"ce3a8738-eb1a-4936-89c5-ddbc045cca9e\") "
Aug 05 17:38:54.382291 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:54.382250    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce3a8738-eb1a-4936-89c5-ddbc045cca9e-tls-certs\") pod \"ce3a8738-eb1a-4936-89c5-ddbc045cca9e\" (UID: \"ce3a8738-eb1a-4936-89c5-ddbc045cca9e\") "
Aug 05 17:38:54.384524 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:54.384492    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce3a8738-eb1a-4936-89c5-ddbc045cca9e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ce3a8738-eb1a-4936-89c5-ddbc045cca9e" (UID: "ce3a8738-eb1a-4936-89c5-ddbc045cca9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:38:54.384640 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:54.384549    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce3a8738-eb1a-4936-89c5-ddbc045cca9e-kube-api-access-h2gq2" (OuterVolumeSpecName: "kube-api-access-h2gq2") pod "ce3a8738-eb1a-4936-89c5-ddbc045cca9e" (UID: "ce3a8738-eb1a-4936-89c5-ddbc045cca9e"). InnerVolumeSpecName "kube-api-access-h2gq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:38:54.483416 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:54.483377    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2gq2\" (UniqueName: \"kubernetes.io/projected/ce3a8738-eb1a-4936-89c5-ddbc045cca9e-kube-api-access-h2gq2\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:38:54.483416 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:54.483410    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce3a8738-eb1a-4936-89c5-ddbc045cca9e-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:38:54.982198 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:54.982161    2561 generic.go:356] "Generic (PLEG): container finished" podID="ce3a8738-eb1a-4936-89c5-ddbc045cca9e" containerID="6c48d7a50393c22c78b91b9bd9f8841522b431ac617fe768c037a3074bbab43f" exitCode=0
Aug 05 17:38:54.982389 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:54.982224    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg"
Aug 05 17:38:54.982389 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:54.982231    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg" event={"ID":"ce3a8738-eb1a-4936-89c5-ddbc045cca9e","Type":"ContainerDied","Data":"6c48d7a50393c22c78b91b9bd9f8841522b431ac617fe768c037a3074bbab43f"}
Aug 05 17:38:54.982389 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:54.982284    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg" event={"ID":"ce3a8738-eb1a-4936-89c5-ddbc045cca9e","Type":"ContainerDied","Data":"b169432bd83f6320072a9365fd8c6f695541de543b4cbd67403fac6d5ee3c3e0"}
Aug 05 17:38:54.982389 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:54.982312    2561 scope.go:117] "RemoveContainer" containerID="6c48d7a50393c22c78b91b9bd9f8841522b431ac617fe768c037a3074bbab43f"
Aug 05 17:38:54.991201 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:54.991179    2561 scope.go:117] "RemoveContainer" containerID="6c48d7a50393c22c78b91b9bd9f8841522b431ac617fe768c037a3074bbab43f"
Aug 05 17:38:54.991422 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:38:54.991405    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c48d7a50393c22c78b91b9bd9f8841522b431ac617fe768c037a3074bbab43f\": container with ID starting with 6c48d7a50393c22c78b91b9bd9f8841522b431ac617fe768c037a3074bbab43f not found: ID does not exist" containerID="6c48d7a50393c22c78b91b9bd9f8841522b431ac617fe768c037a3074bbab43f"
Aug 05 17:38:54.991492 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:54.991428    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c48d7a50393c22c78b91b9bd9f8841522b431ac617fe768c037a3074bbab43f"} err="failed to get container status \"6c48d7a50393c22c78b91b9bd9f8841522b431ac617fe768c037a3074bbab43f\": rpc error: code = NotFound desc = could not find container \"6c48d7a50393c22c78b91b9bd9f8841522b431ac617fe768c037a3074bbab43f\": container with ID starting with 6c48d7a50393c22c78b91b9bd9f8841522b431ac617fe768c037a3074bbab43f not found: ID does not exist"
Aug 05 17:38:55.002588 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:55.002534    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg"]
Aug 05 17:38:55.011234 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:55.011213    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9bt5jg"]
Aug 05 17:38:56.730184 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:38:56.730151    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce3a8738-eb1a-4936-89c5-ddbc045cca9e" path="/var/lib/kubelet/pods/ce3a8738-eb1a-4936-89c5-ddbc045cca9e/volumes"
Aug 05 17:39:07.134053 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.134010    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"]
Aug 05 17:39:07.134546 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.134527    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01a90c65-6320-429d-aca1-867753c6eb4e" containerName="llm-d-routing-sidecar"
Aug 05 17:39:07.134618 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.134550    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="01a90c65-6320-429d-aca1-867753c6eb4e" containerName="llm-d-routing-sidecar"
Aug 05 17:39:07.134618 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.134572    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce3a8738-eb1a-4936-89c5-ddbc045cca9e" containerName="main"
Aug 05 17:39:07.134618 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.134583    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce3a8738-eb1a-4936-89c5-ddbc045cca9e" containerName="main"
Aug 05 17:39:07.134618 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.134602    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07b14313-4de3-4353-92da-e347fe4286bf" containerName="storage-initializer"
Aug 05 17:39:07.134618 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.134610    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="07b14313-4de3-4353-92da-e347fe4286bf" containerName="storage-initializer"
Aug 05 17:39:07.134862 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.134624    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07b14313-4de3-4353-92da-e347fe4286bf" containerName="main"
Aug 05 17:39:07.134862 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.134632    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="07b14313-4de3-4353-92da-e347fe4286bf" containerName="main"
Aug 05 17:39:07.134862 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.134643    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01a90c65-6320-429d-aca1-867753c6eb4e" containerName="main"
Aug 05 17:39:07.134862 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.134651    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="01a90c65-6320-429d-aca1-867753c6eb4e" containerName="main"
Aug 05 17:39:07.134862 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.134734    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="01a90c65-6320-429d-aca1-867753c6eb4e" containerName="main"
Aug 05 17:39:07.134862 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.134748    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="07b14313-4de3-4353-92da-e347fe4286bf" containerName="main"
Aug 05 17:39:07.134862 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.134757    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="01a90c65-6320-429d-aca1-867753c6eb4e" containerName="llm-d-routing-sidecar"
Aug 05 17:39:07.134862 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.134770    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce3a8738-eb1a-4936-89c5-ddbc045cca9e" containerName="main"
Aug 05 17:39:07.137914 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.137893    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"
Aug 05 17:39:07.141893 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.141870    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"default-dockercfg-xwjhd\""
Aug 05 17:39:07.142023 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.141976    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"openshift-service-ca.crt\""
Aug 05 17:39:07.142023 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.142011    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v1-kserve-self-signed-certs\""
Aug 05 17:39:07.142139 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.142035    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"kube-root-ca.crt\""
Aug 05 17:39:07.149481 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.149459    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"]
Aug 05 17:39:07.293614 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.293576    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d66b7d70-6942-47a3-ab45-b92d098637a7-home\") pod \"canary-v1-kserve-65d58ccc9-6xnhl\" (UID: \"d66b7d70-6942-47a3-ab45-b92d098637a7\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"
Aug 05 17:39:07.293614 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.293612    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtgts\" (UniqueName: \"kubernetes.io/projected/d66b7d70-6942-47a3-ab45-b92d098637a7-kube-api-access-gtgts\") pod \"canary-v1-kserve-65d58ccc9-6xnhl\" (UID: \"d66b7d70-6942-47a3-ab45-b92d098637a7\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"
Aug 05 17:39:07.293842 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.293634    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d66b7d70-6942-47a3-ab45-b92d098637a7-model-cache\") pod \"canary-v1-kserve-65d58ccc9-6xnhl\" (UID: \"d66b7d70-6942-47a3-ab45-b92d098637a7\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"
Aug 05 17:39:07.293842 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.293657    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d66b7d70-6942-47a3-ab45-b92d098637a7-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-6xnhl\" (UID: \"d66b7d70-6942-47a3-ab45-b92d098637a7\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"
Aug 05 17:39:07.293842 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.293769    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d66b7d70-6942-47a3-ab45-b92d098637a7-dshm\") pod \"canary-v1-kserve-65d58ccc9-6xnhl\" (UID: \"d66b7d70-6942-47a3-ab45-b92d098637a7\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"
Aug 05 17:39:07.293842 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.293824    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d66b7d70-6942-47a3-ab45-b92d098637a7-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-6xnhl\" (UID: \"d66b7d70-6942-47a3-ab45-b92d098637a7\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"
Aug 05 17:39:07.394397 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.394308    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d66b7d70-6942-47a3-ab45-b92d098637a7-home\") pod \"canary-v1-kserve-65d58ccc9-6xnhl\" (UID: \"d66b7d70-6942-47a3-ab45-b92d098637a7\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"
Aug 05 17:39:07.394397 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.394351    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtgts\" (UniqueName: \"kubernetes.io/projected/d66b7d70-6942-47a3-ab45-b92d098637a7-kube-api-access-gtgts\") pod \"canary-v1-kserve-65d58ccc9-6xnhl\" (UID: \"d66b7d70-6942-47a3-ab45-b92d098637a7\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"
Aug 05 17:39:07.394397 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.394379    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d66b7d70-6942-47a3-ab45-b92d098637a7-model-cache\") pod \"canary-v1-kserve-65d58ccc9-6xnhl\" (UID: \"d66b7d70-6942-47a3-ab45-b92d098637a7\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"
Aug 05 17:39:07.394636 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.394402    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d66b7d70-6942-47a3-ab45-b92d098637a7-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-6xnhl\" (UID: \"d66b7d70-6942-47a3-ab45-b92d098637a7\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"
Aug 05 17:39:07.394636 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.394447    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d66b7d70-6942-47a3-ab45-b92d098637a7-dshm\") pod \"canary-v1-kserve-65d58ccc9-6xnhl\" (UID: \"d66b7d70-6942-47a3-ab45-b92d098637a7\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"
Aug 05 17:39:07.394636 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.394487    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d66b7d70-6942-47a3-ab45-b92d098637a7-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-6xnhl\" (UID: \"d66b7d70-6942-47a3-ab45-b92d098637a7\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"
Aug 05 17:39:07.394818 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.394796    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d66b7d70-6942-47a3-ab45-b92d098637a7-home\") pod \"canary-v1-kserve-65d58ccc9-6xnhl\" (UID: \"d66b7d70-6942-47a3-ab45-b92d098637a7\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"
Aug 05 17:39:07.394854 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.394837    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d66b7d70-6942-47a3-ab45-b92d098637a7-model-cache\") pod \"canary-v1-kserve-65d58ccc9-6xnhl\" (UID: \"d66b7d70-6942-47a3-ab45-b92d098637a7\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"
Aug 05 17:39:07.394898 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.394849    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d66b7d70-6942-47a3-ab45-b92d098637a7-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-6xnhl\" (UID: \"d66b7d70-6942-47a3-ab45-b92d098637a7\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"
Aug 05 17:39:07.396791 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.396774    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d66b7d70-6942-47a3-ab45-b92d098637a7-dshm\") pod \"canary-v1-kserve-65d58ccc9-6xnhl\" (UID: \"d66b7d70-6942-47a3-ab45-b92d098637a7\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"
Aug 05 17:39:07.397063 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.397044    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d66b7d70-6942-47a3-ab45-b92d098637a7-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-6xnhl\" (UID: \"d66b7d70-6942-47a3-ab45-b92d098637a7\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"
Aug 05 17:39:07.403197 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.403174    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtgts\" (UniqueName: \"kubernetes.io/projected/d66b7d70-6942-47a3-ab45-b92d098637a7-kube-api-access-gtgts\") pod \"canary-v1-kserve-65d58ccc9-6xnhl\" (UID: \"d66b7d70-6942-47a3-ab45-b92d098637a7\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"
Aug 05 17:39:07.448912 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.448883    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"
Aug 05 17:39:07.627877 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:07.627847    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"]
Aug 05 17:39:07.629977 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:39:07.629939    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd66b7d70_6942_47a3_ab45_b92d098637a7.slice/crio-5cf0489829b6f2ddea90494b77e68dc521e59e8bb1ff9fec50d14f37057757e6 WatchSource:0}: Error finding container 5cf0489829b6f2ddea90494b77e68dc521e59e8bb1ff9fec50d14f37057757e6: Status 404 returned error can't find the container with id 5cf0489829b6f2ddea90494b77e68dc521e59e8bb1ff9fec50d14f37057757e6
Aug 05 17:39:08.059709 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:08.059667    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl" event={"ID":"d66b7d70-6942-47a3-ab45-b92d098637a7","Type":"ContainerStarted","Data":"5cf0489829b6f2ddea90494b77e68dc521e59e8bb1ff9fec50d14f37057757e6"}
Aug 05 17:39:09.066994 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:09.066945    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl" event={"ID":"d66b7d70-6942-47a3-ab45-b92d098637a7","Type":"ContainerStarted","Data":"ba73c49952da7caa18d082b2212f06c7aba55f53a3063735df3f0ceebf2a07db"}
Aug 05 17:39:09.094914 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:09.094864    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl" podStartSLOduration=1.112227447 podStartE2EDuration="2.094849962s" totalImagesPullingTime="982.622515ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:39:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:39:09.094121298 +0000 UTC m=+1604.838755626" watchObservedRunningTime="2026-08-05 17:39:09.094849962 +0000 UTC m=+1604.839484289"
Aug 05 17:39:09.449391 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:09.449301    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"
Aug 05 17:39:16.453700 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:16.453617    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"]
Aug 05 17:39:16.457372 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:16.457347    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"
Aug 05 17:39:16.462305 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:16.462283    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v2-kserve-self-signed-certs\""
Aug 05 17:39:16.476874 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:16.476845    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"]
Aug 05 17:39:16.573928 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:16.573894    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-rp97d\" (UID: \"6a0d0398-3b45-4e2b-b424-3c11e507cf8d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"
Aug 05 17:39:16.574135 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:16.573945    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-rp97d\" (UID: \"6a0d0398-3b45-4e2b-b424-3c11e507cf8d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"
Aug 05 17:39:16.574135 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:16.574051    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-rp97d\" (UID: \"6a0d0398-3b45-4e2b-b424-3c11e507cf8d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"
Aug 05 17:39:16.574135 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:16.574090    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-dshm\") pod \"canary-v2-kserve-7f65dbbccc-rp97d\" (UID: \"6a0d0398-3b45-4e2b-b424-3c11e507cf8d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"
Aug 05 17:39:16.574135 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:16.574116    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-home\") pod \"canary-v2-kserve-7f65dbbccc-rp97d\" (UID: \"6a0d0398-3b45-4e2b-b424-3c11e507cf8d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"
Aug 05 17:39:16.574310 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:16.574146    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dngdp\" (UniqueName: \"kubernetes.io/projected/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-kube-api-access-dngdp\") pod \"canary-v2-kserve-7f65dbbccc-rp97d\" (UID: \"6a0d0398-3b45-4e2b-b424-3c11e507cf8d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"
Aug 05 17:39:16.675051 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:16.675001    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-rp97d\" (UID: \"6a0d0398-3b45-4e2b-b424-3c11e507cf8d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"
Aug 05 17:39:16.675051 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:16.675067    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-rp97d\" (UID: \"6a0d0398-3b45-4e2b-b424-3c11e507cf8d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"
Aug 05 17:39:16.675302 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:16.675089    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-dshm\") pod \"canary-v2-kserve-7f65dbbccc-rp97d\" (UID: \"6a0d0398-3b45-4e2b-b424-3c11e507cf8d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"
Aug 05 17:39:16.675302 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:16.675111    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-home\") pod \"canary-v2-kserve-7f65dbbccc-rp97d\" (UID: \"6a0d0398-3b45-4e2b-b424-3c11e507cf8d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"
Aug 05 17:39:16.675302 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:16.675139    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dngdp\" (UniqueName: \"kubernetes.io/projected/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-kube-api-access-dngdp\") pod \"canary-v2-kserve-7f65dbbccc-rp97d\" (UID: \"6a0d0398-3b45-4e2b-b424-3c11e507cf8d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"
Aug 05 17:39:16.675302 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:16.675170    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-rp97d\" (UID: \"6a0d0398-3b45-4e2b-b424-3c11e507cf8d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"
Aug 05 17:39:16.675595 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:16.675573    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-rp97d\" (UID: \"6a0d0398-3b45-4e2b-b424-3c11e507cf8d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"
Aug 05 17:39:16.675657 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:16.675631    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-home\") pod \"canary-v2-kserve-7f65dbbccc-rp97d\" (UID: \"6a0d0398-3b45-4e2b-b424-3c11e507cf8d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"
Aug 05 17:39:16.675697 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:16.675673    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-rp97d\" (UID: \"6a0d0398-3b45-4e2b-b424-3c11e507cf8d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"
Aug 05 17:39:16.677398 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:16.677370    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-dshm\") pod \"canary-v2-kserve-7f65dbbccc-rp97d\" (UID: \"6a0d0398-3b45-4e2b-b424-3c11e507cf8d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"
Aug 05 17:39:16.677515 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:16.677485    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-rp97d\" (UID: \"6a0d0398-3b45-4e2b-b424-3c11e507cf8d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"
Aug 05 17:39:16.686479 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:16.686456    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dngdp\" (UniqueName: \"kubernetes.io/projected/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-kube-api-access-dngdp\") pod \"canary-v2-kserve-7f65dbbccc-rp97d\" (UID: \"6a0d0398-3b45-4e2b-b424-3c11e507cf8d\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"
Aug 05 17:39:16.768053 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:16.767977    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"
Aug 05 17:39:16.925158 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:39:16.925123    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0d0398_3b45_4e2b_b424_3c11e507cf8d.slice/crio-b12bffbcc30cdf23e462c90a6542625fd45798be2ede74b29948b1358869cc97 WatchSource:0}: Error finding container b12bffbcc30cdf23e462c90a6542625fd45798be2ede74b29948b1358869cc97: Status 404 returned error can't find the container with id b12bffbcc30cdf23e462c90a6542625fd45798be2ede74b29948b1358869cc97
Aug 05 17:39:16.926149 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:16.926110    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"]
Aug 05 17:39:17.112807 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:17.112775    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d" event={"ID":"6a0d0398-3b45-4e2b-b424-3c11e507cf8d","Type":"ContainerStarted","Data":"9db4becf6c12ed53499d09ece6bdc537fce419e6b3cb5d4d9792d0bf94b34bc9"}
Aug 05 17:39:17.113002 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:17.112825    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d" event={"ID":"6a0d0398-3b45-4e2b-b424-3c11e507cf8d","Type":"ContainerStarted","Data":"b12bffbcc30cdf23e462c90a6542625fd45798be2ede74b29948b1358869cc97"}
Aug 05 17:39:17.149790 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:17.149738    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d" podStartSLOduration=1.14972095 podStartE2EDuration="1.14972095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:39:17.147025516 +0000 UTC m=+1612.891659847" watchObservedRunningTime="2026-08-05 17:39:17.14972095 +0000 UTC m=+1612.894355276"
Aug 05 17:39:17.449944 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:17.449851    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"
Aug 05 17:39:17.462385 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:17.462359    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"
Aug 05 17:39:17.768169 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:17.768095    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"
Aug 05 17:39:18.128455 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:18.128424    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"
Aug 05 17:39:25.973246 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:25.973214    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"]
Aug 05 17:39:25.976812 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:25.976785    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"
Aug 05 17:39:25.979323 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:25.979298    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-6qwjx\""
Aug 05 17:39:25.980366 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:25.980350    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-4fx89\""
Aug 05 17:39:25.980463 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:25.980404    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Aug 05 17:39:25.981210 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:25.981190    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Aug 05 17:39:25.981315 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:25.981220    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Aug 05 17:39:25.991340 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:25.991320    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"]
Aug 05 17:39:26.057851 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:26.057818    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ebb9275-183f-46be-b1af-13db33fc8966-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9\" (UID: \"9ebb9275-183f-46be-b1af-13db33fc8966\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"
Aug 05 17:39:26.058067 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:26.057859    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ebb9275-183f-46be-b1af-13db33fc8966-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9\" (UID: \"9ebb9275-183f-46be-b1af-13db33fc8966\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"
Aug 05 17:39:26.058067 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:26.057877    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ebb9275-183f-46be-b1af-13db33fc8966-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9\" (UID: \"9ebb9275-183f-46be-b1af-13db33fc8966\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"
Aug 05 17:39:26.058067 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:26.057896    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5c6z\" (UniqueName: \"kubernetes.io/projected/9ebb9275-183f-46be-b1af-13db33fc8966-kube-api-access-n5c6z\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9\" (UID: \"9ebb9275-183f-46be-b1af-13db33fc8966\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"
Aug 05 17:39:26.058067 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:26.057995    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ebb9275-183f-46be-b1af-13db33fc8966-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9\" (UID: \"9ebb9275-183f-46be-b1af-13db33fc8966\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"
Aug 05 17:39:26.058067 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:26.058056    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ebb9275-183f-46be-b1af-13db33fc8966-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9\" (UID: \"9ebb9275-183f-46be-b1af-13db33fc8966\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"
Aug 05 17:39:26.158667 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:26.158636    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ebb9275-183f-46be-b1af-13db33fc8966-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9\" (UID: \"9ebb9275-183f-46be-b1af-13db33fc8966\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"
Aug 05 17:39:26.158864 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:26.158680    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ebb9275-183f-46be-b1af-13db33fc8966-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9\" (UID: \"9ebb9275-183f-46be-b1af-13db33fc8966\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"
Aug 05 17:39:26.158864 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:26.158768    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ebb9275-183f-46be-b1af-13db33fc8966-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9\" (UID: \"9ebb9275-183f-46be-b1af-13db33fc8966\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"
Aug 05 17:39:26.158864 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:26.158794    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5c6z\" (UniqueName: \"kubernetes.io/projected/9ebb9275-183f-46be-b1af-13db33fc8966-kube-api-access-n5c6z\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9\" (UID: \"9ebb9275-183f-46be-b1af-13db33fc8966\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"
Aug 05 17:39:26.158864 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:26.158829    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ebb9275-183f-46be-b1af-13db33fc8966-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9\" (UID: \"9ebb9275-183f-46be-b1af-13db33fc8966\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"
Aug 05 17:39:26.159140 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:26.158889    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ebb9275-183f-46be-b1af-13db33fc8966-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9\" (UID: \"9ebb9275-183f-46be-b1af-13db33fc8966\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"
Aug 05 17:39:26.159200 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:26.159162    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ebb9275-183f-46be-b1af-13db33fc8966-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9\" (UID: \"9ebb9275-183f-46be-b1af-13db33fc8966\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"
Aug 05 17:39:26.159249 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:26.159223    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ebb9275-183f-46be-b1af-13db33fc8966-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9\" (UID: \"9ebb9275-183f-46be-b1af-13db33fc8966\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"
Aug 05 17:39:26.159303 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:26.159271    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ebb9275-183f-46be-b1af-13db33fc8966-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9\" (UID: \"9ebb9275-183f-46be-b1af-13db33fc8966\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"
Aug 05 17:39:26.161088 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:26.161064    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ebb9275-183f-46be-b1af-13db33fc8966-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9\" (UID: \"9ebb9275-183f-46be-b1af-13db33fc8966\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"
Aug 05 17:39:26.161330 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:26.161313    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ebb9275-183f-46be-b1af-13db33fc8966-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9\" (UID: \"9ebb9275-183f-46be-b1af-13db33fc8966\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"
Aug 05 17:39:26.170910 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:26.170881    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5c6z\" (UniqueName: \"kubernetes.io/projected/9ebb9275-183f-46be-b1af-13db33fc8966-kube-api-access-n5c6z\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9\" (UID: \"9ebb9275-183f-46be-b1af-13db33fc8966\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"
Aug 05 17:39:26.289869 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:26.289833    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"
Aug 05 17:39:26.444524 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:26.444489    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"]
Aug 05 17:39:26.445520 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:39:26.445494    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ebb9275_183f_46be_b1af_13db33fc8966.slice/crio-14703f8b247891575132a5b1020e005e7fa7f5f299c78736cd506b262a16cc76 WatchSource:0}: Error finding container 14703f8b247891575132a5b1020e005e7fa7f5f299c78736cd506b262a16cc76: Status 404 returned error can't find the container with id 14703f8b247891575132a5b1020e005e7fa7f5f299c78736cd506b262a16cc76
Aug 05 17:39:26.769188 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:26.769105    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"
Aug 05 17:39:26.781379 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:26.781352    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"
Aug 05 17:39:27.169396 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:27.169358    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9" event={"ID":"9ebb9275-183f-46be-b1af-13db33fc8966","Type":"ContainerStarted","Data":"6571a48f2433fb6455e43ae3c2490c9a397c1ead0f0f2d74a26fb2fc66e76898"}
Aug 05 17:39:27.169839 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:27.169431    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9" event={"ID":"9ebb9275-183f-46be-b1af-13db33fc8966","Type":"ContainerStarted","Data":"14703f8b247891575132a5b1020e005e7fa7f5f299c78736cd506b262a16cc76"}
Aug 05 17:39:27.169972 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:27.169929    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"
Aug 05 17:39:27.181519 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:27.181498    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"
Aug 05 17:39:28.177888 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:28.177845    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9" event={"ID":"9ebb9275-183f-46be-b1af-13db33fc8966","Type":"ContainerStarted","Data":"25c78613e2844595aae78c76dc6e9eb169f2c6b835299151b1d4d1181f03ba09"}
Aug 05 17:39:28.202160 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:28.202111    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9" podStartSLOduration=3.202095323 podStartE2EDuration="3.202095323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:39:28.201100539 +0000 UTC m=+1623.945734865" watchObservedRunningTime="2026-08-05 17:39:28.202095323 +0000 UTC m=+1623.946729703"
Aug 05 17:39:28.290382 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:28.290348    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"
Aug 05 17:39:36.290556 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:36.290516    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"
Aug 05 17:39:36.304170 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:36.304138    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"
Aug 05 17:39:36.304356 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:36.304262    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"
Aug 05 17:39:37.235927 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:39:37.235901    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"
Aug 05 17:40:26.457332 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:26.457280    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"]
Aug 05 17:40:26.467198 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:40:26.467166    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 05 17:40:26.467198 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:40:26.467198 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:40:26.467198 ip-10-0-135-151 kubenswrapper[2561]:  > containerID="25c78613e2844595aae78c76dc6e9eb169f2c6b835299151b1d4d1181f03ba09" cmd=["/bin/sleep","15"]
Aug 05 17:40:26.467432 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:26.467214    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9" message="" err=<
Aug 05 17:40:26.467432 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:40:26.467432 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:40:26.467432 ip-10-0-135-151 kubenswrapper[2561]:  >
Aug 05 17:40:26.467432 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:40:26.467225    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 05 17:40:26.467432 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:40:26.467432 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:40:26.467432 ip-10-0-135-151 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9" podUID="9ebb9275-183f-46be-b1af-13db33fc8966" containerName="main" containerID="cri-o://25c78613e2844595aae78c76dc6e9eb169f2c6b835299151b1d4d1181f03ba09"
Aug 05 17:40:26.467432 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:26.467270    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9" podUID="9ebb9275-183f-46be-b1af-13db33fc8966" containerName="main" containerID="cri-o://25c78613e2844595aae78c76dc6e9eb169f2c6b835299151b1d4d1181f03ba09" gracePeriod=60
Aug 05 17:40:26.683415 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:26.683350    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9" podUID="9ebb9275-183f-46be-b1af-13db33fc8966" containerName="llm-d-routing-sidecar" containerID="cri-o://6571a48f2433fb6455e43ae3c2490c9a397c1ead0f0f2d74a26fb2fc66e76898" gracePeriod=60
Aug 05 17:40:26.967047 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:26.967024    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"
Aug 05 17:40:27.011160 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.011131    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ebb9275-183f-46be-b1af-13db33fc8966-dshm\") pod \"9ebb9275-183f-46be-b1af-13db33fc8966\" (UID: \"9ebb9275-183f-46be-b1af-13db33fc8966\") "
Aug 05 17:40:27.011332 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.011182    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ebb9275-183f-46be-b1af-13db33fc8966-model-cache\") pod \"9ebb9275-183f-46be-b1af-13db33fc8966\" (UID: \"9ebb9275-183f-46be-b1af-13db33fc8966\") "
Aug 05 17:40:27.011332 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.011218    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ebb9275-183f-46be-b1af-13db33fc8966-tls-certs\") pod \"9ebb9275-183f-46be-b1af-13db33fc8966\" (UID: \"9ebb9275-183f-46be-b1af-13db33fc8966\") "
Aug 05 17:40:27.011332 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.011249    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ebb9275-183f-46be-b1af-13db33fc8966-home\") pod \"9ebb9275-183f-46be-b1af-13db33fc8966\" (UID: \"9ebb9275-183f-46be-b1af-13db33fc8966\") "
Aug 05 17:40:27.011332 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.011269    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ebb9275-183f-46be-b1af-13db33fc8966-tmp-dir\") pod \"9ebb9275-183f-46be-b1af-13db33fc8966\" (UID: \"9ebb9275-183f-46be-b1af-13db33fc8966\") "
Aug 05 17:40:27.011533 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.011336    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5c6z\" (UniqueName: \"kubernetes.io/projected/9ebb9275-183f-46be-b1af-13db33fc8966-kube-api-access-n5c6z\") pod \"9ebb9275-183f-46be-b1af-13db33fc8966\" (UID: \"9ebb9275-183f-46be-b1af-13db33fc8966\") "
Aug 05 17:40:27.011707 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.011670    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ebb9275-183f-46be-b1af-13db33fc8966-model-cache" (OuterVolumeSpecName: "model-cache") pod "9ebb9275-183f-46be-b1af-13db33fc8966" (UID: "9ebb9275-183f-46be-b1af-13db33fc8966"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:40:27.011801 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.011685    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ebb9275-183f-46be-b1af-13db33fc8966-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9ebb9275-183f-46be-b1af-13db33fc8966" (UID: "9ebb9275-183f-46be-b1af-13db33fc8966"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:40:27.011801 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.011757    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ebb9275-183f-46be-b1af-13db33fc8966-home" (OuterVolumeSpecName: "home") pod "9ebb9275-183f-46be-b1af-13db33fc8966" (UID: "9ebb9275-183f-46be-b1af-13db33fc8966"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:40:27.013536 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.013509    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ebb9275-183f-46be-b1af-13db33fc8966-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9ebb9275-183f-46be-b1af-13db33fc8966" (UID: "9ebb9275-183f-46be-b1af-13db33fc8966"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:40:27.013690 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.013666    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ebb9275-183f-46be-b1af-13db33fc8966-kube-api-access-n5c6z" (OuterVolumeSpecName: "kube-api-access-n5c6z") pod "9ebb9275-183f-46be-b1af-13db33fc8966" (UID: "9ebb9275-183f-46be-b1af-13db33fc8966"). InnerVolumeSpecName "kube-api-access-n5c6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:40:27.013892 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.013872    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ebb9275-183f-46be-b1af-13db33fc8966-dshm" (OuterVolumeSpecName: "dshm") pod "9ebb9275-183f-46be-b1af-13db33fc8966" (UID: "9ebb9275-183f-46be-b1af-13db33fc8966"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:40:27.113067 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.112987    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ebb9275-183f-46be-b1af-13db33fc8966-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:40:27.113067 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.113013    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ebb9275-183f-46be-b1af-13db33fc8966-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:40:27.113067 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.113025    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ebb9275-183f-46be-b1af-13db33fc8966-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:40:27.113067 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.113033    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ebb9275-183f-46be-b1af-13db33fc8966-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:40:27.113067 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.113061    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ebb9275-183f-46be-b1af-13db33fc8966-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:40:27.113067 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.113070    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5c6z\" (UniqueName: \"kubernetes.io/projected/9ebb9275-183f-46be-b1af-13db33fc8966-kube-api-access-n5c6z\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:40:27.501822 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.501729    2561 generic.go:356] "Generic (PLEG): container finished" podID="9ebb9275-183f-46be-b1af-13db33fc8966" containerID="25c78613e2844595aae78c76dc6e9eb169f2c6b835299151b1d4d1181f03ba09" exitCode=0
Aug 05 17:40:27.501822 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.501755    2561 generic.go:356] "Generic (PLEG): container finished" podID="9ebb9275-183f-46be-b1af-13db33fc8966" containerID="6571a48f2433fb6455e43ae3c2490c9a397c1ead0f0f2d74a26fb2fc66e76898" exitCode=0
Aug 05 17:40:27.501822 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.501783    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"
Aug 05 17:40:27.501822 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.501813    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9" event={"ID":"9ebb9275-183f-46be-b1af-13db33fc8966","Type":"ContainerDied","Data":"25c78613e2844595aae78c76dc6e9eb169f2c6b835299151b1d4d1181f03ba09"}
Aug 05 17:40:27.502486 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.501868    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9" event={"ID":"9ebb9275-183f-46be-b1af-13db33fc8966","Type":"ContainerDied","Data":"6571a48f2433fb6455e43ae3c2490c9a397c1ead0f0f2d74a26fb2fc66e76898"}
Aug 05 17:40:27.502486 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.501885    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9" event={"ID":"9ebb9275-183f-46be-b1af-13db33fc8966","Type":"ContainerDied","Data":"14703f8b247891575132a5b1020e005e7fa7f5f299c78736cd506b262a16cc76"}
Aug 05 17:40:27.502486 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.501907    2561 scope.go:117] "RemoveContainer" containerID="25c78613e2844595aae78c76dc6e9eb169f2c6b835299151b1d4d1181f03ba09"
Aug 05 17:40:27.512400 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.512381    2561 scope.go:117] "RemoveContainer" containerID="6571a48f2433fb6455e43ae3c2490c9a397c1ead0f0f2d74a26fb2fc66e76898"
Aug 05 17:40:27.520484 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.520466    2561 scope.go:117] "RemoveContainer" containerID="25c78613e2844595aae78c76dc6e9eb169f2c6b835299151b1d4d1181f03ba09"
Aug 05 17:40:27.520760 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:40:27.520741    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25c78613e2844595aae78c76dc6e9eb169f2c6b835299151b1d4d1181f03ba09\": container with ID starting with 25c78613e2844595aae78c76dc6e9eb169f2c6b835299151b1d4d1181f03ba09 not found: ID does not exist" containerID="25c78613e2844595aae78c76dc6e9eb169f2c6b835299151b1d4d1181f03ba09"
Aug 05 17:40:27.520808 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.520771    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25c78613e2844595aae78c76dc6e9eb169f2c6b835299151b1d4d1181f03ba09"} err="failed to get container status \"25c78613e2844595aae78c76dc6e9eb169f2c6b835299151b1d4d1181f03ba09\": rpc error: code = NotFound desc = could not find container \"25c78613e2844595aae78c76dc6e9eb169f2c6b835299151b1d4d1181f03ba09\": container with ID starting with 25c78613e2844595aae78c76dc6e9eb169f2c6b835299151b1d4d1181f03ba09 not found: ID does not exist"
Aug 05 17:40:27.520808 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.520789    2561 scope.go:117] "RemoveContainer" containerID="6571a48f2433fb6455e43ae3c2490c9a397c1ead0f0f2d74a26fb2fc66e76898"
Aug 05 17:40:27.521036 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:40:27.521015    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6571a48f2433fb6455e43ae3c2490c9a397c1ead0f0f2d74a26fb2fc66e76898\": container with ID starting with 6571a48f2433fb6455e43ae3c2490c9a397c1ead0f0f2d74a26fb2fc66e76898 not found: ID does not exist" containerID="6571a48f2433fb6455e43ae3c2490c9a397c1ead0f0f2d74a26fb2fc66e76898"
Aug 05 17:40:27.521107 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.521042    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6571a48f2433fb6455e43ae3c2490c9a397c1ead0f0f2d74a26fb2fc66e76898"} err="failed to get container status \"6571a48f2433fb6455e43ae3c2490c9a397c1ead0f0f2d74a26fb2fc66e76898\": rpc error: code = NotFound desc = could not find container \"6571a48f2433fb6455e43ae3c2490c9a397c1ead0f0f2d74a26fb2fc66e76898\": container with ID starting with 6571a48f2433fb6455e43ae3c2490c9a397c1ead0f0f2d74a26fb2fc66e76898 not found: ID does not exist"
Aug 05 17:40:27.521107 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.521057    2561 scope.go:117] "RemoveContainer" containerID="25c78613e2844595aae78c76dc6e9eb169f2c6b835299151b1d4d1181f03ba09"
Aug 05 17:40:27.521267 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.521248    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25c78613e2844595aae78c76dc6e9eb169f2c6b835299151b1d4d1181f03ba09"} err="failed to get container status \"25c78613e2844595aae78c76dc6e9eb169f2c6b835299151b1d4d1181f03ba09\": rpc error: code = NotFound desc = could not find container \"25c78613e2844595aae78c76dc6e9eb169f2c6b835299151b1d4d1181f03ba09\": container with ID starting with 25c78613e2844595aae78c76dc6e9eb169f2c6b835299151b1d4d1181f03ba09 not found: ID does not exist"
Aug 05 17:40:27.521338 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.521270    2561 scope.go:117] "RemoveContainer" containerID="6571a48f2433fb6455e43ae3c2490c9a397c1ead0f0f2d74a26fb2fc66e76898"
Aug 05 17:40:27.521477 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.521460    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6571a48f2433fb6455e43ae3c2490c9a397c1ead0f0f2d74a26fb2fc66e76898"} err="failed to get container status \"6571a48f2433fb6455e43ae3c2490c9a397c1ead0f0f2d74a26fb2fc66e76898\": rpc error: code = NotFound desc = could not find container \"6571a48f2433fb6455e43ae3c2490c9a397c1ead0f0f2d74a26fb2fc66e76898\": container with ID starting with 6571a48f2433fb6455e43ae3c2490c9a397c1ead0f0f2d74a26fb2fc66e76898 not found: ID does not exist"
Aug 05 17:40:27.531243 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.531222    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"]
Aug 05 17:40:27.533416 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:27.533394    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-zkhk9"]
Aug 05 17:40:28.730789 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:28.730758    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ebb9275-183f-46be-b1af-13db33fc8966" path="/var/lib/kubelet/pods/9ebb9275-183f-46be-b1af-13db33fc8966/volumes"
Aug 05 17:40:47.495696 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.495619    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"]
Aug 05 17:40:47.496122 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.496003    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ebb9275-183f-46be-b1af-13db33fc8966" containerName="llm-d-routing-sidecar"
Aug 05 17:40:47.496122 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.496014    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ebb9275-183f-46be-b1af-13db33fc8966" containerName="llm-d-routing-sidecar"
Aug 05 17:40:47.496122 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.496027    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ebb9275-183f-46be-b1af-13db33fc8966" containerName="main"
Aug 05 17:40:47.496122 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.496032    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ebb9275-183f-46be-b1af-13db33fc8966" containerName="main"
Aug 05 17:40:47.496122 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.496088    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ebb9275-183f-46be-b1af-13db33fc8966" containerName="llm-d-routing-sidecar"
Aug 05 17:40:47.496122 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.496097    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ebb9275-183f-46be-b1af-13db33fc8966" containerName="main"
Aug 05 17:40:47.500710 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.500680    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:40:47.506120 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.506100    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Aug 05 17:40:47.506849 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.506828    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Aug 05 17:40:47.507580 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.507557    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Aug 05 17:40:47.507580 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.507575    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Aug 05 17:40:47.507803 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.507781    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-qbppr\""
Aug 05 17:40:47.508239 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.507842    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Aug 05 17:40:47.524511 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.524484    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"]
Aug 05 17:40:47.598652 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.598617    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba31cb21-558f-4079-b090-b6a82d850e60-model-cache\") pod \"tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:40:47.598842 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.598673    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba31cb21-558f-4079-b090-b6a82d850e60-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:40:47.598842 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.598709    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba31cb21-558f-4079-b090-b6a82d850e60-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:40:47.598842 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.598774    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba31cb21-558f-4079-b090-b6a82d850e60-home\") pod \"tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:40:47.598842 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.598799    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba31cb21-558f-4079-b090-b6a82d850e60-dshm\") pod \"tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:40:47.598842 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.598823    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba31cb21-558f-4079-b090-b6a82d850e60-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:40:47.599068 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.598851    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh2mg\" (UniqueName: \"kubernetes.io/projected/ba31cb21-558f-4079-b090-b6a82d850e60-kube-api-access-zh2mg\") pod \"tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:40:47.599068 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.598890    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba31cb21-558f-4079-b090-b6a82d850e60-tls-certs\") pod \"tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:40:47.700026 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.699988    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba31cb21-558f-4079-b090-b6a82d850e60-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:40:47.700221 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.700036    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba31cb21-558f-4079-b090-b6a82d850e60-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:40:47.700221 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.700119    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba31cb21-558f-4079-b090-b6a82d850e60-home\") pod \"tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:40:47.700221 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.700153    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba31cb21-558f-4079-b090-b6a82d850e60-dshm\") pod \"tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:40:47.700221 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.700191    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba31cb21-558f-4079-b090-b6a82d850e60-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:40:47.700221 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.700217    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh2mg\" (UniqueName: \"kubernetes.io/projected/ba31cb21-558f-4079-b090-b6a82d850e60-kube-api-access-zh2mg\") pod \"tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:40:47.700501 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.700275    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba31cb21-558f-4079-b090-b6a82d850e60-tls-certs\") pod \"tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:40:47.700501 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.700311    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba31cb21-558f-4079-b090-b6a82d850e60-model-cache\") pod \"tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:40:47.700501 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.700397    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba31cb21-558f-4079-b090-b6a82d850e60-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:40:47.700501 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.700471    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba31cb21-558f-4079-b090-b6a82d850e60-home\") pod \"tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:40:47.700644 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.700625    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba31cb21-558f-4079-b090-b6a82d850e60-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:40:47.700741 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.700718    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba31cb21-558f-4079-b090-b6a82d850e60-model-cache\") pod \"tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:40:47.700811 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.700764    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba31cb21-558f-4079-b090-b6a82d850e60-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:40:47.702500 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.702481    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba31cb21-558f-4079-b090-b6a82d850e60-dshm\") pod \"tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:40:47.702808 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.702788    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba31cb21-558f-4079-b090-b6a82d850e60-tls-certs\") pod \"tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:40:47.711036 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.711016    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh2mg\" (UniqueName: \"kubernetes.io/projected/ba31cb21-558f-4079-b090-b6a82d850e60-kube-api-access-zh2mg\") pod \"tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:40:47.811467 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.811431    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:40:47.842697 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.842667    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr"]
Aug 05 17:40:47.846505 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.846486    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr"
Aug 05 17:40:47.867924 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:47.867878    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr"]
Aug 05 17:40:48.003435 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:48.003406    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d61b0ff-74a9-4961-bcf0-4c79b289b20e-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr\" (UID: \"9d61b0ff-74a9-4961-bcf0-4c79b289b20e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr"
Aug 05 17:40:48.003435 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:48.003438    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d61b0ff-74a9-4961-bcf0-4c79b289b20e-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr\" (UID: \"9d61b0ff-74a9-4961-bcf0-4c79b289b20e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr"
Aug 05 17:40:48.003664 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:48.003457    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d61b0ff-74a9-4961-bcf0-4c79b289b20e-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr\" (UID: \"9d61b0ff-74a9-4961-bcf0-4c79b289b20e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr"
Aug 05 17:40:48.003664 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:48.003592    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d61b0ff-74a9-4961-bcf0-4c79b289b20e-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr\" (UID: \"9d61b0ff-74a9-4961-bcf0-4c79b289b20e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr"
Aug 05 17:40:48.011380 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:48.011355    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"]
Aug 05 17:40:48.013421 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:40:48.013392    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba31cb21_558f_4079_b090_b6a82d850e60.slice/crio-e51f5acba34c94c15af5487cee9c98be51abe00c6dd5115198934f3b27f45763 WatchSource:0}: Error finding container e51f5acba34c94c15af5487cee9c98be51abe00c6dd5115198934f3b27f45763: Status 404 returned error can't find the container with id e51f5acba34c94c15af5487cee9c98be51abe00c6dd5115198934f3b27f45763
Aug 05 17:40:48.105120 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:48.105079    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d61b0ff-74a9-4961-bcf0-4c79b289b20e-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr\" (UID: \"9d61b0ff-74a9-4961-bcf0-4c79b289b20e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr"
Aug 05 17:40:48.105327 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:48.105198    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d61b0ff-74a9-4961-bcf0-4c79b289b20e-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr\" (UID: \"9d61b0ff-74a9-4961-bcf0-4c79b289b20e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr"
Aug 05 17:40:48.105327 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:48.105237    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d61b0ff-74a9-4961-bcf0-4c79b289b20e-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr\" (UID: \"9d61b0ff-74a9-4961-bcf0-4c79b289b20e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr"
Aug 05 17:40:48.105327 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:48.105264    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d61b0ff-74a9-4961-bcf0-4c79b289b20e-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr\" (UID: \"9d61b0ff-74a9-4961-bcf0-4c79b289b20e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr"
Aug 05 17:40:48.105618 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:48.105596    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d61b0ff-74a9-4961-bcf0-4c79b289b20e-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr\" (UID: \"9d61b0ff-74a9-4961-bcf0-4c79b289b20e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr"
Aug 05 17:40:48.105702 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:48.105620    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d61b0ff-74a9-4961-bcf0-4c79b289b20e-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr\" (UID: \"9d61b0ff-74a9-4961-bcf0-4c79b289b20e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr"
Aug 05 17:40:48.105873 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:48.105852    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d61b0ff-74a9-4961-bcf0-4c79b289b20e-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr\" (UID: \"9d61b0ff-74a9-4961-bcf0-4c79b289b20e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr"
Aug 05 17:40:48.107817 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:48.107795    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d61b0ff-74a9-4961-bcf0-4c79b289b20e-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr\" (UID: \"9d61b0ff-74a9-4961-bcf0-4c79b289b20e\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr"
Aug 05 17:40:48.156752 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:48.156718    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr"
Aug 05 17:40:48.327830 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:48.327805    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr"]
Aug 05 17:40:48.330311 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:40:48.330290    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d61b0ff_74a9_4961_bcf0_4c79b289b20e.slice/crio-6d880773e82334fa75fb04c3901d62900e205b8ef47d2927027617cd839d0484 WatchSource:0}: Error finding container 6d880773e82334fa75fb04c3901d62900e205b8ef47d2927027617cd839d0484: Status 404 returned error can't find the container with id 6d880773e82334fa75fb04c3901d62900e205b8ef47d2927027617cd839d0484
Aug 05 17:40:48.621762 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:48.621661    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n" event={"ID":"ba31cb21-558f-4079-b090-b6a82d850e60","Type":"ContainerStarted","Data":"f4a0d3afa4d639faa8f07f687499b535c28d42721311886af73f6cdeee422863"}
Aug 05 17:40:48.621762 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:48.621745    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n" event={"ID":"ba31cb21-558f-4079-b090-b6a82d850e60","Type":"ContainerStarted","Data":"e51f5acba34c94c15af5487cee9c98be51abe00c6dd5115198934f3b27f45763"}
Aug 05 17:40:48.623731 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:48.623694    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr" event={"ID":"9d61b0ff-74a9-4961-bcf0-4c79b289b20e","Type":"ContainerStarted","Data":"d44745a876a3ea46db95dc89dcaac4c40899f6ff98e25b0654480969616c3e7d"}
Aug 05 17:40:48.623870 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:48.623737    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr" event={"ID":"9d61b0ff-74a9-4961-bcf0-4c79b289b20e","Type":"ContainerStarted","Data":"6d880773e82334fa75fb04c3901d62900e205b8ef47d2927027617cd839d0484"}
Aug 05 17:40:50.635779 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:50.635734    2561 generic.go:356] "Generic (PLEG): container finished" podID="ba31cb21-558f-4079-b090-b6a82d850e60" containerID="f4a0d3afa4d639faa8f07f687499b535c28d42721311886af73f6cdeee422863" exitCode=0
Aug 05 17:40:50.636181 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:50.635806    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n" event={"ID":"ba31cb21-558f-4079-b090-b6a82d850e60","Type":"ContainerDied","Data":"f4a0d3afa4d639faa8f07f687499b535c28d42721311886af73f6cdeee422863"}
Aug 05 17:40:51.643315 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:51.643284    2561 generic.go:356] "Generic (PLEG): container finished" podID="9d61b0ff-74a9-4961-bcf0-4c79b289b20e" containerID="d44745a876a3ea46db95dc89dcaac4c40899f6ff98e25b0654480969616c3e7d" exitCode=0
Aug 05 17:40:51.643732 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:51.643367    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr" event={"ID":"9d61b0ff-74a9-4961-bcf0-4c79b289b20e","Type":"ContainerDied","Data":"d44745a876a3ea46db95dc89dcaac4c40899f6ff98e25b0654480969616c3e7d"}
Aug 05 17:40:51.644659 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:51.644638    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 05 17:40:51.645716 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:51.645688    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n" event={"ID":"ba31cb21-558f-4079-b090-b6a82d850e60","Type":"ContainerStarted","Data":"1c022f129ffa18687d1588b1fe956e1d23e6f3a1c1dc28296eb82062b32953f8"}
Aug 05 17:40:51.699625 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:51.699563    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n" podStartSLOduration=2.201211671 podStartE2EDuration="4.699543368s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.498331697s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:40:51.698553684 +0000 UTC m=+1707.443188050" watchObservedRunningTime="2026-08-05 17:40:51.699543368 +0000 UTC m=+1707.444177695"
Aug 05 17:40:51.812079 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:51.812041    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:40:57.811936 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:57.811897    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:40:57.829201 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:57.829176    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:40:58.723308 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:40:58.723278    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:41:38.988003 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:41:38.987966    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr" event={"ID":"9d61b0ff-74a9-4961-bcf0-4c79b289b20e","Type":"ContainerStarted","Data":"4517688dabc03dc1403cedca4f22be9aa1d5d1442342c7f7308a9bd7643844fd"}
Aug 05 17:41:38.988521 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:41:38.988184    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr"
Aug 05 17:41:39.014141 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:41:39.014091    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr" podStartSLOduration=2.744809167 podStartE2EDuration="52.014075627s" totalImagesPullingTime="46.714924644s" totalInitContainerRuntime="2.554341816s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:40:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:41:39.01004336 +0000 UTC m=+1754.754677669" watchObservedRunningTime="2026-08-05 17:41:39.014075627 +0000 UTC m=+1754.758709954"
Aug 05 17:41:48.101548 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:41:48.101500    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"]
Aug 05 17:41:49.097713 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:41:49.097664    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"]
Aug 05 17:42:03.120847 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:03.120799    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d" podUID="6a0d0398-3b45-4e2b-b424-3c11e507cf8d" containerName="main" containerID="cri-o://9db4becf6c12ed53499d09ece6bdc537fce419e6b3cb5d4d9792d0bf94b34bc9" gracePeriod=45
Aug 05 17:42:03.170426 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:03.170379    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d" podUID="6a0d0398-3b45-4e2b-b424-3c11e507cf8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Aug 05 17:42:03.915388 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:03.915357    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"
Aug 05 17:42:04.059136 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.059107    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-model-cache\") pod \"6a0d0398-3b45-4e2b-b424-3c11e507cf8d\" (UID: \"6a0d0398-3b45-4e2b-b424-3c11e507cf8d\") "
Aug 05 17:42:04.059332 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.059151    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-home\") pod \"6a0d0398-3b45-4e2b-b424-3c11e507cf8d\" (UID: \"6a0d0398-3b45-4e2b-b424-3c11e507cf8d\") "
Aug 05 17:42:04.059332 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.059208    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-dshm\") pod \"6a0d0398-3b45-4e2b-b424-3c11e507cf8d\" (UID: \"6a0d0398-3b45-4e2b-b424-3c11e507cf8d\") "
Aug 05 17:42:04.059332 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.059228    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dngdp\" (UniqueName: \"kubernetes.io/projected/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-kube-api-access-dngdp\") pod \"6a0d0398-3b45-4e2b-b424-3c11e507cf8d\" (UID: \"6a0d0398-3b45-4e2b-b424-3c11e507cf8d\") "
Aug 05 17:42:04.059332 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.059284    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-tls-certs\") pod \"6a0d0398-3b45-4e2b-b424-3c11e507cf8d\" (UID: \"6a0d0398-3b45-4e2b-b424-3c11e507cf8d\") "
Aug 05 17:42:04.059332 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.059315    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-tmp-dir\") pod \"6a0d0398-3b45-4e2b-b424-3c11e507cf8d\" (UID: \"6a0d0398-3b45-4e2b-b424-3c11e507cf8d\") "
Aug 05 17:42:04.059660 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.059384    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-model-cache" (OuterVolumeSpecName: "model-cache") pod "6a0d0398-3b45-4e2b-b424-3c11e507cf8d" (UID: "6a0d0398-3b45-4e2b-b424-3c11e507cf8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:42:04.059660 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.059480    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-home" (OuterVolumeSpecName: "home") pod "6a0d0398-3b45-4e2b-b424-3c11e507cf8d" (UID: "6a0d0398-3b45-4e2b-b424-3c11e507cf8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:42:04.059660 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.059572    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6a0d0398-3b45-4e2b-b424-3c11e507cf8d" (UID: "6a0d0398-3b45-4e2b-b424-3c11e507cf8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:42:04.059660 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.059582    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:42:04.059660 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.059597    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:42:04.061539 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.061495    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a0d0398-3b45-4e2b-b424-3c11e507cf8d" (UID: "6a0d0398-3b45-4e2b-b424-3c11e507cf8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:42:04.061646 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.061585    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-kube-api-access-dngdp" (OuterVolumeSpecName: "kube-api-access-dngdp") pod "6a0d0398-3b45-4e2b-b424-3c11e507cf8d" (UID: "6a0d0398-3b45-4e2b-b424-3c11e507cf8d"). InnerVolumeSpecName "kube-api-access-dngdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:42:04.061885 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.061863    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-dshm" (OuterVolumeSpecName: "dshm") pod "6a0d0398-3b45-4e2b-b424-3c11e507cf8d" (UID: "6a0d0398-3b45-4e2b-b424-3c11e507cf8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:42:04.117342 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.117293    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl" podUID="d66b7d70-6942-47a3-ab45-b92d098637a7" containerName="main" containerID="cri-o://ba73c49952da7caa18d082b2212f06c7aba55f53a3063735df3f0ceebf2a07db" gracePeriod=45
Aug 05 17:42:04.122405 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.122371    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl" podUID="d66b7d70-6942-47a3-ab45-b92d098637a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": EOF"
Aug 05 17:42:04.145854 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.145817    2561 generic.go:356] "Generic (PLEG): container finished" podID="6a0d0398-3b45-4e2b-b424-3c11e507cf8d" containerID="9db4becf6c12ed53499d09ece6bdc537fce419e6b3cb5d4d9792d0bf94b34bc9" exitCode=0
Aug 05 17:42:04.146028 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.145979    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"
Aug 05 17:42:04.146095 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.146048    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d" event={"ID":"6a0d0398-3b45-4e2b-b424-3c11e507cf8d","Type":"ContainerDied","Data":"9db4becf6c12ed53499d09ece6bdc537fce419e6b3cb5d4d9792d0bf94b34bc9"}
Aug 05 17:42:04.146152 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.146135    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d" event={"ID":"6a0d0398-3b45-4e2b-b424-3c11e507cf8d","Type":"ContainerDied","Data":"b12bffbcc30cdf23e462c90a6542625fd45798be2ede74b29948b1358869cc97"}
Aug 05 17:42:04.146201 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.146164    2561 scope.go:117] "RemoveContainer" containerID="9db4becf6c12ed53499d09ece6bdc537fce419e6b3cb5d4d9792d0bf94b34bc9"
Aug 05 17:42:04.160720 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.160696    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:42:04.160720 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.160720    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dngdp\" (UniqueName: \"kubernetes.io/projected/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-kube-api-access-dngdp\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:42:04.160881 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.160734    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:42:04.160881 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.160772    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a0d0398-3b45-4e2b-b424-3c11e507cf8d-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:42:04.197088 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.197060    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"]
Aug 05 17:42:04.199488 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.199465    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-rp97d"]
Aug 05 17:42:04.229849 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.229826    2561 scope.go:117] "RemoveContainer" containerID="9db4becf6c12ed53499d09ece6bdc537fce419e6b3cb5d4d9792d0bf94b34bc9"
Aug 05 17:42:04.230267 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:42:04.230245    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9db4becf6c12ed53499d09ece6bdc537fce419e6b3cb5d4d9792d0bf94b34bc9\": container with ID starting with 9db4becf6c12ed53499d09ece6bdc537fce419e6b3cb5d4d9792d0bf94b34bc9 not found: ID does not exist" containerID="9db4becf6c12ed53499d09ece6bdc537fce419e6b3cb5d4d9792d0bf94b34bc9"
Aug 05 17:42:04.230346 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.230278    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9db4becf6c12ed53499d09ece6bdc537fce419e6b3cb5d4d9792d0bf94b34bc9"} err="failed to get container status \"9db4becf6c12ed53499d09ece6bdc537fce419e6b3cb5d4d9792d0bf94b34bc9\": rpc error: code = NotFound desc = could not find container \"9db4becf6c12ed53499d09ece6bdc537fce419e6b3cb5d4d9792d0bf94b34bc9\": container with ID starting with 9db4becf6c12ed53499d09ece6bdc537fce419e6b3cb5d4d9792d0bf94b34bc9 not found: ID does not exist"
Aug 05 17:42:04.408991 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.408965    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"
Aug 05 17:42:04.565639 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.565606    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d66b7d70-6942-47a3-ab45-b92d098637a7-model-cache\") pod \"d66b7d70-6942-47a3-ab45-b92d098637a7\" (UID: \"d66b7d70-6942-47a3-ab45-b92d098637a7\") "
Aug 05 17:42:04.565639 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.565644    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d66b7d70-6942-47a3-ab45-b92d098637a7-tmp-dir\") pod \"d66b7d70-6942-47a3-ab45-b92d098637a7\" (UID: \"d66b7d70-6942-47a3-ab45-b92d098637a7\") "
Aug 05 17:42:04.565918 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.565688    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d66b7d70-6942-47a3-ab45-b92d098637a7-tls-certs\") pod \"d66b7d70-6942-47a3-ab45-b92d098637a7\" (UID: \"d66b7d70-6942-47a3-ab45-b92d098637a7\") "
Aug 05 17:42:04.565918 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.565727    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtgts\" (UniqueName: \"kubernetes.io/projected/d66b7d70-6942-47a3-ab45-b92d098637a7-kube-api-access-gtgts\") pod \"d66b7d70-6942-47a3-ab45-b92d098637a7\" (UID: \"d66b7d70-6942-47a3-ab45-b92d098637a7\") "
Aug 05 17:42:04.565918 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.565815    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d66b7d70-6942-47a3-ab45-b92d098637a7-home\") pod \"d66b7d70-6942-47a3-ab45-b92d098637a7\" (UID: \"d66b7d70-6942-47a3-ab45-b92d098637a7\") "
Aug 05 17:42:04.565918 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.565840    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d66b7d70-6942-47a3-ab45-b92d098637a7-dshm\") pod \"d66b7d70-6942-47a3-ab45-b92d098637a7\" (UID: \"d66b7d70-6942-47a3-ab45-b92d098637a7\") "
Aug 05 17:42:04.566169 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.565978    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d66b7d70-6942-47a3-ab45-b92d098637a7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d66b7d70-6942-47a3-ab45-b92d098637a7" (UID: "d66b7d70-6942-47a3-ab45-b92d098637a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:42:04.566169 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.565974    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d66b7d70-6942-47a3-ab45-b92d098637a7-model-cache" (OuterVolumeSpecName: "model-cache") pod "d66b7d70-6942-47a3-ab45-b92d098637a7" (UID: "d66b7d70-6942-47a3-ab45-b92d098637a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:42:04.566169 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.566139    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d66b7d70-6942-47a3-ab45-b92d098637a7-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:42:04.566169 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.566155    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d66b7d70-6942-47a3-ab45-b92d098637a7-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:42:04.566379 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.566205    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d66b7d70-6942-47a3-ab45-b92d098637a7-home" (OuterVolumeSpecName: "home") pod "d66b7d70-6942-47a3-ab45-b92d098637a7" (UID: "d66b7d70-6942-47a3-ab45-b92d098637a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:42:04.567974 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.567915    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d66b7d70-6942-47a3-ab45-b92d098637a7-dshm" (OuterVolumeSpecName: "dshm") pod "d66b7d70-6942-47a3-ab45-b92d098637a7" (UID: "d66b7d70-6942-47a3-ab45-b92d098637a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:42:04.568116 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.568094    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d66b7d70-6942-47a3-ab45-b92d098637a7-kube-api-access-gtgts" (OuterVolumeSpecName: "kube-api-access-gtgts") pod "d66b7d70-6942-47a3-ab45-b92d098637a7" (UID: "d66b7d70-6942-47a3-ab45-b92d098637a7"). InnerVolumeSpecName "kube-api-access-gtgts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:42:04.568238 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.568222    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d66b7d70-6942-47a3-ab45-b92d098637a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d66b7d70-6942-47a3-ab45-b92d098637a7" (UID: "d66b7d70-6942-47a3-ab45-b92d098637a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:42:04.666894 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.666811    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d66b7d70-6942-47a3-ab45-b92d098637a7-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:42:04.666894 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.666842    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtgts\" (UniqueName: \"kubernetes.io/projected/d66b7d70-6942-47a3-ab45-b92d098637a7-kube-api-access-gtgts\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:42:04.666894 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.666856    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d66b7d70-6942-47a3-ab45-b92d098637a7-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:42:04.666894 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.666868    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d66b7d70-6942-47a3-ab45-b92d098637a7-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:42:04.734808 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:04.734773    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a0d0398-3b45-4e2b-b424-3c11e507cf8d" path="/var/lib/kubelet/pods/6a0d0398-3b45-4e2b-b424-3c11e507cf8d/volumes"
Aug 05 17:42:05.157081 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:05.157045    2561 generic.go:356] "Generic (PLEG): container finished" podID="d66b7d70-6942-47a3-ab45-b92d098637a7" containerID="ba73c49952da7caa18d082b2212f06c7aba55f53a3063735df3f0ceebf2a07db" exitCode=0
Aug 05 17:42:05.157597 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:05.157110    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"
Aug 05 17:42:05.157597 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:05.157137    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl" event={"ID":"d66b7d70-6942-47a3-ab45-b92d098637a7","Type":"ContainerDied","Data":"ba73c49952da7caa18d082b2212f06c7aba55f53a3063735df3f0ceebf2a07db"}
Aug 05 17:42:05.157597 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:05.157192    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl" event={"ID":"d66b7d70-6942-47a3-ab45-b92d098637a7","Type":"ContainerDied","Data":"5cf0489829b6f2ddea90494b77e68dc521e59e8bb1ff9fec50d14f37057757e6"}
Aug 05 17:42:05.157597 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:05.157215    2561 scope.go:117] "RemoveContainer" containerID="ba73c49952da7caa18d082b2212f06c7aba55f53a3063735df3f0ceebf2a07db"
Aug 05 17:42:05.168425 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:05.168377    2561 scope.go:117] "RemoveContainer" containerID="ba73c49952da7caa18d082b2212f06c7aba55f53a3063735df3f0ceebf2a07db"
Aug 05 17:42:05.168613 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:42:05.168593    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba73c49952da7caa18d082b2212f06c7aba55f53a3063735df3f0ceebf2a07db\": container with ID starting with ba73c49952da7caa18d082b2212f06c7aba55f53a3063735df3f0ceebf2a07db not found: ID does not exist" containerID="ba73c49952da7caa18d082b2212f06c7aba55f53a3063735df3f0ceebf2a07db"
Aug 05 17:42:05.168676 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:05.168624    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba73c49952da7caa18d082b2212f06c7aba55f53a3063735df3f0ceebf2a07db"} err="failed to get container status \"ba73c49952da7caa18d082b2212f06c7aba55f53a3063735df3f0ceebf2a07db\": rpc error: code = NotFound desc = could not find container \"ba73c49952da7caa18d082b2212f06c7aba55f53a3063735df3f0ceebf2a07db\": container with ID starting with ba73c49952da7caa18d082b2212f06c7aba55f53a3063735df3f0ceebf2a07db not found: ID does not exist"
Aug 05 17:42:05.177445 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:05.177418    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"]
Aug 05 17:42:05.182805 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:05.182784    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-6xnhl"]
Aug 05 17:42:06.730243 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:06.730204    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d66b7d70-6942-47a3-ab45-b92d098637a7" path="/var/lib/kubelet/pods/d66b7d70-6942-47a3-ab45-b92d098637a7/volumes"
Aug 05 17:42:10.005335 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:10.005295    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr"
Aug 05 17:42:14.045020 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:14.044986    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr_9d61b0ff-74a9-4961-bcf0-4c79b289b20e/main/0.log"
Aug 05 17:42:36.697220 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:36.697178    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"]
Aug 05 17:42:36.704501 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:36.704474    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr"]
Aug 05 17:42:36.704679 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:36.704660    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr" podUID="9d61b0ff-74a9-4961-bcf0-4c79b289b20e" containerName="main" containerID="cri-o://4517688dabc03dc1403cedca4f22be9aa1d5d1442342c7f7308a9bd7643844fd" gracePeriod=30
Aug 05 17:42:36.707313 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:42:36.707278    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 05 17:42:36.707313 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:42:36.707313 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:42:36.707313 ip-10-0-135-151 kubenswrapper[2561]:  > containerID="1c022f129ffa18687d1588b1fe956e1d23e6f3a1c1dc28296eb82062b32953f8" cmd=["/bin/sleep","15"]
Aug 05 17:42:36.707575 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:36.707327    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n" message="" err=<
Aug 05 17:42:36.707575 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:42:36.707575 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:42:36.707575 ip-10-0-135-151 kubenswrapper[2561]:  >
Aug 05 17:42:36.707575 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:42:36.707340    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 05 17:42:36.707575 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:42:36.707575 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:42:36.707575 ip-10-0-135-151 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n" podUID="ba31cb21-558f-4079-b090-b6a82d850e60" containerName="main" containerID="cri-o://1c022f129ffa18687d1588b1fe956e1d23e6f3a1c1dc28296eb82062b32953f8"
Aug 05 17:42:36.707575 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:36.707386    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n" podUID="ba31cb21-558f-4079-b090-b6a82d850e60" containerName="main" containerID="cri-o://1c022f129ffa18687d1588b1fe956e1d23e6f3a1c1dc28296eb82062b32953f8" gracePeriod=60
Aug 05 17:42:37.063772 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.063746    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:42:37.153312 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.153283    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba31cb21-558f-4079-b090-b6a82d850e60-tmp-dir\") pod \"ba31cb21-558f-4079-b090-b6a82d850e60\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") "
Aug 05 17:42:37.153463 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.153334    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba31cb21-558f-4079-b090-b6a82d850e60-home\") pod \"ba31cb21-558f-4079-b090-b6a82d850e60\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") "
Aug 05 17:42:37.153463 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.153356    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba31cb21-558f-4079-b090-b6a82d850e60-cabundle-cert\") pod \"ba31cb21-558f-4079-b090-b6a82d850e60\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") "
Aug 05 17:42:37.153463 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.153390    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba31cb21-558f-4079-b090-b6a82d850e60-dshm\") pod \"ba31cb21-558f-4079-b090-b6a82d850e60\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") "
Aug 05 17:42:37.153463 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.153439    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba31cb21-558f-4079-b090-b6a82d850e60-model-cache\") pod \"ba31cb21-558f-4079-b090-b6a82d850e60\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") "
Aug 05 17:42:37.153677 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.153488    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba31cb21-558f-4079-b090-b6a82d850e60-kserve-provision-location\") pod \"ba31cb21-558f-4079-b090-b6a82d850e60\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") "
Aug 05 17:42:37.153677 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.153515    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba31cb21-558f-4079-b090-b6a82d850e60-tls-certs\") pod \"ba31cb21-558f-4079-b090-b6a82d850e60\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") "
Aug 05 17:42:37.153677 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.153542    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zh2mg\" (UniqueName: \"kubernetes.io/projected/ba31cb21-558f-4079-b090-b6a82d850e60-kube-api-access-zh2mg\") pod \"ba31cb21-558f-4079-b090-b6a82d850e60\" (UID: \"ba31cb21-558f-4079-b090-b6a82d850e60\") "
Aug 05 17:42:37.153677 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.153627    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba31cb21-558f-4079-b090-b6a82d850e60-home" (OuterVolumeSpecName: "home") pod "ba31cb21-558f-4079-b090-b6a82d850e60" (UID: "ba31cb21-558f-4079-b090-b6a82d850e60"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:42:37.153677 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.153606    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba31cb21-558f-4079-b090-b6a82d850e60-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ba31cb21-558f-4079-b090-b6a82d850e60" (UID: "ba31cb21-558f-4079-b090-b6a82d850e60"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:42:37.153937 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.153740    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba31cb21-558f-4079-b090-b6a82d850e60-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ba31cb21-558f-4079-b090-b6a82d850e60" (UID: "ba31cb21-558f-4079-b090-b6a82d850e60"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:42:37.153937 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.153745    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba31cb21-558f-4079-b090-b6a82d850e60-model-cache" (OuterVolumeSpecName: "model-cache") pod "ba31cb21-558f-4079-b090-b6a82d850e60" (UID: "ba31cb21-558f-4079-b090-b6a82d850e60"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:42:37.153937 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.153878    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba31cb21-558f-4079-b090-b6a82d850e60-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:42:37.153937 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.153910    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba31cb21-558f-4079-b090-b6a82d850e60-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:42:37.153937 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.153923    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba31cb21-558f-4079-b090-b6a82d850e60-cabundle-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:42:37.153937 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.153935    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba31cb21-558f-4079-b090-b6a82d850e60-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:42:37.157110 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.157084    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba31cb21-558f-4079-b090-b6a82d850e60-dshm" (OuterVolumeSpecName: "dshm") pod "ba31cb21-558f-4079-b090-b6a82d850e60" (UID: "ba31cb21-558f-4079-b090-b6a82d850e60"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:42:37.159316 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.159297    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba31cb21-558f-4079-b090-b6a82d850e60-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba31cb21-558f-4079-b090-b6a82d850e60" (UID: "ba31cb21-558f-4079-b090-b6a82d850e60"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:42:37.159404 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.159379    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba31cb21-558f-4079-b090-b6a82d850e60-kube-api-access-zh2mg" (OuterVolumeSpecName: "kube-api-access-zh2mg") pod "ba31cb21-558f-4079-b090-b6a82d850e60" (UID: "ba31cb21-558f-4079-b090-b6a82d850e60"). InnerVolumeSpecName "kube-api-access-zh2mg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:42:37.179180 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.179151    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba31cb21-558f-4079-b090-b6a82d850e60-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ba31cb21-558f-4079-b090-b6a82d850e60" (UID: "ba31cb21-558f-4079-b090-b6a82d850e60"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:42:37.255355 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.255274    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba31cb21-558f-4079-b090-b6a82d850e60-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:42:37.255355 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.255302    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba31cb21-558f-4079-b090-b6a82d850e60-kserve-provision-location\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:42:37.255355 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.255313    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba31cb21-558f-4079-b090-b6a82d850e60-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:42:37.255355 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.255322    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh2mg\" (UniqueName: \"kubernetes.io/projected/ba31cb21-558f-4079-b090-b6a82d850e60-kube-api-access-zh2mg\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:42:37.341399 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.341354    2561 generic.go:356] "Generic (PLEG): container finished" podID="ba31cb21-558f-4079-b090-b6a82d850e60" containerID="1c022f129ffa18687d1588b1fe956e1d23e6f3a1c1dc28296eb82062b32953f8" exitCode=0
Aug 05 17:42:37.341593 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.341415    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"
Aug 05 17:42:37.341593 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.341476    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n" event={"ID":"ba31cb21-558f-4079-b090-b6a82d850e60","Type":"ContainerDied","Data":"1c022f129ffa18687d1588b1fe956e1d23e6f3a1c1dc28296eb82062b32953f8"}
Aug 05 17:42:37.341593 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.341569    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n" event={"ID":"ba31cb21-558f-4079-b090-b6a82d850e60","Type":"ContainerDied","Data":"e51f5acba34c94c15af5487cee9c98be51abe00c6dd5115198934f3b27f45763"}
Aug 05 17:42:37.341769 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.341601    2561 scope.go:117] "RemoveContainer" containerID="1c022f129ffa18687d1588b1fe956e1d23e6f3a1c1dc28296eb82062b32953f8"
Aug 05 17:42:37.344451 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.344425    2561 generic.go:356] "Generic (PLEG): container finished" podID="9d61b0ff-74a9-4961-bcf0-4c79b289b20e" containerID="4517688dabc03dc1403cedca4f22be9aa1d5d1442342c7f7308a9bd7643844fd" exitCode=0
Aug 05 17:42:37.344578 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.344455    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr" event={"ID":"9d61b0ff-74a9-4961-bcf0-4c79b289b20e","Type":"ContainerDied","Data":"4517688dabc03dc1403cedca4f22be9aa1d5d1442342c7f7308a9bd7643844fd"}
Aug 05 17:42:37.354644 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.354628    2561 scope.go:117] "RemoveContainer" containerID="f4a0d3afa4d639faa8f07f687499b535c28d42721311886af73f6cdeee422863"
Aug 05 17:42:37.365826 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.365809    2561 scope.go:117] "RemoveContainer" containerID="1c022f129ffa18687d1588b1fe956e1d23e6f3a1c1dc28296eb82062b32953f8"
Aug 05 17:42:37.366128 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:42:37.366102    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c022f129ffa18687d1588b1fe956e1d23e6f3a1c1dc28296eb82062b32953f8\": container with ID starting with 1c022f129ffa18687d1588b1fe956e1d23e6f3a1c1dc28296eb82062b32953f8 not found: ID does not exist" containerID="1c022f129ffa18687d1588b1fe956e1d23e6f3a1c1dc28296eb82062b32953f8"
Aug 05 17:42:37.366214 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.366139    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c022f129ffa18687d1588b1fe956e1d23e6f3a1c1dc28296eb82062b32953f8"} err="failed to get container status \"1c022f129ffa18687d1588b1fe956e1d23e6f3a1c1dc28296eb82062b32953f8\": rpc error: code = NotFound desc = could not find container \"1c022f129ffa18687d1588b1fe956e1d23e6f3a1c1dc28296eb82062b32953f8\": container with ID starting with 1c022f129ffa18687d1588b1fe956e1d23e6f3a1c1dc28296eb82062b32953f8 not found: ID does not exist"
Aug 05 17:42:37.366214 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.366162    2561 scope.go:117] "RemoveContainer" containerID="f4a0d3afa4d639faa8f07f687499b535c28d42721311886af73f6cdeee422863"
Aug 05 17:42:37.366440 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:42:37.366407    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4a0d3afa4d639faa8f07f687499b535c28d42721311886af73f6cdeee422863\": container with ID starting with f4a0d3afa4d639faa8f07f687499b535c28d42721311886af73f6cdeee422863 not found: ID does not exist" containerID="f4a0d3afa4d639faa8f07f687499b535c28d42721311886af73f6cdeee422863"
Aug 05 17:42:37.366440 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.366429    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4a0d3afa4d639faa8f07f687499b535c28d42721311886af73f6cdeee422863"} err="failed to get container status \"f4a0d3afa4d639faa8f07f687499b535c28d42721311886af73f6cdeee422863\": rpc error: code = NotFound desc = could not find container \"f4a0d3afa4d639faa8f07f687499b535c28d42721311886af73f6cdeee422863\": container with ID starting with f4a0d3afa4d639faa8f07f687499b535c28d42721311886af73f6cdeee422863 not found: ID does not exist"
Aug 05 17:42:37.366538 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:42:37.366518    2561 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba31cb21_558f_4079_b090_b6a82d850e60.slice/crio-e51f5acba34c94c15af5487cee9c98be51abe00c6dd5115198934f3b27f45763\": RecentStats: unable to find data in memory cache]"
Aug 05 17:42:37.367699 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.367679    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr"
Aug 05 17:42:37.375174 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.375154    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"]
Aug 05 17:42:37.381829 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.381796    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-58cc87dd64-bwp6n"]
Aug 05 17:42:37.457662 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.457631    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d61b0ff-74a9-4961-bcf0-4c79b289b20e-tmp-dir\") pod \"9d61b0ff-74a9-4961-bcf0-4c79b289b20e\" (UID: \"9d61b0ff-74a9-4961-bcf0-4c79b289b20e\") "
Aug 05 17:42:37.457848 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.457697    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d61b0ff-74a9-4961-bcf0-4c79b289b20e-tls-certs\") pod \"9d61b0ff-74a9-4961-bcf0-4c79b289b20e\" (UID: \"9d61b0ff-74a9-4961-bcf0-4c79b289b20e\") "
Aug 05 17:42:37.457848 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.457760    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d61b0ff-74a9-4961-bcf0-4c79b289b20e-kserve-provision-location\") pod \"9d61b0ff-74a9-4961-bcf0-4c79b289b20e\" (UID: \"9d61b0ff-74a9-4961-bcf0-4c79b289b20e\") "
Aug 05 17:42:37.457848 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.457785    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d61b0ff-74a9-4961-bcf0-4c79b289b20e-cabundle-cert\") pod \"9d61b0ff-74a9-4961-bcf0-4c79b289b20e\" (UID: \"9d61b0ff-74a9-4961-bcf0-4c79b289b20e\") "
Aug 05 17:42:37.458098 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.458073    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d61b0ff-74a9-4961-bcf0-4c79b289b20e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9d61b0ff-74a9-4961-bcf0-4c79b289b20e" (UID: "9d61b0ff-74a9-4961-bcf0-4c79b289b20e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:42:37.458192 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.458171    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d61b0ff-74a9-4961-bcf0-4c79b289b20e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9d61b0ff-74a9-4961-bcf0-4c79b289b20e" (UID: "9d61b0ff-74a9-4961-bcf0-4c79b289b20e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:42:37.459865 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.459849    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d61b0ff-74a9-4961-bcf0-4c79b289b20e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9d61b0ff-74a9-4961-bcf0-4c79b289b20e" (UID: "9d61b0ff-74a9-4961-bcf0-4c79b289b20e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:42:37.482553 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.482513    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d61b0ff-74a9-4961-bcf0-4c79b289b20e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9d61b0ff-74a9-4961-bcf0-4c79b289b20e" (UID: "9d61b0ff-74a9-4961-bcf0-4c79b289b20e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:42:37.559292 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.559259    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d61b0ff-74a9-4961-bcf0-4c79b289b20e-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:42:37.559292 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.559285    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d61b0ff-74a9-4961-bcf0-4c79b289b20e-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:42:37.559292 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.559294    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d61b0ff-74a9-4961-bcf0-4c79b289b20e-kserve-provision-location\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:42:37.559519 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:37.559304    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d61b0ff-74a9-4961-bcf0-4c79b289b20e-cabundle-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:42:38.352879 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:38.352852    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr"
Aug 05 17:42:38.353401 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:38.352888    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr" event={"ID":"9d61b0ff-74a9-4961-bcf0-4c79b289b20e","Type":"ContainerDied","Data":"6d880773e82334fa75fb04c3901d62900e205b8ef47d2927027617cd839d0484"}
Aug 05 17:42:38.353401 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:38.352985    2561 scope.go:117] "RemoveContainer" containerID="4517688dabc03dc1403cedca4f22be9aa1d5d1442342c7f7308a9bd7643844fd"
Aug 05 17:42:38.365627 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:38.365606    2561 scope.go:117] "RemoveContainer" containerID="d44745a876a3ea46db95dc89dcaac4c40899f6ff98e25b0654480969616c3e7d"
Aug 05 17:42:38.381303 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:38.381275    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr"]
Aug 05 17:42:38.386467 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:38.386436    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5f86876695-cpcwr"]
Aug 05 17:42:38.730175 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:38.730079    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d61b0ff-74a9-4961-bcf0-4c79b289b20e" path="/var/lib/kubelet/pods/9d61b0ff-74a9-4961-bcf0-4c79b289b20e/volumes"
Aug 05 17:42:38.730533 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:38.730517    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba31cb21-558f-4079-b090-b6a82d850e60" path="/var/lib/kubelet/pods/ba31cb21-558f-4079-b090-b6a82d850e60/volumes"
Aug 05 17:42:51.815586 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:51.815554    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"]
Aug 05 17:42:51.816054 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:51.815901    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d61b0ff-74a9-4961-bcf0-4c79b289b20e" containerName="main"
Aug 05 17:42:51.816054 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:51.815912    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d61b0ff-74a9-4961-bcf0-4c79b289b20e" containerName="main"
Aug 05 17:42:51.816054 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:51.815922    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a0d0398-3b45-4e2b-b424-3c11e507cf8d" containerName="main"
Aug 05 17:42:51.816054 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:51.815927    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a0d0398-3b45-4e2b-b424-3c11e507cf8d" containerName="main"
Aug 05 17:42:51.816054 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:51.815933    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d61b0ff-74a9-4961-bcf0-4c79b289b20e" containerName="storage-initializer"
Aug 05 17:42:51.816054 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:51.815939    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d61b0ff-74a9-4961-bcf0-4c79b289b20e" containerName="storage-initializer"
Aug 05 17:42:51.816054 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:51.815945    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba31cb21-558f-4079-b090-b6a82d850e60" containerName="main"
Aug 05 17:42:51.816054 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:51.815967    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba31cb21-558f-4079-b090-b6a82d850e60" containerName="main"
Aug 05 17:42:51.816054 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:51.815980    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba31cb21-558f-4079-b090-b6a82d850e60" containerName="storage-initializer"
Aug 05 17:42:51.816054 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:51.815985    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba31cb21-558f-4079-b090-b6a82d850e60" containerName="storage-initializer"
Aug 05 17:42:51.816054 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:51.815999    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d66b7d70-6942-47a3-ab45-b92d098637a7" containerName="main"
Aug 05 17:42:51.816054 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:51.816004    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d66b7d70-6942-47a3-ab45-b92d098637a7" containerName="main"
Aug 05 17:42:51.816421 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:51.816068    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba31cb21-558f-4079-b090-b6a82d850e60" containerName="main"
Aug 05 17:42:51.816421 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:51.816076    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d61b0ff-74a9-4961-bcf0-4c79b289b20e" containerName="main"
Aug 05 17:42:51.816421 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:51.816082    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a0d0398-3b45-4e2b-b424-3c11e507cf8d" containerName="main"
Aug 05 17:42:51.816421 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:51.816089    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d66b7d70-6942-47a3-ab45-b92d098637a7" containerName="main"
Aug 05 17:42:51.821094 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:51.821074    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"
Aug 05 17:42:51.824173 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:51.824151    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"default-dockercfg-zjdwg\""
Aug 05 17:42:51.824405 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:51.824174    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"kube-root-ca.crt\""
Aug 05 17:42:51.825431 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:51.825406    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v1-kserve-self-signed-certs\""
Aug 05 17:42:51.825572 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:51.825448    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"openshift-service-ca.crt\""
Aug 05 17:42:51.835573 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:51.835548    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"]
Aug 05 17:42:51.992062 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:51.992024    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-home\") pod \"test-div-v1-kserve-fb566d667-9n4mf\" (UID: \"42bf4ac5-0faa-48ed-8dc4-2efd761d0745\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"
Aug 05 17:42:51.992232 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:51.992072    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cbmh\" (UniqueName: \"kubernetes.io/projected/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-kube-api-access-9cbmh\") pod \"test-div-v1-kserve-fb566d667-9n4mf\" (UID: \"42bf4ac5-0faa-48ed-8dc4-2efd761d0745\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"
Aug 05 17:42:51.992232 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:51.992161    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-tls-certs\") pod \"test-div-v1-kserve-fb566d667-9n4mf\" (UID: \"42bf4ac5-0faa-48ed-8dc4-2efd761d0745\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"
Aug 05 17:42:51.992232 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:51.992198    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-dshm\") pod \"test-div-v1-kserve-fb566d667-9n4mf\" (UID: \"42bf4ac5-0faa-48ed-8dc4-2efd761d0745\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"
Aug 05 17:42:51.992339 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:51.992241    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-model-cache\") pod \"test-div-v1-kserve-fb566d667-9n4mf\" (UID: \"42bf4ac5-0faa-48ed-8dc4-2efd761d0745\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"
Aug 05 17:42:51.992339 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:51.992256    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-9n4mf\" (UID: \"42bf4ac5-0faa-48ed-8dc4-2efd761d0745\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"
Aug 05 17:42:52.092767 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:52.092686    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cbmh\" (UniqueName: \"kubernetes.io/projected/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-kube-api-access-9cbmh\") pod \"test-div-v1-kserve-fb566d667-9n4mf\" (UID: \"42bf4ac5-0faa-48ed-8dc4-2efd761d0745\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"
Aug 05 17:42:52.092767 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:52.092728    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-tls-certs\") pod \"test-div-v1-kserve-fb566d667-9n4mf\" (UID: \"42bf4ac5-0faa-48ed-8dc4-2efd761d0745\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"
Aug 05 17:42:52.092767 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:52.092752    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-dshm\") pod \"test-div-v1-kserve-fb566d667-9n4mf\" (UID: \"42bf4ac5-0faa-48ed-8dc4-2efd761d0745\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"
Aug 05 17:42:52.093058 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:52.092795    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-model-cache\") pod \"test-div-v1-kserve-fb566d667-9n4mf\" (UID: \"42bf4ac5-0faa-48ed-8dc4-2efd761d0745\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"
Aug 05 17:42:52.093058 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:52.092816    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-9n4mf\" (UID: \"42bf4ac5-0faa-48ed-8dc4-2efd761d0745\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"
Aug 05 17:42:52.093058 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:52.092868    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-home\") pod \"test-div-v1-kserve-fb566d667-9n4mf\" (UID: \"42bf4ac5-0faa-48ed-8dc4-2efd761d0745\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"
Aug 05 17:42:52.093320 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:52.093290    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-model-cache\") pod \"test-div-v1-kserve-fb566d667-9n4mf\" (UID: \"42bf4ac5-0faa-48ed-8dc4-2efd761d0745\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"
Aug 05 17:42:52.093449 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:52.093317    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-home\") pod \"test-div-v1-kserve-fb566d667-9n4mf\" (UID: \"42bf4ac5-0faa-48ed-8dc4-2efd761d0745\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"
Aug 05 17:42:52.093449 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:52.093376    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-9n4mf\" (UID: \"42bf4ac5-0faa-48ed-8dc4-2efd761d0745\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"
Aug 05 17:42:52.095118 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:52.095097    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-dshm\") pod \"test-div-v1-kserve-fb566d667-9n4mf\" (UID: \"42bf4ac5-0faa-48ed-8dc4-2efd761d0745\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"
Aug 05 17:42:52.095389 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:52.095370    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-tls-certs\") pod \"test-div-v1-kserve-fb566d667-9n4mf\" (UID: \"42bf4ac5-0faa-48ed-8dc4-2efd761d0745\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"
Aug 05 17:42:52.101947 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:52.101922    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cbmh\" (UniqueName: \"kubernetes.io/projected/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-kube-api-access-9cbmh\") pod \"test-div-v1-kserve-fb566d667-9n4mf\" (UID: \"42bf4ac5-0faa-48ed-8dc4-2efd761d0745\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"
Aug 05 17:42:52.131778 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:52.131750    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"
Aug 05 17:42:52.275172 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:52.275110    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"]
Aug 05 17:42:52.277396 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:42:52.277366    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42bf4ac5_0faa_48ed_8dc4_2efd761d0745.slice/crio-312c662845d3c5f435a7fb97835d8649c8f4364c5fc242804147984ccc746d83 WatchSource:0}: Error finding container 312c662845d3c5f435a7fb97835d8649c8f4364c5fc242804147984ccc746d83: Status 404 returned error can't find the container with id 312c662845d3c5f435a7fb97835d8649c8f4364c5fc242804147984ccc746d83
Aug 05 17:42:52.431531 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:52.431491    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf" event={"ID":"42bf4ac5-0faa-48ed-8dc4-2efd761d0745","Type":"ContainerStarted","Data":"fa7a908968e3274208df6715a1371e10783ea23c69b9c062a55862fa725eb44e"}
Aug 05 17:42:52.431721 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:52.431560    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf" event={"ID":"42bf4ac5-0faa-48ed-8dc4-2efd761d0745","Type":"ContainerStarted","Data":"312c662845d3c5f435a7fb97835d8649c8f4364c5fc242804147984ccc746d83"}
Aug 05 17:42:53.132263 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:53.132230    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"
Aug 05 17:42:57.624071 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:57.624003    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf" podStartSLOduration=6.623980575 podStartE2EDuration="6.623980575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:42:52.44982266 +0000 UTC m=+1828.194456996" watchObservedRunningTime="2026-08-05 17:42:57.623980575 +0000 UTC m=+1833.368614899"
Aug 05 17:42:57.625656 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:57.625633    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"]
Aug 05 17:42:57.629730 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:57.629706    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"
Aug 05 17:42:57.633406 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:57.633382    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v2-kserve-self-signed-certs\""
Aug 05 17:42:57.640389 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:57.640366    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"]
Aug 05 17:42:57.744108 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:57.744079    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39052355-6867-474c-b998-3ce37d87ea5b-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-2lc8f\" (UID: \"39052355-6867-474c-b998-3ce37d87ea5b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"
Aug 05 17:42:57.744108 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:57.744110    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39052355-6867-474c-b998-3ce37d87ea5b-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-2lc8f\" (UID: \"39052355-6867-474c-b998-3ce37d87ea5b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"
Aug 05 17:42:57.744322 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:57.744147    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39052355-6867-474c-b998-3ce37d87ea5b-home\") pod \"test-div-v2-kserve-5f6fdb6b77-2lc8f\" (UID: \"39052355-6867-474c-b998-3ce37d87ea5b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"
Aug 05 17:42:57.744322 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:57.744181    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39052355-6867-474c-b998-3ce37d87ea5b-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-2lc8f\" (UID: \"39052355-6867-474c-b998-3ce37d87ea5b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"
Aug 05 17:42:57.744322 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:57.744211    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39052355-6867-474c-b998-3ce37d87ea5b-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-2lc8f\" (UID: \"39052355-6867-474c-b998-3ce37d87ea5b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"
Aug 05 17:42:57.744322 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:57.744229    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r96v5\" (UniqueName: \"kubernetes.io/projected/39052355-6867-474c-b998-3ce37d87ea5b-kube-api-access-r96v5\") pod \"test-div-v2-kserve-5f6fdb6b77-2lc8f\" (UID: \"39052355-6867-474c-b998-3ce37d87ea5b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"
Aug 05 17:42:57.845399 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:57.845360    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39052355-6867-474c-b998-3ce37d87ea5b-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-2lc8f\" (UID: \"39052355-6867-474c-b998-3ce37d87ea5b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"
Aug 05 17:42:57.845399 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:57.845394    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39052355-6867-474c-b998-3ce37d87ea5b-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-2lc8f\" (UID: \"39052355-6867-474c-b998-3ce37d87ea5b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"
Aug 05 17:42:57.845658 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:57.845441    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39052355-6867-474c-b998-3ce37d87ea5b-home\") pod \"test-div-v2-kserve-5f6fdb6b77-2lc8f\" (UID: \"39052355-6867-474c-b998-3ce37d87ea5b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"
Aug 05 17:42:57.845658 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:57.845469    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39052355-6867-474c-b998-3ce37d87ea5b-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-2lc8f\" (UID: \"39052355-6867-474c-b998-3ce37d87ea5b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"
Aug 05 17:42:57.845658 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:57.845493    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39052355-6867-474c-b998-3ce37d87ea5b-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-2lc8f\" (UID: \"39052355-6867-474c-b998-3ce37d87ea5b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"
Aug 05 17:42:57.845658 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:57.845581    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r96v5\" (UniqueName: \"kubernetes.io/projected/39052355-6867-474c-b998-3ce37d87ea5b-kube-api-access-r96v5\") pod \"test-div-v2-kserve-5f6fdb6b77-2lc8f\" (UID: \"39052355-6867-474c-b998-3ce37d87ea5b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"
Aug 05 17:42:57.845876 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:57.845852    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39052355-6867-474c-b998-3ce37d87ea5b-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-2lc8f\" (UID: \"39052355-6867-474c-b998-3ce37d87ea5b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"
Aug 05 17:42:57.846045 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:57.846022    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39052355-6867-474c-b998-3ce37d87ea5b-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-2lc8f\" (UID: \"39052355-6867-474c-b998-3ce37d87ea5b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"
Aug 05 17:42:57.846216 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:57.846194    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39052355-6867-474c-b998-3ce37d87ea5b-home\") pod \"test-div-v2-kserve-5f6fdb6b77-2lc8f\" (UID: \"39052355-6867-474c-b998-3ce37d87ea5b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"
Aug 05 17:42:57.848017 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:57.847997    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39052355-6867-474c-b998-3ce37d87ea5b-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-2lc8f\" (UID: \"39052355-6867-474c-b998-3ce37d87ea5b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"
Aug 05 17:42:57.848129 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:57.848021    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39052355-6867-474c-b998-3ce37d87ea5b-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-2lc8f\" (UID: \"39052355-6867-474c-b998-3ce37d87ea5b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"
Aug 05 17:42:57.854506 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:57.854485    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r96v5\" (UniqueName: \"kubernetes.io/projected/39052355-6867-474c-b998-3ce37d87ea5b-kube-api-access-r96v5\") pod \"test-div-v2-kserve-5f6fdb6b77-2lc8f\" (UID: \"39052355-6867-474c-b998-3ce37d87ea5b\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"
Aug 05 17:42:57.940877 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:57.940804    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"
Aug 05 17:42:58.087508 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:58.087480    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"]
Aug 05 17:42:58.089688 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:42:58.089655    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39052355_6867_474c_b998_3ce37d87ea5b.slice/crio-a05bcbc03c221cbb7b33067da9955f044f7e596bcb376715a61582dca03b5467 WatchSource:0}: Error finding container a05bcbc03c221cbb7b33067da9955f044f7e596bcb376715a61582dca03b5467: Status 404 returned error can't find the container with id a05bcbc03c221cbb7b33067da9955f044f7e596bcb376715a61582dca03b5467
Aug 05 17:42:58.471447 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:58.471416    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f" event={"ID":"39052355-6867-474c-b998-3ce37d87ea5b","Type":"ContainerStarted","Data":"d303baa690ab7c24ce37d3924b3344222879bfd1abd51cde435806623ca354bb"}
Aug 05 17:42:58.471661 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:58.471463    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f" event={"ID":"39052355-6867-474c-b998-3ce37d87ea5b","Type":"ContainerStarted","Data":"a05bcbc03c221cbb7b33067da9955f044f7e596bcb376715a61582dca03b5467"}
Aug 05 17:42:58.492804 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:58.492754    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f" podStartSLOduration=1.492738685 podStartE2EDuration="1.492738685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:42:58.491273583 +0000 UTC m=+1834.235907922" watchObservedRunningTime="2026-08-05 17:42:58.492738685 +0000 UTC m=+1834.237373013"
Aug 05 17:42:58.941871 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:42:58.941826    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"
Aug 05 17:43:02.132164 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:02.132127    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"
Aug 05 17:43:02.144425 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:02.144363    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"
Aug 05 17:43:02.504397 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:02.504316    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"
Aug 05 17:43:07.941760 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:07.941729    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"
Aug 05 17:43:07.958646 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:07.958621    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"
Aug 05 17:43:08.536239 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:08.536207    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"
Aug 05 17:43:20.619557 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.619526    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"]
Aug 05 17:43:20.623425 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.623403    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:43:20.626118 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.626062    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Aug 05 17:43:20.626118 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.626069    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Aug 05 17:43:20.627024 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.627004    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Aug 05 17:43:20.627134 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.627055    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Aug 05 17:43:20.627134 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.627056    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-jwjml\""
Aug 05 17:43:20.627255 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.627014    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Aug 05 17:43:20.635787 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.635767    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"]
Aug 05 17:43:20.745771 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.745737    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b75d-796f-4f66-a8dd-feb930be691d-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-68595dd9c-85979\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:43:20.745771 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.745771    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bf0b75d-796f-4f66-a8dd-feb930be691d-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-68595dd9c-85979\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:43:20.746035 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.745839    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf0b75d-796f-4f66-a8dd-feb930be691d-tls-certs\") pod \"tokenizer-migration-path-test-kserve-68595dd9c-85979\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:43:20.746035 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.745871    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b75d-796f-4f66-a8dd-feb930be691d-home\") pod \"tokenizer-migration-path-test-kserve-68595dd9c-85979\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:43:20.746035 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.745939    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b75d-796f-4f66-a8dd-feb930be691d-model-cache\") pod \"tokenizer-migration-path-test-kserve-68595dd9c-85979\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:43:20.746035 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.746013    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9t6n\" (UniqueName: \"kubernetes.io/projected/5bf0b75d-796f-4f66-a8dd-feb930be691d-kube-api-access-d9t6n\") pod \"tokenizer-migration-path-test-kserve-68595dd9c-85979\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:43:20.746211 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.746043    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b75d-796f-4f66-a8dd-feb930be691d-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-68595dd9c-85979\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:43:20.746211 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.746070    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b75d-796f-4f66-a8dd-feb930be691d-dshm\") pod \"tokenizer-migration-path-test-kserve-68595dd9c-85979\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:43:20.846691 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.846655    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf0b75d-796f-4f66-a8dd-feb930be691d-tls-certs\") pod \"tokenizer-migration-path-test-kserve-68595dd9c-85979\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:43:20.846854 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.846698    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b75d-796f-4f66-a8dd-feb930be691d-home\") pod \"tokenizer-migration-path-test-kserve-68595dd9c-85979\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:43:20.846854 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.846763    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b75d-796f-4f66-a8dd-feb930be691d-model-cache\") pod \"tokenizer-migration-path-test-kserve-68595dd9c-85979\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:43:20.846854 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.846805    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9t6n\" (UniqueName: \"kubernetes.io/projected/5bf0b75d-796f-4f66-a8dd-feb930be691d-kube-api-access-d9t6n\") pod \"tokenizer-migration-path-test-kserve-68595dd9c-85979\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:43:20.846854 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.846829    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b75d-796f-4f66-a8dd-feb930be691d-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-68595dd9c-85979\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:43:20.846854 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.846851    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b75d-796f-4f66-a8dd-feb930be691d-dshm\") pod \"tokenizer-migration-path-test-kserve-68595dd9c-85979\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:43:20.847155 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.846917    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b75d-796f-4f66-a8dd-feb930be691d-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-68595dd9c-85979\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:43:20.847155 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.846941    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bf0b75d-796f-4f66-a8dd-feb930be691d-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-68595dd9c-85979\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:43:20.847266 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.847201    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b75d-796f-4f66-a8dd-feb930be691d-home\") pod \"tokenizer-migration-path-test-kserve-68595dd9c-85979\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:43:20.847266 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.847253    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b75d-796f-4f66-a8dd-feb930be691d-model-cache\") pod \"tokenizer-migration-path-test-kserve-68595dd9c-85979\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:43:20.847423 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.847402    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b75d-796f-4f66-a8dd-feb930be691d-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-68595dd9c-85979\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:43:20.847560 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.847538    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b75d-796f-4f66-a8dd-feb930be691d-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-68595dd9c-85979\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:43:20.847678 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.847660    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bf0b75d-796f-4f66-a8dd-feb930be691d-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-68595dd9c-85979\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:43:20.850711 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.849675    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf0b75d-796f-4f66-a8dd-feb930be691d-tls-certs\") pod \"tokenizer-migration-path-test-kserve-68595dd9c-85979\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:43:20.850711 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.849481    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b75d-796f-4f66-a8dd-feb930be691d-dshm\") pod \"tokenizer-migration-path-test-kserve-68595dd9c-85979\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:43:20.861452 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.860400    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9t6n\" (UniqueName: \"kubernetes.io/projected/5bf0b75d-796f-4f66-a8dd-feb930be691d-kube-api-access-d9t6n\") pod \"tokenizer-migration-path-test-kserve-68595dd9c-85979\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:43:20.896102 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.896031    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw"]
Aug 05 17:43:20.901026 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.901002    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw"
Aug 05 17:43:20.910222 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.910193    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw"]
Aug 05 17:43:20.934414 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.934392    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:43:20.947700 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.947669    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47fa6056-d5ca-45ac-a281-14743d147bf3-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw\" (UID: \"47fa6056-d5ca-45ac-a281-14743d147bf3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw"
Aug 05 17:43:20.947802 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.947709    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47fa6056-d5ca-45ac-a281-14743d147bf3-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw\" (UID: \"47fa6056-d5ca-45ac-a281-14743d147bf3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw"
Aug 05 17:43:20.947853 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.947826    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47fa6056-d5ca-45ac-a281-14743d147bf3-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw\" (UID: \"47fa6056-d5ca-45ac-a281-14743d147bf3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw"
Aug 05 17:43:20.947895 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:20.947881    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47fa6056-d5ca-45ac-a281-14743d147bf3-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw\" (UID: \"47fa6056-d5ca-45ac-a281-14743d147bf3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw"
Aug 05 17:43:21.048580 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:21.048540    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47fa6056-d5ca-45ac-a281-14743d147bf3-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw\" (UID: \"47fa6056-d5ca-45ac-a281-14743d147bf3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw"
Aug 05 17:43:21.048758 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:21.048589    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47fa6056-d5ca-45ac-a281-14743d147bf3-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw\" (UID: \"47fa6056-d5ca-45ac-a281-14743d147bf3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw"
Aug 05 17:43:21.048826 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:21.048763    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47fa6056-d5ca-45ac-a281-14743d147bf3-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw\" (UID: \"47fa6056-d5ca-45ac-a281-14743d147bf3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw"
Aug 05 17:43:21.048826 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:21.048800    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47fa6056-d5ca-45ac-a281-14743d147bf3-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw\" (UID: \"47fa6056-d5ca-45ac-a281-14743d147bf3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw"
Aug 05 17:43:21.049013 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:21.048993    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47fa6056-d5ca-45ac-a281-14743d147bf3-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw\" (UID: \"47fa6056-d5ca-45ac-a281-14743d147bf3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw"
Aug 05 17:43:21.049171 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:21.049154    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47fa6056-d5ca-45ac-a281-14743d147bf3-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw\" (UID: \"47fa6056-d5ca-45ac-a281-14743d147bf3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw"
Aug 05 17:43:21.049338 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:21.049310    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47fa6056-d5ca-45ac-a281-14743d147bf3-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw\" (UID: \"47fa6056-d5ca-45ac-a281-14743d147bf3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw"
Aug 05 17:43:21.051317 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:21.051295    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47fa6056-d5ca-45ac-a281-14743d147bf3-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw\" (UID: \"47fa6056-d5ca-45ac-a281-14743d147bf3\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw"
Aug 05 17:43:21.071480 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:21.071450    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"]
Aug 05 17:43:21.072552 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:43:21.072531    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf0b75d_796f_4f66_a8dd_feb930be691d.slice/crio-8c35bcd3670db424497bc399d69a0f1a9b5786dec219ffa0eab0a155cd5ae942 WatchSource:0}: Error finding container 8c35bcd3670db424497bc399d69a0f1a9b5786dec219ffa0eab0a155cd5ae942: Status 404 returned error can't find the container with id 8c35bcd3670db424497bc399d69a0f1a9b5786dec219ffa0eab0a155cd5ae942
Aug 05 17:43:21.216334 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:21.216249    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw"
Aug 05 17:43:21.364707 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:21.364676    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw"]
Aug 05 17:43:21.365840 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:43:21.365811    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47fa6056_d5ca_45ac_a281_14743d147bf3.slice/crio-632300e662af17b73f90b8a6fba8106c2d2a6672574f38f614a78d1d0433183a WatchSource:0}: Error finding container 632300e662af17b73f90b8a6fba8106c2d2a6672574f38f614a78d1d0433183a: Status 404 returned error can't find the container with id 632300e662af17b73f90b8a6fba8106c2d2a6672574f38f614a78d1d0433183a
Aug 05 17:43:21.597136 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:21.597090    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw" event={"ID":"47fa6056-d5ca-45ac-a281-14743d147bf3","Type":"ContainerStarted","Data":"b6a7bf607dcf6149d522d2c9849a4103fb00907adeed81adc529298ece2857d6"}
Aug 05 17:43:21.597342 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:21.597152    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw" event={"ID":"47fa6056-d5ca-45ac-a281-14743d147bf3","Type":"ContainerStarted","Data":"632300e662af17b73f90b8a6fba8106c2d2a6672574f38f614a78d1d0433183a"}
Aug 05 17:43:21.598787 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:21.598759    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979" event={"ID":"5bf0b75d-796f-4f66-a8dd-feb930be691d","Type":"ContainerStarted","Data":"4a8b34468f5bcf494aa8c660e2421a31510fcffbab1154b741a18adbff5028d0"}
Aug 05 17:43:21.598929 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:21.598807    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979" event={"ID":"5bf0b75d-796f-4f66-a8dd-feb930be691d","Type":"ContainerStarted","Data":"8c35bcd3670db424497bc399d69a0f1a9b5786dec219ffa0eab0a155cd5ae942"}
Aug 05 17:43:24.618408 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:24.618379    2561 generic.go:356] "Generic (PLEG): container finished" podID="47fa6056-d5ca-45ac-a281-14743d147bf3" containerID="b6a7bf607dcf6149d522d2c9849a4103fb00907adeed81adc529298ece2857d6" exitCode=0
Aug 05 17:43:24.618880 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:24.618443    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw" event={"ID":"47fa6056-d5ca-45ac-a281-14743d147bf3","Type":"ContainerDied","Data":"b6a7bf607dcf6149d522d2c9849a4103fb00907adeed81adc529298ece2857d6"}
Aug 05 17:43:24.620261 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:24.620241    2561 generic.go:356] "Generic (PLEG): container finished" podID="5bf0b75d-796f-4f66-a8dd-feb930be691d" containerID="4a8b34468f5bcf494aa8c660e2421a31510fcffbab1154b741a18adbff5028d0" exitCode=0
Aug 05 17:43:24.620376 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:24.620286    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979" event={"ID":"5bf0b75d-796f-4f66-a8dd-feb930be691d","Type":"ContainerDied","Data":"4a8b34468f5bcf494aa8c660e2421a31510fcffbab1154b741a18adbff5028d0"}
Aug 05 17:43:25.627988 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:25.627937    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979" event={"ID":"5bf0b75d-796f-4f66-a8dd-feb930be691d","Type":"ContainerStarted","Data":"14a0423b36cdbb08872511050db4bf0608eec13e8510c6a410585192927924ed"}
Aug 05 17:43:25.629826 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:25.629802    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw" event={"ID":"47fa6056-d5ca-45ac-a281-14743d147bf3","Type":"ContainerStarted","Data":"e85afa2fd9220123ba843e6d5a248e1b6b1325b262c196d78c3f03615c9b8039"}
Aug 05 17:43:25.630033 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:25.630015    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw"
Aug 05 17:43:25.655377 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:25.655328    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979" podStartSLOduration=3.117276027 podStartE2EDuration="5.655312834s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.538036807s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:43:25.654531291 +0000 UTC m=+1861.399165642" watchObservedRunningTime="2026-08-05 17:43:25.655312834 +0000 UTC m=+1861.399947160"
Aug 05 17:43:25.676441 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:25.676394    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw" podStartSLOduration=3.180572614 podStartE2EDuration="5.676383456s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.495810842s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:43:25.674143973 +0000 UTC m=+1861.418778300" watchObservedRunningTime="2026-08-05 17:43:25.676383456 +0000 UTC m=+1861.421017782"
Aug 05 17:43:25.934979 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:25.934891    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:43:30.935033 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:30.934984    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:43:30.947733 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:30.947708    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:43:31.677579 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:31.677553    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:43:56.646351 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:43:56.646271    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw"
Aug 05 17:44:01.842516 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:01.842481    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw_47fa6056-d5ca-45ac-a281-14743d147bf3/main/0.log"
Aug 05 17:44:02.971759 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:02.971727    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"]
Aug 05 17:44:02.978432 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:02.978407    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw"]
Aug 05 17:44:02.978640 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:02.978617    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw" podUID="47fa6056-d5ca-45ac-a281-14743d147bf3" containerName="main" containerID="cri-o://e85afa2fd9220123ba843e6d5a248e1b6b1325b262c196d78c3f03615c9b8039" gracePeriod=30
Aug 05 17:44:02.981927 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:44:02.981899    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 05 17:44:02.981927 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:44:02.981927 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:44:02.981927 ip-10-0-135-151 kubenswrapper[2561]:  > containerID="14a0423b36cdbb08872511050db4bf0608eec13e8510c6a410585192927924ed" cmd=["/bin/sleep","15"]
Aug 05 17:44:02.982161 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:02.981947    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979" message="" err=<
Aug 05 17:44:02.982161 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:44:02.982161 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:44:02.982161 ip-10-0-135-151 kubenswrapper[2561]:  >
Aug 05 17:44:02.982161 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:44:02.981984    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 05 17:44:02.982161 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:44:02.982161 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:44:02.982161 ip-10-0-135-151 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979" podUID="5bf0b75d-796f-4f66-a8dd-feb930be691d" containerName="main" containerID="cri-o://14a0423b36cdbb08872511050db4bf0608eec13e8510c6a410585192927924ed"
Aug 05 17:44:02.982161 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:02.982023    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979" podUID="5bf0b75d-796f-4f66-a8dd-feb930be691d" containerName="main" containerID="cri-o://14a0423b36cdbb08872511050db4bf0608eec13e8510c6a410585192927924ed" gracePeriod=60
Aug 05 17:44:03.340059 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.340034    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:44:03.408704 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.408669    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b75d-796f-4f66-a8dd-feb930be691d-model-cache\") pod \"5bf0b75d-796f-4f66-a8dd-feb930be691d\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") "
Aug 05 17:44:03.408898 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.408721    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b75d-796f-4f66-a8dd-feb930be691d-tmp-dir\") pod \"5bf0b75d-796f-4f66-a8dd-feb930be691d\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") "
Aug 05 17:44:03.408898 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.408772    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bf0b75d-796f-4f66-a8dd-feb930be691d-cabundle-cert\") pod \"5bf0b75d-796f-4f66-a8dd-feb930be691d\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") "
Aug 05 17:44:03.408898 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.408808    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b75d-796f-4f66-a8dd-feb930be691d-home\") pod \"5bf0b75d-796f-4f66-a8dd-feb930be691d\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") "
Aug 05 17:44:03.408898 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.408842    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9t6n\" (UniqueName: \"kubernetes.io/projected/5bf0b75d-796f-4f66-a8dd-feb930be691d-kube-api-access-d9t6n\") pod \"5bf0b75d-796f-4f66-a8dd-feb930be691d\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") "
Aug 05 17:44:03.409150 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.408942    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf0b75d-796f-4f66-a8dd-feb930be691d-model-cache" (OuterVolumeSpecName: "model-cache") pod "5bf0b75d-796f-4f66-a8dd-feb930be691d" (UID: "5bf0b75d-796f-4f66-a8dd-feb930be691d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:44:03.409150 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.408976    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b75d-796f-4f66-a8dd-feb930be691d-dshm\") pod \"5bf0b75d-796f-4f66-a8dd-feb930be691d\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") "
Aug 05 17:44:03.409150 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.408976    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf0b75d-796f-4f66-a8dd-feb930be691d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5bf0b75d-796f-4f66-a8dd-feb930be691d" (UID: "5bf0b75d-796f-4f66-a8dd-feb930be691d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:44:03.409150 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.409035    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf0b75d-796f-4f66-a8dd-feb930be691d-tls-certs\") pod \"5bf0b75d-796f-4f66-a8dd-feb930be691d\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") "
Aug 05 17:44:03.409150 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.409049    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf0b75d-796f-4f66-a8dd-feb930be691d-home" (OuterVolumeSpecName: "home") pod "5bf0b75d-796f-4f66-a8dd-feb930be691d" (UID: "5bf0b75d-796f-4f66-a8dd-feb930be691d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:44:03.409150 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.409112    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b75d-796f-4f66-a8dd-feb930be691d-kserve-provision-location\") pod \"5bf0b75d-796f-4f66-a8dd-feb930be691d\" (UID: \"5bf0b75d-796f-4f66-a8dd-feb930be691d\") "
Aug 05 17:44:03.409441 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.409160    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bf0b75d-796f-4f66-a8dd-feb930be691d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5bf0b75d-796f-4f66-a8dd-feb930be691d" (UID: "5bf0b75d-796f-4f66-a8dd-feb930be691d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:44:03.409502 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.409445    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b75d-796f-4f66-a8dd-feb930be691d-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:44:03.409502 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.409470    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b75d-796f-4f66-a8dd-feb930be691d-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:44:03.409502 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.409483    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bf0b75d-796f-4f66-a8dd-feb930be691d-cabundle-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:44:03.409502 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.409496    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b75d-796f-4f66-a8dd-feb930be691d-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:44:03.411353 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.411321    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf0b75d-796f-4f66-a8dd-feb930be691d-dshm" (OuterVolumeSpecName: "dshm") pod "5bf0b75d-796f-4f66-a8dd-feb930be691d" (UID: "5bf0b75d-796f-4f66-a8dd-feb930be691d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:44:03.411708 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.411682    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf0b75d-796f-4f66-a8dd-feb930be691d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5bf0b75d-796f-4f66-a8dd-feb930be691d" (UID: "5bf0b75d-796f-4f66-a8dd-feb930be691d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:44:03.411708 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.411697    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf0b75d-796f-4f66-a8dd-feb930be691d-kube-api-access-d9t6n" (OuterVolumeSpecName: "kube-api-access-d9t6n") pod "5bf0b75d-796f-4f66-a8dd-feb930be691d" (UID: "5bf0b75d-796f-4f66-a8dd-feb930be691d"). InnerVolumeSpecName "kube-api-access-d9t6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:44:03.432283 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.432252    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf0b75d-796f-4f66-a8dd-feb930be691d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5bf0b75d-796f-4f66-a8dd-feb930be691d" (UID: "5bf0b75d-796f-4f66-a8dd-feb930be691d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:44:03.510511 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.510454    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b75d-796f-4f66-a8dd-feb930be691d-kserve-provision-location\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:44:03.510511 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.510480    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9t6n\" (UniqueName: \"kubernetes.io/projected/5bf0b75d-796f-4f66-a8dd-feb930be691d-kube-api-access-d9t6n\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:44:03.510511 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.510491    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf0b75d-796f-4f66-a8dd-feb930be691d-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:44:03.510511 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.510501    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf0b75d-796f-4f66-a8dd-feb930be691d-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:44:03.626813 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.626791    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw"
Aug 05 17:44:03.712495 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.712471    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47fa6056-d5ca-45ac-a281-14743d147bf3-tls-certs\") pod \"47fa6056-d5ca-45ac-a281-14743d147bf3\" (UID: \"47fa6056-d5ca-45ac-a281-14743d147bf3\") "
Aug 05 17:44:03.712628 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.712517    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47fa6056-d5ca-45ac-a281-14743d147bf3-kserve-provision-location\") pod \"47fa6056-d5ca-45ac-a281-14743d147bf3\" (UID: \"47fa6056-d5ca-45ac-a281-14743d147bf3\") "
Aug 05 17:44:03.712628 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.712553    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47fa6056-d5ca-45ac-a281-14743d147bf3-cabundle-cert\") pod \"47fa6056-d5ca-45ac-a281-14743d147bf3\" (UID: \"47fa6056-d5ca-45ac-a281-14743d147bf3\") "
Aug 05 17:44:03.712628 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.712585    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47fa6056-d5ca-45ac-a281-14743d147bf3-tmp-dir\") pod \"47fa6056-d5ca-45ac-a281-14743d147bf3\" (UID: \"47fa6056-d5ca-45ac-a281-14743d147bf3\") "
Aug 05 17:44:03.712991 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.712940    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47fa6056-d5ca-45ac-a281-14743d147bf3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "47fa6056-d5ca-45ac-a281-14743d147bf3" (UID: "47fa6056-d5ca-45ac-a281-14743d147bf3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:44:03.712991 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.712983    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47fa6056-d5ca-45ac-a281-14743d147bf3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "47fa6056-d5ca-45ac-a281-14743d147bf3" (UID: "47fa6056-d5ca-45ac-a281-14743d147bf3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:44:03.714504 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.714475    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47fa6056-d5ca-45ac-a281-14743d147bf3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "47fa6056-d5ca-45ac-a281-14743d147bf3" (UID: "47fa6056-d5ca-45ac-a281-14743d147bf3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:44:03.735071 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.735048    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47fa6056-d5ca-45ac-a281-14743d147bf3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "47fa6056-d5ca-45ac-a281-14743d147bf3" (UID: "47fa6056-d5ca-45ac-a281-14743d147bf3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:44:03.813673 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.813651    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47fa6056-d5ca-45ac-a281-14743d147bf3-kserve-provision-location\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:44:03.813673 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.813674    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47fa6056-d5ca-45ac-a281-14743d147bf3-cabundle-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:44:03.813830 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.813685    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47fa6056-d5ca-45ac-a281-14743d147bf3-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:44:03.813830 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.813693    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47fa6056-d5ca-45ac-a281-14743d147bf3-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:44:03.844949 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.844922    2561 generic.go:356] "Generic (PLEG): container finished" podID="5bf0b75d-796f-4f66-a8dd-feb930be691d" containerID="14a0423b36cdbb08872511050db4bf0608eec13e8510c6a410585192927924ed" exitCode=0
Aug 05 17:44:03.845110 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.844997    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"
Aug 05 17:44:03.845110 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.845032    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979" event={"ID":"5bf0b75d-796f-4f66-a8dd-feb930be691d","Type":"ContainerDied","Data":"14a0423b36cdbb08872511050db4bf0608eec13e8510c6a410585192927924ed"}
Aug 05 17:44:03.845110 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.845081    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979" event={"ID":"5bf0b75d-796f-4f66-a8dd-feb930be691d","Type":"ContainerDied","Data":"8c35bcd3670db424497bc399d69a0f1a9b5786dec219ffa0eab0a155cd5ae942"}
Aug 05 17:44:03.845266 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.845112    2561 scope.go:117] "RemoveContainer" containerID="14a0423b36cdbb08872511050db4bf0608eec13e8510c6a410585192927924ed"
Aug 05 17:44:03.847149 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.847122    2561 generic.go:356] "Generic (PLEG): container finished" podID="47fa6056-d5ca-45ac-a281-14743d147bf3" containerID="e85afa2fd9220123ba843e6d5a248e1b6b1325b262c196d78c3f03615c9b8039" exitCode=0
Aug 05 17:44:03.847285 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.847257    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw" event={"ID":"47fa6056-d5ca-45ac-a281-14743d147bf3","Type":"ContainerDied","Data":"e85afa2fd9220123ba843e6d5a248e1b6b1325b262c196d78c3f03615c9b8039"}
Aug 05 17:44:03.847337 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.847317    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw" event={"ID":"47fa6056-d5ca-45ac-a281-14743d147bf3","Type":"ContainerDied","Data":"632300e662af17b73f90b8a6fba8106c2d2a6672574f38f614a78d1d0433183a"}
Aug 05 17:44:03.847396 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.847368    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw"
Aug 05 17:44:03.858224 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.857346    2561 scope.go:117] "RemoveContainer" containerID="4a8b34468f5bcf494aa8c660e2421a31510fcffbab1154b741a18adbff5028d0"
Aug 05 17:44:03.866188 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.866171    2561 scope.go:117] "RemoveContainer" containerID="14a0423b36cdbb08872511050db4bf0608eec13e8510c6a410585192927924ed"
Aug 05 17:44:03.866462 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:44:03.866440    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a0423b36cdbb08872511050db4bf0608eec13e8510c6a410585192927924ed\": container with ID starting with 14a0423b36cdbb08872511050db4bf0608eec13e8510c6a410585192927924ed not found: ID does not exist" containerID="14a0423b36cdbb08872511050db4bf0608eec13e8510c6a410585192927924ed"
Aug 05 17:44:03.866557 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.866468    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a0423b36cdbb08872511050db4bf0608eec13e8510c6a410585192927924ed"} err="failed to get container status \"14a0423b36cdbb08872511050db4bf0608eec13e8510c6a410585192927924ed\": rpc error: code = NotFound desc = could not find container \"14a0423b36cdbb08872511050db4bf0608eec13e8510c6a410585192927924ed\": container with ID starting with 14a0423b36cdbb08872511050db4bf0608eec13e8510c6a410585192927924ed not found: ID does not exist"
Aug 05 17:44:03.866557 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.866486    2561 scope.go:117] "RemoveContainer" containerID="4a8b34468f5bcf494aa8c660e2421a31510fcffbab1154b741a18adbff5028d0"
Aug 05 17:44:03.866758 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:44:03.866740    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a8b34468f5bcf494aa8c660e2421a31510fcffbab1154b741a18adbff5028d0\": container with ID starting with 4a8b34468f5bcf494aa8c660e2421a31510fcffbab1154b741a18adbff5028d0 not found: ID does not exist" containerID="4a8b34468f5bcf494aa8c660e2421a31510fcffbab1154b741a18adbff5028d0"
Aug 05 17:44:03.866807 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.866764    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a8b34468f5bcf494aa8c660e2421a31510fcffbab1154b741a18adbff5028d0"} err="failed to get container status \"4a8b34468f5bcf494aa8c660e2421a31510fcffbab1154b741a18adbff5028d0\": rpc error: code = NotFound desc = could not find container \"4a8b34468f5bcf494aa8c660e2421a31510fcffbab1154b741a18adbff5028d0\": container with ID starting with 4a8b34468f5bcf494aa8c660e2421a31510fcffbab1154b741a18adbff5028d0 not found: ID does not exist"
Aug 05 17:44:03.866807 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.866780    2561 scope.go:117] "RemoveContainer" containerID="e85afa2fd9220123ba843e6d5a248e1b6b1325b262c196d78c3f03615c9b8039"
Aug 05 17:44:03.874062 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.873990    2561 scope.go:117] "RemoveContainer" containerID="b6a7bf607dcf6149d522d2c9849a4103fb00907adeed81adc529298ece2857d6"
Aug 05 17:44:03.880724 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.880705    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw"]
Aug 05 17:44:03.881419 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.881405    2561 scope.go:117] "RemoveContainer" containerID="e85afa2fd9220123ba843e6d5a248e1b6b1325b262c196d78c3f03615c9b8039"
Aug 05 17:44:03.881624 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:44:03.881605    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e85afa2fd9220123ba843e6d5a248e1b6b1325b262c196d78c3f03615c9b8039\": container with ID starting with e85afa2fd9220123ba843e6d5a248e1b6b1325b262c196d78c3f03615c9b8039 not found: ID does not exist" containerID="e85afa2fd9220123ba843e6d5a248e1b6b1325b262c196d78c3f03615c9b8039"
Aug 05 17:44:03.881676 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.881630    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e85afa2fd9220123ba843e6d5a248e1b6b1325b262c196d78c3f03615c9b8039"} err="failed to get container status \"e85afa2fd9220123ba843e6d5a248e1b6b1325b262c196d78c3f03615c9b8039\": rpc error: code = NotFound desc = could not find container \"e85afa2fd9220123ba843e6d5a248e1b6b1325b262c196d78c3f03615c9b8039\": container with ID starting with e85afa2fd9220123ba843e6d5a248e1b6b1325b262c196d78c3f03615c9b8039 not found: ID does not exist"
Aug 05 17:44:03.881676 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.881646    2561 scope.go:117] "RemoveContainer" containerID="b6a7bf607dcf6149d522d2c9849a4103fb00907adeed81adc529298ece2857d6"
Aug 05 17:44:03.881872 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:44:03.881855    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6a7bf607dcf6149d522d2c9849a4103fb00907adeed81adc529298ece2857d6\": container with ID starting with b6a7bf607dcf6149d522d2c9849a4103fb00907adeed81adc529298ece2857d6 not found: ID does not exist" containerID="b6a7bf607dcf6149d522d2c9849a4103fb00907adeed81adc529298ece2857d6"
Aug 05 17:44:03.881924 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.881880    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6a7bf607dcf6149d522d2c9849a4103fb00907adeed81adc529298ece2857d6"} err="failed to get container status \"b6a7bf607dcf6149d522d2c9849a4103fb00907adeed81adc529298ece2857d6\": rpc error: code = NotFound desc = could not find container \"b6a7bf607dcf6149d522d2c9849a4103fb00907adeed81adc529298ece2857d6\": container with ID starting with b6a7bf607dcf6149d522d2c9849a4103fb00907adeed81adc529298ece2857d6 not found: ID does not exist"
Aug 05 17:44:03.887032 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.887013    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb84d69-kzctw"]
Aug 05 17:44:03.899156 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.899135    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"]
Aug 05 17:44:03.901368 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:03.901350    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68595dd9c-85979"]
Aug 05 17:44:04.730134 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:04.730106    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47fa6056-d5ca-45ac-a281-14743d147bf3" path="/var/lib/kubelet/pods/47fa6056-d5ca-45ac-a281-14743d147bf3/volumes"
Aug 05 17:44:04.730550 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:04.730537    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bf0b75d-796f-4f66-a8dd-feb930be691d" path="/var/lib/kubelet/pods/5bf0b75d-796f-4f66-a8dd-feb930be691d/volumes"
Aug 05 17:44:05.873029 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:05.872993    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"]
Aug 05 17:44:06.222152 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:06.222068    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"]
Aug 05 17:44:20.890930 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:20.890872    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf" podUID="42bf4ac5-0faa-48ed-8dc4-2efd761d0745" containerName="main" containerID="cri-o://fa7a908968e3274208df6715a1371e10783ea23c69b9c062a55862fa725eb44e" gracePeriod=45
Aug 05 17:44:21.148896 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.148835    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"
Aug 05 17:44:21.240946 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.240905    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f" podUID="39052355-6867-474c-b998-3ce37d87ea5b" containerName="main" containerID="cri-o://d303baa690ab7c24ce37d3924b3344222879bfd1abd51cde435806623ca354bb" gracePeriod=45
Aug 05 17:44:21.248531 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.248506    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-model-cache\") pod \"42bf4ac5-0faa-48ed-8dc4-2efd761d0745\" (UID: \"42bf4ac5-0faa-48ed-8dc4-2efd761d0745\") "
Aug 05 17:44:21.248640 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.248573    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-dshm\") pod \"42bf4ac5-0faa-48ed-8dc4-2efd761d0745\" (UID: \"42bf4ac5-0faa-48ed-8dc4-2efd761d0745\") "
Aug 05 17:44:21.248640 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.248613    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-home\") pod \"42bf4ac5-0faa-48ed-8dc4-2efd761d0745\" (UID: \"42bf4ac5-0faa-48ed-8dc4-2efd761d0745\") "
Aug 05 17:44:21.248753 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.248644    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cbmh\" (UniqueName: \"kubernetes.io/projected/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-kube-api-access-9cbmh\") pod \"42bf4ac5-0faa-48ed-8dc4-2efd761d0745\" (UID: \"42bf4ac5-0faa-48ed-8dc4-2efd761d0745\") "
Aug 05 17:44:21.248753 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.248668    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-tls-certs\") pod \"42bf4ac5-0faa-48ed-8dc4-2efd761d0745\" (UID: \"42bf4ac5-0faa-48ed-8dc4-2efd761d0745\") "
Aug 05 17:44:21.248856 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.248749    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-model-cache" (OuterVolumeSpecName: "model-cache") pod "42bf4ac5-0faa-48ed-8dc4-2efd761d0745" (UID: "42bf4ac5-0faa-48ed-8dc4-2efd761d0745"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:44:21.248856 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.248763    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-tmp-dir\") pod \"42bf4ac5-0faa-48ed-8dc4-2efd761d0745\" (UID: \"42bf4ac5-0faa-48ed-8dc4-2efd761d0745\") "
Aug 05 17:44:21.248990 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.248853    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-home" (OuterVolumeSpecName: "home") pod "42bf4ac5-0faa-48ed-8dc4-2efd761d0745" (UID: "42bf4ac5-0faa-48ed-8dc4-2efd761d0745"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:44:21.249098 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.249072    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "42bf4ac5-0faa-48ed-8dc4-2efd761d0745" (UID: "42bf4ac5-0faa-48ed-8dc4-2efd761d0745"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:44:21.249098 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.249087    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:44:21.249276 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.249105    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:44:21.250825 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.250802    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "42bf4ac5-0faa-48ed-8dc4-2efd761d0745" (UID: "42bf4ac5-0faa-48ed-8dc4-2efd761d0745"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:44:21.250921 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.250874    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-dshm" (OuterVolumeSpecName: "dshm") pod "42bf4ac5-0faa-48ed-8dc4-2efd761d0745" (UID: "42bf4ac5-0faa-48ed-8dc4-2efd761d0745"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:44:21.251106 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.251088    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-kube-api-access-9cbmh" (OuterVolumeSpecName: "kube-api-access-9cbmh") pod "42bf4ac5-0faa-48ed-8dc4-2efd761d0745" (UID: "42bf4ac5-0faa-48ed-8dc4-2efd761d0745"). InnerVolumeSpecName "kube-api-access-9cbmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:44:21.349615 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.349595    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:44:21.349697 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.349617    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:44:21.349697 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.349631    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cbmh\" (UniqueName: \"kubernetes.io/projected/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-kube-api-access-9cbmh\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:44:21.349697 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.349643    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42bf4ac5-0faa-48ed-8dc4-2efd761d0745-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:44:21.473802 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.473778    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"
Aug 05 17:44:21.551803 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.551774    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39052355-6867-474c-b998-3ce37d87ea5b-tmp-dir\") pod \"39052355-6867-474c-b998-3ce37d87ea5b\" (UID: \"39052355-6867-474c-b998-3ce37d87ea5b\") "
Aug 05 17:44:21.551910 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.551836    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39052355-6867-474c-b998-3ce37d87ea5b-dshm\") pod \"39052355-6867-474c-b998-3ce37d87ea5b\" (UID: \"39052355-6867-474c-b998-3ce37d87ea5b\") "
Aug 05 17:44:21.551910 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.551874    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39052355-6867-474c-b998-3ce37d87ea5b-home\") pod \"39052355-6867-474c-b998-3ce37d87ea5b\" (UID: \"39052355-6867-474c-b998-3ce37d87ea5b\") "
Aug 05 17:44:21.551910 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.551893    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39052355-6867-474c-b998-3ce37d87ea5b-model-cache\") pod \"39052355-6867-474c-b998-3ce37d87ea5b\" (UID: \"39052355-6867-474c-b998-3ce37d87ea5b\") "
Aug 05 17:44:21.552195 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.551927    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39052355-6867-474c-b998-3ce37d87ea5b-tls-certs\") pod \"39052355-6867-474c-b998-3ce37d87ea5b\" (UID: \"39052355-6867-474c-b998-3ce37d87ea5b\") "
Aug 05 17:44:21.552195 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.551971    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r96v5\" (UniqueName: \"kubernetes.io/projected/39052355-6867-474c-b998-3ce37d87ea5b-kube-api-access-r96v5\") pod \"39052355-6867-474c-b998-3ce37d87ea5b\" (UID: \"39052355-6867-474c-b998-3ce37d87ea5b\") "
Aug 05 17:44:21.552195 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.552095    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39052355-6867-474c-b998-3ce37d87ea5b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "39052355-6867-474c-b998-3ce37d87ea5b" (UID: "39052355-6867-474c-b998-3ce37d87ea5b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:44:21.552195 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.552176    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39052355-6867-474c-b998-3ce37d87ea5b-home" (OuterVolumeSpecName: "home") pod "39052355-6867-474c-b998-3ce37d87ea5b" (UID: "39052355-6867-474c-b998-3ce37d87ea5b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:44:21.552195 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.552190    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39052355-6867-474c-b998-3ce37d87ea5b-model-cache" (OuterVolumeSpecName: "model-cache") pod "39052355-6867-474c-b998-3ce37d87ea5b" (UID: "39052355-6867-474c-b998-3ce37d87ea5b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:44:21.552411 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.552261    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39052355-6867-474c-b998-3ce37d87ea5b-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:44:21.552411 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.552279    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39052355-6867-474c-b998-3ce37d87ea5b-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:44:21.552411 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.552292    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39052355-6867-474c-b998-3ce37d87ea5b-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:44:21.554043 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.554021    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39052355-6867-474c-b998-3ce37d87ea5b-kube-api-access-r96v5" (OuterVolumeSpecName: "kube-api-access-r96v5") pod "39052355-6867-474c-b998-3ce37d87ea5b" (UID: "39052355-6867-474c-b998-3ce37d87ea5b"). InnerVolumeSpecName "kube-api-access-r96v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:44:21.554361 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.554333    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39052355-6867-474c-b998-3ce37d87ea5b-dshm" (OuterVolumeSpecName: "dshm") pod "39052355-6867-474c-b998-3ce37d87ea5b" (UID: "39052355-6867-474c-b998-3ce37d87ea5b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:44:21.554361 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.554334    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39052355-6867-474c-b998-3ce37d87ea5b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "39052355-6867-474c-b998-3ce37d87ea5b" (UID: "39052355-6867-474c-b998-3ce37d87ea5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:44:21.652995 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.652948    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39052355-6867-474c-b998-3ce37d87ea5b-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:44:21.652995 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.652993    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39052355-6867-474c-b998-3ce37d87ea5b-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:44:21.653133 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.653008    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r96v5\" (UniqueName: \"kubernetes.io/projected/39052355-6867-474c-b998-3ce37d87ea5b-kube-api-access-r96v5\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:44:21.954753 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.954722    2561 generic.go:356] "Generic (PLEG): container finished" podID="39052355-6867-474c-b998-3ce37d87ea5b" containerID="d303baa690ab7c24ce37d3924b3344222879bfd1abd51cde435806623ca354bb" exitCode=0
Aug 05 17:44:21.955156 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.954781    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"
Aug 05 17:44:21.955156 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.954811    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f" event={"ID":"39052355-6867-474c-b998-3ce37d87ea5b","Type":"ContainerDied","Data":"d303baa690ab7c24ce37d3924b3344222879bfd1abd51cde435806623ca354bb"}
Aug 05 17:44:21.955156 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.954891    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f" event={"ID":"39052355-6867-474c-b998-3ce37d87ea5b","Type":"ContainerDied","Data":"a05bcbc03c221cbb7b33067da9955f044f7e596bcb376715a61582dca03b5467"}
Aug 05 17:44:21.955156 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.954918    2561 scope.go:117] "RemoveContainer" containerID="d303baa690ab7c24ce37d3924b3344222879bfd1abd51cde435806623ca354bb"
Aug 05 17:44:21.956316 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.956295    2561 generic.go:356] "Generic (PLEG): container finished" podID="42bf4ac5-0faa-48ed-8dc4-2efd761d0745" containerID="fa7a908968e3274208df6715a1371e10783ea23c69b9c062a55862fa725eb44e" exitCode=0
Aug 05 17:44:21.956399 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.956347    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"
Aug 05 17:44:21.956399 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.956359    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf" event={"ID":"42bf4ac5-0faa-48ed-8dc4-2efd761d0745","Type":"ContainerDied","Data":"fa7a908968e3274208df6715a1371e10783ea23c69b9c062a55862fa725eb44e"}
Aug 05 17:44:21.956490 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.956397    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf" event={"ID":"42bf4ac5-0faa-48ed-8dc4-2efd761d0745","Type":"ContainerDied","Data":"312c662845d3c5f435a7fb97835d8649c8f4364c5fc242804147984ccc746d83"}
Aug 05 17:44:21.965900 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.965883    2561 scope.go:117] "RemoveContainer" containerID="d303baa690ab7c24ce37d3924b3344222879bfd1abd51cde435806623ca354bb"
Aug 05 17:44:21.966235 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:44:21.966217    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d303baa690ab7c24ce37d3924b3344222879bfd1abd51cde435806623ca354bb\": container with ID starting with d303baa690ab7c24ce37d3924b3344222879bfd1abd51cde435806623ca354bb not found: ID does not exist" containerID="d303baa690ab7c24ce37d3924b3344222879bfd1abd51cde435806623ca354bb"
Aug 05 17:44:21.966326 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.966245    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d303baa690ab7c24ce37d3924b3344222879bfd1abd51cde435806623ca354bb"} err="failed to get container status \"d303baa690ab7c24ce37d3924b3344222879bfd1abd51cde435806623ca354bb\": rpc error: code = NotFound desc = could not find container \"d303baa690ab7c24ce37d3924b3344222879bfd1abd51cde435806623ca354bb\": container with ID starting with d303baa690ab7c24ce37d3924b3344222879bfd1abd51cde435806623ca354bb not found: ID does not exist"
Aug 05 17:44:21.966326 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.966267    2561 scope.go:117] "RemoveContainer" containerID="fa7a908968e3274208df6715a1371e10783ea23c69b9c062a55862fa725eb44e"
Aug 05 17:44:21.974096 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.974039    2561 scope.go:117] "RemoveContainer" containerID="fa7a908968e3274208df6715a1371e10783ea23c69b9c062a55862fa725eb44e"
Aug 05 17:44:21.974284 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:44:21.974268    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa7a908968e3274208df6715a1371e10783ea23c69b9c062a55862fa725eb44e\": container with ID starting with fa7a908968e3274208df6715a1371e10783ea23c69b9c062a55862fa725eb44e not found: ID does not exist" containerID="fa7a908968e3274208df6715a1371e10783ea23c69b9c062a55862fa725eb44e"
Aug 05 17:44:21.974326 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.974288    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa7a908968e3274208df6715a1371e10783ea23c69b9c062a55862fa725eb44e"} err="failed to get container status \"fa7a908968e3274208df6715a1371e10783ea23c69b9c062a55862fa725eb44e\": rpc error: code = NotFound desc = could not find container \"fa7a908968e3274208df6715a1371e10783ea23c69b9c062a55862fa725eb44e\": container with ID starting with fa7a908968e3274208df6715a1371e10783ea23c69b9c062a55862fa725eb44e not found: ID does not exist"
Aug 05 17:44:21.981662 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.981642    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"]
Aug 05 17:44:21.987348 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.987329    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-2lc8f"]
Aug 05 17:44:22.000004 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:21.999983    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"]
Aug 05 17:44:22.003890 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:22.003867    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-9n4mf"]
Aug 05 17:44:22.730614 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:22.730583    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39052355-6867-474c-b998-3ce37d87ea5b" path="/var/lib/kubelet/pods/39052355-6867-474c-b998-3ce37d87ea5b/volumes"
Aug 05 17:44:22.730971 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:22.730945    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42bf4ac5-0faa-48ed-8dc4-2efd761d0745" path="/var/lib/kubelet/pods/42bf4ac5-0faa-48ed-8dc4-2efd761d0745/volumes"
Aug 05 17:44:51.724847 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.724817    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"]
Aug 05 17:44:51.725290 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.725163    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf0b75d-796f-4f66-a8dd-feb930be691d" containerName="storage-initializer"
Aug 05 17:44:51.725290 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.725175    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf0b75d-796f-4f66-a8dd-feb930be691d" containerName="storage-initializer"
Aug 05 17:44:51.725290 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.725186    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39052355-6867-474c-b998-3ce37d87ea5b" containerName="main"
Aug 05 17:44:51.725290 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.725192    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="39052355-6867-474c-b998-3ce37d87ea5b" containerName="main"
Aug 05 17:44:51.725290 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.725199    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47fa6056-d5ca-45ac-a281-14743d147bf3" containerName="storage-initializer"
Aug 05 17:44:51.725290 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.725206    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="47fa6056-d5ca-45ac-a281-14743d147bf3" containerName="storage-initializer"
Aug 05 17:44:51.725290 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.725224    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bf0b75d-796f-4f66-a8dd-feb930be691d" containerName="main"
Aug 05 17:44:51.725290 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.725229    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bf0b75d-796f-4f66-a8dd-feb930be691d" containerName="main"
Aug 05 17:44:51.725290 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.725237    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47fa6056-d5ca-45ac-a281-14743d147bf3" containerName="main"
Aug 05 17:44:51.725290 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.725241    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="47fa6056-d5ca-45ac-a281-14743d147bf3" containerName="main"
Aug 05 17:44:51.725290 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.725248    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42bf4ac5-0faa-48ed-8dc4-2efd761d0745" containerName="main"
Aug 05 17:44:51.725290 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.725253    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="42bf4ac5-0faa-48ed-8dc4-2efd761d0745" containerName="main"
Aug 05 17:44:51.725807 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.725300    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bf0b75d-796f-4f66-a8dd-feb930be691d" containerName="main"
Aug 05 17:44:51.725807 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.725309    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="39052355-6867-474c-b998-3ce37d87ea5b" containerName="main"
Aug 05 17:44:51.725807 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.725316    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="47fa6056-d5ca-45ac-a281-14743d147bf3" containerName="main"
Aug 05 17:44:51.725807 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.725321    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="42bf4ac5-0faa-48ed-8dc4-2efd761d0745" containerName="main"
Aug 05 17:44:51.729755 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.729739    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"
Aug 05 17:44:51.732918 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.732897    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\""
Aug 05 17:44:51.734140 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.734120    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\""
Aug 05 17:44:51.734253 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.734181    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-8rtd9\""
Aug 05 17:44:51.734253 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.734122    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v1-kserve-self-signed-certs\""
Aug 05 17:44:51.740917 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.740896    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"]
Aug 05 17:44:51.884054 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.884028    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e33b507e-e986-48f1-96b8-1f09a32c0531-tls-certs\") pod \"stop-v1-kserve-5657b88c95-f2tbv\" (UID: \"e33b507e-e986-48f1-96b8-1f09a32c0531\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"
Aug 05 17:44:51.884054 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.884057    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e33b507e-e986-48f1-96b8-1f09a32c0531-home\") pod \"stop-v1-kserve-5657b88c95-f2tbv\" (UID: \"e33b507e-e986-48f1-96b8-1f09a32c0531\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"
Aug 05 17:44:51.884261 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.884084    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dwh8\" (UniqueName: \"kubernetes.io/projected/e33b507e-e986-48f1-96b8-1f09a32c0531-kube-api-access-5dwh8\") pod \"stop-v1-kserve-5657b88c95-f2tbv\" (UID: \"e33b507e-e986-48f1-96b8-1f09a32c0531\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"
Aug 05 17:44:51.884261 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.884169    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e33b507e-e986-48f1-96b8-1f09a32c0531-dshm\") pod \"stop-v1-kserve-5657b88c95-f2tbv\" (UID: \"e33b507e-e986-48f1-96b8-1f09a32c0531\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"
Aug 05 17:44:51.884343 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.884284    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e33b507e-e986-48f1-96b8-1f09a32c0531-model-cache\") pod \"stop-v1-kserve-5657b88c95-f2tbv\" (UID: \"e33b507e-e986-48f1-96b8-1f09a32c0531\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"
Aug 05 17:44:51.884389 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.884360    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e33b507e-e986-48f1-96b8-1f09a32c0531-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-f2tbv\" (UID: \"e33b507e-e986-48f1-96b8-1f09a32c0531\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"
Aug 05 17:44:51.985586 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.985505    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e33b507e-e986-48f1-96b8-1f09a32c0531-dshm\") pod \"stop-v1-kserve-5657b88c95-f2tbv\" (UID: \"e33b507e-e986-48f1-96b8-1f09a32c0531\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"
Aug 05 17:44:51.985586 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.985543    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e33b507e-e986-48f1-96b8-1f09a32c0531-model-cache\") pod \"stop-v1-kserve-5657b88c95-f2tbv\" (UID: \"e33b507e-e986-48f1-96b8-1f09a32c0531\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"
Aug 05 17:44:51.985792 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.985674    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e33b507e-e986-48f1-96b8-1f09a32c0531-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-f2tbv\" (UID: \"e33b507e-e986-48f1-96b8-1f09a32c0531\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"
Aug 05 17:44:51.985792 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.985762    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e33b507e-e986-48f1-96b8-1f09a32c0531-tls-certs\") pod \"stop-v1-kserve-5657b88c95-f2tbv\" (UID: \"e33b507e-e986-48f1-96b8-1f09a32c0531\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"
Aug 05 17:44:51.985792 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.985788    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e33b507e-e986-48f1-96b8-1f09a32c0531-home\") pod \"stop-v1-kserve-5657b88c95-f2tbv\" (UID: \"e33b507e-e986-48f1-96b8-1f09a32c0531\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"
Aug 05 17:44:51.985973 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.985834    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dwh8\" (UniqueName: \"kubernetes.io/projected/e33b507e-e986-48f1-96b8-1f09a32c0531-kube-api-access-5dwh8\") pod \"stop-v1-kserve-5657b88c95-f2tbv\" (UID: \"e33b507e-e986-48f1-96b8-1f09a32c0531\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"
Aug 05 17:44:51.986073 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.986051    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e33b507e-e986-48f1-96b8-1f09a32c0531-model-cache\") pod \"stop-v1-kserve-5657b88c95-f2tbv\" (UID: \"e33b507e-e986-48f1-96b8-1f09a32c0531\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"
Aug 05 17:44:51.986129 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.986094    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e33b507e-e986-48f1-96b8-1f09a32c0531-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-f2tbv\" (UID: \"e33b507e-e986-48f1-96b8-1f09a32c0531\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"
Aug 05 17:44:51.986209 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.986189    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e33b507e-e986-48f1-96b8-1f09a32c0531-home\") pod \"stop-v1-kserve-5657b88c95-f2tbv\" (UID: \"e33b507e-e986-48f1-96b8-1f09a32c0531\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"
Aug 05 17:44:51.987732 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.987705    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e33b507e-e986-48f1-96b8-1f09a32c0531-dshm\") pod \"stop-v1-kserve-5657b88c95-f2tbv\" (UID: \"e33b507e-e986-48f1-96b8-1f09a32c0531\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"
Aug 05 17:44:51.988187 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.988171    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e33b507e-e986-48f1-96b8-1f09a32c0531-tls-certs\") pod \"stop-v1-kserve-5657b88c95-f2tbv\" (UID: \"e33b507e-e986-48f1-96b8-1f09a32c0531\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"
Aug 05 17:44:51.996017 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:51.995998    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dwh8\" (UniqueName: \"kubernetes.io/projected/e33b507e-e986-48f1-96b8-1f09a32c0531-kube-api-access-5dwh8\") pod \"stop-v1-kserve-5657b88c95-f2tbv\" (UID: \"e33b507e-e986-48f1-96b8-1f09a32c0531\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"
Aug 05 17:44:52.039631 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:52.039609    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"
Aug 05 17:44:52.205254 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:52.205231    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"]
Aug 05 17:44:52.207400 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:44:52.207369    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode33b507e_e986_48f1_96b8_1f09a32c0531.slice/crio-cea1ddb308fbfa94ccf8f5cd16e149f4b0f1eb959e584ff798549927ee4106e2 WatchSource:0}: Error finding container cea1ddb308fbfa94ccf8f5cd16e149f4b0f1eb959e584ff798549927ee4106e2: Status 404 returned error can't find the container with id cea1ddb308fbfa94ccf8f5cd16e149f4b0f1eb959e584ff798549927ee4106e2
Aug 05 17:44:53.119278 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:53.119240    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv" event={"ID":"e33b507e-e986-48f1-96b8-1f09a32c0531","Type":"ContainerStarted","Data":"950d57482262e6cb92b0c81d9f025f96f6ea2c194981f4ff7b7f91e373bcc02b"}
Aug 05 17:44:53.119747 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:53.119302    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv" event={"ID":"e33b507e-e986-48f1-96b8-1f09a32c0531","Type":"ContainerStarted","Data":"cea1ddb308fbfa94ccf8f5cd16e149f4b0f1eb959e584ff798549927ee4106e2"}
Aug 05 17:44:53.171609 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:53.171545    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv" podStartSLOduration=2.171523633 podStartE2EDuration="2.171523633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:44:53.170642534 +0000 UTC m=+1948.915276863" watchObservedRunningTime="2026-08-05 17:44:53.171523633 +0000 UTC m=+1948.916157962"
Aug 05 17:44:54.040188 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:44:54.040158    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"
Aug 05 17:45:02.040594 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:02.040561    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"
Aug 05 17:45:02.056868 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:02.056845    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"
Aug 05 17:45:02.176868 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:02.176800    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"
Aug 05 17:45:04.635456 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:04.635425    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"]
Aug 05 17:45:04.639073 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:04.639049    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"
Aug 05 17:45:04.642353 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:04.642335    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v2-kserve-self-signed-certs\""
Aug 05 17:45:04.659352 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:04.659330    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"]
Aug 05 17:45:04.689442 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:04.689418    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da118eaa-599e-4803-9292-911301ccfebb-dshm\") pod \"stop-v2-kserve-7f68747c45-gqbxr\" (UID: \"da118eaa-599e-4803-9292-911301ccfebb\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"
Aug 05 17:45:04.689554 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:04.689451    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da118eaa-599e-4803-9292-911301ccfebb-model-cache\") pod \"stop-v2-kserve-7f68747c45-gqbxr\" (UID: \"da118eaa-599e-4803-9292-911301ccfebb\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"
Aug 05 17:45:04.689554 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:04.689487    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da118eaa-599e-4803-9292-911301ccfebb-home\") pod \"stop-v2-kserve-7f68747c45-gqbxr\" (UID: \"da118eaa-599e-4803-9292-911301ccfebb\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"
Aug 05 17:45:04.689554 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:04.689513    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da118eaa-599e-4803-9292-911301ccfebb-tls-certs\") pod \"stop-v2-kserve-7f68747c45-gqbxr\" (UID: \"da118eaa-599e-4803-9292-911301ccfebb\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"
Aug 05 17:45:04.689733 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:04.689601    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4czpd\" (UniqueName: \"kubernetes.io/projected/da118eaa-599e-4803-9292-911301ccfebb-kube-api-access-4czpd\") pod \"stop-v2-kserve-7f68747c45-gqbxr\" (UID: \"da118eaa-599e-4803-9292-911301ccfebb\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"
Aug 05 17:45:04.689733 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:04.689663    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da118eaa-599e-4803-9292-911301ccfebb-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-gqbxr\" (UID: \"da118eaa-599e-4803-9292-911301ccfebb\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"
Aug 05 17:45:04.790496 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:04.790473    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4czpd\" (UniqueName: \"kubernetes.io/projected/da118eaa-599e-4803-9292-911301ccfebb-kube-api-access-4czpd\") pod \"stop-v2-kserve-7f68747c45-gqbxr\" (UID: \"da118eaa-599e-4803-9292-911301ccfebb\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"
Aug 05 17:45:04.790616 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:04.790507    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da118eaa-599e-4803-9292-911301ccfebb-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-gqbxr\" (UID: \"da118eaa-599e-4803-9292-911301ccfebb\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"
Aug 05 17:45:04.790616 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:04.790552    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da118eaa-599e-4803-9292-911301ccfebb-dshm\") pod \"stop-v2-kserve-7f68747c45-gqbxr\" (UID: \"da118eaa-599e-4803-9292-911301ccfebb\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"
Aug 05 17:45:04.790616 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:04.790576    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da118eaa-599e-4803-9292-911301ccfebb-model-cache\") pod \"stop-v2-kserve-7f68747c45-gqbxr\" (UID: \"da118eaa-599e-4803-9292-911301ccfebb\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"
Aug 05 17:45:04.790616 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:04.790611    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da118eaa-599e-4803-9292-911301ccfebb-home\") pod \"stop-v2-kserve-7f68747c45-gqbxr\" (UID: \"da118eaa-599e-4803-9292-911301ccfebb\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"
Aug 05 17:45:04.790819 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:04.790632    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da118eaa-599e-4803-9292-911301ccfebb-tls-certs\") pod \"stop-v2-kserve-7f68747c45-gqbxr\" (UID: \"da118eaa-599e-4803-9292-911301ccfebb\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"
Aug 05 17:45:04.790932 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:04.790908    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da118eaa-599e-4803-9292-911301ccfebb-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-gqbxr\" (UID: \"da118eaa-599e-4803-9292-911301ccfebb\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"
Aug 05 17:45:04.791147 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:04.791121    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da118eaa-599e-4803-9292-911301ccfebb-model-cache\") pod \"stop-v2-kserve-7f68747c45-gqbxr\" (UID: \"da118eaa-599e-4803-9292-911301ccfebb\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"
Aug 05 17:45:04.791239 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:04.791158    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da118eaa-599e-4803-9292-911301ccfebb-home\") pod \"stop-v2-kserve-7f68747c45-gqbxr\" (UID: \"da118eaa-599e-4803-9292-911301ccfebb\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"
Aug 05 17:45:04.792723 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:04.792705    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da118eaa-599e-4803-9292-911301ccfebb-dshm\") pod \"stop-v2-kserve-7f68747c45-gqbxr\" (UID: \"da118eaa-599e-4803-9292-911301ccfebb\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"
Aug 05 17:45:04.793036 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:04.793017    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da118eaa-599e-4803-9292-911301ccfebb-tls-certs\") pod \"stop-v2-kserve-7f68747c45-gqbxr\" (UID: \"da118eaa-599e-4803-9292-911301ccfebb\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"
Aug 05 17:45:04.800611 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:04.800578    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4czpd\" (UniqueName: \"kubernetes.io/projected/da118eaa-599e-4803-9292-911301ccfebb-kube-api-access-4czpd\") pod \"stop-v2-kserve-7f68747c45-gqbxr\" (UID: \"da118eaa-599e-4803-9292-911301ccfebb\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"
Aug 05 17:45:04.950097 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:04.950035    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"
Aug 05 17:45:05.095692 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:05.095646    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"]
Aug 05 17:45:05.097282 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:45:05.097254    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda118eaa_599e_4803_9292_911301ccfebb.slice/crio-96f48ea6ad8452c5c73b4e21c2789a83ad01c875b575b4a2483bb14dd9ed79e4 WatchSource:0}: Error finding container 96f48ea6ad8452c5c73b4e21c2789a83ad01c875b575b4a2483bb14dd9ed79e4: Status 404 returned error can't find the container with id 96f48ea6ad8452c5c73b4e21c2789a83ad01c875b575b4a2483bb14dd9ed79e4
Aug 05 17:45:05.184821 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:05.184793    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr" event={"ID":"da118eaa-599e-4803-9292-911301ccfebb","Type":"ContainerStarted","Data":"6ffe19a078d98490f56a0490940a131674cda5eff899eeab3413ebffcee0345e"}
Aug 05 17:45:05.184969 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:05.184849    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr" event={"ID":"da118eaa-599e-4803-9292-911301ccfebb","Type":"ContainerStarted","Data":"96f48ea6ad8452c5c73b4e21c2789a83ad01c875b575b4a2483bb14dd9ed79e4"}
Aug 05 17:45:05.218446 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:05.218358    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr" podStartSLOduration=1.218339814 podStartE2EDuration="1.218339814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:45:05.216251036 +0000 UTC m=+1960.960885361" watchObservedRunningTime="2026-08-05 17:45:05.218339814 +0000 UTC m=+1960.962974141"
Aug 05 17:45:05.950351 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:05.950316    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"
Aug 05 17:45:14.950641 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:14.950609    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"
Aug 05 17:45:14.962908 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:14.962883    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"
Aug 05 17:45:15.254477 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:15.254399    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"
Aug 05 17:45:17.085080 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.085008    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"]
Aug 05 17:45:17.088785 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.088765    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:17.092612 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.092593    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Aug 05 17:45:17.093348 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.093332    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-2nw7j\""
Aug 05 17:45:17.094503 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.094483    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Aug 05 17:45:17.094503 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.094496    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Aug 05 17:45:17.094661 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.094502    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Aug 05 17:45:17.094776 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.094762    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Aug 05 17:45:17.105899 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.105880    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"]
Aug 05 17:45:17.188685 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.188660    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:17.188782 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.188695    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-home\") pod \"tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:17.188782 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.188729    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:17.188782 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.188766    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-dshm\") pod \"tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:17.188782 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.188780    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:17.188935 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.188800    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-tls-certs\") pod \"tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:17.188935 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.188838    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlmgz\" (UniqueName: \"kubernetes.io/projected/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-kube-api-access-qlmgz\") pod \"tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:17.188935 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.188876    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-model-cache\") pod \"tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:17.289462 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.289439    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-model-cache\") pod \"tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:17.289598 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.289477    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:17.289598 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.289500    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-home\") pod \"tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:17.289598 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.289523    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:17.289598 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.289542    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-dshm\") pod \"tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:17.289598 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.289561    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:17.289836 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.289705    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-tls-certs\") pod \"tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:17.289836 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.289773    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlmgz\" (UniqueName: \"kubernetes.io/projected/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-kube-api-access-qlmgz\") pod \"tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:17.289980 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.289931    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-home\") pod \"tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:17.290066 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.290001    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:17.290128 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.290093    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:17.290303 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.290283    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:17.290348 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.290329    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-model-cache\") pod \"tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:17.291760 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.291732    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-dshm\") pod \"tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:17.292060 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.292041    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-tls-certs\") pod \"tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:17.303620 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.303601    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlmgz\" (UniqueName: \"kubernetes.io/projected/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-kube-api-access-qlmgz\") pod \"tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:17.401529 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.401461    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:17.451120 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.451088    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9"]
Aug 05 17:45:17.454758 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.454740    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9"
Aug 05 17:45:17.479649 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.479614    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9"]
Aug 05 17:45:17.491678 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.491640    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a7808af-ef24-48f4-85a8-cd688b5f277b-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9\" (UID: \"0a7808af-ef24-48f4-85a8-cd688b5f277b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9"
Aug 05 17:45:17.491830 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.491699    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a7808af-ef24-48f4-85a8-cd688b5f277b-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9\" (UID: \"0a7808af-ef24-48f4-85a8-cd688b5f277b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9"
Aug 05 17:45:17.491830 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.491750    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a7808af-ef24-48f4-85a8-cd688b5f277b-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9\" (UID: \"0a7808af-ef24-48f4-85a8-cd688b5f277b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9"
Aug 05 17:45:17.491830 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.491779    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a7808af-ef24-48f4-85a8-cd688b5f277b-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9\" (UID: \"0a7808af-ef24-48f4-85a8-cd688b5f277b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9"
Aug 05 17:45:17.565065 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.565040    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"]
Aug 05 17:45:17.566336 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:45:17.566312    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a4cf0d_20a1_44d8_9652_abb6fc2ae7d5.slice/crio-323eb0113e2179d4595a7bd5e6189e202789ba2295a2dd9d50acc8c596f42acf WatchSource:0}: Error finding container 323eb0113e2179d4595a7bd5e6189e202789ba2295a2dd9d50acc8c596f42acf: Status 404 returned error can't find the container with id 323eb0113e2179d4595a7bd5e6189e202789ba2295a2dd9d50acc8c596f42acf
Aug 05 17:45:17.592293 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.592267    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a7808af-ef24-48f4-85a8-cd688b5f277b-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9\" (UID: \"0a7808af-ef24-48f4-85a8-cd688b5f277b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9"
Aug 05 17:45:17.592384 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.592311    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a7808af-ef24-48f4-85a8-cd688b5f277b-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9\" (UID: \"0a7808af-ef24-48f4-85a8-cd688b5f277b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9"
Aug 05 17:45:17.592384 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.592353    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a7808af-ef24-48f4-85a8-cd688b5f277b-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9\" (UID: \"0a7808af-ef24-48f4-85a8-cd688b5f277b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9"
Aug 05 17:45:17.592459 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.592383    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a7808af-ef24-48f4-85a8-cd688b5f277b-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9\" (UID: \"0a7808af-ef24-48f4-85a8-cd688b5f277b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9"
Aug 05 17:45:17.592735 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.592713    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a7808af-ef24-48f4-85a8-cd688b5f277b-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9\" (UID: \"0a7808af-ef24-48f4-85a8-cd688b5f277b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9"
Aug 05 17:45:17.592781 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.592760    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a7808af-ef24-48f4-85a8-cd688b5f277b-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9\" (UID: \"0a7808af-ef24-48f4-85a8-cd688b5f277b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9"
Aug 05 17:45:17.593005 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.592984    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a7808af-ef24-48f4-85a8-cd688b5f277b-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9\" (UID: \"0a7808af-ef24-48f4-85a8-cd688b5f277b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9"
Aug 05 17:45:17.594772 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.594754    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a7808af-ef24-48f4-85a8-cd688b5f277b-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9\" (UID: \"0a7808af-ef24-48f4-85a8-cd688b5f277b\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9"
Aug 05 17:45:17.772519 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.772442    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9"
Aug 05 17:45:17.910831 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:17.910801    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9"]
Aug 05 17:45:17.911766 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:45:17.911739    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a7808af_ef24_48f4_85a8_cd688b5f277b.slice/crio-74d330ec2cdcc83a29d7a20974f96c139d89dd21f14e6095d8ffef977afb1fc5 WatchSource:0}: Error finding container 74d330ec2cdcc83a29d7a20974f96c139d89dd21f14e6095d8ffef977afb1fc5: Status 404 returned error can't find the container with id 74d330ec2cdcc83a29d7a20974f96c139d89dd21f14e6095d8ffef977afb1fc5
Aug 05 17:45:18.265456 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:18.265413    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9" event={"ID":"0a7808af-ef24-48f4-85a8-cd688b5f277b","Type":"ContainerStarted","Data":"2359f00ff4b77090f717c71094cd0ad7e56328357e11f5f687eaf6de1f9039cc"}
Aug 05 17:45:18.265976 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:18.265475    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9" event={"ID":"0a7808af-ef24-48f4-85a8-cd688b5f277b","Type":"ContainerStarted","Data":"74d330ec2cdcc83a29d7a20974f96c139d89dd21f14e6095d8ffef977afb1fc5"}
Aug 05 17:45:18.267702 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:18.267588    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn" event={"ID":"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5","Type":"ContainerStarted","Data":"70755e7c4d674678848b3b861491f6428202f2e14f66c8664b991cc788362067"}
Aug 05 17:45:18.267702 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:18.267644    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn" event={"ID":"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5","Type":"ContainerStarted","Data":"323eb0113e2179d4595a7bd5e6189e202789ba2295a2dd9d50acc8c596f42acf"}
Aug 05 17:45:20.294393 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:20.294362    2561 generic.go:356] "Generic (PLEG): container finished" podID="48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5" containerID="70755e7c4d674678848b3b861491f6428202f2e14f66c8664b991cc788362067" exitCode=0
Aug 05 17:45:20.294763 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:20.294439    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn" event={"ID":"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5","Type":"ContainerDied","Data":"70755e7c4d674678848b3b861491f6428202f2e14f66c8664b991cc788362067"}
Aug 05 17:45:21.301406 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:21.301375    2561 generic.go:356] "Generic (PLEG): container finished" podID="0a7808af-ef24-48f4-85a8-cd688b5f277b" containerID="2359f00ff4b77090f717c71094cd0ad7e56328357e11f5f687eaf6de1f9039cc" exitCode=0
Aug 05 17:45:21.301837 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:21.301451    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9" event={"ID":"0a7808af-ef24-48f4-85a8-cd688b5f277b","Type":"ContainerDied","Data":"2359f00ff4b77090f717c71094cd0ad7e56328357e11f5f687eaf6de1f9039cc"}
Aug 05 17:45:21.303378 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:21.303357    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn" event={"ID":"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5","Type":"ContainerStarted","Data":"47e1b1b76147975301decbb72b58419e103e1c0b7ddfa09c0fbb2f41d9cb4b77"}
Aug 05 17:45:21.361617 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:21.361571    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn" podStartSLOduration=1.864094557 podStartE2EDuration="4.361555213s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.497460656s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:45:21.358748001 +0000 UTC m=+1977.103382326" watchObservedRunningTime="2026-08-05 17:45:21.361555213 +0000 UTC m=+1977.106189539"
Aug 05 17:45:21.402261 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:21.402230    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:22.314545 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:22.314507    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9" event={"ID":"0a7808af-ef24-48f4-85a8-cd688b5f277b","Type":"ContainerStarted","Data":"949fc7a98e7d86590f4966d95144ebde8501c138370e4efb685b4016adc58777"}
Aug 05 17:45:22.315108 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:22.315050    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9"
Aug 05 17:45:22.341900 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:22.341850    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9" podStartSLOduration=2.84755574 podStartE2EDuration="5.341837327s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.494281587s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:45:22.341716316 +0000 UTC m=+1978.086350661" watchObservedRunningTime="2026-08-05 17:45:22.341837327 +0000 UTC m=+1978.086471668"
Aug 05 17:45:27.401996 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:27.401940    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:27.414716 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:27.414696    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:28.368323 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:28.368285    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:53.332891 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:53.332860    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9"
Aug 05 17:45:56.320589 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:56.320559    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9_0a7808af-ef24-48f4-85a8-cd688b5f277b/main/0.log"
Aug 05 17:45:56.772096 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:56.772020    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"]
Aug 05 17:45:56.781786 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:45:56.781748    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 05 17:45:56.781786 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:45:56.781786 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:45:56.781786 ip-10-0-135-151 kubenswrapper[2561]:  > containerID="47e1b1b76147975301decbb72b58419e103e1c0b7ddfa09c0fbb2f41d9cb4b77" cmd=["/bin/sleep","15"]
Aug 05 17:45:56.782202 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:56.781811    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn" message="" err=<
Aug 05 17:45:56.782202 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:45:56.782202 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:45:56.782202 ip-10-0-135-151 kubenswrapper[2561]:  >
Aug 05 17:45:56.782202 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:45:56.781829    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 05 17:45:56.782202 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:45:56.782202 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:45:56.782202 ip-10-0-135-151 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn" podUID="48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5" containerName="main" containerID="cri-o://47e1b1b76147975301decbb72b58419e103e1c0b7ddfa09c0fbb2f41d9cb4b77"
Aug 05 17:45:56.782202 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:56.781882    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn" podUID="48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5" containerName="main" containerID="cri-o://47e1b1b76147975301decbb72b58419e103e1c0b7ddfa09c0fbb2f41d9cb4b77" gracePeriod=60
Aug 05 17:45:56.783734 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:56.783712    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9"]
Aug 05 17:45:56.783937 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:56.783916    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9" podUID="0a7808af-ef24-48f4-85a8-cd688b5f277b" containerName="main" containerID="cri-o://949fc7a98e7d86590f4966d95144ebde8501c138370e4efb685b4016adc58777" gracePeriod=30
Aug 05 17:45:57.145623 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.145600    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:57.334002 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.333923    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-tmp-dir\") pod \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") "
Aug 05 17:45:57.334399 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.334002    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-dshm\") pod \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") "
Aug 05 17:45:57.334399 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.334032    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-kserve-provision-location\") pod \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") "
Aug 05 17:45:57.334399 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.334050    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-model-cache\") pod \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") "
Aug 05 17:45:57.334399 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.334101    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-home\") pod \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") "
Aug 05 17:45:57.334399 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.334158    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlmgz\" (UniqueName: \"kubernetes.io/projected/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-kube-api-access-qlmgz\") pod \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") "
Aug 05 17:45:57.334399 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.334187    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-cabundle-cert\") pod \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") "
Aug 05 17:45:57.334399 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.334223    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-tls-certs\") pod \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\" (UID: \"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5\") "
Aug 05 17:45:57.334399 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.334264    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5" (UID: "48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:45:57.334399 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.334380    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-model-cache" (OuterVolumeSpecName: "model-cache") pod "48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5" (UID: "48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:45:57.334897 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.334396    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-home" (OuterVolumeSpecName: "home") pod "48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5" (UID: "48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:45:57.334897 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.334607    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:45:57.334897 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.334629    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:45:57.334897 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.334642    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:45:57.334897 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.334718    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5" (UID: "48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:45:57.336361 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.336339    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5" (UID: "48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:45:57.336476 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.336436    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-dshm" (OuterVolumeSpecName: "dshm") pod "48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5" (UID: "48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:45:57.336753 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.336727    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-kube-api-access-qlmgz" (OuterVolumeSpecName: "kube-api-access-qlmgz") pod "48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5" (UID: "48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5"). InnerVolumeSpecName "kube-api-access-qlmgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:45:57.361675 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.361643    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5" (UID: "48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:45:57.424724 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.424696    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9"
Aug 05 17:45:57.435758 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.435732    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:45:57.435758 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.435754    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-kserve-provision-location\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:45:57.435995 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.435765    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlmgz\" (UniqueName: \"kubernetes.io/projected/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-kube-api-access-qlmgz\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:45:57.435995 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.435775    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-cabundle-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:45:57.435995 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.435783    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:45:57.524825 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.524790    2561 generic.go:356] "Generic (PLEG): container finished" podID="0a7808af-ef24-48f4-85a8-cd688b5f277b" containerID="949fc7a98e7d86590f4966d95144ebde8501c138370e4efb685b4016adc58777" exitCode=0
Aug 05 17:45:57.525047 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.524851    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9"
Aug 05 17:45:57.525047 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.524881    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9" event={"ID":"0a7808af-ef24-48f4-85a8-cd688b5f277b","Type":"ContainerDied","Data":"949fc7a98e7d86590f4966d95144ebde8501c138370e4efb685b4016adc58777"}
Aug 05 17:45:57.525047 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.524932    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9" event={"ID":"0a7808af-ef24-48f4-85a8-cd688b5f277b","Type":"ContainerDied","Data":"74d330ec2cdcc83a29d7a20974f96c139d89dd21f14e6095d8ffef977afb1fc5"}
Aug 05 17:45:57.525047 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.524978    2561 scope.go:117] "RemoveContainer" containerID="949fc7a98e7d86590f4966d95144ebde8501c138370e4efb685b4016adc58777"
Aug 05 17:45:57.527072 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.527046    2561 generic.go:356] "Generic (PLEG): container finished" podID="48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5" containerID="47e1b1b76147975301decbb72b58419e103e1c0b7ddfa09c0fbb2f41d9cb4b77" exitCode=0
Aug 05 17:45:57.527206 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.527079    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn" event={"ID":"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5","Type":"ContainerDied","Data":"47e1b1b76147975301decbb72b58419e103e1c0b7ddfa09c0fbb2f41d9cb4b77"}
Aug 05 17:45:57.527206 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.527114    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"
Aug 05 17:45:57.527206 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.527141    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn" event={"ID":"48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5","Type":"ContainerDied","Data":"323eb0113e2179d4595a7bd5e6189e202789ba2295a2dd9d50acc8c596f42acf"}
Aug 05 17:45:57.535318 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.535302    2561 scope.go:117] "RemoveContainer" containerID="2359f00ff4b77090f717c71094cd0ad7e56328357e11f5f687eaf6de1f9039cc"
Aug 05 17:45:57.536035 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.536012    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a7808af-ef24-48f4-85a8-cd688b5f277b-cabundle-cert\") pod \"0a7808af-ef24-48f4-85a8-cd688b5f277b\" (UID: \"0a7808af-ef24-48f4-85a8-cd688b5f277b\") "
Aug 05 17:45:57.536145 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.536098    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a7808af-ef24-48f4-85a8-cd688b5f277b-kserve-provision-location\") pod \"0a7808af-ef24-48f4-85a8-cd688b5f277b\" (UID: \"0a7808af-ef24-48f4-85a8-cd688b5f277b\") "
Aug 05 17:45:57.536145 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.536124    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a7808af-ef24-48f4-85a8-cd688b5f277b-tls-certs\") pod \"0a7808af-ef24-48f4-85a8-cd688b5f277b\" (UID: \"0a7808af-ef24-48f4-85a8-cd688b5f277b\") "
Aug 05 17:45:57.536822 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.536177    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a7808af-ef24-48f4-85a8-cd688b5f277b-tmp-dir\") pod \"0a7808af-ef24-48f4-85a8-cd688b5f277b\" (UID: \"0a7808af-ef24-48f4-85a8-cd688b5f277b\") "
Aug 05 17:45:57.536822 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.536441    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a7808af-ef24-48f4-85a8-cd688b5f277b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0a7808af-ef24-48f4-85a8-cd688b5f277b" (UID: "0a7808af-ef24-48f4-85a8-cd688b5f277b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:45:57.536822 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.536555    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a7808af-ef24-48f4-85a8-cd688b5f277b-cabundle-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:45:57.537011 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.536843    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7808af-ef24-48f4-85a8-cd688b5f277b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0a7808af-ef24-48f4-85a8-cd688b5f277b" (UID: "0a7808af-ef24-48f4-85a8-cd688b5f277b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:45:57.538754 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.538730    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a7808af-ef24-48f4-85a8-cd688b5f277b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0a7808af-ef24-48f4-85a8-cd688b5f277b" (UID: "0a7808af-ef24-48f4-85a8-cd688b5f277b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:45:57.547097 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.546043    2561 scope.go:117] "RemoveContainer" containerID="949fc7a98e7d86590f4966d95144ebde8501c138370e4efb685b4016adc58777"
Aug 05 17:45:57.547097 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:45:57.546335    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"949fc7a98e7d86590f4966d95144ebde8501c138370e4efb685b4016adc58777\": container with ID starting with 949fc7a98e7d86590f4966d95144ebde8501c138370e4efb685b4016adc58777 not found: ID does not exist" containerID="949fc7a98e7d86590f4966d95144ebde8501c138370e4efb685b4016adc58777"
Aug 05 17:45:57.547097 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.546370    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"949fc7a98e7d86590f4966d95144ebde8501c138370e4efb685b4016adc58777"} err="failed to get container status \"949fc7a98e7d86590f4966d95144ebde8501c138370e4efb685b4016adc58777\": rpc error: code = NotFound desc = could not find container \"949fc7a98e7d86590f4966d95144ebde8501c138370e4efb685b4016adc58777\": container with ID starting with 949fc7a98e7d86590f4966d95144ebde8501c138370e4efb685b4016adc58777 not found: ID does not exist"
Aug 05 17:45:57.547097 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.546390    2561 scope.go:117] "RemoveContainer" containerID="2359f00ff4b77090f717c71094cd0ad7e56328357e11f5f687eaf6de1f9039cc"
Aug 05 17:45:57.547097 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:45:57.546669    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2359f00ff4b77090f717c71094cd0ad7e56328357e11f5f687eaf6de1f9039cc\": container with ID starting with 2359f00ff4b77090f717c71094cd0ad7e56328357e11f5f687eaf6de1f9039cc not found: ID does not exist" containerID="2359f00ff4b77090f717c71094cd0ad7e56328357e11f5f687eaf6de1f9039cc"
Aug 05 17:45:57.547097 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.546719    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2359f00ff4b77090f717c71094cd0ad7e56328357e11f5f687eaf6de1f9039cc"} err="failed to get container status \"2359f00ff4b77090f717c71094cd0ad7e56328357e11f5f687eaf6de1f9039cc\": rpc error: code = NotFound desc = could not find container \"2359f00ff4b77090f717c71094cd0ad7e56328357e11f5f687eaf6de1f9039cc\": container with ID starting with 2359f00ff4b77090f717c71094cd0ad7e56328357e11f5f687eaf6de1f9039cc not found: ID does not exist"
Aug 05 17:45:57.547097 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.546741    2561 scope.go:117] "RemoveContainer" containerID="47e1b1b76147975301decbb72b58419e103e1c0b7ddfa09c0fbb2f41d9cb4b77"
Aug 05 17:45:57.557472 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.557441    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"]
Aug 05 17:45:57.562892 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.562803    2561 scope.go:117] "RemoveContainer" containerID="70755e7c4d674678848b3b861491f6428202f2e14f66c8664b991cc788362067"
Aug 05 17:45:57.564365 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.564343    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cf4bd5bc-7rsqn"]
Aug 05 17:45:57.567968 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.567924    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7808af-ef24-48f4-85a8-cd688b5f277b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0a7808af-ef24-48f4-85a8-cd688b5f277b" (UID: "0a7808af-ef24-48f4-85a8-cd688b5f277b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:45:57.571130 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.571114    2561 scope.go:117] "RemoveContainer" containerID="47e1b1b76147975301decbb72b58419e103e1c0b7ddfa09c0fbb2f41d9cb4b77"
Aug 05 17:45:57.571374 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:45:57.571354    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47e1b1b76147975301decbb72b58419e103e1c0b7ddfa09c0fbb2f41d9cb4b77\": container with ID starting with 47e1b1b76147975301decbb72b58419e103e1c0b7ddfa09c0fbb2f41d9cb4b77 not found: ID does not exist" containerID="47e1b1b76147975301decbb72b58419e103e1c0b7ddfa09c0fbb2f41d9cb4b77"
Aug 05 17:45:57.571445 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.571383    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47e1b1b76147975301decbb72b58419e103e1c0b7ddfa09c0fbb2f41d9cb4b77"} err="failed to get container status \"47e1b1b76147975301decbb72b58419e103e1c0b7ddfa09c0fbb2f41d9cb4b77\": rpc error: code = NotFound desc = could not find container \"47e1b1b76147975301decbb72b58419e103e1c0b7ddfa09c0fbb2f41d9cb4b77\": container with ID starting with 47e1b1b76147975301decbb72b58419e103e1c0b7ddfa09c0fbb2f41d9cb4b77 not found: ID does not exist"
Aug 05 17:45:57.571445 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.571406    2561 scope.go:117] "RemoveContainer" containerID="70755e7c4d674678848b3b861491f6428202f2e14f66c8664b991cc788362067"
Aug 05 17:45:57.571672 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:45:57.571656    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70755e7c4d674678848b3b861491f6428202f2e14f66c8664b991cc788362067\": container with ID starting with 70755e7c4d674678848b3b861491f6428202f2e14f66c8664b991cc788362067 not found: ID does not exist" containerID="70755e7c4d674678848b3b861491f6428202f2e14f66c8664b991cc788362067"
Aug 05 17:45:57.571712 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.571677    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70755e7c4d674678848b3b861491f6428202f2e14f66c8664b991cc788362067"} err="failed to get container status \"70755e7c4d674678848b3b861491f6428202f2e14f66c8664b991cc788362067\": rpc error: code = NotFound desc = could not find container \"70755e7c4d674678848b3b861491f6428202f2e14f66c8664b991cc788362067\": container with ID starting with 70755e7c4d674678848b3b861491f6428202f2e14f66c8664b991cc788362067 not found: ID does not exist"
Aug 05 17:45:57.637481 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.637422    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a7808af-ef24-48f4-85a8-cd688b5f277b-kserve-provision-location\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:45:57.637481 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.637445    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a7808af-ef24-48f4-85a8-cd688b5f277b-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:45:57.637481 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.637460    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a7808af-ef24-48f4-85a8-cd688b5f277b-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:45:57.859529 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.859503    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9"]
Aug 05 17:45:57.867034 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.867011    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-6dbfcddcf7-m5vp9"]
Aug 05 17:45:57.900345 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:57.900325    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"]
Aug 05 17:45:58.729757 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:58.729723    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a7808af-ef24-48f4-85a8-cd688b5f277b" path="/var/lib/kubelet/pods/0a7808af-ef24-48f4-85a8-cd688b5f277b/volumes"
Aug 05 17:45:58.730212 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:45:58.730187    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5" path="/var/lib/kubelet/pods/48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5/volumes"
Aug 05 17:46:12.918481 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:12.918436    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr" podUID="da118eaa-599e-4803-9292-911301ccfebb" containerName="main" containerID="cri-o://6ffe19a078d98490f56a0490940a131674cda5eff899eeab3413ebffcee0345e" gracePeriod=45
Aug 05 17:46:13.167717 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:13.167695    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"
Aug 05 17:46:13.252219 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:13.252157    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da118eaa-599e-4803-9292-911301ccfebb-home\") pod \"da118eaa-599e-4803-9292-911301ccfebb\" (UID: \"da118eaa-599e-4803-9292-911301ccfebb\") "
Aug 05 17:46:13.252219 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:13.252187    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da118eaa-599e-4803-9292-911301ccfebb-tls-certs\") pod \"da118eaa-599e-4803-9292-911301ccfebb\" (UID: \"da118eaa-599e-4803-9292-911301ccfebb\") "
Aug 05 17:46:13.252219 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:13.252208    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da118eaa-599e-4803-9292-911301ccfebb-model-cache\") pod \"da118eaa-599e-4803-9292-911301ccfebb\" (UID: \"da118eaa-599e-4803-9292-911301ccfebb\") "
Aug 05 17:46:13.252447 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:13.252228    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da118eaa-599e-4803-9292-911301ccfebb-tmp-dir\") pod \"da118eaa-599e-4803-9292-911301ccfebb\" (UID: \"da118eaa-599e-4803-9292-911301ccfebb\") "
Aug 05 17:46:13.252447 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:13.252293    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4czpd\" (UniqueName: \"kubernetes.io/projected/da118eaa-599e-4803-9292-911301ccfebb-kube-api-access-4czpd\") pod \"da118eaa-599e-4803-9292-911301ccfebb\" (UID: \"da118eaa-599e-4803-9292-911301ccfebb\") "
Aug 05 17:46:13.252447 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:13.252337    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da118eaa-599e-4803-9292-911301ccfebb-dshm\") pod \"da118eaa-599e-4803-9292-911301ccfebb\" (UID: \"da118eaa-599e-4803-9292-911301ccfebb\") "
Aug 05 17:46:13.252606 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:13.252471    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da118eaa-599e-4803-9292-911301ccfebb-home" (OuterVolumeSpecName: "home") pod "da118eaa-599e-4803-9292-911301ccfebb" (UID: "da118eaa-599e-4803-9292-911301ccfebb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:46:13.252606 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:13.252496    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da118eaa-599e-4803-9292-911301ccfebb-model-cache" (OuterVolumeSpecName: "model-cache") pod "da118eaa-599e-4803-9292-911301ccfebb" (UID: "da118eaa-599e-4803-9292-911301ccfebb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:46:13.252606 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:13.252544    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da118eaa-599e-4803-9292-911301ccfebb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "da118eaa-599e-4803-9292-911301ccfebb" (UID: "da118eaa-599e-4803-9292-911301ccfebb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:46:13.252741 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:13.252657    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da118eaa-599e-4803-9292-911301ccfebb-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:46:13.252741 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:13.252674    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da118eaa-599e-4803-9292-911301ccfebb-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:46:13.252741 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:13.252689    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da118eaa-599e-4803-9292-911301ccfebb-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:46:13.254453 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:13.254431    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da118eaa-599e-4803-9292-911301ccfebb-dshm" (OuterVolumeSpecName: "dshm") pod "da118eaa-599e-4803-9292-911301ccfebb" (UID: "da118eaa-599e-4803-9292-911301ccfebb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:46:13.254737 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:13.254708    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da118eaa-599e-4803-9292-911301ccfebb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "da118eaa-599e-4803-9292-911301ccfebb" (UID: "da118eaa-599e-4803-9292-911301ccfebb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:46:13.254737 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:13.254720    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da118eaa-599e-4803-9292-911301ccfebb-kube-api-access-4czpd" (OuterVolumeSpecName: "kube-api-access-4czpd") pod "da118eaa-599e-4803-9292-911301ccfebb" (UID: "da118eaa-599e-4803-9292-911301ccfebb"). InnerVolumeSpecName "kube-api-access-4czpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:46:13.353115 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:13.353092    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da118eaa-599e-4803-9292-911301ccfebb-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:46:13.353115 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:13.353112    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4czpd\" (UniqueName: \"kubernetes.io/projected/da118eaa-599e-4803-9292-911301ccfebb-kube-api-access-4czpd\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:46:13.353271 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:13.353122    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da118eaa-599e-4803-9292-911301ccfebb-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:46:13.614563 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:13.614535    2561 generic.go:356] "Generic (PLEG): container finished" podID="da118eaa-599e-4803-9292-911301ccfebb" containerID="6ffe19a078d98490f56a0490940a131674cda5eff899eeab3413ebffcee0345e" exitCode=0
Aug 05 17:46:13.614701 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:13.614580    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"
Aug 05 17:46:13.614701 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:13.614616    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr" event={"ID":"da118eaa-599e-4803-9292-911301ccfebb","Type":"ContainerDied","Data":"6ffe19a078d98490f56a0490940a131674cda5eff899eeab3413ebffcee0345e"}
Aug 05 17:46:13.614701 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:13.614668    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr" event={"ID":"da118eaa-599e-4803-9292-911301ccfebb","Type":"ContainerDied","Data":"96f48ea6ad8452c5c73b4e21c2789a83ad01c875b575b4a2483bb14dd9ed79e4"}
Aug 05 17:46:13.614701 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:13.614690    2561 scope.go:117] "RemoveContainer" containerID="6ffe19a078d98490f56a0490940a131674cda5eff899eeab3413ebffcee0345e"
Aug 05 17:46:13.623351 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:13.623333    2561 scope.go:117] "RemoveContainer" containerID="6ffe19a078d98490f56a0490940a131674cda5eff899eeab3413ebffcee0345e"
Aug 05 17:46:13.623587 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:46:13.623572    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ffe19a078d98490f56a0490940a131674cda5eff899eeab3413ebffcee0345e\": container with ID starting with 6ffe19a078d98490f56a0490940a131674cda5eff899eeab3413ebffcee0345e not found: ID does not exist" containerID="6ffe19a078d98490f56a0490940a131674cda5eff899eeab3413ebffcee0345e"
Aug 05 17:46:13.623640 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:13.623594    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ffe19a078d98490f56a0490940a131674cda5eff899eeab3413ebffcee0345e"} err="failed to get container status \"6ffe19a078d98490f56a0490940a131674cda5eff899eeab3413ebffcee0345e\": rpc error: code = NotFound desc = could not find container \"6ffe19a078d98490f56a0490940a131674cda5eff899eeab3413ebffcee0345e\": container with ID starting with 6ffe19a078d98490f56a0490940a131674cda5eff899eeab3413ebffcee0345e not found: ID does not exist"
Aug 05 17:46:13.637399 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:13.637374    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"]
Aug 05 17:46:13.641606 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:13.641587    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gqbxr"]
Aug 05 17:46:14.729333 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:14.729298    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da118eaa-599e-4803-9292-911301ccfebb" path="/var/lib/kubelet/pods/da118eaa-599e-4803-9292-911301ccfebb/volumes"
Aug 05 17:46:24.369110 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:24.369076    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"]
Aug 05 17:46:39.387172 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:39.387130    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv" podUID="e33b507e-e986-48f1-96b8-1f09a32c0531" containerName="main" containerID="cri-o://950d57482262e6cb92b0c81d9f025f96f6ea2c194981f4ff7b7f91e373bcc02b" gracePeriod=45
Aug 05 17:46:39.658096 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:39.658071    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"
Aug 05 17:46:39.758134 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:39.758102    2561 generic.go:356] "Generic (PLEG): container finished" podID="e33b507e-e986-48f1-96b8-1f09a32c0531" containerID="950d57482262e6cb92b0c81d9f025f96f6ea2c194981f4ff7b7f91e373bcc02b" exitCode=0
Aug 05 17:46:39.758134 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:39.758139    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv" event={"ID":"e33b507e-e986-48f1-96b8-1f09a32c0531","Type":"ContainerDied","Data":"950d57482262e6cb92b0c81d9f025f96f6ea2c194981f4ff7b7f91e373bcc02b"}
Aug 05 17:46:39.758366 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:39.758148    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"
Aug 05 17:46:39.758366 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:39.758170    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv" event={"ID":"e33b507e-e986-48f1-96b8-1f09a32c0531","Type":"ContainerDied","Data":"cea1ddb308fbfa94ccf8f5cd16e149f4b0f1eb959e584ff798549927ee4106e2"}
Aug 05 17:46:39.758366 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:39.758192    2561 scope.go:117] "RemoveContainer" containerID="950d57482262e6cb92b0c81d9f025f96f6ea2c194981f4ff7b7f91e373bcc02b"
Aug 05 17:46:39.760328 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:39.760307    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e33b507e-e986-48f1-96b8-1f09a32c0531-tmp-dir\") pod \"e33b507e-e986-48f1-96b8-1f09a32c0531\" (UID: \"e33b507e-e986-48f1-96b8-1f09a32c0531\") "
Aug 05 17:46:39.760430 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:39.760357    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e33b507e-e986-48f1-96b8-1f09a32c0531-tls-certs\") pod \"e33b507e-e986-48f1-96b8-1f09a32c0531\" (UID: \"e33b507e-e986-48f1-96b8-1f09a32c0531\") "
Aug 05 17:46:39.760430 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:39.760392    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e33b507e-e986-48f1-96b8-1f09a32c0531-home\") pod \"e33b507e-e986-48f1-96b8-1f09a32c0531\" (UID: \"e33b507e-e986-48f1-96b8-1f09a32c0531\") "
Aug 05 17:46:39.760544 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:39.760437    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e33b507e-e986-48f1-96b8-1f09a32c0531-model-cache\") pod \"e33b507e-e986-48f1-96b8-1f09a32c0531\" (UID: \"e33b507e-e986-48f1-96b8-1f09a32c0531\") "
Aug 05 17:46:39.760544 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:39.760462    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e33b507e-e986-48f1-96b8-1f09a32c0531-dshm\") pod \"e33b507e-e986-48f1-96b8-1f09a32c0531\" (UID: \"e33b507e-e986-48f1-96b8-1f09a32c0531\") "
Aug 05 17:46:39.760544 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:39.760515    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dwh8\" (UniqueName: \"kubernetes.io/projected/e33b507e-e986-48f1-96b8-1f09a32c0531-kube-api-access-5dwh8\") pod \"e33b507e-e986-48f1-96b8-1f09a32c0531\" (UID: \"e33b507e-e986-48f1-96b8-1f09a32c0531\") "
Aug 05 17:46:39.760699 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:39.760537    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33b507e-e986-48f1-96b8-1f09a32c0531-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e33b507e-e986-48f1-96b8-1f09a32c0531" (UID: "e33b507e-e986-48f1-96b8-1f09a32c0531"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:46:39.760699 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:39.760687    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33b507e-e986-48f1-96b8-1f09a32c0531-model-cache" (OuterVolumeSpecName: "model-cache") pod "e33b507e-e986-48f1-96b8-1f09a32c0531" (UID: "e33b507e-e986-48f1-96b8-1f09a32c0531"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:46:39.760808 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:39.760700    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33b507e-e986-48f1-96b8-1f09a32c0531-home" (OuterVolumeSpecName: "home") pod "e33b507e-e986-48f1-96b8-1f09a32c0531" (UID: "e33b507e-e986-48f1-96b8-1f09a32c0531"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:46:39.760878 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:39.760802    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e33b507e-e986-48f1-96b8-1f09a32c0531-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:46:39.760878 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:39.760820    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e33b507e-e986-48f1-96b8-1f09a32c0531-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:46:39.760878 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:39.760832    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e33b507e-e986-48f1-96b8-1f09a32c0531-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:46:39.762561 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:39.762538    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e33b507e-e986-48f1-96b8-1f09a32c0531-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e33b507e-e986-48f1-96b8-1f09a32c0531" (UID: "e33b507e-e986-48f1-96b8-1f09a32c0531"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:46:39.762561 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:39.762541    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e33b507e-e986-48f1-96b8-1f09a32c0531-kube-api-access-5dwh8" (OuterVolumeSpecName: "kube-api-access-5dwh8") pod "e33b507e-e986-48f1-96b8-1f09a32c0531" (UID: "e33b507e-e986-48f1-96b8-1f09a32c0531"). InnerVolumeSpecName "kube-api-access-5dwh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:46:39.762697 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:39.762580    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33b507e-e986-48f1-96b8-1f09a32c0531-dshm" (OuterVolumeSpecName: "dshm") pod "e33b507e-e986-48f1-96b8-1f09a32c0531" (UID: "e33b507e-e986-48f1-96b8-1f09a32c0531"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:46:39.766904 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:39.766887    2561 scope.go:117] "RemoveContainer" containerID="950d57482262e6cb92b0c81d9f025f96f6ea2c194981f4ff7b7f91e373bcc02b"
Aug 05 17:46:39.767194 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:46:39.767173    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"950d57482262e6cb92b0c81d9f025f96f6ea2c194981f4ff7b7f91e373bcc02b\": container with ID starting with 950d57482262e6cb92b0c81d9f025f96f6ea2c194981f4ff7b7f91e373bcc02b not found: ID does not exist" containerID="950d57482262e6cb92b0c81d9f025f96f6ea2c194981f4ff7b7f91e373bcc02b"
Aug 05 17:46:39.767276 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:39.767204    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"950d57482262e6cb92b0c81d9f025f96f6ea2c194981f4ff7b7f91e373bcc02b"} err="failed to get container status \"950d57482262e6cb92b0c81d9f025f96f6ea2c194981f4ff7b7f91e373bcc02b\": rpc error: code = NotFound desc = could not find container \"950d57482262e6cb92b0c81d9f025f96f6ea2c194981f4ff7b7f91e373bcc02b\": container with ID starting with 950d57482262e6cb92b0c81d9f025f96f6ea2c194981f4ff7b7f91e373bcc02b not found: ID does not exist"
Aug 05 17:46:39.861755 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:39.861728    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dwh8\" (UniqueName: \"kubernetes.io/projected/e33b507e-e986-48f1-96b8-1f09a32c0531-kube-api-access-5dwh8\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:46:39.861755 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:39.861752    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e33b507e-e986-48f1-96b8-1f09a32c0531-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:46:39.861902 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:39.861761    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e33b507e-e986-48f1-96b8-1f09a32c0531-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:46:40.090765 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:40.090735    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"]
Aug 05 17:46:40.095564 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:40.095540    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-f2tbv"]
Aug 05 17:46:40.730201 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:40.730169    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e33b507e-e986-48f1-96b8-1f09a32c0531" path="/var/lib/kubelet/pods/e33b507e-e986-48f1-96b8-1f09a32c0531/volumes"
Aug 05 17:46:49.139656 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.139569    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"]
Aug 05 17:46:49.140205 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.140105    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a7808af-ef24-48f4-85a8-cd688b5f277b" containerName="storage-initializer"
Aug 05 17:46:49.140205 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.140126    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a7808af-ef24-48f4-85a8-cd688b5f277b" containerName="storage-initializer"
Aug 05 17:46:49.140205 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.140143    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a7808af-ef24-48f4-85a8-cd688b5f277b" containerName="main"
Aug 05 17:46:49.140205 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.140152    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a7808af-ef24-48f4-85a8-cd688b5f277b" containerName="main"
Aug 05 17:46:49.140205 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.140170    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e33b507e-e986-48f1-96b8-1f09a32c0531" containerName="main"
Aug 05 17:46:49.140205 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.140178    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e33b507e-e986-48f1-96b8-1f09a32c0531" containerName="main"
Aug 05 17:46:49.140205 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.140187    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da118eaa-599e-4803-9292-911301ccfebb" containerName="main"
Aug 05 17:46:49.140205 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.140195    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="da118eaa-599e-4803-9292-911301ccfebb" containerName="main"
Aug 05 17:46:49.140616 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.140215    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5" containerName="storage-initializer"
Aug 05 17:46:49.140616 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.140223    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5" containerName="storage-initializer"
Aug 05 17:46:49.140616 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.140240    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5" containerName="main"
Aug 05 17:46:49.140616 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.140248    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5" containerName="main"
Aug 05 17:46:49.140616 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.140327    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="48a4cf0d-20a1-44d8-9652-abb6fc2ae7d5" containerName="main"
Aug 05 17:46:49.140616 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.140339    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e33b507e-e986-48f1-96b8-1f09a32c0531" containerName="main"
Aug 05 17:46:49.140616 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.140348    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="da118eaa-599e-4803-9292-911301ccfebb" containerName="main"
Aug 05 17:46:49.140616 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.140356    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a7808af-ef24-48f4-85a8-cd688b5f277b" containerName="main"
Aug 05 17:46:49.144846 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.144825    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"
Aug 05 17:46:49.151912 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.151892    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Aug 05 17:46:49.153000 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.152983    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Aug 05 17:46:49.153071 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.153028    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Aug 05 17:46:49.153178 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.153160    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-9v97x\""
Aug 05 17:46:49.162493 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.162471    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"]
Aug 05 17:46:49.242101 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.242073    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b72cf28-5494-450c-9a1c-50d61a61efba-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2\" (UID: \"3b72cf28-5494-450c-9a1c-50d61a61efba\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"
Aug 05 17:46:49.242229 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.242111    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b72cf28-5494-450c-9a1c-50d61a61efba-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2\" (UID: \"3b72cf28-5494-450c-9a1c-50d61a61efba\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"
Aug 05 17:46:49.242229 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.242143    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b72cf28-5494-450c-9a1c-50d61a61efba-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2\" (UID: \"3b72cf28-5494-450c-9a1c-50d61a61efba\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"
Aug 05 17:46:49.242229 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.242192    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv25t\" (UniqueName: \"kubernetes.io/projected/3b72cf28-5494-450c-9a1c-50d61a61efba-kube-api-access-lv25t\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2\" (UID: \"3b72cf28-5494-450c-9a1c-50d61a61efba\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"
Aug 05 17:46:49.242342 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.242256    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b72cf28-5494-450c-9a1c-50d61a61efba-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2\" (UID: \"3b72cf28-5494-450c-9a1c-50d61a61efba\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"
Aug 05 17:46:49.242342 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.242282    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b72cf28-5494-450c-9a1c-50d61a61efba-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2\" (UID: \"3b72cf28-5494-450c-9a1c-50d61a61efba\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"
Aug 05 17:46:49.342647 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.342613    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b72cf28-5494-450c-9a1c-50d61a61efba-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2\" (UID: \"3b72cf28-5494-450c-9a1c-50d61a61efba\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"
Aug 05 17:46:49.342819 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.342656    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b72cf28-5494-450c-9a1c-50d61a61efba-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2\" (UID: \"3b72cf28-5494-450c-9a1c-50d61a61efba\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"
Aug 05 17:46:49.342819 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.342680    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b72cf28-5494-450c-9a1c-50d61a61efba-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2\" (UID: \"3b72cf28-5494-450c-9a1c-50d61a61efba\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"
Aug 05 17:46:49.342819 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.342698    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv25t\" (UniqueName: \"kubernetes.io/projected/3b72cf28-5494-450c-9a1c-50d61a61efba-kube-api-access-lv25t\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2\" (UID: \"3b72cf28-5494-450c-9a1c-50d61a61efba\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"
Aug 05 17:46:49.342819 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.342724    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b72cf28-5494-450c-9a1c-50d61a61efba-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2\" (UID: \"3b72cf28-5494-450c-9a1c-50d61a61efba\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"
Aug 05 17:46:49.342819 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.342750    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b72cf28-5494-450c-9a1c-50d61a61efba-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2\" (UID: \"3b72cf28-5494-450c-9a1c-50d61a61efba\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"
Aug 05 17:46:49.343133 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.343084    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b72cf28-5494-450c-9a1c-50d61a61efba-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2\" (UID: \"3b72cf28-5494-450c-9a1c-50d61a61efba\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"
Aug 05 17:46:49.343133 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.343096    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b72cf28-5494-450c-9a1c-50d61a61efba-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2\" (UID: \"3b72cf28-5494-450c-9a1c-50d61a61efba\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"
Aug 05 17:46:49.343211 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.343129    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b72cf28-5494-450c-9a1c-50d61a61efba-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2\" (UID: \"3b72cf28-5494-450c-9a1c-50d61a61efba\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"
Aug 05 17:46:49.344918 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.344895    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b72cf28-5494-450c-9a1c-50d61a61efba-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2\" (UID: \"3b72cf28-5494-450c-9a1c-50d61a61efba\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"
Aug 05 17:46:49.345214 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.345195    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b72cf28-5494-450c-9a1c-50d61a61efba-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2\" (UID: \"3b72cf28-5494-450c-9a1c-50d61a61efba\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"
Aug 05 17:46:49.355329 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.355301    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv25t\" (UniqueName: \"kubernetes.io/projected/3b72cf28-5494-450c-9a1c-50d61a61efba-kube-api-access-lv25t\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2\" (UID: \"3b72cf28-5494-450c-9a1c-50d61a61efba\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"
Aug 05 17:46:49.455343 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.455265    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"
Aug 05 17:46:49.595126 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.595089    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"]
Aug 05 17:46:49.600506 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:46:49.600482    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b72cf28_5494_450c_9a1c_50d61a61efba.slice/crio-7c3eaae11321a36b752c3ddb593ca93098008ea6d2bbe8f66b815aee2dc4d099 WatchSource:0}: Error finding container 7c3eaae11321a36b752c3ddb593ca93098008ea6d2bbe8f66b815aee2dc4d099: Status 404 returned error can't find the container with id 7c3eaae11321a36b752c3ddb593ca93098008ea6d2bbe8f66b815aee2dc4d099
Aug 05 17:46:49.809978 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.809921    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2" event={"ID":"3b72cf28-5494-450c-9a1c-50d61a61efba","Type":"ContainerStarted","Data":"2c8a9bb5859229cc1ff74b4df841e42318e7063df586868e0a1e8dfe55bd6a56"}
Aug 05 17:46:49.810153 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.810014    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2" event={"ID":"3b72cf28-5494-450c-9a1c-50d61a61efba","Type":"ContainerStarted","Data":"7c3eaae11321a36b752c3ddb593ca93098008ea6d2bbe8f66b815aee2dc4d099"}
Aug 05 17:46:49.830394 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:49.830341    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2" podStartSLOduration=0.830324908 podStartE2EDuration="830.324908ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:46:49.828882506 +0000 UTC m=+2065.573516834" watchObservedRunningTime="2026-08-05 17:46:49.830324908 +0000 UTC m=+2065.574959235"
Aug 05 17:46:50.456323 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:50.456294    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"
Aug 05 17:46:59.456143 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:59.456105    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"
Aug 05 17:46:59.468297 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:59.468275    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"
Aug 05 17:46:59.869885 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:46:59.869857    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"
Aug 05 17:47:03.319016 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:03.318972    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"]
Aug 05 17:47:03.322986 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:03.322915    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"
Aug 05 17:47:03.325850 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:03.325831    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-wdwlb\""
Aug 05 17:47:03.325987 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:03.325894    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Aug 05 17:47:03.326714 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:03.326691    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Aug 05 17:47:03.326854 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:03.326836    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v1-kserve-self-signed-certs\""
Aug 05 17:47:03.333373 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:03.333355    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"]
Aug 05 17:47:03.352673 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:03.352644    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2f9x\" (UniqueName: \"kubernetes.io/projected/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-kube-api-access-b2f9x\") pod \"decom-v1-kserve-6cb797fb75-qhz7v\" (UID: \"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"
Aug 05 17:47:03.352783 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:03.352682    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-qhz7v\" (UID: \"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"
Aug 05 17:47:03.352783 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:03.352700    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-home\") pod \"decom-v1-kserve-6cb797fb75-qhz7v\" (UID: \"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"
Aug 05 17:47:03.352783 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:03.352774    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-model-cache\") pod \"decom-v1-kserve-6cb797fb75-qhz7v\" (UID: \"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"
Aug 05 17:47:03.352910 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:03.352815    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-qhz7v\" (UID: \"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"
Aug 05 17:47:03.352910 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:03.352869    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-dshm\") pod \"decom-v1-kserve-6cb797fb75-qhz7v\" (UID: \"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"
Aug 05 17:47:03.454047 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:03.454020    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-dshm\") pod \"decom-v1-kserve-6cb797fb75-qhz7v\" (UID: \"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"
Aug 05 17:47:03.454206 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:03.454063    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2f9x\" (UniqueName: \"kubernetes.io/projected/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-kube-api-access-b2f9x\") pod \"decom-v1-kserve-6cb797fb75-qhz7v\" (UID: \"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"
Aug 05 17:47:03.454206 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:03.454129    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-qhz7v\" (UID: \"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"
Aug 05 17:47:03.454206 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:03.454153    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-home\") pod \"decom-v1-kserve-6cb797fb75-qhz7v\" (UID: \"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"
Aug 05 17:47:03.454367 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:03.454230    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-model-cache\") pod \"decom-v1-kserve-6cb797fb75-qhz7v\" (UID: \"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"
Aug 05 17:47:03.454367 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:03.454300    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-qhz7v\" (UID: \"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"
Aug 05 17:47:03.454596 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:03.454574    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-qhz7v\" (UID: \"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"
Aug 05 17:47:03.454696 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:03.454607    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-home\") pod \"decom-v1-kserve-6cb797fb75-qhz7v\" (UID: \"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"
Aug 05 17:47:03.454772 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:03.454751    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-model-cache\") pod \"decom-v1-kserve-6cb797fb75-qhz7v\" (UID: \"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"
Aug 05 17:47:03.456279 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:03.456258    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-dshm\") pod \"decom-v1-kserve-6cb797fb75-qhz7v\" (UID: \"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"
Aug 05 17:47:03.456649 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:03.456632    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-qhz7v\" (UID: \"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"
Aug 05 17:47:03.461839 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:03.461817    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2f9x\" (UniqueName: \"kubernetes.io/projected/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-kube-api-access-b2f9x\") pod \"decom-v1-kserve-6cb797fb75-qhz7v\" (UID: \"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"
Aug 05 17:47:03.634416 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:03.634341    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"
Aug 05 17:47:03.769421 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:03.769398    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"]
Aug 05 17:47:03.771106 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:47:03.771080    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e6f5470_3ba3_4a25_9a39_38f68a8ed4da.slice/crio-e466ca44661efcc40be5b13e262a0b0fbf1f5fb5831e5d8e5ee0c8bb4778b89c WatchSource:0}: Error finding container e466ca44661efcc40be5b13e262a0b0fbf1f5fb5831e5d8e5ee0c8bb4778b89c: Status 404 returned error can't find the container with id e466ca44661efcc40be5b13e262a0b0fbf1f5fb5831e5d8e5ee0c8bb4778b89c
Aug 05 17:47:03.882667 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:03.882630    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v" event={"ID":"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da","Type":"ContainerStarted","Data":"3e4c06b7d6244d8792ccd6baa1b48eb3083277a3e67e5235bf1dc013eaa27dac"}
Aug 05 17:47:03.882852 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:03.882682    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v" event={"ID":"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da","Type":"ContainerStarted","Data":"e466ca44661efcc40be5b13e262a0b0fbf1f5fb5831e5d8e5ee0c8bb4778b89c"}
Aug 05 17:47:03.939348 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:03.939241    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v" podStartSLOduration=0.939225781 podStartE2EDuration="939.225781ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:47:03.937157523 +0000 UTC m=+2079.681791878" watchObservedRunningTime="2026-08-05 17:47:03.939225781 +0000 UTC m=+2079.683860107"
Aug 05 17:47:04.635095 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:04.635066    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"
Aug 05 17:47:08.808464 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:08.808429    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"]
Aug 05 17:47:08.812559 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:08.812539    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"
Aug 05 17:47:08.815195 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:08.815174    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v2-kserve-self-signed-certs\""
Aug 05 17:47:08.822398 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:08.822373    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"]
Aug 05 17:47:08.897707 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:08.897674    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf2003a6-64db-4b6d-9290-21a2daa9c70b-home\") pod \"decom-v2-kserve-77f867d787-66d8v\" (UID: \"cf2003a6-64db-4b6d-9290-21a2daa9c70b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"
Aug 05 17:47:08.897855 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:08.897725    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8ncr\" (UniqueName: \"kubernetes.io/projected/cf2003a6-64db-4b6d-9290-21a2daa9c70b-kube-api-access-j8ncr\") pod \"decom-v2-kserve-77f867d787-66d8v\" (UID: \"cf2003a6-64db-4b6d-9290-21a2daa9c70b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"
Aug 05 17:47:08.897855 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:08.897782    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf2003a6-64db-4b6d-9290-21a2daa9c70b-dshm\") pod \"decom-v2-kserve-77f867d787-66d8v\" (UID: \"cf2003a6-64db-4b6d-9290-21a2daa9c70b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"
Aug 05 17:47:08.897855 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:08.897805    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf2003a6-64db-4b6d-9290-21a2daa9c70b-model-cache\") pod \"decom-v2-kserve-77f867d787-66d8v\" (UID: \"cf2003a6-64db-4b6d-9290-21a2daa9c70b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"
Aug 05 17:47:08.897855 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:08.897847    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf2003a6-64db-4b6d-9290-21a2daa9c70b-tmp-dir\") pod \"decom-v2-kserve-77f867d787-66d8v\" (UID: \"cf2003a6-64db-4b6d-9290-21a2daa9c70b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"
Aug 05 17:47:08.898024 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:08.897872    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf2003a6-64db-4b6d-9290-21a2daa9c70b-tls-certs\") pod \"decom-v2-kserve-77f867d787-66d8v\" (UID: \"cf2003a6-64db-4b6d-9290-21a2daa9c70b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"
Aug 05 17:47:08.999160 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:08.999128    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf2003a6-64db-4b6d-9290-21a2daa9c70b-dshm\") pod \"decom-v2-kserve-77f867d787-66d8v\" (UID: \"cf2003a6-64db-4b6d-9290-21a2daa9c70b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"
Aug 05 17:47:08.999160 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:08.999162    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf2003a6-64db-4b6d-9290-21a2daa9c70b-model-cache\") pod \"decom-v2-kserve-77f867d787-66d8v\" (UID: \"cf2003a6-64db-4b6d-9290-21a2daa9c70b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"
Aug 05 17:47:08.999417 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:08.999197    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf2003a6-64db-4b6d-9290-21a2daa9c70b-tmp-dir\") pod \"decom-v2-kserve-77f867d787-66d8v\" (UID: \"cf2003a6-64db-4b6d-9290-21a2daa9c70b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"
Aug 05 17:47:08.999417 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:08.999231    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf2003a6-64db-4b6d-9290-21a2daa9c70b-tls-certs\") pod \"decom-v2-kserve-77f867d787-66d8v\" (UID: \"cf2003a6-64db-4b6d-9290-21a2daa9c70b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"
Aug 05 17:47:08.999417 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:08.999291    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf2003a6-64db-4b6d-9290-21a2daa9c70b-home\") pod \"decom-v2-kserve-77f867d787-66d8v\" (UID: \"cf2003a6-64db-4b6d-9290-21a2daa9c70b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"
Aug 05 17:47:08.999417 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:08.999350    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8ncr\" (UniqueName: \"kubernetes.io/projected/cf2003a6-64db-4b6d-9290-21a2daa9c70b-kube-api-access-j8ncr\") pod \"decom-v2-kserve-77f867d787-66d8v\" (UID: \"cf2003a6-64db-4b6d-9290-21a2daa9c70b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"
Aug 05 17:47:08.999633 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:08.999598    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf2003a6-64db-4b6d-9290-21a2daa9c70b-model-cache\") pod \"decom-v2-kserve-77f867d787-66d8v\" (UID: \"cf2003a6-64db-4b6d-9290-21a2daa9c70b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"
Aug 05 17:47:08.999722 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:08.999694    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf2003a6-64db-4b6d-9290-21a2daa9c70b-tmp-dir\") pod \"decom-v2-kserve-77f867d787-66d8v\" (UID: \"cf2003a6-64db-4b6d-9290-21a2daa9c70b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"
Aug 05 17:47:08.999782 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:08.999741    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf2003a6-64db-4b6d-9290-21a2daa9c70b-home\") pod \"decom-v2-kserve-77f867d787-66d8v\" (UID: \"cf2003a6-64db-4b6d-9290-21a2daa9c70b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"
Aug 05 17:47:09.001526 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:09.001502    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf2003a6-64db-4b6d-9290-21a2daa9c70b-dshm\") pod \"decom-v2-kserve-77f867d787-66d8v\" (UID: \"cf2003a6-64db-4b6d-9290-21a2daa9c70b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"
Aug 05 17:47:09.001796 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:09.001775    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf2003a6-64db-4b6d-9290-21a2daa9c70b-tls-certs\") pod \"decom-v2-kserve-77f867d787-66d8v\" (UID: \"cf2003a6-64db-4b6d-9290-21a2daa9c70b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"
Aug 05 17:47:09.007455 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:09.007432    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8ncr\" (UniqueName: \"kubernetes.io/projected/cf2003a6-64db-4b6d-9290-21a2daa9c70b-kube-api-access-j8ncr\") pod \"decom-v2-kserve-77f867d787-66d8v\" (UID: \"cf2003a6-64db-4b6d-9290-21a2daa9c70b\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"
Aug 05 17:47:09.124250 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:09.124170    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"
Aug 05 17:47:09.263050 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:09.263024    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"]
Aug 05 17:47:09.264747 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:47:09.264707    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf2003a6_64db_4b6d_9290_21a2daa9c70b.slice/crio-df97c0efb2e34863ff35dbbd6a1accaa86eadc279b3b19351b09afce86c7515c WatchSource:0}: Error finding container df97c0efb2e34863ff35dbbd6a1accaa86eadc279b3b19351b09afce86c7515c: Status 404 returned error can't find the container with id df97c0efb2e34863ff35dbbd6a1accaa86eadc279b3b19351b09afce86c7515c
Aug 05 17:47:09.916333 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:09.916300    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v" event={"ID":"cf2003a6-64db-4b6d-9290-21a2daa9c70b","Type":"ContainerStarted","Data":"f0013ce9575afaa0a62fd9ae7a9a5b5431bc96c02d4449587cc7f3e8518b463d"}
Aug 05 17:47:09.916735 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:09.916351    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v" event={"ID":"cf2003a6-64db-4b6d-9290-21a2daa9c70b","Type":"ContainerStarted","Data":"df97c0efb2e34863ff35dbbd6a1accaa86eadc279b3b19351b09afce86c7515c"}
Aug 05 17:47:09.939352 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:09.939303    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v" podStartSLOduration=1.939289566 podStartE2EDuration="1.939289566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:47:09.938178374 +0000 UTC m=+2085.682812703" watchObservedRunningTime="2026-08-05 17:47:09.939289566 +0000 UTC m=+2085.683923892"
Aug 05 17:47:10.124381 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:10.124344    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"
Aug 05 17:47:13.634774 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:13.634739    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"
Aug 05 17:47:13.647297 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:13.647267    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"
Aug 05 17:47:13.949347 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:13.949259    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"
Aug 05 17:47:19.125191 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:19.125160    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"
Aug 05 17:47:19.137452 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:19.137426    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"
Aug 05 17:47:19.986552 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:19.986521    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"
Aug 05 17:47:48.911191 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:48.911156    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"]
Aug 05 17:47:48.920791 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:47:48.920755    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 05 17:47:48.920791 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:47:48.920791 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:47:48.920791 ip-10-0-135-151 kubenswrapper[2561]:  > containerID="2c8a9bb5859229cc1ff74b4df841e42318e7063df586868e0a1e8dfe55bd6a56" cmd=["/bin/sleep","15"]
Aug 05 17:47:48.921111 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:48.920812    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2" message="" err=<
Aug 05 17:47:48.921111 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:47:48.921111 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:47:48.921111 ip-10-0-135-151 kubenswrapper[2561]:  >
Aug 05 17:47:48.921111 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:47:48.920831    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 05 17:47:48.921111 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:47:48.921111 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:47:48.921111 ip-10-0-135-151 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2" podUID="3b72cf28-5494-450c-9a1c-50d61a61efba" containerName="main" containerID="cri-o://2c8a9bb5859229cc1ff74b4df841e42318e7063df586868e0a1e8dfe55bd6a56"
Aug 05 17:47:48.921111 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:48.920873    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2" podUID="3b72cf28-5494-450c-9a1c-50d61a61efba" containerName="main" containerID="cri-o://2c8a9bb5859229cc1ff74b4df841e42318e7063df586868e0a1e8dfe55bd6a56" gracePeriod=60
Aug 05 17:47:49.275990 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:49.275964    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"
Aug 05 17:47:49.357808 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:49.357776    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b72cf28-5494-450c-9a1c-50d61a61efba-home\") pod \"3b72cf28-5494-450c-9a1c-50d61a61efba\" (UID: \"3b72cf28-5494-450c-9a1c-50d61a61efba\") "
Aug 05 17:47:49.357808 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:49.357808    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b72cf28-5494-450c-9a1c-50d61a61efba-tmp-dir\") pod \"3b72cf28-5494-450c-9a1c-50d61a61efba\" (UID: \"3b72cf28-5494-450c-9a1c-50d61a61efba\") "
Aug 05 17:47:49.358129 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:49.357848    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b72cf28-5494-450c-9a1c-50d61a61efba-model-cache\") pod \"3b72cf28-5494-450c-9a1c-50d61a61efba\" (UID: \"3b72cf28-5494-450c-9a1c-50d61a61efba\") "
Aug 05 17:47:49.358129 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:49.357878    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lv25t\" (UniqueName: \"kubernetes.io/projected/3b72cf28-5494-450c-9a1c-50d61a61efba-kube-api-access-lv25t\") pod \"3b72cf28-5494-450c-9a1c-50d61a61efba\" (UID: \"3b72cf28-5494-450c-9a1c-50d61a61efba\") "
Aug 05 17:47:49.358129 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:49.357911    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b72cf28-5494-450c-9a1c-50d61a61efba-tls-certs\") pod \"3b72cf28-5494-450c-9a1c-50d61a61efba\" (UID: \"3b72cf28-5494-450c-9a1c-50d61a61efba\") "
Aug 05 17:47:49.358129 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:49.357946    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b72cf28-5494-450c-9a1c-50d61a61efba-dshm\") pod \"3b72cf28-5494-450c-9a1c-50d61a61efba\" (UID: \"3b72cf28-5494-450c-9a1c-50d61a61efba\") "
Aug 05 17:47:49.358129 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:49.358104    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b72cf28-5494-450c-9a1c-50d61a61efba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3b72cf28-5494-450c-9a1c-50d61a61efba" (UID: "3b72cf28-5494-450c-9a1c-50d61a61efba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:47:49.358129 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:49.358119    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b72cf28-5494-450c-9a1c-50d61a61efba-home" (OuterVolumeSpecName: "home") pod "3b72cf28-5494-450c-9a1c-50d61a61efba" (UID: "3b72cf28-5494-450c-9a1c-50d61a61efba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:47:49.358408 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:49.358177    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b72cf28-5494-450c-9a1c-50d61a61efba-model-cache" (OuterVolumeSpecName: "model-cache") pod "3b72cf28-5494-450c-9a1c-50d61a61efba" (UID: "3b72cf28-5494-450c-9a1c-50d61a61efba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:47:49.358408 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:49.358346    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b72cf28-5494-450c-9a1c-50d61a61efba-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:47:49.358408 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:49.358363    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b72cf28-5494-450c-9a1c-50d61a61efba-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:47:49.358408 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:49.358372    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b72cf28-5494-450c-9a1c-50d61a61efba-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:47:49.360137 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:49.360114    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b72cf28-5494-450c-9a1c-50d61a61efba-dshm" (OuterVolumeSpecName: "dshm") pod "3b72cf28-5494-450c-9a1c-50d61a61efba" (UID: "3b72cf28-5494-450c-9a1c-50d61a61efba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:47:49.360137 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:49.360125    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b72cf28-5494-450c-9a1c-50d61a61efba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3b72cf28-5494-450c-9a1c-50d61a61efba" (UID: "3b72cf28-5494-450c-9a1c-50d61a61efba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:47:49.360281 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:49.360226    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b72cf28-5494-450c-9a1c-50d61a61efba-kube-api-access-lv25t" (OuterVolumeSpecName: "kube-api-access-lv25t") pod "3b72cf28-5494-450c-9a1c-50d61a61efba" (UID: "3b72cf28-5494-450c-9a1c-50d61a61efba"). InnerVolumeSpecName "kube-api-access-lv25t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:47:49.459131 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:49.459074    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b72cf28-5494-450c-9a1c-50d61a61efba-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:47:49.459131 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:49.459092    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b72cf28-5494-450c-9a1c-50d61a61efba-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:47:49.459131 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:49.459105    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv25t\" (UniqueName: \"kubernetes.io/projected/3b72cf28-5494-450c-9a1c-50d61a61efba-kube-api-access-lv25t\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:47:50.130963 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:50.130918    2561 generic.go:356] "Generic (PLEG): container finished" podID="3b72cf28-5494-450c-9a1c-50d61a61efba" containerID="2c8a9bb5859229cc1ff74b4df841e42318e7063df586868e0a1e8dfe55bd6a56" exitCode=0
Aug 05 17:47:50.131455 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:50.130987    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"
Aug 05 17:47:50.131455 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:50.131006    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2" event={"ID":"3b72cf28-5494-450c-9a1c-50d61a61efba","Type":"ContainerDied","Data":"2c8a9bb5859229cc1ff74b4df841e42318e7063df586868e0a1e8dfe55bd6a56"}
Aug 05 17:47:50.131455 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:50.131066    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2" event={"ID":"3b72cf28-5494-450c-9a1c-50d61a61efba","Type":"ContainerDied","Data":"7c3eaae11321a36b752c3ddb593ca93098008ea6d2bbe8f66b815aee2dc4d099"}
Aug 05 17:47:50.131455 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:50.131096    2561 scope.go:117] "RemoveContainer" containerID="2c8a9bb5859229cc1ff74b4df841e42318e7063df586868e0a1e8dfe55bd6a56"
Aug 05 17:47:50.140595 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:50.140580    2561 scope.go:117] "RemoveContainer" containerID="2c8a9bb5859229cc1ff74b4df841e42318e7063df586868e0a1e8dfe55bd6a56"
Aug 05 17:47:50.140913 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:47:50.140894    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c8a9bb5859229cc1ff74b4df841e42318e7063df586868e0a1e8dfe55bd6a56\": container with ID starting with 2c8a9bb5859229cc1ff74b4df841e42318e7063df586868e0a1e8dfe55bd6a56 not found: ID does not exist" containerID="2c8a9bb5859229cc1ff74b4df841e42318e7063df586868e0a1e8dfe55bd6a56"
Aug 05 17:47:50.141015 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:50.140918    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c8a9bb5859229cc1ff74b4df841e42318e7063df586868e0a1e8dfe55bd6a56"} err="failed to get container status \"2c8a9bb5859229cc1ff74b4df841e42318e7063df586868e0a1e8dfe55bd6a56\": rpc error: code = NotFound desc = could not find container \"2c8a9bb5859229cc1ff74b4df841e42318e7063df586868e0a1e8dfe55bd6a56\": container with ID starting with 2c8a9bb5859229cc1ff74b4df841e42318e7063df586868e0a1e8dfe55bd6a56 not found: ID does not exist"
Aug 05 17:47:50.175653 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:50.175628    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"]
Aug 05 17:47:50.181475 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:50.181454    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-clzm2"]
Aug 05 17:47:50.731420 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:47:50.731384    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b72cf28-5494-450c-9a1c-50d61a61efba" path="/var/lib/kubelet/pods/3b72cf28-5494-450c-9a1c-50d61a61efba/volumes"
Aug 05 17:48:01.312599 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:01.312565    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"]
Aug 05 17:48:16.331489 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:16.331428    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v" podUID="cf2003a6-64db-4b6d-9290-21a2daa9c70b" containerName="main" containerID="cri-o://f0013ce9575afaa0a62fd9ae7a9a5b5431bc96c02d4449587cc7f3e8518b463d" gracePeriod=45
Aug 05 17:48:16.570112 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:16.570089    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"
Aug 05 17:48:16.685180 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:16.685120    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf2003a6-64db-4b6d-9290-21a2daa9c70b-home\") pod \"cf2003a6-64db-4b6d-9290-21a2daa9c70b\" (UID: \"cf2003a6-64db-4b6d-9290-21a2daa9c70b\") "
Aug 05 17:48:16.685180 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:16.685175    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf2003a6-64db-4b6d-9290-21a2daa9c70b-tmp-dir\") pod \"cf2003a6-64db-4b6d-9290-21a2daa9c70b\" (UID: \"cf2003a6-64db-4b6d-9290-21a2daa9c70b\") "
Aug 05 17:48:16.685324 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:16.685228    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf2003a6-64db-4b6d-9290-21a2daa9c70b-dshm\") pod \"cf2003a6-64db-4b6d-9290-21a2daa9c70b\" (UID: \"cf2003a6-64db-4b6d-9290-21a2daa9c70b\") "
Aug 05 17:48:16.685324 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:16.685248    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf2003a6-64db-4b6d-9290-21a2daa9c70b-tls-certs\") pod \"cf2003a6-64db-4b6d-9290-21a2daa9c70b\" (UID: \"cf2003a6-64db-4b6d-9290-21a2daa9c70b\") "
Aug 05 17:48:16.685324 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:16.685280    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf2003a6-64db-4b6d-9290-21a2daa9c70b-model-cache\") pod \"cf2003a6-64db-4b6d-9290-21a2daa9c70b\" (UID: \"cf2003a6-64db-4b6d-9290-21a2daa9c70b\") "
Aug 05 17:48:16.685484 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:16.685370    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2003a6-64db-4b6d-9290-21a2daa9c70b-home" (OuterVolumeSpecName: "home") pod "cf2003a6-64db-4b6d-9290-21a2daa9c70b" (UID: "cf2003a6-64db-4b6d-9290-21a2daa9c70b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:48:16.685484 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:16.685383    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8ncr\" (UniqueName: \"kubernetes.io/projected/cf2003a6-64db-4b6d-9290-21a2daa9c70b-kube-api-access-j8ncr\") pod \"cf2003a6-64db-4b6d-9290-21a2daa9c70b\" (UID: \"cf2003a6-64db-4b6d-9290-21a2daa9c70b\") "
Aug 05 17:48:16.685484 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:16.685471    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2003a6-64db-4b6d-9290-21a2daa9c70b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cf2003a6-64db-4b6d-9290-21a2daa9c70b" (UID: "cf2003a6-64db-4b6d-9290-21a2daa9c70b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:48:16.685609 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:16.685502    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2003a6-64db-4b6d-9290-21a2daa9c70b-model-cache" (OuterVolumeSpecName: "model-cache") pod "cf2003a6-64db-4b6d-9290-21a2daa9c70b" (UID: "cf2003a6-64db-4b6d-9290-21a2daa9c70b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:48:16.685666 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:16.685653    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf2003a6-64db-4b6d-9290-21a2daa9c70b-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:48:16.685733 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:16.685667    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf2003a6-64db-4b6d-9290-21a2daa9c70b-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:48:16.685733 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:16.685675    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf2003a6-64db-4b6d-9290-21a2daa9c70b-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:48:16.687394 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:16.687365    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf2003a6-64db-4b6d-9290-21a2daa9c70b-dshm" (OuterVolumeSpecName: "dshm") pod "cf2003a6-64db-4b6d-9290-21a2daa9c70b" (UID: "cf2003a6-64db-4b6d-9290-21a2daa9c70b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:48:16.687394 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:16.687380    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf2003a6-64db-4b6d-9290-21a2daa9c70b-kube-api-access-j8ncr" (OuterVolumeSpecName: "kube-api-access-j8ncr") pod "cf2003a6-64db-4b6d-9290-21a2daa9c70b" (UID: "cf2003a6-64db-4b6d-9290-21a2daa9c70b"). InnerVolumeSpecName "kube-api-access-j8ncr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:48:16.687533 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:16.687392    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf2003a6-64db-4b6d-9290-21a2daa9c70b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cf2003a6-64db-4b6d-9290-21a2daa9c70b" (UID: "cf2003a6-64db-4b6d-9290-21a2daa9c70b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:48:16.786584 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:16.786563    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf2003a6-64db-4b6d-9290-21a2daa9c70b-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:48:16.786584 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:16.786583    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf2003a6-64db-4b6d-9290-21a2daa9c70b-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:48:16.786719 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:16.786592    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8ncr\" (UniqueName: \"kubernetes.io/projected/cf2003a6-64db-4b6d-9290-21a2daa9c70b-kube-api-access-j8ncr\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:48:17.279684 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:17.279650    2561 generic.go:356] "Generic (PLEG): container finished" podID="cf2003a6-64db-4b6d-9290-21a2daa9c70b" containerID="f0013ce9575afaa0a62fd9ae7a9a5b5431bc96c02d4449587cc7f3e8518b463d" exitCode=0
Aug 05 17:48:17.279871 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:17.279699    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"
Aug 05 17:48:17.279871 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:17.279732    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v" event={"ID":"cf2003a6-64db-4b6d-9290-21a2daa9c70b","Type":"ContainerDied","Data":"f0013ce9575afaa0a62fd9ae7a9a5b5431bc96c02d4449587cc7f3e8518b463d"}
Aug 05 17:48:17.279871 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:17.279783    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v" event={"ID":"cf2003a6-64db-4b6d-9290-21a2daa9c70b","Type":"ContainerDied","Data":"df97c0efb2e34863ff35dbbd6a1accaa86eadc279b3b19351b09afce86c7515c"}
Aug 05 17:48:17.279871 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:17.279804    2561 scope.go:117] "RemoveContainer" containerID="f0013ce9575afaa0a62fd9ae7a9a5b5431bc96c02d4449587cc7f3e8518b463d"
Aug 05 17:48:17.288320 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:17.288296    2561 scope.go:117] "RemoveContainer" containerID="f0013ce9575afaa0a62fd9ae7a9a5b5431bc96c02d4449587cc7f3e8518b463d"
Aug 05 17:48:17.288555 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:48:17.288536    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0013ce9575afaa0a62fd9ae7a9a5b5431bc96c02d4449587cc7f3e8518b463d\": container with ID starting with f0013ce9575afaa0a62fd9ae7a9a5b5431bc96c02d4449587cc7f3e8518b463d not found: ID does not exist" containerID="f0013ce9575afaa0a62fd9ae7a9a5b5431bc96c02d4449587cc7f3e8518b463d"
Aug 05 17:48:17.288605 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:17.288564    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0013ce9575afaa0a62fd9ae7a9a5b5431bc96c02d4449587cc7f3e8518b463d"} err="failed to get container status \"f0013ce9575afaa0a62fd9ae7a9a5b5431bc96c02d4449587cc7f3e8518b463d\": rpc error: code = NotFound desc = could not find container \"f0013ce9575afaa0a62fd9ae7a9a5b5431bc96c02d4449587cc7f3e8518b463d\": container with ID starting with f0013ce9575afaa0a62fd9ae7a9a5b5431bc96c02d4449587cc7f3e8518b463d not found: ID does not exist"
Aug 05 17:48:17.300149 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:17.300129    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"]
Aug 05 17:48:17.302514 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:17.302494    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-66d8v"]
Aug 05 17:48:18.729943 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:18.729908    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf2003a6-64db-4b6d-9290-21a2daa9c70b" path="/var/lib/kubelet/pods/cf2003a6-64db-4b6d-9290-21a2daa9c70b/volumes"
Aug 05 17:48:22.647449 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:22.647414    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"]
Aug 05 17:48:35.446941 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.446907    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"]
Aug 05 17:48:35.447451 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.447431    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf2003a6-64db-4b6d-9290-21a2daa9c70b" containerName="main"
Aug 05 17:48:35.447528 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.447454    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf2003a6-64db-4b6d-9290-21a2daa9c70b" containerName="main"
Aug 05 17:48:35.447528 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.447479    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b72cf28-5494-450c-9a1c-50d61a61efba" containerName="main"
Aug 05 17:48:35.447528 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.447487    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b72cf28-5494-450c-9a1c-50d61a61efba" containerName="main"
Aug 05 17:48:35.447678 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.447613    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf2003a6-64db-4b6d-9290-21a2daa9c70b" containerName="main"
Aug 05 17:48:35.447678 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.447627    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b72cf28-5494-450c-9a1c-50d61a61efba" containerName="main"
Aug 05 17:48:35.450490 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.450471    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"
Aug 05 17:48:35.453545 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.453526    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-h6f25\""
Aug 05 17:48:35.453669 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.453527    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Aug 05 17:48:35.454460 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.454438    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Aug 05 17:48:35.454568 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.454552    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Aug 05 17:48:35.463335 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.463315    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"]
Aug 05 17:48:35.528860 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.528834    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf\" (UID: \"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"
Aug 05 17:48:35.529033 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.528878    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf\" (UID: \"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"
Aug 05 17:48:35.529033 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.528903    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf\" (UID: \"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"
Aug 05 17:48:35.529033 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.528997    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf\" (UID: \"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"
Aug 05 17:48:35.529192 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.529055    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsh5n\" (UniqueName: \"kubernetes.io/projected/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-kube-api-access-xsh5n\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf\" (UID: \"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"
Aug 05 17:48:35.529192 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.529098    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf\" (UID: \"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"
Aug 05 17:48:35.629702 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.629668    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf\" (UID: \"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"
Aug 05 17:48:35.629904 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.629711    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf\" (UID: \"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"
Aug 05 17:48:35.629904 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.629765    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf\" (UID: \"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"
Aug 05 17:48:35.629904 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.629822    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsh5n\" (UniqueName: \"kubernetes.io/projected/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-kube-api-access-xsh5n\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf\" (UID: \"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"
Aug 05 17:48:35.629904 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.629856    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf\" (UID: \"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"
Aug 05 17:48:35.630209 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.629916    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf\" (UID: \"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"
Aug 05 17:48:35.630322 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.630292    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf\" (UID: \"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"
Aug 05 17:48:35.630384 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.630324    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf\" (UID: \"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"
Aug 05 17:48:35.630503 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.630485    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf\" (UID: \"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"
Aug 05 17:48:35.631964 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.631930    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf\" (UID: \"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"
Aug 05 17:48:35.632485 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.632466    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf\" (UID: \"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"
Aug 05 17:48:35.640924 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.640900    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsh5n\" (UniqueName: \"kubernetes.io/projected/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-kube-api-access-xsh5n\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf\" (UID: \"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"
Aug 05 17:48:35.760293 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.760207    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"
Aug 05 17:48:35.908685 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:35.908664    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"]
Aug 05 17:48:35.910451 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:48:35.910428    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9264de5_8ee1_4f1b_9f78_1c6315aa37b4.slice/crio-5fb2dcc6cc39fd5253c837e6c7005c77c5041fb214c71c9351af3d0cfb8677ad WatchSource:0}: Error finding container 5fb2dcc6cc39fd5253c837e6c7005c77c5041fb214c71c9351af3d0cfb8677ad: Status 404 returned error can't find the container with id 5fb2dcc6cc39fd5253c837e6c7005c77c5041fb214c71c9351af3d0cfb8677ad
Aug 05 17:48:36.383435 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:36.383400    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf" event={"ID":"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4","Type":"ContainerStarted","Data":"b174415b97a33cb1bced362886e2095bc4997bb54ed16182f5f669c83f8e5ef0"}
Aug 05 17:48:36.383605 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:36.383458    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf" event={"ID":"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4","Type":"ContainerStarted","Data":"5fb2dcc6cc39fd5253c837e6c7005c77c5041fb214c71c9351af3d0cfb8677ad"}
Aug 05 17:48:36.403370 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:36.403322    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf" podStartSLOduration=1.403306455 podStartE2EDuration="1.403306455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:48:36.402634482 +0000 UTC m=+2172.147268809" watchObservedRunningTime="2026-08-05 17:48:36.403306455 +0000 UTC m=+2172.147940825"
Aug 05 17:48:36.760484 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:36.760406    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"
Aug 05 17:48:37.666439 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:37.666392    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v" podUID="6e6f5470-3ba3-4a25-9a39-38f68a8ed4da" containerName="main" containerID="cri-o://3e4c06b7d6244d8792ccd6baa1b48eb3083277a3e67e5235bf1dc013eaa27dac" gracePeriod=45
Aug 05 17:48:37.915608 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:37.915585    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"
Aug 05 17:48:38.050645 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:38.050611    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-model-cache\") pod \"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da\" (UID: \"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da\") "
Aug 05 17:48:38.050853 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:38.050666    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-tmp-dir\") pod \"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da\" (UID: \"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da\") "
Aug 05 17:48:38.050853 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:38.050755    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2f9x\" (UniqueName: \"kubernetes.io/projected/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-kube-api-access-b2f9x\") pod \"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da\" (UID: \"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da\") "
Aug 05 17:48:38.050853 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:38.050784    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-dshm\") pod \"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da\" (UID: \"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da\") "
Aug 05 17:48:38.050853 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:38.050831    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-tls-certs\") pod \"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da\" (UID: \"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da\") "
Aug 05 17:48:38.051120 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:38.050860    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-home\") pod \"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da\" (UID: \"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da\") "
Aug 05 17:48:38.051120 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:38.050909    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-model-cache" (OuterVolumeSpecName: "model-cache") pod "6e6f5470-3ba3-4a25-9a39-38f68a8ed4da" (UID: "6e6f5470-3ba3-4a25-9a39-38f68a8ed4da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:48:38.051230 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:38.051142    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:48:38.051392 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:38.051368    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-home" (OuterVolumeSpecName: "home") pod "6e6f5470-3ba3-4a25-9a39-38f68a8ed4da" (UID: "6e6f5470-3ba3-4a25-9a39-38f68a8ed4da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:48:38.051564 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:38.051535    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6e6f5470-3ba3-4a25-9a39-38f68a8ed4da" (UID: "6e6f5470-3ba3-4a25-9a39-38f68a8ed4da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:48:38.053034 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:38.053005    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-kube-api-access-b2f9x" (OuterVolumeSpecName: "kube-api-access-b2f9x") pod "6e6f5470-3ba3-4a25-9a39-38f68a8ed4da" (UID: "6e6f5470-3ba3-4a25-9a39-38f68a8ed4da"). InnerVolumeSpecName "kube-api-access-b2f9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:48:38.053506 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:38.053467    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-dshm" (OuterVolumeSpecName: "dshm") pod "6e6f5470-3ba3-4a25-9a39-38f68a8ed4da" (UID: "6e6f5470-3ba3-4a25-9a39-38f68a8ed4da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:48:38.053598 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:38.053573    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6e6f5470-3ba3-4a25-9a39-38f68a8ed4da" (UID: "6e6f5470-3ba3-4a25-9a39-38f68a8ed4da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:48:38.152624 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:38.152597    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:48:38.152720 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:38.152628    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2f9x\" (UniqueName: \"kubernetes.io/projected/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-kube-api-access-b2f9x\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:48:38.152720 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:38.152642    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:48:38.152720 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:38.152653    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:48:38.152720 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:38.152664    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:48:38.394253 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:38.394173    2561 generic.go:356] "Generic (PLEG): container finished" podID="6e6f5470-3ba3-4a25-9a39-38f68a8ed4da" containerID="3e4c06b7d6244d8792ccd6baa1b48eb3083277a3e67e5235bf1dc013eaa27dac" exitCode=0
Aug 05 17:48:38.394253 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:38.394223    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"
Aug 05 17:48:38.394427 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:38.394257    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v" event={"ID":"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da","Type":"ContainerDied","Data":"3e4c06b7d6244d8792ccd6baa1b48eb3083277a3e67e5235bf1dc013eaa27dac"}
Aug 05 17:48:38.394427 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:38.394308    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v" event={"ID":"6e6f5470-3ba3-4a25-9a39-38f68a8ed4da","Type":"ContainerDied","Data":"e466ca44661efcc40be5b13e262a0b0fbf1f5fb5831e5d8e5ee0c8bb4778b89c"}
Aug 05 17:48:38.394427 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:38.394330    2561 scope.go:117] "RemoveContainer" containerID="3e4c06b7d6244d8792ccd6baa1b48eb3083277a3e67e5235bf1dc013eaa27dac"
Aug 05 17:48:38.403067 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:38.403052    2561 scope.go:117] "RemoveContainer" containerID="3e4c06b7d6244d8792ccd6baa1b48eb3083277a3e67e5235bf1dc013eaa27dac"
Aug 05 17:48:38.403308 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:48:38.403291    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e4c06b7d6244d8792ccd6baa1b48eb3083277a3e67e5235bf1dc013eaa27dac\": container with ID starting with 3e4c06b7d6244d8792ccd6baa1b48eb3083277a3e67e5235bf1dc013eaa27dac not found: ID does not exist" containerID="3e4c06b7d6244d8792ccd6baa1b48eb3083277a3e67e5235bf1dc013eaa27dac"
Aug 05 17:48:38.403385 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:38.403316    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e4c06b7d6244d8792ccd6baa1b48eb3083277a3e67e5235bf1dc013eaa27dac"} err="failed to get container status \"3e4c06b7d6244d8792ccd6baa1b48eb3083277a3e67e5235bf1dc013eaa27dac\": rpc error: code = NotFound desc = could not find container \"3e4c06b7d6244d8792ccd6baa1b48eb3083277a3e67e5235bf1dc013eaa27dac\": container with ID starting with 3e4c06b7d6244d8792ccd6baa1b48eb3083277a3e67e5235bf1dc013eaa27dac not found: ID does not exist"
Aug 05 17:48:38.416296 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:38.416271    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"]
Aug 05 17:48:38.422438 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:38.422420    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-qhz7v"]
Aug 05 17:48:38.729710 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:38.729630    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e6f5470-3ba3-4a25-9a39-38f68a8ed4da" path="/var/lib/kubelet/pods/6e6f5470-3ba3-4a25-9a39-38f68a8ed4da/volumes"
Aug 05 17:48:45.760938 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:45.760902    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"
Aug 05 17:48:45.773128 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:45.773103    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"
Aug 05 17:48:46.409428 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.409392    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"]
Aug 05 17:48:46.409756 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.409743    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e6f5470-3ba3-4a25-9a39-38f68a8ed4da" containerName="main"
Aug 05 17:48:46.409800 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.409757    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e6f5470-3ba3-4a25-9a39-38f68a8ed4da" containerName="main"
Aug 05 17:48:46.409832 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.409819    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e6f5470-3ba3-4a25-9a39-38f68a8ed4da" containerName="main"
Aug 05 17:48:46.414308 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.414288    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"
Aug 05 17:48:46.417109 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.417085    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\""
Aug 05 17:48:46.418255 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.418237    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\""
Aug 05 17:48:46.418343 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.418306    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v1-kserve-self-signed-certs\""
Aug 05 17:48:46.418343 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.418318    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-fbmpb\""
Aug 05 17:48:46.425296 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.425274    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"]
Aug 05 17:48:46.447476 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.447454    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"
Aug 05 17:48:46.516134 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.516104    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mfmq\" (UniqueName: \"kubernetes.io/projected/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-kube-api-access-7mfmq\") pod \"leave-v1-kserve-65684d4f5-tt7c2\" (UID: \"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"
Aug 05 17:48:46.516289 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.516143    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-home\") pod \"leave-v1-kserve-65684d4f5-tt7c2\" (UID: \"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"
Aug 05 17:48:46.516289 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.516170    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-tls-certs\") pod \"leave-v1-kserve-65684d4f5-tt7c2\" (UID: \"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"
Aug 05 17:48:46.516289 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.516227    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-model-cache\") pod \"leave-v1-kserve-65684d4f5-tt7c2\" (UID: \"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"
Aug 05 17:48:46.516431 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.516414    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-dshm\") pod \"leave-v1-kserve-65684d4f5-tt7c2\" (UID: \"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"
Aug 05 17:48:46.516469 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.516459    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-tt7c2\" (UID: \"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"
Aug 05 17:48:46.617237 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.617209    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-dshm\") pod \"leave-v1-kserve-65684d4f5-tt7c2\" (UID: \"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"
Aug 05 17:48:46.617391 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.617241    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-tt7c2\" (UID: \"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"
Aug 05 17:48:46.617391 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.617274    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mfmq\" (UniqueName: \"kubernetes.io/projected/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-kube-api-access-7mfmq\") pod \"leave-v1-kserve-65684d4f5-tt7c2\" (UID: \"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"
Aug 05 17:48:46.617391 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.617299    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-home\") pod \"leave-v1-kserve-65684d4f5-tt7c2\" (UID: \"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"
Aug 05 17:48:46.617391 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.617313    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-tls-certs\") pod \"leave-v1-kserve-65684d4f5-tt7c2\" (UID: \"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"
Aug 05 17:48:46.617391 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.617345    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-model-cache\") pod \"leave-v1-kserve-65684d4f5-tt7c2\" (UID: \"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"
Aug 05 17:48:46.617738 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.617717    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-home\") pod \"leave-v1-kserve-65684d4f5-tt7c2\" (UID: \"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"
Aug 05 17:48:46.617809 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.617756    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-model-cache\") pod \"leave-v1-kserve-65684d4f5-tt7c2\" (UID: \"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"
Aug 05 17:48:46.617809 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.617779    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-tt7c2\" (UID: \"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"
Aug 05 17:48:46.619344 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.619327    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-dshm\") pod \"leave-v1-kserve-65684d4f5-tt7c2\" (UID: \"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"
Aug 05 17:48:46.619647 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.619629    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-tls-certs\") pod \"leave-v1-kserve-65684d4f5-tt7c2\" (UID: \"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"
Aug 05 17:48:46.626565 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.626544    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mfmq\" (UniqueName: \"kubernetes.io/projected/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-kube-api-access-7mfmq\") pod \"leave-v1-kserve-65684d4f5-tt7c2\" (UID: \"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"
Aug 05 17:48:46.725494 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.725421    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"
Aug 05 17:48:46.859781 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:46.859753    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"]
Aug 05 17:48:46.860992 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:48:46.860964    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9249e33_6fe2_48a9_b3d6_0ea0e8038b64.slice/crio-b51cb355aa12bfa25a64e854bdf29965e30fe8c63bce0a2b0d85c2778cef84c6 WatchSource:0}: Error finding container b51cb355aa12bfa25a64e854bdf29965e30fe8c63bce0a2b0d85c2778cef84c6: Status 404 returned error can't find the container with id b51cb355aa12bfa25a64e854bdf29965e30fe8c63bce0a2b0d85c2778cef84c6
Aug 05 17:48:47.443969 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:47.443922    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2" event={"ID":"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64","Type":"ContainerStarted","Data":"af6f444734a465e674235a76eaa713508a6a1cf16a74d6b3e7f6e19f941e67b5"}
Aug 05 17:48:47.444162 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:47.444003    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2" event={"ID":"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64","Type":"ContainerStarted","Data":"b51cb355aa12bfa25a64e854bdf29965e30fe8c63bce0a2b0d85c2778cef84c6"}
Aug 05 17:48:47.474904 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:47.474858    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2" podStartSLOduration=1.474844074 podStartE2EDuration="1.474844074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:48:47.474269486 +0000 UTC m=+2183.218903813" watchObservedRunningTime="2026-08-05 17:48:47.474844074 +0000 UTC m=+2183.219478400"
Aug 05 17:48:47.726068 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:47.725998    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"
Aug 05 17:48:52.405508 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:52.405475    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"]
Aug 05 17:48:52.409191 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:52.409175    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"
Aug 05 17:48:52.411605 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:52.411583    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v2-kserve-self-signed-certs\""
Aug 05 17:48:52.417638 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:52.417619    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"]
Aug 05 17:48:52.569346 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:52.569315    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ed12546-f37b-40a1-a340-3f22c4fd76ec-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-dt9tv\" (UID: \"2ed12546-f37b-40a1-a340-3f22c4fd76ec\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"
Aug 05 17:48:52.569508 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:52.569358    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ed12546-f37b-40a1-a340-3f22c4fd76ec-home\") pod \"leave-v2-kserve-556b4dddf9-dt9tv\" (UID: \"2ed12546-f37b-40a1-a340-3f22c4fd76ec\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"
Aug 05 17:48:52.569508 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:52.569380    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ed12546-f37b-40a1-a340-3f22c4fd76ec-dshm\") pod \"leave-v2-kserve-556b4dddf9-dt9tv\" (UID: \"2ed12546-f37b-40a1-a340-3f22c4fd76ec\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"
Aug 05 17:48:52.569508 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:52.569481    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ed12546-f37b-40a1-a340-3f22c4fd76ec-model-cache\") pod \"leave-v2-kserve-556b4dddf9-dt9tv\" (UID: \"2ed12546-f37b-40a1-a340-3f22c4fd76ec\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"
Aug 05 17:48:52.569616 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:52.569510    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7qqc\" (UniqueName: \"kubernetes.io/projected/2ed12546-f37b-40a1-a340-3f22c4fd76ec-kube-api-access-z7qqc\") pod \"leave-v2-kserve-556b4dddf9-dt9tv\" (UID: \"2ed12546-f37b-40a1-a340-3f22c4fd76ec\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"
Aug 05 17:48:52.569616 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:52.569550    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ed12546-f37b-40a1-a340-3f22c4fd76ec-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-dt9tv\" (UID: \"2ed12546-f37b-40a1-a340-3f22c4fd76ec\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"
Aug 05 17:48:52.670666 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:52.670598    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ed12546-f37b-40a1-a340-3f22c4fd76ec-model-cache\") pod \"leave-v2-kserve-556b4dddf9-dt9tv\" (UID: \"2ed12546-f37b-40a1-a340-3f22c4fd76ec\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"
Aug 05 17:48:52.670666 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:52.670628    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7qqc\" (UniqueName: \"kubernetes.io/projected/2ed12546-f37b-40a1-a340-3f22c4fd76ec-kube-api-access-z7qqc\") pod \"leave-v2-kserve-556b4dddf9-dt9tv\" (UID: \"2ed12546-f37b-40a1-a340-3f22c4fd76ec\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"
Aug 05 17:48:52.670666 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:52.670657    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ed12546-f37b-40a1-a340-3f22c4fd76ec-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-dt9tv\" (UID: \"2ed12546-f37b-40a1-a340-3f22c4fd76ec\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"
Aug 05 17:48:52.670971 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:52.670696    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ed12546-f37b-40a1-a340-3f22c4fd76ec-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-dt9tv\" (UID: \"2ed12546-f37b-40a1-a340-3f22c4fd76ec\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"
Aug 05 17:48:52.670971 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:52.670729    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ed12546-f37b-40a1-a340-3f22c4fd76ec-home\") pod \"leave-v2-kserve-556b4dddf9-dt9tv\" (UID: \"2ed12546-f37b-40a1-a340-3f22c4fd76ec\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"
Aug 05 17:48:52.670971 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:52.670762    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ed12546-f37b-40a1-a340-3f22c4fd76ec-dshm\") pod \"leave-v2-kserve-556b4dddf9-dt9tv\" (UID: \"2ed12546-f37b-40a1-a340-3f22c4fd76ec\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"
Aug 05 17:48:52.671160 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:52.671036    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ed12546-f37b-40a1-a340-3f22c4fd76ec-model-cache\") pod \"leave-v2-kserve-556b4dddf9-dt9tv\" (UID: \"2ed12546-f37b-40a1-a340-3f22c4fd76ec\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"
Aug 05 17:48:52.671160 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:52.671104    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ed12546-f37b-40a1-a340-3f22c4fd76ec-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-dt9tv\" (UID: \"2ed12546-f37b-40a1-a340-3f22c4fd76ec\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"
Aug 05 17:48:52.671160 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:52.671112    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ed12546-f37b-40a1-a340-3f22c4fd76ec-home\") pod \"leave-v2-kserve-556b4dddf9-dt9tv\" (UID: \"2ed12546-f37b-40a1-a340-3f22c4fd76ec\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"
Aug 05 17:48:52.673113 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:52.673084    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ed12546-f37b-40a1-a340-3f22c4fd76ec-dshm\") pod \"leave-v2-kserve-556b4dddf9-dt9tv\" (UID: \"2ed12546-f37b-40a1-a340-3f22c4fd76ec\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"
Aug 05 17:48:52.673287 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:52.673270    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ed12546-f37b-40a1-a340-3f22c4fd76ec-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-dt9tv\" (UID: \"2ed12546-f37b-40a1-a340-3f22c4fd76ec\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"
Aug 05 17:48:52.679024 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:52.679005    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7qqc\" (UniqueName: \"kubernetes.io/projected/2ed12546-f37b-40a1-a340-3f22c4fd76ec-kube-api-access-z7qqc\") pod \"leave-v2-kserve-556b4dddf9-dt9tv\" (UID: \"2ed12546-f37b-40a1-a340-3f22c4fd76ec\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"
Aug 05 17:48:52.719851 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:52.719826    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"
Aug 05 17:48:52.870894 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:52.870857    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"]
Aug 05 17:48:52.872983 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:48:52.872443    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed12546_f37b_40a1_a340_3f22c4fd76ec.slice/crio-11c45d177622124c95972eeba52f8d62d3bda9f7dfee5b27d5ea22ec42891640 WatchSource:0}: Error finding container 11c45d177622124c95972eeba52f8d62d3bda9f7dfee5b27d5ea22ec42891640: Status 404 returned error can't find the container with id 11c45d177622124c95972eeba52f8d62d3bda9f7dfee5b27d5ea22ec42891640
Aug 05 17:48:53.477140 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:53.477103    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv" event={"ID":"2ed12546-f37b-40a1-a340-3f22c4fd76ec","Type":"ContainerStarted","Data":"90221df1258213b905dcf5f20e1d164bd1211f32c22faa038d49988ba4474bfd"}
Aug 05 17:48:53.477511 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:53.477152    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv" event={"ID":"2ed12546-f37b-40a1-a340-3f22c4fd76ec","Type":"ContainerStarted","Data":"11c45d177622124c95972eeba52f8d62d3bda9f7dfee5b27d5ea22ec42891640"}
Aug 05 17:48:53.497642 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:53.497595    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv" podStartSLOduration=1.497580996 podStartE2EDuration="1.497580996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:48:53.495743944 +0000 UTC m=+2189.240378299" watchObservedRunningTime="2026-08-05 17:48:53.497580996 +0000 UTC m=+2189.242215325"
Aug 05 17:48:53.720404 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:53.720375    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"
Aug 05 17:48:56.731148 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:56.731122    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"
Aug 05 17:48:56.739095 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:56.739069    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"
Aug 05 17:48:57.511347 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:48:57.511316    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"
Aug 05 17:49:02.720924 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:49:02.720884    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"
Aug 05 17:49:02.734115 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:49:02.734091    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"
Aug 05 17:49:03.544772 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:49:03.544745    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"
Aug 05 17:50:04.656541 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:04.656505    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt"]
Aug 05 17:50:04.659713 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:04.659693    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt"
Aug 05 17:50:04.663405 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:04.663382    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-tvtm9\""
Aug 05 17:50:04.663542 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:04.663468    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Aug 05 17:50:04.672561 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:04.672539    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt"]
Aug 05 17:50:04.686594 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:04.686574    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9042d2a7-c199-4e93-94ea-46246c967ce9-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt\" (UID: \"9042d2a7-c199-4e93-94ea-46246c967ce9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt"
Aug 05 17:50:04.686692 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:04.686626    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tp2t\" (UniqueName: \"kubernetes.io/projected/9042d2a7-c199-4e93-94ea-46246c967ce9-kube-api-access-5tp2t\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt\" (UID: \"9042d2a7-c199-4e93-94ea-46246c967ce9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt"
Aug 05 17:50:04.787620 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:04.787588    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9042d2a7-c199-4e93-94ea-46246c967ce9-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt\" (UID: \"9042d2a7-c199-4e93-94ea-46246c967ce9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt"
Aug 05 17:50:04.787773 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:04.787649    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tp2t\" (UniqueName: \"kubernetes.io/projected/9042d2a7-c199-4e93-94ea-46246c967ce9-kube-api-access-5tp2t\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt\" (UID: \"9042d2a7-c199-4e93-94ea-46246c967ce9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt"
Aug 05 17:50:04.790068 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:04.790044    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9042d2a7-c199-4e93-94ea-46246c967ce9-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt\" (UID: \"9042d2a7-c199-4e93-94ea-46246c967ce9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt"
Aug 05 17:50:04.797593 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:04.797567    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tp2t\" (UniqueName: \"kubernetes.io/projected/9042d2a7-c199-4e93-94ea-46246c967ce9-kube-api-access-5tp2t\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt\" (UID: \"9042d2a7-c199-4e93-94ea-46246c967ce9\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt"
Aug 05 17:50:04.971035 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:04.970970    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt"
Aug 05 17:50:05.108734 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:05.108705    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt"]
Aug 05 17:50:05.110212 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:50:05.110181    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9042d2a7_c199_4e93_94ea_46246c967ce9.slice/crio-9b62039f83688a4c2ddcbdc9dbdbc3326643e95246dd84f2a43c414f3847ff04 WatchSource:0}: Error finding container 9b62039f83688a4c2ddcbdc9dbdbc3326643e95246dd84f2a43c414f3847ff04: Status 404 returned error can't find the container with id 9b62039f83688a4c2ddcbdc9dbdbc3326643e95246dd84f2a43c414f3847ff04
Aug 05 17:50:05.869273 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:05.869240    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt" event={"ID":"9042d2a7-c199-4e93-94ea-46246c967ce9","Type":"ContainerStarted","Data":"d03f109a36090580cf32a57a25b204d6757f06706a9074236f43f09e553b0710"}
Aug 05 17:50:05.869658 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:05.869302    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt"
Aug 05 17:50:05.869658 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:05.869320    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt" event={"ID":"9042d2a7-c199-4e93-94ea-46246c967ce9","Type":"ContainerStarted","Data":"9b62039f83688a4c2ddcbdc9dbdbc3326643e95246dd84f2a43c414f3847ff04"}
Aug 05 17:50:05.888680 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:05.888629    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt" podStartSLOduration=1.888613795 podStartE2EDuration="1.888613795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:50:05.886885894 +0000 UTC m=+2261.631520239" watchObservedRunningTime="2026-08-05 17:50:05.888613795 +0000 UTC m=+2261.633248122"
Aug 05 17:50:36.877814 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:36.877786    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt"
Aug 05 17:50:41.050352 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:41.050303    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt"]
Aug 05 17:50:46.877483 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:46.877444    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt" podUID="9042d2a7-c199-4e93-94ea-46246c967ce9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 05 17:50:47.923828 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:47.923794    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"]
Aug 05 17:50:48.688470 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:48.688441    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"]
Aug 05 17:50:56.071526 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:56.071468    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt" podUID="9042d2a7-c199-4e93-94ea-46246c967ce9" containerName="main" containerID="cri-o://d03f109a36090580cf32a57a25b204d6757f06706a9074236f43f09e553b0710" gracePeriod=45
Aug 05 17:50:56.320765 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:56.320739    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt"
Aug 05 17:50:56.451854 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:56.451756    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tp2t\" (UniqueName: \"kubernetes.io/projected/9042d2a7-c199-4e93-94ea-46246c967ce9-kube-api-access-5tp2t\") pod \"9042d2a7-c199-4e93-94ea-46246c967ce9\" (UID: \"9042d2a7-c199-4e93-94ea-46246c967ce9\") "
Aug 05 17:50:56.451854 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:56.451816    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9042d2a7-c199-4e93-94ea-46246c967ce9-tls-certs\") pod \"9042d2a7-c199-4e93-94ea-46246c967ce9\" (UID: \"9042d2a7-c199-4e93-94ea-46246c967ce9\") "
Aug 05 17:50:56.454180 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:56.454150    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9042d2a7-c199-4e93-94ea-46246c967ce9-kube-api-access-5tp2t" (OuterVolumeSpecName: "kube-api-access-5tp2t") pod "9042d2a7-c199-4e93-94ea-46246c967ce9" (UID: "9042d2a7-c199-4e93-94ea-46246c967ce9"). InnerVolumeSpecName "kube-api-access-5tp2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:50:56.454304 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:56.454228    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9042d2a7-c199-4e93-94ea-46246c967ce9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9042d2a7-c199-4e93-94ea-46246c967ce9" (UID: "9042d2a7-c199-4e93-94ea-46246c967ce9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:50:56.553208 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:56.553173    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tp2t\" (UniqueName: \"kubernetes.io/projected/9042d2a7-c199-4e93-94ea-46246c967ce9-kube-api-access-5tp2t\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:50:56.553208 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:56.553203    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9042d2a7-c199-4e93-94ea-46246c967ce9-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:50:57.158074 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:57.158036    2561 generic.go:356] "Generic (PLEG): container finished" podID="9042d2a7-c199-4e93-94ea-46246c967ce9" containerID="d03f109a36090580cf32a57a25b204d6757f06706a9074236f43f09e553b0710" exitCode=0
Aug 05 17:50:57.158529 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:57.158094    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt"
Aug 05 17:50:57.158529 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:57.158125    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt" event={"ID":"9042d2a7-c199-4e93-94ea-46246c967ce9","Type":"ContainerDied","Data":"d03f109a36090580cf32a57a25b204d6757f06706a9074236f43f09e553b0710"}
Aug 05 17:50:57.158529 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:57.158179    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt" event={"ID":"9042d2a7-c199-4e93-94ea-46246c967ce9","Type":"ContainerDied","Data":"9b62039f83688a4c2ddcbdc9dbdbc3326643e95246dd84f2a43c414f3847ff04"}
Aug 05 17:50:57.158529 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:57.158195    2561 scope.go:117] "RemoveContainer" containerID="d03f109a36090580cf32a57a25b204d6757f06706a9074236f43f09e553b0710"
Aug 05 17:50:57.167440 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:57.167420    2561 scope.go:117] "RemoveContainer" containerID="d03f109a36090580cf32a57a25b204d6757f06706a9074236f43f09e553b0710"
Aug 05 17:50:57.167697 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:50:57.167677    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d03f109a36090580cf32a57a25b204d6757f06706a9074236f43f09e553b0710\": container with ID starting with d03f109a36090580cf32a57a25b204d6757f06706a9074236f43f09e553b0710 not found: ID does not exist" containerID="d03f109a36090580cf32a57a25b204d6757f06706a9074236f43f09e553b0710"
Aug 05 17:50:57.167755 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:57.167706    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d03f109a36090580cf32a57a25b204d6757f06706a9074236f43f09e553b0710"} err="failed to get container status \"d03f109a36090580cf32a57a25b204d6757f06706a9074236f43f09e553b0710\": rpc error: code = NotFound desc = could not find container \"d03f109a36090580cf32a57a25b204d6757f06706a9074236f43f09e553b0710\": container with ID starting with d03f109a36090580cf32a57a25b204d6757f06706a9074236f43f09e553b0710 not found: ID does not exist"
Aug 05 17:50:57.175825 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:57.175803    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt"]
Aug 05 17:50:57.182654 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:57.182630    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche5hnlt"]
Aug 05 17:50:58.729785 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:50:58.729755    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9042d2a7-c199-4e93-94ea-46246c967ce9" path="/var/lib/kubelet/pods/9042d2a7-c199-4e93-94ea-46246c967ce9/volumes"
Aug 05 17:51:00.990208 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:00.990163    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"]
Aug 05 17:51:00.990694 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:00.990677    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9042d2a7-c199-4e93-94ea-46246c967ce9" containerName="main"
Aug 05 17:51:00.990743 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:00.990698    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9042d2a7-c199-4e93-94ea-46246c967ce9" containerName="main"
Aug 05 17:51:00.990784 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:00.990764    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9042d2a7-c199-4e93-94ea-46246c967ce9" containerName="main"
Aug 05 17:51:00.995488 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:00.995465    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"
Aug 05 17:51:00.999881 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:00.999858    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-f25g4\""
Aug 05 17:51:01.000045 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:00.999858    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v2-kserve-self-signed-certs\""
Aug 05 17:51:01.000045 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:00.999912    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\""
Aug 05 17:51:01.000045 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:00.999912    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\""
Aug 05 17:51:01.005417 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:01.005392    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"]
Aug 05 17:51:01.095172 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:01.095134    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb8dc525-2780-4db2-b6b2-fd06db2ec294-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-whmh7\" (UID: \"bb8dc525-2780-4db2-b6b2-fd06db2ec294\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"
Aug 05 17:51:01.095172 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:01.095173    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l95zf\" (UniqueName: \"kubernetes.io/projected/bb8dc525-2780-4db2-b6b2-fd06db2ec294-kube-api-access-l95zf\") pod \"tri-v2-kserve-dcb554bf5-whmh7\" (UID: \"bb8dc525-2780-4db2-b6b2-fd06db2ec294\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"
Aug 05 17:51:01.095383 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:01.095247    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb8dc525-2780-4db2-b6b2-fd06db2ec294-model-cache\") pod \"tri-v2-kserve-dcb554bf5-whmh7\" (UID: \"bb8dc525-2780-4db2-b6b2-fd06db2ec294\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"
Aug 05 17:51:01.095383 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:01.095281    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb8dc525-2780-4db2-b6b2-fd06db2ec294-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-whmh7\" (UID: \"bb8dc525-2780-4db2-b6b2-fd06db2ec294\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"
Aug 05 17:51:01.095383 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:01.095370    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb8dc525-2780-4db2-b6b2-fd06db2ec294-home\") pod \"tri-v2-kserve-dcb554bf5-whmh7\" (UID: \"bb8dc525-2780-4db2-b6b2-fd06db2ec294\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"
Aug 05 17:51:01.095484 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:01.095405    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb8dc525-2780-4db2-b6b2-fd06db2ec294-dshm\") pod \"tri-v2-kserve-dcb554bf5-whmh7\" (UID: \"bb8dc525-2780-4db2-b6b2-fd06db2ec294\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"
Aug 05 17:51:01.196217 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:01.196183    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb8dc525-2780-4db2-b6b2-fd06db2ec294-dshm\") pod \"tri-v2-kserve-dcb554bf5-whmh7\" (UID: \"bb8dc525-2780-4db2-b6b2-fd06db2ec294\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"
Aug 05 17:51:01.196415 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:01.196232    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb8dc525-2780-4db2-b6b2-fd06db2ec294-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-whmh7\" (UID: \"bb8dc525-2780-4db2-b6b2-fd06db2ec294\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"
Aug 05 17:51:01.196415 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:01.196253    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l95zf\" (UniqueName: \"kubernetes.io/projected/bb8dc525-2780-4db2-b6b2-fd06db2ec294-kube-api-access-l95zf\") pod \"tri-v2-kserve-dcb554bf5-whmh7\" (UID: \"bb8dc525-2780-4db2-b6b2-fd06db2ec294\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"
Aug 05 17:51:01.196415 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:01.196285    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb8dc525-2780-4db2-b6b2-fd06db2ec294-model-cache\") pod \"tri-v2-kserve-dcb554bf5-whmh7\" (UID: \"bb8dc525-2780-4db2-b6b2-fd06db2ec294\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"
Aug 05 17:51:01.196415 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:01.196302    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb8dc525-2780-4db2-b6b2-fd06db2ec294-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-whmh7\" (UID: \"bb8dc525-2780-4db2-b6b2-fd06db2ec294\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"
Aug 05 17:51:01.196635 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:01.196472    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb8dc525-2780-4db2-b6b2-fd06db2ec294-home\") pod \"tri-v2-kserve-dcb554bf5-whmh7\" (UID: \"bb8dc525-2780-4db2-b6b2-fd06db2ec294\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"
Aug 05 17:51:01.196684 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:01.196631    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb8dc525-2780-4db2-b6b2-fd06db2ec294-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-whmh7\" (UID: \"bb8dc525-2780-4db2-b6b2-fd06db2ec294\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"
Aug 05 17:51:01.196738 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:01.196718    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb8dc525-2780-4db2-b6b2-fd06db2ec294-model-cache\") pod \"tri-v2-kserve-dcb554bf5-whmh7\" (UID: \"bb8dc525-2780-4db2-b6b2-fd06db2ec294\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"
Aug 05 17:51:01.196796 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:01.196749    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb8dc525-2780-4db2-b6b2-fd06db2ec294-home\") pod \"tri-v2-kserve-dcb554bf5-whmh7\" (UID: \"bb8dc525-2780-4db2-b6b2-fd06db2ec294\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"
Aug 05 17:51:01.198591 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:01.198568    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb8dc525-2780-4db2-b6b2-fd06db2ec294-dshm\") pod \"tri-v2-kserve-dcb554bf5-whmh7\" (UID: \"bb8dc525-2780-4db2-b6b2-fd06db2ec294\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"
Aug 05 17:51:01.198786 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:01.198768    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb8dc525-2780-4db2-b6b2-fd06db2ec294-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-whmh7\" (UID: \"bb8dc525-2780-4db2-b6b2-fd06db2ec294\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"
Aug 05 17:51:01.205369 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:01.205340    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l95zf\" (UniqueName: \"kubernetes.io/projected/bb8dc525-2780-4db2-b6b2-fd06db2ec294-kube-api-access-l95zf\") pod \"tri-v2-kserve-dcb554bf5-whmh7\" (UID: \"bb8dc525-2780-4db2-b6b2-fd06db2ec294\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"
Aug 05 17:51:01.307593 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:01.307548    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"
Aug 05 17:51:01.457692 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:51:01.457663    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb8dc525_2780_4db2_b6b2_fd06db2ec294.slice/crio-aef8f24db4b379e947ee4c1a7f5138b8faee6b405d4bb1fd239996b97121a706 WatchSource:0}: Error finding container aef8f24db4b379e947ee4c1a7f5138b8faee6b405d4bb1fd239996b97121a706: Status 404 returned error can't find the container with id aef8f24db4b379e947ee4c1a7f5138b8faee6b405d4bb1fd239996b97121a706
Aug 05 17:51:01.457898 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:01.457870    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"]
Aug 05 17:51:02.188543 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:02.188459    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7" event={"ID":"bb8dc525-2780-4db2-b6b2-fd06db2ec294","Type":"ContainerStarted","Data":"d1ac9ad4eba196511015f09e3c05dabbc6b4374e33c44818cbf97f803446a6aa"}
Aug 05 17:51:02.188543 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:02.188512    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7" event={"ID":"bb8dc525-2780-4db2-b6b2-fd06db2ec294","Type":"ContainerStarted","Data":"aef8f24db4b379e947ee4c1a7f5138b8faee6b405d4bb1fd239996b97121a706"}
Aug 05 17:51:02.211931 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:02.211880    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7" podStartSLOduration=2.21186412 podStartE2EDuration="2.21186412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:51:02.210194047 +0000 UTC m=+2317.954828374" watchObservedRunningTime="2026-08-05 17:51:02.21186412 +0000 UTC m=+2317.956498466"
Aug 05 17:51:02.308049 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:02.308019    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"
Aug 05 17:51:02.806208 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:02.806174    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"]
Aug 05 17:51:02.815716 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:51:02.815681    2561 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 05 17:51:02.815716 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:51:02.815716 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:51:02.815716 ip-10-0-135-151 kubenswrapper[2561]:  > containerID="b174415b97a33cb1bced362886e2095bc4997bb54ed16182f5f669c83f8e5ef0" cmd=["/bin/sleep","15"]
Aug 05 17:51:02.816114 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:02.815731    2561 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf" message="" err=<
Aug 05 17:51:02.816114 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:51:02.816114 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:51:02.816114 ip-10-0-135-151 kubenswrapper[2561]:  >
Aug 05 17:51:02.816114 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:51:02.815743    2561 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 05 17:51:02.816114 ip-10-0-135-151 kubenswrapper[2561]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 05 17:51:02.816114 ip-10-0-135-151 kubenswrapper[2561]:         , stdout: , stderr: , exit code -1
Aug 05 17:51:02.816114 ip-10-0-135-151 kubenswrapper[2561]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf" podUID="e9264de5-8ee1-4f1b-9f78-1c6315aa37b4" containerName="main" containerID="cri-o://b174415b97a33cb1bced362886e2095bc4997bb54ed16182f5f669c83f8e5ef0"
Aug 05 17:51:02.816114 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:02.815777    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf" podUID="e9264de5-8ee1-4f1b-9f78-1c6315aa37b4" containerName="main" containerID="cri-o://b174415b97a33cb1bced362886e2095bc4997bb54ed16182f5f669c83f8e5ef0" gracePeriod=60
Aug 05 17:51:02.942074 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:02.942036    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2" podUID="f9249e33-6fe2-48a9-b3d6-0ea0e8038b64" containerName="main" containerID="cri-o://af6f444734a465e674235a76eaa713508a6a1cf16a74d6b3e7f6e19f941e67b5" gracePeriod=45
Aug 05 17:51:03.195556 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.195523    2561 generic.go:356] "Generic (PLEG): container finished" podID="e9264de5-8ee1-4f1b-9f78-1c6315aa37b4" containerID="b174415b97a33cb1bced362886e2095bc4997bb54ed16182f5f669c83f8e5ef0" exitCode=0
Aug 05 17:51:03.195924 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.195586    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf" event={"ID":"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4","Type":"ContainerDied","Data":"b174415b97a33cb1bced362886e2095bc4997bb54ed16182f5f669c83f8e5ef0"}
Aug 05 17:51:03.197452 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.197431    2561 generic.go:356] "Generic (PLEG): container finished" podID="f9249e33-6fe2-48a9-b3d6-0ea0e8038b64" containerID="af6f444734a465e674235a76eaa713508a6a1cf16a74d6b3e7f6e19f941e67b5" exitCode=0
Aug 05 17:51:03.197569 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.197507    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2" event={"ID":"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64","Type":"ContainerDied","Data":"af6f444734a465e674235a76eaa713508a6a1cf16a74d6b3e7f6e19f941e67b5"}
Aug 05 17:51:03.217369 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.217349    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"
Aug 05 17:51:03.220234 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.220217    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"
Aug 05 17:51:03.316033 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.315996    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-model-cache\") pod \"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4\" (UID: \"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4\") "
Aug 05 17:51:03.316206 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.316041    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsh5n\" (UniqueName: \"kubernetes.io/projected/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-kube-api-access-xsh5n\") pod \"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4\" (UID: \"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4\") "
Aug 05 17:51:03.316206 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.316085    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-home\") pod \"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4\" (UID: \"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4\") "
Aug 05 17:51:03.316206 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.316143    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-tmp-dir\") pod \"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4\" (UID: \"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4\") "
Aug 05 17:51:03.316206 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.316191    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-model-cache\") pod \"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64\" (UID: \"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64\") "
Aug 05 17:51:03.316412 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.316221    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-tls-certs\") pod \"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4\" (UID: \"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4\") "
Aug 05 17:51:03.316412 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.316268    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-model-cache" (OuterVolumeSpecName: "model-cache") pod "e9264de5-8ee1-4f1b-9f78-1c6315aa37b4" (UID: "e9264de5-8ee1-4f1b-9f78-1c6315aa37b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:51:03.316412 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.316280    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-home\") pod \"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64\" (UID: \"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64\") "
Aug 05 17:51:03.316412 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.316358    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-tls-certs\") pod \"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64\" (UID: \"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64\") "
Aug 05 17:51:03.316412 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.316374    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-home" (OuterVolumeSpecName: "home") pod "e9264de5-8ee1-4f1b-9f78-1c6315aa37b4" (UID: "e9264de5-8ee1-4f1b-9f78-1c6315aa37b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:51:03.316412 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.316395    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mfmq\" (UniqueName: \"kubernetes.io/projected/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-kube-api-access-7mfmq\") pod \"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64\" (UID: \"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64\") "
Aug 05 17:51:03.316719 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.316448    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-dshm\") pod \"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4\" (UID: \"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4\") "
Aug 05 17:51:03.316719 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.316474    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-dshm\") pod \"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64\" (UID: \"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64\") "
Aug 05 17:51:03.316719 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.316501    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-tmp-dir\") pod \"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64\" (UID: \"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64\") "
Aug 05 17:51:03.316719 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.316425    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e9264de5-8ee1-4f1b-9f78-1c6315aa37b4" (UID: "e9264de5-8ee1-4f1b-9f78-1c6315aa37b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:51:03.316719 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.316516    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-model-cache" (OuterVolumeSpecName: "model-cache") pod "f9249e33-6fe2-48a9-b3d6-0ea0e8038b64" (UID: "f9249e33-6fe2-48a9-b3d6-0ea0e8038b64"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:51:03.316719 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.316495    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-home" (OuterVolumeSpecName: "home") pod "f9249e33-6fe2-48a9-b3d6-0ea0e8038b64" (UID: "f9249e33-6fe2-48a9-b3d6-0ea0e8038b64"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:51:03.317076 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.316859    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:51:03.317076 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.316885    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:51:03.317076 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.316899    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:51:03.317076 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.316949    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:51:03.317076 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.317038    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:51:03.317605 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.317579    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f9249e33-6fe2-48a9-b3d6-0ea0e8038b64" (UID: "f9249e33-6fe2-48a9-b3d6-0ea0e8038b64"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:51:03.319262 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.319226    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-kube-api-access-xsh5n" (OuterVolumeSpecName: "kube-api-access-xsh5n") pod "e9264de5-8ee1-4f1b-9f78-1c6315aa37b4" (UID: "e9264de5-8ee1-4f1b-9f78-1c6315aa37b4"). InnerVolumeSpecName "kube-api-access-xsh5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:51:03.319262 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.319223    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-dshm" (OuterVolumeSpecName: "dshm") pod "f9249e33-6fe2-48a9-b3d6-0ea0e8038b64" (UID: "f9249e33-6fe2-48a9-b3d6-0ea0e8038b64"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:51:03.319262 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.319243    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-kube-api-access-7mfmq" (OuterVolumeSpecName: "kube-api-access-7mfmq") pod "f9249e33-6fe2-48a9-b3d6-0ea0e8038b64" (UID: "f9249e33-6fe2-48a9-b3d6-0ea0e8038b64"). InnerVolumeSpecName "kube-api-access-7mfmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:51:03.319530 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.319508    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f9249e33-6fe2-48a9-b3d6-0ea0e8038b64" (UID: "f9249e33-6fe2-48a9-b3d6-0ea0e8038b64"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:51:03.319600 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.319514    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e9264de5-8ee1-4f1b-9f78-1c6315aa37b4" (UID: "e9264de5-8ee1-4f1b-9f78-1c6315aa37b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:51:03.319640 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.319595    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-dshm" (OuterVolumeSpecName: "dshm") pod "e9264de5-8ee1-4f1b-9f78-1c6315aa37b4" (UID: "e9264de5-8ee1-4f1b-9f78-1c6315aa37b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:51:03.418123 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.418043    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:51:03.418123 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.418070    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:51:03.418123 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.418080    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mfmq\" (UniqueName: \"kubernetes.io/projected/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-kube-api-access-7mfmq\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:51:03.418123 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.418089    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:51:03.418123 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.418097    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:51:03.418123 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.418104    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:51:03.418123 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.418113    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsh5n\" (UniqueName: \"kubernetes.io/projected/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4-kube-api-access-xsh5n\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:51:03.708845 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.708726    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv" podUID="2ed12546-f37b-40a1-a340-3f22c4fd76ec" containerName="main" containerID="cri-o://90221df1258213b905dcf5f20e1d164bd1211f32c22faa038d49988ba4474bfd" gracePeriod=45
Aug 05 17:51:03.962920 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:03.962862    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"
Aug 05 17:51:04.025074 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.025043    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7qqc\" (UniqueName: \"kubernetes.io/projected/2ed12546-f37b-40a1-a340-3f22c4fd76ec-kube-api-access-z7qqc\") pod \"2ed12546-f37b-40a1-a340-3f22c4fd76ec\" (UID: \"2ed12546-f37b-40a1-a340-3f22c4fd76ec\") "
Aug 05 17:51:04.025261 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.025092    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ed12546-f37b-40a1-a340-3f22c4fd76ec-tls-certs\") pod \"2ed12546-f37b-40a1-a340-3f22c4fd76ec\" (UID: \"2ed12546-f37b-40a1-a340-3f22c4fd76ec\") "
Aug 05 17:51:04.025261 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.025135    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ed12546-f37b-40a1-a340-3f22c4fd76ec-model-cache\") pod \"2ed12546-f37b-40a1-a340-3f22c4fd76ec\" (UID: \"2ed12546-f37b-40a1-a340-3f22c4fd76ec\") "
Aug 05 17:51:04.025261 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.025174    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ed12546-f37b-40a1-a340-3f22c4fd76ec-home\") pod \"2ed12546-f37b-40a1-a340-3f22c4fd76ec\" (UID: \"2ed12546-f37b-40a1-a340-3f22c4fd76ec\") "
Aug 05 17:51:04.025261 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.025194    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ed12546-f37b-40a1-a340-3f22c4fd76ec-tmp-dir\") pod \"2ed12546-f37b-40a1-a340-3f22c4fd76ec\" (UID: \"2ed12546-f37b-40a1-a340-3f22c4fd76ec\") "
Aug 05 17:51:04.025261 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.025255    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ed12546-f37b-40a1-a340-3f22c4fd76ec-dshm\") pod \"2ed12546-f37b-40a1-a340-3f22c4fd76ec\" (UID: \"2ed12546-f37b-40a1-a340-3f22c4fd76ec\") "
Aug 05 17:51:04.026138 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.025449    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed12546-f37b-40a1-a340-3f22c4fd76ec-model-cache" (OuterVolumeSpecName: "model-cache") pod "2ed12546-f37b-40a1-a340-3f22c4fd76ec" (UID: "2ed12546-f37b-40a1-a340-3f22c4fd76ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:51:04.026138 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.025481    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed12546-f37b-40a1-a340-3f22c4fd76ec-home" (OuterVolumeSpecName: "home") pod "2ed12546-f37b-40a1-a340-3f22c4fd76ec" (UID: "2ed12546-f37b-40a1-a340-3f22c4fd76ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:51:04.026138 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.025514    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed12546-f37b-40a1-a340-3f22c4fd76ec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2ed12546-f37b-40a1-a340-3f22c4fd76ec" (UID: "2ed12546-f37b-40a1-a340-3f22c4fd76ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:51:04.026138 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.025633    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ed12546-f37b-40a1-a340-3f22c4fd76ec-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:51:04.026138 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.025653    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ed12546-f37b-40a1-a340-3f22c4fd76ec-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:51:04.026138 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.025667    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ed12546-f37b-40a1-a340-3f22c4fd76ec-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:51:04.027407 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.027376    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed12546-f37b-40a1-a340-3f22c4fd76ec-dshm" (OuterVolumeSpecName: "dshm") pod "2ed12546-f37b-40a1-a340-3f22c4fd76ec" (UID: "2ed12546-f37b-40a1-a340-3f22c4fd76ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:51:04.027407 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.027388    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ed12546-f37b-40a1-a340-3f22c4fd76ec-kube-api-access-z7qqc" (OuterVolumeSpecName: "kube-api-access-z7qqc") pod "2ed12546-f37b-40a1-a340-3f22c4fd76ec" (UID: "2ed12546-f37b-40a1-a340-3f22c4fd76ec"). InnerVolumeSpecName "kube-api-access-z7qqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:51:04.027545 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.027415    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ed12546-f37b-40a1-a340-3f22c4fd76ec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2ed12546-f37b-40a1-a340-3f22c4fd76ec" (UID: "2ed12546-f37b-40a1-a340-3f22c4fd76ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:51:04.126444 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.126406    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ed12546-f37b-40a1-a340-3f22c4fd76ec-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:51:04.126444 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.126438    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7qqc\" (UniqueName: \"kubernetes.io/projected/2ed12546-f37b-40a1-a340-3f22c4fd76ec-kube-api-access-z7qqc\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:51:04.126444 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.126452    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ed12546-f37b-40a1-a340-3f22c4fd76ec-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:51:04.204054 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.204017    2561 generic.go:356] "Generic (PLEG): container finished" podID="2ed12546-f37b-40a1-a340-3f22c4fd76ec" containerID="90221df1258213b905dcf5f20e1d164bd1211f32c22faa038d49988ba4474bfd" exitCode=0
Aug 05 17:51:04.204523 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.204081    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"
Aug 05 17:51:04.204523 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.204106    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv" event={"ID":"2ed12546-f37b-40a1-a340-3f22c4fd76ec","Type":"ContainerDied","Data":"90221df1258213b905dcf5f20e1d164bd1211f32c22faa038d49988ba4474bfd"}
Aug 05 17:51:04.204523 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.204165    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv" event={"ID":"2ed12546-f37b-40a1-a340-3f22c4fd76ec","Type":"ContainerDied","Data":"11c45d177622124c95972eeba52f8d62d3bda9f7dfee5b27d5ea22ec42891640"}
Aug 05 17:51:04.204523 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.204196    2561 scope.go:117] "RemoveContainer" containerID="90221df1258213b905dcf5f20e1d164bd1211f32c22faa038d49988ba4474bfd"
Aug 05 17:51:04.205586 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.205563    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"
Aug 05 17:51:04.205720 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.205620    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2" event={"ID":"f9249e33-6fe2-48a9-b3d6-0ea0e8038b64","Type":"ContainerDied","Data":"b51cb355aa12bfa25a64e854bdf29965e30fe8c63bce0a2b0d85c2778cef84c6"}
Aug 05 17:51:04.207266 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.207243    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"
Aug 05 17:51:04.207266 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.207242    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf" event={"ID":"e9264de5-8ee1-4f1b-9f78-1c6315aa37b4","Type":"ContainerDied","Data":"5fb2dcc6cc39fd5253c837e6c7005c77c5041fb214c71c9351af3d0cfb8677ad"}
Aug 05 17:51:04.219277 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.219125    2561 scope.go:117] "RemoveContainer" containerID="90221df1258213b905dcf5f20e1d164bd1211f32c22faa038d49988ba4474bfd"
Aug 05 17:51:04.219472 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:51:04.219445    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90221df1258213b905dcf5f20e1d164bd1211f32c22faa038d49988ba4474bfd\": container with ID starting with 90221df1258213b905dcf5f20e1d164bd1211f32c22faa038d49988ba4474bfd not found: ID does not exist" containerID="90221df1258213b905dcf5f20e1d164bd1211f32c22faa038d49988ba4474bfd"
Aug 05 17:51:04.219557 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.219486    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90221df1258213b905dcf5f20e1d164bd1211f32c22faa038d49988ba4474bfd"} err="failed to get container status \"90221df1258213b905dcf5f20e1d164bd1211f32c22faa038d49988ba4474bfd\": rpc error: code = NotFound desc = could not find container \"90221df1258213b905dcf5f20e1d164bd1211f32c22faa038d49988ba4474bfd\": container with ID starting with 90221df1258213b905dcf5f20e1d164bd1211f32c22faa038d49988ba4474bfd not found: ID does not exist"
Aug 05 17:51:04.219557 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.219502    2561 scope.go:117] "RemoveContainer" containerID="af6f444734a465e674235a76eaa713508a6a1cf16a74d6b3e7f6e19f941e67b5"
Aug 05 17:51:04.227213 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.227197    2561 scope.go:117] "RemoveContainer" containerID="b174415b97a33cb1bced362886e2095bc4997bb54ed16182f5f669c83f8e5ef0"
Aug 05 17:51:04.233556 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.233535    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"]
Aug 05 17:51:04.237363 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.237345    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-dt9tv"]
Aug 05 17:51:04.248488 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.248466    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"]
Aug 05 17:51:04.253479 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.253461    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-fzjdf"]
Aug 05 17:51:04.263875 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.263854    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"]
Aug 05 17:51:04.267640 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.267617    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-tt7c2"]
Aug 05 17:51:04.729749 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.729716    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ed12546-f37b-40a1-a340-3f22c4fd76ec" path="/var/lib/kubelet/pods/2ed12546-f37b-40a1-a340-3f22c4fd76ec/volumes"
Aug 05 17:51:04.730136 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.730121    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9264de5-8ee1-4f1b-9f78-1c6315aa37b4" path="/var/lib/kubelet/pods/e9264de5-8ee1-4f1b-9f78-1c6315aa37b4/volumes"
Aug 05 17:51:04.730461 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:04.730449    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9249e33-6fe2-48a9-b3d6-0ea0e8038b64" path="/var/lib/kubelet/pods/f9249e33-6fe2-48a9-b3d6-0ea0e8038b64/volumes"
Aug 05 17:51:11.308822 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:11.308780    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"
Aug 05 17:51:11.321071 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:11.321045    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"
Aug 05 17:51:12.270123 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:12.270096    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"
Aug 05 17:51:17.897575 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:17.897496    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"]
Aug 05 17:51:17.897946 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:17.897848    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9249e33-6fe2-48a9-b3d6-0ea0e8038b64" containerName="main"
Aug 05 17:51:17.897946 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:17.897859    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9249e33-6fe2-48a9-b3d6-0ea0e8038b64" containerName="main"
Aug 05 17:51:17.897946 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:17.897867    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ed12546-f37b-40a1-a340-3f22c4fd76ec" containerName="main"
Aug 05 17:51:17.897946 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:17.897874    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ed12546-f37b-40a1-a340-3f22c4fd76ec" containerName="main"
Aug 05 17:51:17.897946 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:17.897893    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9264de5-8ee1-4f1b-9f78-1c6315aa37b4" containerName="main"
Aug 05 17:51:17.897946 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:17.897899    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9264de5-8ee1-4f1b-9f78-1c6315aa37b4" containerName="main"
Aug 05 17:51:17.898250 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:17.897983    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9264de5-8ee1-4f1b-9f78-1c6315aa37b4" containerName="main"
Aug 05 17:51:17.898250 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:17.897998    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9249e33-6fe2-48a9-b3d6-0ea0e8038b64" containerName="main"
Aug 05 17:51:17.898250 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:17.898007    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ed12546-f37b-40a1-a340-3f22c4fd76ec" containerName="main"
Aug 05 17:51:17.902893 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:17.902872    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"
Aug 05 17:51:17.905315 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:17.905296    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v3-kserve-self-signed-certs\""
Aug 05 17:51:17.912921 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:17.912896    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"]
Aug 05 17:51:18.046879 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:18.046844    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-5xfwl\" (UID: \"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"
Aug 05 17:51:18.046879 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:18.046880    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-tls-certs\") pod \"tri-v3-kserve-798c88c66c-5xfwl\" (UID: \"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"
Aug 05 17:51:18.047124 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:18.046905    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5gj6\" (UniqueName: \"kubernetes.io/projected/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-kube-api-access-z5gj6\") pod \"tri-v3-kserve-798c88c66c-5xfwl\" (UID: \"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"
Aug 05 17:51:18.047124 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:18.046980    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-dshm\") pod \"tri-v3-kserve-798c88c66c-5xfwl\" (UID: \"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"
Aug 05 17:51:18.047124 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:18.047034    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-home\") pod \"tri-v3-kserve-798c88c66c-5xfwl\" (UID: \"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"
Aug 05 17:51:18.047124 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:18.047085    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-model-cache\") pod \"tri-v3-kserve-798c88c66c-5xfwl\" (UID: \"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"
Aug 05 17:51:18.148142 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:18.148058    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5gj6\" (UniqueName: \"kubernetes.io/projected/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-kube-api-access-z5gj6\") pod \"tri-v3-kserve-798c88c66c-5xfwl\" (UID: \"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"
Aug 05 17:51:18.148142 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:18.148107    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-dshm\") pod \"tri-v3-kserve-798c88c66c-5xfwl\" (UID: \"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"
Aug 05 17:51:18.148366 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:18.148148    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-home\") pod \"tri-v3-kserve-798c88c66c-5xfwl\" (UID: \"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"
Aug 05 17:51:18.148366 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:18.148203    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-model-cache\") pod \"tri-v3-kserve-798c88c66c-5xfwl\" (UID: \"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"
Aug 05 17:51:18.148366 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:18.148253    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-5xfwl\" (UID: \"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"
Aug 05 17:51:18.148366 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:18.148286    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-tls-certs\") pod \"tri-v3-kserve-798c88c66c-5xfwl\" (UID: \"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"
Aug 05 17:51:18.148647 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:18.148626    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-home\") pod \"tri-v3-kserve-798c88c66c-5xfwl\" (UID: \"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"
Aug 05 17:51:18.148731 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:18.148692    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-model-cache\") pod \"tri-v3-kserve-798c88c66c-5xfwl\" (UID: \"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"
Aug 05 17:51:18.148828 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:18.148789    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-5xfwl\" (UID: \"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"
Aug 05 17:51:18.150554 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:18.150534    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-dshm\") pod \"tri-v3-kserve-798c88c66c-5xfwl\" (UID: \"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"
Aug 05 17:51:18.150975 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:18.150933    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-tls-certs\") pod \"tri-v3-kserve-798c88c66c-5xfwl\" (UID: \"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"
Aug 05 17:51:18.156298 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:18.156277    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5gj6\" (UniqueName: \"kubernetes.io/projected/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-kube-api-access-z5gj6\") pod \"tri-v3-kserve-798c88c66c-5xfwl\" (UID: \"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"
Aug 05 17:51:18.213792 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:18.213760    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"
Aug 05 17:51:18.355121 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:18.355081    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"]
Aug 05 17:51:18.355984 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:51:18.355938    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0b6821_bb7b_4c01_82ab_675d6c88dbfd.slice/crio-1eae7d90ce5f2852ef6044d597b39c56609070be3689f309dd9560c1b2c2fba4 WatchSource:0}: Error finding container 1eae7d90ce5f2852ef6044d597b39c56609070be3689f309dd9560c1b2c2fba4: Status 404 returned error can't find the container with id 1eae7d90ce5f2852ef6044d597b39c56609070be3689f309dd9560c1b2c2fba4
Aug 05 17:51:19.298454 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:19.298416    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl" event={"ID":"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd","Type":"ContainerStarted","Data":"ac3dceaf765257ec9937481259404fde93984a00f82e0e4fd52ea31e7259293d"}
Aug 05 17:51:19.298913 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:19.298496    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl" event={"ID":"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd","Type":"ContainerStarted","Data":"1eae7d90ce5f2852ef6044d597b39c56609070be3689f309dd9560c1b2c2fba4"}
Aug 05 17:51:19.318350 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:19.318300    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl" podStartSLOduration=2.318284363 podStartE2EDuration="2.318284363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:51:19.316624529 +0000 UTC m=+2335.061258858" watchObservedRunningTime="2026-08-05 17:51:19.318284363 +0000 UTC m=+2335.062918689"
Aug 05 17:51:20.214352 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:20.214322    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"
Aug 05 17:51:28.214318 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:28.214277    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"
Aug 05 17:51:28.226920 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:28.226890    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"
Aug 05 17:51:28.366228 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:28.366197    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"
Aug 05 17:51:58.534555 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:58.534516    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89"]
Aug 05 17:51:58.539768 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:58.539743    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89"
Aug 05 17:51:58.543405 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:58.543379    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-jjmgx\""
Aug 05 17:51:58.543598 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:58.543381    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Aug 05 17:51:58.543598 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:58.543493    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Aug 05 17:51:58.543598 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:58.543447    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-54kk4\""
Aug 05 17:51:58.543598 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:58.543442    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Aug 05 17:51:58.547652 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:58.547600    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89"]
Aug 05 17:51:58.677140 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:58.677106    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cec9a23-caf7-4a40-bb63-84fdca30af40-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89\" (UID: \"6cec9a23-caf7-4a40-bb63-84fdca30af40\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89"
Aug 05 17:51:58.677352 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:58.677160    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj9s9\" (UniqueName: \"kubernetes.io/projected/6cec9a23-caf7-4a40-bb63-84fdca30af40-kube-api-access-gj9s9\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89\" (UID: \"6cec9a23-caf7-4a40-bb63-84fdca30af40\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89"
Aug 05 17:51:58.778689 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:58.778649    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cec9a23-caf7-4a40-bb63-84fdca30af40-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89\" (UID: \"6cec9a23-caf7-4a40-bb63-84fdca30af40\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89"
Aug 05 17:51:58.778877 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:58.778708    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj9s9\" (UniqueName: \"kubernetes.io/projected/6cec9a23-caf7-4a40-bb63-84fdca30af40-kube-api-access-gj9s9\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89\" (UID: \"6cec9a23-caf7-4a40-bb63-84fdca30af40\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89"
Aug 05 17:51:58.781182 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:58.781158    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cec9a23-caf7-4a40-bb63-84fdca30af40-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89\" (UID: \"6cec9a23-caf7-4a40-bb63-84fdca30af40\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89"
Aug 05 17:51:58.787864 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:58.787800    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj9s9\" (UniqueName: \"kubernetes.io/projected/6cec9a23-caf7-4a40-bb63-84fdca30af40-kube-api-access-gj9s9\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89\" (UID: \"6cec9a23-caf7-4a40-bb63-84fdca30af40\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89"
Aug 05 17:51:58.852779 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:58.852742    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89"
Aug 05 17:51:58.988962 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:58.988922    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89"]
Aug 05 17:51:58.991060 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:51:58.991028    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cec9a23_caf7_4a40_bb63_84fdca30af40.slice/crio-6ebe672e524370f4ed8b25274ca32315ef088e3eae0632a7a960cf515f154df0 WatchSource:0}: Error finding container 6ebe672e524370f4ed8b25274ca32315ef088e3eae0632a7a960cf515f154df0: Status 404 returned error can't find the container with id 6ebe672e524370f4ed8b25274ca32315ef088e3eae0632a7a960cf515f154df0
Aug 05 17:51:59.521426 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:59.521324    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89" event={"ID":"6cec9a23-caf7-4a40-bb63-84fdca30af40","Type":"ContainerStarted","Data":"5f329a265073acacdb44668a3b08fa1ff7e02d845a6f2bda46c5b98020251f77"}
Aug 05 17:51:59.521426 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:59.521394    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89"
Aug 05 17:51:59.521426 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:59.521417    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89" event={"ID":"6cec9a23-caf7-4a40-bb63-84fdca30af40","Type":"ContainerStarted","Data":"6ebe672e524370f4ed8b25274ca32315ef088e3eae0632a7a960cf515f154df0"}
Aug 05 17:51:59.539995 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:51:59.539925    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89" podStartSLOduration=1.5399109549999999 podStartE2EDuration="1.539910955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:51:59.538353789 +0000 UTC m=+2375.282988119" watchObservedRunningTime="2026-08-05 17:51:59.539910955 +0000 UTC m=+2375.284545281"
Aug 05 17:52:30.528824 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:30.528792    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89"
Aug 05 17:52:41.793585 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:41.793547    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"]
Aug 05 17:52:41.798697 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:41.798669    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"]
Aug 05 17:52:56.812844 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:56.812735    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7" podUID="bb8dc525-2780-4db2-b6b2-fd06db2ec294" containerName="main" containerID="cri-o://d1ac9ad4eba196511015f09e3c05dabbc6b4374e33c44818cbf97f803446a6aa" gracePeriod=45
Aug 05 17:52:56.817363 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:56.817297    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl" podUID="ad0b6821-bb7b-4c01-82ab-675d6c88dbfd" containerName="main" containerID="cri-o://ac3dceaf765257ec9937481259404fde93984a00f82e0e4fd52ea31e7259293d" gracePeriod=45
Aug 05 17:52:57.093510 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.093480    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"
Aug 05 17:52:57.096525 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.096502    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"
Aug 05 17:52:57.214070 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.214039    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-tmp-dir\") pod \"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd\" (UID: \"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd\") "
Aug 05 17:52:57.214070 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.214076    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5gj6\" (UniqueName: \"kubernetes.io/projected/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-kube-api-access-z5gj6\") pod \"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd\" (UID: \"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd\") "
Aug 05 17:52:57.214315 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.214098    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-home\") pod \"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd\" (UID: \"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd\") "
Aug 05 17:52:57.214315 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.214124    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb8dc525-2780-4db2-b6b2-fd06db2ec294-tmp-dir\") pod \"bb8dc525-2780-4db2-b6b2-fd06db2ec294\" (UID: \"bb8dc525-2780-4db2-b6b2-fd06db2ec294\") "
Aug 05 17:52:57.214315 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.214139    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-tls-certs\") pod \"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd\" (UID: \"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd\") "
Aug 05 17:52:57.214315 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.214164    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb8dc525-2780-4db2-b6b2-fd06db2ec294-model-cache\") pod \"bb8dc525-2780-4db2-b6b2-fd06db2ec294\" (UID: \"bb8dc525-2780-4db2-b6b2-fd06db2ec294\") "
Aug 05 17:52:57.214315 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.214186    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-model-cache\") pod \"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd\" (UID: \"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd\") "
Aug 05 17:52:57.214315 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.214215    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l95zf\" (UniqueName: \"kubernetes.io/projected/bb8dc525-2780-4db2-b6b2-fd06db2ec294-kube-api-access-l95zf\") pod \"bb8dc525-2780-4db2-b6b2-fd06db2ec294\" (UID: \"bb8dc525-2780-4db2-b6b2-fd06db2ec294\") "
Aug 05 17:52:57.214315 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.214249    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb8dc525-2780-4db2-b6b2-fd06db2ec294-dshm\") pod \"bb8dc525-2780-4db2-b6b2-fd06db2ec294\" (UID: \"bb8dc525-2780-4db2-b6b2-fd06db2ec294\") "
Aug 05 17:52:57.214643 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.214369    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ad0b6821-bb7b-4c01-82ab-675d6c88dbfd" (UID: "ad0b6821-bb7b-4c01-82ab-675d6c88dbfd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:52:57.214643 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.214418    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-home" (OuterVolumeSpecName: "home") pod "ad0b6821-bb7b-4c01-82ab-675d6c88dbfd" (UID: "ad0b6821-bb7b-4c01-82ab-675d6c88dbfd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:52:57.214643 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.214420    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb8dc525-2780-4db2-b6b2-fd06db2ec294-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bb8dc525-2780-4db2-b6b2-fd06db2ec294" (UID: "bb8dc525-2780-4db2-b6b2-fd06db2ec294"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:52:57.214643 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.214459    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-dshm\") pod \"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd\" (UID: \"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd\") "
Aug 05 17:52:57.214643 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.214470    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb8dc525-2780-4db2-b6b2-fd06db2ec294-model-cache" (OuterVolumeSpecName: "model-cache") pod "bb8dc525-2780-4db2-b6b2-fd06db2ec294" (UID: "bb8dc525-2780-4db2-b6b2-fd06db2ec294"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:52:57.214643 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.214569    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-model-cache" (OuterVolumeSpecName: "model-cache") pod "ad0b6821-bb7b-4c01-82ab-675d6c88dbfd" (UID: "ad0b6821-bb7b-4c01-82ab-675d6c88dbfd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:52:57.214643 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.214589    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb8dc525-2780-4db2-b6b2-fd06db2ec294-tls-certs\") pod \"bb8dc525-2780-4db2-b6b2-fd06db2ec294\" (UID: \"bb8dc525-2780-4db2-b6b2-fd06db2ec294\") "
Aug 05 17:52:57.215064 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.214651    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb8dc525-2780-4db2-b6b2-fd06db2ec294-home\") pod \"bb8dc525-2780-4db2-b6b2-fd06db2ec294\" (UID: \"bb8dc525-2780-4db2-b6b2-fd06db2ec294\") "
Aug 05 17:52:57.215064 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.214926    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:52:57.215064 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.214947    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:52:57.215064 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.214994    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb8dc525-2780-4db2-b6b2-fd06db2ec294-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:52:57.215064 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.215007    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb8dc525-2780-4db2-b6b2-fd06db2ec294-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:52:57.215064 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.215019    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:52:57.215064 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.215026    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb8dc525-2780-4db2-b6b2-fd06db2ec294-home" (OuterVolumeSpecName: "home") pod "bb8dc525-2780-4db2-b6b2-fd06db2ec294" (UID: "bb8dc525-2780-4db2-b6b2-fd06db2ec294"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:52:57.216542 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.216485    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb8dc525-2780-4db2-b6b2-fd06db2ec294-dshm" (OuterVolumeSpecName: "dshm") pod "bb8dc525-2780-4db2-b6b2-fd06db2ec294" (UID: "bb8dc525-2780-4db2-b6b2-fd06db2ec294"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:52:57.216694 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.216671    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb8dc525-2780-4db2-b6b2-fd06db2ec294-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb8dc525-2780-4db2-b6b2-fd06db2ec294" (UID: "bb8dc525-2780-4db2-b6b2-fd06db2ec294"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:52:57.216766 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.216703    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-kube-api-access-z5gj6" (OuterVolumeSpecName: "kube-api-access-z5gj6") pod "ad0b6821-bb7b-4c01-82ab-675d6c88dbfd" (UID: "ad0b6821-bb7b-4c01-82ab-675d6c88dbfd"). InnerVolumeSpecName "kube-api-access-z5gj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:52:57.216766 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.216722    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb8dc525-2780-4db2-b6b2-fd06db2ec294-kube-api-access-l95zf" (OuterVolumeSpecName: "kube-api-access-l95zf") pod "bb8dc525-2780-4db2-b6b2-fd06db2ec294" (UID: "bb8dc525-2780-4db2-b6b2-fd06db2ec294"). InnerVolumeSpecName "kube-api-access-l95zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:52:57.216844 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.216818    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-dshm" (OuterVolumeSpecName: "dshm") pod "ad0b6821-bb7b-4c01-82ab-675d6c88dbfd" (UID: "ad0b6821-bb7b-4c01-82ab-675d6c88dbfd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:52:57.216924 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.216910    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ad0b6821-bb7b-4c01-82ab-675d6c88dbfd" (UID: "ad0b6821-bb7b-4c01-82ab-675d6c88dbfd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:52:57.315565 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.315524    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb8dc525-2780-4db2-b6b2-fd06db2ec294-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:52:57.315565 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.315557    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb8dc525-2780-4db2-b6b2-fd06db2ec294-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:52:57.315565 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.315567    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5gj6\" (UniqueName: \"kubernetes.io/projected/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-kube-api-access-z5gj6\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:52:57.315565 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.315578    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:52:57.315846 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.315587    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l95zf\" (UniqueName: \"kubernetes.io/projected/bb8dc525-2780-4db2-b6b2-fd06db2ec294-kube-api-access-l95zf\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:52:57.315846 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.315594    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb8dc525-2780-4db2-b6b2-fd06db2ec294-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:52:57.315846 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.315603    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:52:57.849848 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.849816    2561 generic.go:356] "Generic (PLEG): container finished" podID="bb8dc525-2780-4db2-b6b2-fd06db2ec294" containerID="d1ac9ad4eba196511015f09e3c05dabbc6b4374e33c44818cbf97f803446a6aa" exitCode=0
Aug 05 17:52:57.850313 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.849856    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"
Aug 05 17:52:57.850313 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.849907    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7" event={"ID":"bb8dc525-2780-4db2-b6b2-fd06db2ec294","Type":"ContainerDied","Data":"d1ac9ad4eba196511015f09e3c05dabbc6b4374e33c44818cbf97f803446a6aa"}
Aug 05 17:52:57.850313 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.849990    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7" event={"ID":"bb8dc525-2780-4db2-b6b2-fd06db2ec294","Type":"ContainerDied","Data":"aef8f24db4b379e947ee4c1a7f5138b8faee6b405d4bb1fd239996b97121a706"}
Aug 05 17:52:57.850313 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.850021    2561 scope.go:117] "RemoveContainer" containerID="d1ac9ad4eba196511015f09e3c05dabbc6b4374e33c44818cbf97f803446a6aa"
Aug 05 17:52:57.851483 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.851398    2561 generic.go:356] "Generic (PLEG): container finished" podID="ad0b6821-bb7b-4c01-82ab-675d6c88dbfd" containerID="ac3dceaf765257ec9937481259404fde93984a00f82e0e4fd52ea31e7259293d" exitCode=0
Aug 05 17:52:57.851483 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.851429    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl" event={"ID":"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd","Type":"ContainerDied","Data":"ac3dceaf765257ec9937481259404fde93984a00f82e0e4fd52ea31e7259293d"}
Aug 05 17:52:57.851483 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.851468    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl" event={"ID":"ad0b6821-bb7b-4c01-82ab-675d6c88dbfd","Type":"ContainerDied","Data":"1eae7d90ce5f2852ef6044d597b39c56609070be3689f309dd9560c1b2c2fba4"}
Aug 05 17:52:57.851668 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.851505    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"
Aug 05 17:52:57.859836 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.859820    2561 scope.go:117] "RemoveContainer" containerID="d1ac9ad4eba196511015f09e3c05dabbc6b4374e33c44818cbf97f803446a6aa"
Aug 05 17:52:57.860095 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:52:57.860078    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1ac9ad4eba196511015f09e3c05dabbc6b4374e33c44818cbf97f803446a6aa\": container with ID starting with d1ac9ad4eba196511015f09e3c05dabbc6b4374e33c44818cbf97f803446a6aa not found: ID does not exist" containerID="d1ac9ad4eba196511015f09e3c05dabbc6b4374e33c44818cbf97f803446a6aa"
Aug 05 17:52:57.860187 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.860103    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1ac9ad4eba196511015f09e3c05dabbc6b4374e33c44818cbf97f803446a6aa"} err="failed to get container status \"d1ac9ad4eba196511015f09e3c05dabbc6b4374e33c44818cbf97f803446a6aa\": rpc error: code = NotFound desc = could not find container \"d1ac9ad4eba196511015f09e3c05dabbc6b4374e33c44818cbf97f803446a6aa\": container with ID starting with d1ac9ad4eba196511015f09e3c05dabbc6b4374e33c44818cbf97f803446a6aa not found: ID does not exist"
Aug 05 17:52:57.860187 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.860122    2561 scope.go:117] "RemoveContainer" containerID="ac3dceaf765257ec9937481259404fde93984a00f82e0e4fd52ea31e7259293d"
Aug 05 17:52:57.868028 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.868013    2561 scope.go:117] "RemoveContainer" containerID="ac3dceaf765257ec9937481259404fde93984a00f82e0e4fd52ea31e7259293d"
Aug 05 17:52:57.868261 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:52:57.868244    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac3dceaf765257ec9937481259404fde93984a00f82e0e4fd52ea31e7259293d\": container with ID starting with ac3dceaf765257ec9937481259404fde93984a00f82e0e4fd52ea31e7259293d not found: ID does not exist" containerID="ac3dceaf765257ec9937481259404fde93984a00f82e0e4fd52ea31e7259293d"
Aug 05 17:52:57.868314 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.868270    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac3dceaf765257ec9937481259404fde93984a00f82e0e4fd52ea31e7259293d"} err="failed to get container status \"ac3dceaf765257ec9937481259404fde93984a00f82e0e4fd52ea31e7259293d\": rpc error: code = NotFound desc = could not find container \"ac3dceaf765257ec9937481259404fde93984a00f82e0e4fd52ea31e7259293d\": container with ID starting with ac3dceaf765257ec9937481259404fde93984a00f82e0e4fd52ea31e7259293d not found: ID does not exist"
Aug 05 17:52:57.875031 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.875010    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"]
Aug 05 17:52:57.878522 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.878504    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5xfwl"]
Aug 05 17:52:57.888722 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.888703    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"]
Aug 05 17:52:57.892175 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:57.892155    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-whmh7"]
Aug 05 17:52:58.730067 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:58.730036    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad0b6821-bb7b-4c01-82ab-675d6c88dbfd" path="/var/lib/kubelet/pods/ad0b6821-bb7b-4c01-82ab-675d6c88dbfd/volumes"
Aug 05 17:52:58.730404 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:58.730392    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb8dc525-2780-4db2-b6b2-fd06db2ec294" path="/var/lib/kubelet/pods/bb8dc525-2780-4db2-b6b2-fd06db2ec294/volumes"
Aug 05 17:52:59.794746 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:59.794703    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"]
Aug 05 17:52:59.795300 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:59.795284    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb8dc525-2780-4db2-b6b2-fd06db2ec294" containerName="main"
Aug 05 17:52:59.795345 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:59.795304    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb8dc525-2780-4db2-b6b2-fd06db2ec294" containerName="main"
Aug 05 17:52:59.795345 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:59.795331    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad0b6821-bb7b-4c01-82ab-675d6c88dbfd" containerName="main"
Aug 05 17:52:59.795345 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:59.795339    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad0b6821-bb7b-4c01-82ab-675d6c88dbfd" containerName="main"
Aug 05 17:52:59.795486 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:59.795422    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad0b6821-bb7b-4c01-82ab-675d6c88dbfd" containerName="main"
Aug 05 17:52:59.795486 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:59.795433    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb8dc525-2780-4db2-b6b2-fd06db2ec294" containerName="main"
Aug 05 17:52:59.800383 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:59.800363    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"
Aug 05 17:52:59.804106 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:59.803675    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v1-kserve-self-signed-certs\""
Aug 05 17:52:59.804106 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:59.803706    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Aug 05 17:52:59.804106 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:59.803819    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-b6w9s\""
Aug 05 17:52:59.804106 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:59.803825    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Aug 05 17:52:59.807321 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:59.807301    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"]
Aug 05 17:52:59.938919 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:59.938886    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/190862fe-5e08-4934-a4f8-02ae51ef1619-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-526wx\" (UID: \"190862fe-5e08-4934-a4f8-02ae51ef1619\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"
Aug 05 17:52:59.938919 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:59.938926    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/190862fe-5e08-4934-a4f8-02ae51ef1619-home\") pod \"late-v1-kserve-56cbc8456c-526wx\" (UID: \"190862fe-5e08-4934-a4f8-02ae51ef1619\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"
Aug 05 17:52:59.939153 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:59.938973    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhzw8\" (UniqueName: \"kubernetes.io/projected/190862fe-5e08-4934-a4f8-02ae51ef1619-kube-api-access-lhzw8\") pod \"late-v1-kserve-56cbc8456c-526wx\" (UID: \"190862fe-5e08-4934-a4f8-02ae51ef1619\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"
Aug 05 17:52:59.939153 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:59.939060    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/190862fe-5e08-4934-a4f8-02ae51ef1619-dshm\") pod \"late-v1-kserve-56cbc8456c-526wx\" (UID: \"190862fe-5e08-4934-a4f8-02ae51ef1619\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"
Aug 05 17:52:59.939153 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:59.939086    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/190862fe-5e08-4934-a4f8-02ae51ef1619-model-cache\") pod \"late-v1-kserve-56cbc8456c-526wx\" (UID: \"190862fe-5e08-4934-a4f8-02ae51ef1619\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"
Aug 05 17:52:59.939153 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:52:59.939151    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/190862fe-5e08-4934-a4f8-02ae51ef1619-tls-certs\") pod \"late-v1-kserve-56cbc8456c-526wx\" (UID: \"190862fe-5e08-4934-a4f8-02ae51ef1619\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"
Aug 05 17:53:00.040188 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:00.040156    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/190862fe-5e08-4934-a4f8-02ae51ef1619-tls-certs\") pod \"late-v1-kserve-56cbc8456c-526wx\" (UID: \"190862fe-5e08-4934-a4f8-02ae51ef1619\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"
Aug 05 17:53:00.040397 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:00.040200    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/190862fe-5e08-4934-a4f8-02ae51ef1619-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-526wx\" (UID: \"190862fe-5e08-4934-a4f8-02ae51ef1619\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"
Aug 05 17:53:00.040397 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:00.040222    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/190862fe-5e08-4934-a4f8-02ae51ef1619-home\") pod \"late-v1-kserve-56cbc8456c-526wx\" (UID: \"190862fe-5e08-4934-a4f8-02ae51ef1619\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"
Aug 05 17:53:00.040397 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:00.040254    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhzw8\" (UniqueName: \"kubernetes.io/projected/190862fe-5e08-4934-a4f8-02ae51ef1619-kube-api-access-lhzw8\") pod \"late-v1-kserve-56cbc8456c-526wx\" (UID: \"190862fe-5e08-4934-a4f8-02ae51ef1619\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"
Aug 05 17:53:00.040397 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:00.040307    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/190862fe-5e08-4934-a4f8-02ae51ef1619-dshm\") pod \"late-v1-kserve-56cbc8456c-526wx\" (UID: \"190862fe-5e08-4934-a4f8-02ae51ef1619\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"
Aug 05 17:53:00.040397 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:00.040331    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/190862fe-5e08-4934-a4f8-02ae51ef1619-model-cache\") pod \"late-v1-kserve-56cbc8456c-526wx\" (UID: \"190862fe-5e08-4934-a4f8-02ae51ef1619\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"
Aug 05 17:53:00.040659 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:00.040595    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/190862fe-5e08-4934-a4f8-02ae51ef1619-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-526wx\" (UID: \"190862fe-5e08-4934-a4f8-02ae51ef1619\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"
Aug 05 17:53:00.040709 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:00.040684    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/190862fe-5e08-4934-a4f8-02ae51ef1619-model-cache\") pod \"late-v1-kserve-56cbc8456c-526wx\" (UID: \"190862fe-5e08-4934-a4f8-02ae51ef1619\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"
Aug 05 17:53:00.040891 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:00.040875    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/190862fe-5e08-4934-a4f8-02ae51ef1619-home\") pod \"late-v1-kserve-56cbc8456c-526wx\" (UID: \"190862fe-5e08-4934-a4f8-02ae51ef1619\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"
Aug 05 17:53:00.042559 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:00.042529    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/190862fe-5e08-4934-a4f8-02ae51ef1619-dshm\") pod \"late-v1-kserve-56cbc8456c-526wx\" (UID: \"190862fe-5e08-4934-a4f8-02ae51ef1619\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"
Aug 05 17:53:00.042805 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:00.042788    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/190862fe-5e08-4934-a4f8-02ae51ef1619-tls-certs\") pod \"late-v1-kserve-56cbc8456c-526wx\" (UID: \"190862fe-5e08-4934-a4f8-02ae51ef1619\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"
Aug 05 17:53:00.049714 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:00.049650    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhzw8\" (UniqueName: \"kubernetes.io/projected/190862fe-5e08-4934-a4f8-02ae51ef1619-kube-api-access-lhzw8\") pod \"late-v1-kserve-56cbc8456c-526wx\" (UID: \"190862fe-5e08-4934-a4f8-02ae51ef1619\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"
Aug 05 17:53:00.112080 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:00.112046    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"
Aug 05 17:53:00.247496 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:00.247467    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"]
Aug 05 17:53:00.249608 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:53:00.249571    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod190862fe_5e08_4934_a4f8_02ae51ef1619.slice/crio-e2becd0c2d4332adf3c84ea318b986da98bd9fa8ed935b879b82c450ecf70265 WatchSource:0}: Error finding container e2becd0c2d4332adf3c84ea318b986da98bd9fa8ed935b879b82c450ecf70265: Status 404 returned error can't find the container with id e2becd0c2d4332adf3c84ea318b986da98bd9fa8ed935b879b82c450ecf70265
Aug 05 17:53:00.871305 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:00.871276    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx" event={"ID":"190862fe-5e08-4934-a4f8-02ae51ef1619","Type":"ContainerStarted","Data":"27daf3e701b9c128472d05eea2de0b008f12163d588f9f62d6494f1977960a93"}
Aug 05 17:53:00.871802 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:00.871334    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx" event={"ID":"190862fe-5e08-4934-a4f8-02ae51ef1619","Type":"ContainerStarted","Data":"e2becd0c2d4332adf3c84ea318b986da98bd9fa8ed935b879b82c450ecf70265"}
Aug 05 17:53:00.890386 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:00.890337    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx" podStartSLOduration=1.890322638 podStartE2EDuration="1.890322638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:53:00.888793843 +0000 UTC m=+2436.633428170" watchObservedRunningTime="2026-08-05 17:53:00.890322638 +0000 UTC m=+2436.634956965"
Aug 05 17:53:01.112967 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:01.112918    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"
Aug 05 17:53:10.112410 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:10.112373    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"
Aug 05 17:53:10.124481 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:10.124450    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"
Aug 05 17:53:10.939295 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:10.939264    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"
Aug 05 17:53:31.616891 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:31.616854    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"]
Aug 05 17:53:31.620551 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:31.620528    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"
Aug 05 17:53:31.623113 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:31.623086    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v2-kserve-self-signed-certs\""
Aug 05 17:53:31.630726 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:31.630704    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"]
Aug 05 17:53:31.718067 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:31.718026    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-dshm\") pod \"late-v2-kserve-86766546f-ws2cn\" (UID: \"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"
Aug 05 17:53:31.718248 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:31.718078    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-tmp-dir\") pod \"late-v2-kserve-86766546f-ws2cn\" (UID: \"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"
Aug 05 17:53:31.718248 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:31.718136    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fq2x\" (UniqueName: \"kubernetes.io/projected/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-kube-api-access-7fq2x\") pod \"late-v2-kserve-86766546f-ws2cn\" (UID: \"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"
Aug 05 17:53:31.718248 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:31.718173    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-model-cache\") pod \"late-v2-kserve-86766546f-ws2cn\" (UID: \"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"
Aug 05 17:53:31.718248 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:31.718246    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-home\") pod \"late-v2-kserve-86766546f-ws2cn\" (UID: \"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"
Aug 05 17:53:31.718394 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:31.718274    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-tls-certs\") pod \"late-v2-kserve-86766546f-ws2cn\" (UID: \"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"
Aug 05 17:53:31.819539 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:31.819500    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-home\") pod \"late-v2-kserve-86766546f-ws2cn\" (UID: \"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"
Aug 05 17:53:31.819539 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:31.819537    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-tls-certs\") pod \"late-v2-kserve-86766546f-ws2cn\" (UID: \"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"
Aug 05 17:53:31.819801 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:31.819602    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-dshm\") pod \"late-v2-kserve-86766546f-ws2cn\" (UID: \"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"
Aug 05 17:53:31.819801 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:31.819635    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-tmp-dir\") pod \"late-v2-kserve-86766546f-ws2cn\" (UID: \"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"
Aug 05 17:53:31.819801 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:31.819760    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fq2x\" (UniqueName: \"kubernetes.io/projected/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-kube-api-access-7fq2x\") pod \"late-v2-kserve-86766546f-ws2cn\" (UID: \"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"
Aug 05 17:53:31.819999 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:31.819814    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-model-cache\") pod \"late-v2-kserve-86766546f-ws2cn\" (UID: \"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"
Aug 05 17:53:31.820072 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:31.820041    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-tmp-dir\") pod \"late-v2-kserve-86766546f-ws2cn\" (UID: \"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"
Aug 05 17:53:31.820329 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:31.820304    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-model-cache\") pod \"late-v2-kserve-86766546f-ws2cn\" (UID: \"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"
Aug 05 17:53:31.820425 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:31.820352    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-home\") pod \"late-v2-kserve-86766546f-ws2cn\" (UID: \"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"
Aug 05 17:53:31.822449 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:31.822428    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-dshm\") pod \"late-v2-kserve-86766546f-ws2cn\" (UID: \"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"
Aug 05 17:53:31.822724 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:31.822704    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-tls-certs\") pod \"late-v2-kserve-86766546f-ws2cn\" (UID: \"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"
Aug 05 17:53:31.828025 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:31.828004    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fq2x\" (UniqueName: \"kubernetes.io/projected/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-kube-api-access-7fq2x\") pod \"late-v2-kserve-86766546f-ws2cn\" (UID: \"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"
Aug 05 17:53:31.931887 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:31.931806    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"
Aug 05 17:53:32.097648 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:32.097617    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"]
Aug 05 17:53:32.098413 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:53:32.098387    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b2e3afc_07f6_4ead_8ffe_8f8f55b97fbb.slice/crio-57c5747cf5ddfd24606cde0f381dc7657fc7e0f1aac2002186159be8ac06960e WatchSource:0}: Error finding container 57c5747cf5ddfd24606cde0f381dc7657fc7e0f1aac2002186159be8ac06960e: Status 404 returned error can't find the container with id 57c5747cf5ddfd24606cde0f381dc7657fc7e0f1aac2002186159be8ac06960e
Aug 05 17:53:33.046818 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:33.046773    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn" event={"ID":"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb","Type":"ContainerStarted","Data":"b902c880aea167935043c9277d9595cd5a62bab30554d71d01cd143177ec9015"}
Aug 05 17:53:33.047443 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:33.046845    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn" event={"ID":"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb","Type":"ContainerStarted","Data":"57c5747cf5ddfd24606cde0f381dc7657fc7e0f1aac2002186159be8ac06960e"}
Aug 05 17:53:33.067349 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:33.067301    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn" podStartSLOduration=2.067286998 podStartE2EDuration="2.067286998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:53:33.064820918 +0000 UTC m=+2468.809455245" watchObservedRunningTime="2026-08-05 17:53:33.067286998 +0000 UTC m=+2468.811921325"
Aug 05 17:53:33.932682 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:33.932644    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"
Aug 05 17:53:41.932008 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:41.931978    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"
Aug 05 17:53:41.947685 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:41.947654    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"
Aug 05 17:53:42.110536 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:42.110509    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"
Aug 05 17:53:47.339017 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:47.338984    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89"]
Aug 05 17:53:50.528758 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:53:50.528720    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89" podUID="6cec9a23-caf7-4a40-bb63-84fdca30af40" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 05 17:54:00.528523 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:00.528480    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89" podUID="6cec9a23-caf7-4a40-bb63-84fdca30af40" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 05 17:54:02.364169 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:02.364113    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89" podUID="6cec9a23-caf7-4a40-bb63-84fdca30af40" containerName="main" containerID="cri-o://5f329a265073acacdb44668a3b08fa1ff7e02d845a6f2bda46c5b98020251f77" gracePeriod=45
Aug 05 17:54:02.610635 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:02.610611    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89"
Aug 05 17:54:02.796920 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:02.796882    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gj9s9\" (UniqueName: \"kubernetes.io/projected/6cec9a23-caf7-4a40-bb63-84fdca30af40-kube-api-access-gj9s9\") pod \"6cec9a23-caf7-4a40-bb63-84fdca30af40\" (UID: \"6cec9a23-caf7-4a40-bb63-84fdca30af40\") "
Aug 05 17:54:02.797132 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:02.796928    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cec9a23-caf7-4a40-bb63-84fdca30af40-tls-certs\") pod \"6cec9a23-caf7-4a40-bb63-84fdca30af40\" (UID: \"6cec9a23-caf7-4a40-bb63-84fdca30af40\") "
Aug 05 17:54:02.799121 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:02.799092    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cec9a23-caf7-4a40-bb63-84fdca30af40-kube-api-access-gj9s9" (OuterVolumeSpecName: "kube-api-access-gj9s9") pod "6cec9a23-caf7-4a40-bb63-84fdca30af40" (UID: "6cec9a23-caf7-4a40-bb63-84fdca30af40"). InnerVolumeSpecName "kube-api-access-gj9s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:54:02.799238 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:02.799160    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cec9a23-caf7-4a40-bb63-84fdca30af40-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6cec9a23-caf7-4a40-bb63-84fdca30af40" (UID: "6cec9a23-caf7-4a40-bb63-84fdca30af40"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:54:02.897879 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:02.897840    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj9s9\" (UniqueName: \"kubernetes.io/projected/6cec9a23-caf7-4a40-bb63-84fdca30af40-kube-api-access-gj9s9\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:54:02.897879 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:02.897876    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cec9a23-caf7-4a40-bb63-84fdca30af40-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:54:03.205656 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:03.205560    2561 generic.go:356] "Generic (PLEG): container finished" podID="6cec9a23-caf7-4a40-bb63-84fdca30af40" containerID="5f329a265073acacdb44668a3b08fa1ff7e02d845a6f2bda46c5b98020251f77" exitCode=0
Aug 05 17:54:03.205656 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:03.205612    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89"
Aug 05 17:54:03.205938 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:03.205612    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89" event={"ID":"6cec9a23-caf7-4a40-bb63-84fdca30af40","Type":"ContainerDied","Data":"5f329a265073acacdb44668a3b08fa1ff7e02d845a6f2bda46c5b98020251f77"}
Aug 05 17:54:03.205938 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:03.205719    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89" event={"ID":"6cec9a23-caf7-4a40-bb63-84fdca30af40","Type":"ContainerDied","Data":"6ebe672e524370f4ed8b25274ca32315ef088e3eae0632a7a960cf515f154df0"}
Aug 05 17:54:03.205938 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:03.205745    2561 scope.go:117] "RemoveContainer" containerID="5f329a265073acacdb44668a3b08fa1ff7e02d845a6f2bda46c5b98020251f77"
Aug 05 17:54:03.217172 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:03.217143    2561 scope.go:117] "RemoveContainer" containerID="5f329a265073acacdb44668a3b08fa1ff7e02d845a6f2bda46c5b98020251f77"
Aug 05 17:54:03.217599 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:54:03.217569    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f329a265073acacdb44668a3b08fa1ff7e02d845a6f2bda46c5b98020251f77\": container with ID starting with 5f329a265073acacdb44668a3b08fa1ff7e02d845a6f2bda46c5b98020251f77 not found: ID does not exist" containerID="5f329a265073acacdb44668a3b08fa1ff7e02d845a6f2bda46c5b98020251f77"
Aug 05 17:54:03.217689 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:03.217608    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f329a265073acacdb44668a3b08fa1ff7e02d845a6f2bda46c5b98020251f77"} err="failed to get container status \"5f329a265073acacdb44668a3b08fa1ff7e02d845a6f2bda46c5b98020251f77\": rpc error: code = NotFound desc = could not find container \"5f329a265073acacdb44668a3b08fa1ff7e02d845a6f2bda46c5b98020251f77\": container with ID starting with 5f329a265073acacdb44668a3b08fa1ff7e02d845a6f2bda46c5b98020251f77 not found: ID does not exist"
Aug 05 17:54:03.240357 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:03.240332    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89"]
Aug 05 17:54:03.246763 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:03.246742    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schemtz89"]
Aug 05 17:54:04.729776 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:04.729741    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cec9a23-caf7-4a40-bb63-84fdca30af40" path="/var/lib/kubelet/pods/6cec9a23-caf7-4a40-bb63-84fdca30af40/volumes"
Aug 05 17:54:18.410292 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.410211    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"]
Aug 05 17:54:18.410646 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.410617    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cec9a23-caf7-4a40-bb63-84fdca30af40" containerName="main"
Aug 05 17:54:18.410646 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.410630    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cec9a23-caf7-4a40-bb63-84fdca30af40" containerName="main"
Aug 05 17:54:18.410722 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.410703    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cec9a23-caf7-4a40-bb63-84fdca30af40" containerName="main"
Aug 05 17:54:18.415093 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.415075    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:54:18.417776 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.417754    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Aug 05 17:54:18.417909 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.417822    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Aug 05 17:54:18.417909 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.417845    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-lcqqf\""
Aug 05 17:54:18.417909 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.417875    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Aug 05 17:54:18.417909 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.417822    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Aug 05 17:54:18.418119 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.417829    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Aug 05 17:54:18.426752 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.426732    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"]
Aug 05 17:54:18.527595 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.527559    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:54:18.527595 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.527600    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:54:18.527895 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.527616    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:54:18.527895 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.527632    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9x67\" (UniqueName: \"kubernetes.io/projected/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-kube-api-access-t9x67\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:54:18.527895 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.527723    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:54:18.527895 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.527786    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:54:18.527895 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.527828    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:54:18.527895 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.527866    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:54:18.628843 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.628808    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:54:18.629074 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.628868    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:54:18.629074 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.628894    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:54:18.629074 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.628915    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9x67\" (UniqueName: \"kubernetes.io/projected/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-kube-api-access-t9x67\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:54:18.629074 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.628977    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:54:18.629074 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.629045    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:54:18.629367 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.629240    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:54:18.629367 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.629289    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:54:18.629367 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.629300    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:54:18.629367 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.629249    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:54:18.629580 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.629484    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:54:18.629644 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.629606    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:54:18.629799 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.629778    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:54:18.631305 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.631281    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:54:18.631502 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.631485    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:54:18.637233 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.637212    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9x67\" (UniqueName: \"kubernetes.io/projected/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-kube-api-access-t9x67\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:54:18.727294 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.727211    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:54:18.897219 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:18.897195    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"]
Aug 05 17:54:18.899134 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:54:18.899108    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod053d0b97_e4db_4147_90ea_fd5e0f35fbcf.slice/crio-458bd29a56ff1baa2ce2b73c203821f862d04c02649ee89dbf99e0a8666f33f2 WatchSource:0}: Error finding container 458bd29a56ff1baa2ce2b73c203821f862d04c02649ee89dbf99e0a8666f33f2: Status 404 returned error can't find the container with id 458bd29a56ff1baa2ce2b73c203821f862d04c02649ee89dbf99e0a8666f33f2
Aug 05 17:54:19.294340 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:19.294299    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9" event={"ID":"053d0b97-e4db-4147-90ea-fd5e0f35fbcf","Type":"ContainerStarted","Data":"f8c383fa940e986b5297dc8a1f4cde014797f50db7acbe9c9019e892c846ed30"}
Aug 05 17:54:19.294549 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:19.294363    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9" event={"ID":"053d0b97-e4db-4147-90ea-fd5e0f35fbcf","Type":"ContainerStarted","Data":"458bd29a56ff1baa2ce2b73c203821f862d04c02649ee89dbf99e0a8666f33f2"}
Aug 05 17:54:19.607646 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:19.607548    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"]
Aug 05 17:54:19.613157 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:19.613117    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"]
Aug 05 17:54:22.312946 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:22.312911    2561 generic.go:356] "Generic (PLEG): container finished" podID="053d0b97-e4db-4147-90ea-fd5e0f35fbcf" containerID="f8c383fa940e986b5297dc8a1f4cde014797f50db7acbe9c9019e892c846ed30" exitCode=0
Aug 05 17:54:22.313374 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:22.312994    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9" event={"ID":"053d0b97-e4db-4147-90ea-fd5e0f35fbcf","Type":"ContainerDied","Data":"f8c383fa940e986b5297dc8a1f4cde014797f50db7acbe9c9019e892c846ed30"}
Aug 05 17:54:23.321022 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:23.320985    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9" event={"ID":"053d0b97-e4db-4147-90ea-fd5e0f35fbcf","Type":"ContainerStarted","Data":"2f51dcdcbf8148dddbc9397491f1f7a9961f9255d36939b6faf69dff85c60308"}
Aug 05 17:54:23.347263 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:23.347201    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9" podStartSLOduration=2.533868582 podStartE2EDuration="5.347179575s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.813310993s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:54:23.344875567 +0000 UTC m=+2519.089509912" watchObservedRunningTime="2026-08-05 17:54:23.347179575 +0000 UTC m=+2519.091813903"
Aug 05 17:54:23.727788 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:23.727693    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:54:28.730062 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:28.730028    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:54:28.730471 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:28.730282    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9" podUID="053d0b97-e4db-4147-90ea-fd5e0f35fbcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Aug 05 17:54:34.626833 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:34.626767    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx" podUID="190862fe-5e08-4934-a4f8-02ae51ef1619" containerName="main" containerID="cri-o://27daf3e701b9c128472d05eea2de0b008f12163d588f9f62d6494f1977960a93" gracePeriod=45
Aug 05 17:54:34.631897 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:34.631852    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn" podUID="8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb" containerName="main" containerID="cri-o://b902c880aea167935043c9277d9595cd5a62bab30554d71d01cd143177ec9015" gracePeriod=45
Aug 05 17:54:34.943350 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:34.943321    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"
Aug 05 17:54:34.946240 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:34.946220    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"
Aug 05 17:54:35.084198 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.084161    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/190862fe-5e08-4934-a4f8-02ae51ef1619-home\") pod \"190862fe-5e08-4934-a4f8-02ae51ef1619\" (UID: \"190862fe-5e08-4934-a4f8-02ae51ef1619\") "
Aug 05 17:54:35.084198 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.084203    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-model-cache\") pod \"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb\" (UID: \"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb\") "
Aug 05 17:54:35.084446 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.084228    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-dshm\") pod \"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb\" (UID: \"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb\") "
Aug 05 17:54:35.084446 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.084255    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/190862fe-5e08-4934-a4f8-02ae51ef1619-model-cache\") pod \"190862fe-5e08-4934-a4f8-02ae51ef1619\" (UID: \"190862fe-5e08-4934-a4f8-02ae51ef1619\") "
Aug 05 17:54:35.084446 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.084286    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fq2x\" (UniqueName: \"kubernetes.io/projected/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-kube-api-access-7fq2x\") pod \"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb\" (UID: \"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb\") "
Aug 05 17:54:35.084446 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.084310    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-tls-certs\") pod \"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb\" (UID: \"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb\") "
Aug 05 17:54:35.084446 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.084343    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-home\") pod \"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb\" (UID: \"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb\") "
Aug 05 17:54:35.084446 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.084368    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-tmp-dir\") pod \"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb\" (UID: \"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb\") "
Aug 05 17:54:35.084446 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.084428    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhzw8\" (UniqueName: \"kubernetes.io/projected/190862fe-5e08-4934-a4f8-02ae51ef1619-kube-api-access-lhzw8\") pod \"190862fe-5e08-4934-a4f8-02ae51ef1619\" (UID: \"190862fe-5e08-4934-a4f8-02ae51ef1619\") "
Aug 05 17:54:35.084786 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.084453    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/190862fe-5e08-4934-a4f8-02ae51ef1619-dshm\") pod \"190862fe-5e08-4934-a4f8-02ae51ef1619\" (UID: \"190862fe-5e08-4934-a4f8-02ae51ef1619\") "
Aug 05 17:54:35.084786 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.084453    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190862fe-5e08-4934-a4f8-02ae51ef1619-home" (OuterVolumeSpecName: "home") pod "190862fe-5e08-4934-a4f8-02ae51ef1619" (UID: "190862fe-5e08-4934-a4f8-02ae51ef1619"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:54:35.084786 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.084488    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-model-cache" (OuterVolumeSpecName: "model-cache") pod "8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb" (UID: "8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:54:35.084786 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.084511    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/190862fe-5e08-4934-a4f8-02ae51ef1619-tmp-dir\") pod \"190862fe-5e08-4934-a4f8-02ae51ef1619\" (UID: \"190862fe-5e08-4934-a4f8-02ae51ef1619\") "
Aug 05 17:54:35.084786 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.084537    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/190862fe-5e08-4934-a4f8-02ae51ef1619-tls-certs\") pod \"190862fe-5e08-4934-a4f8-02ae51ef1619\" (UID: \"190862fe-5e08-4934-a4f8-02ae51ef1619\") "
Aug 05 17:54:35.084786 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.084532    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190862fe-5e08-4934-a4f8-02ae51ef1619-model-cache" (OuterVolumeSpecName: "model-cache") pod "190862fe-5e08-4934-a4f8-02ae51ef1619" (UID: "190862fe-5e08-4934-a4f8-02ae51ef1619"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:54:35.084786 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.084616    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-home" (OuterVolumeSpecName: "home") pod "8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb" (UID: "8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:54:35.084786 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.084768    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190862fe-5e08-4934-a4f8-02ae51ef1619-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "190862fe-5e08-4934-a4f8-02ae51ef1619" (UID: "190862fe-5e08-4934-a4f8-02ae51ef1619"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:54:35.085301 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.084840    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/190862fe-5e08-4934-a4f8-02ae51ef1619-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:54:35.085301 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.084859    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/190862fe-5e08-4934-a4f8-02ae51ef1619-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:54:35.085301 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.084872    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:54:35.085301 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.084885    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/190862fe-5e08-4934-a4f8-02ae51ef1619-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:54:35.085301 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.084896    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:54:35.085301 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.084896    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb" (UID: "8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:54:35.087155 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.087129    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/190862fe-5e08-4934-a4f8-02ae51ef1619-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "190862fe-5e08-4934-a4f8-02ae51ef1619" (UID: "190862fe-5e08-4934-a4f8-02ae51ef1619"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:54:35.087275 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.087234    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/190862fe-5e08-4934-a4f8-02ae51ef1619-kube-api-access-lhzw8" (OuterVolumeSpecName: "kube-api-access-lhzw8") pod "190862fe-5e08-4934-a4f8-02ae51ef1619" (UID: "190862fe-5e08-4934-a4f8-02ae51ef1619"). InnerVolumeSpecName "kube-api-access-lhzw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:54:35.087349 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.087309    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb" (UID: "8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:54:35.087349 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.087322    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-kube-api-access-7fq2x" (OuterVolumeSpecName: "kube-api-access-7fq2x") pod "8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb" (UID: "8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb"). InnerVolumeSpecName "kube-api-access-7fq2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:54:35.087349 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.087329    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-dshm" (OuterVolumeSpecName: "dshm") pod "8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb" (UID: "8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:54:35.087485 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.087468    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190862fe-5e08-4934-a4f8-02ae51ef1619-dshm" (OuterVolumeSpecName: "dshm") pod "190862fe-5e08-4934-a4f8-02ae51ef1619" (UID: "190862fe-5e08-4934-a4f8-02ae51ef1619"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:54:35.109977 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.109934    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"]
Aug 05 17:54:35.110511 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.110494    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="190862fe-5e08-4934-a4f8-02ae51ef1619" containerName="main"
Aug 05 17:54:35.110556 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.110516    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="190862fe-5e08-4934-a4f8-02ae51ef1619" containerName="main"
Aug 05 17:54:35.110556 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.110535    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb" containerName="main"
Aug 05 17:54:35.110556 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.110546    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb" containerName="main"
Aug 05 17:54:35.110667 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.110656    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="190862fe-5e08-4934-a4f8-02ae51ef1619" containerName="main"
Aug 05 17:54:35.110705 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.110671    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb" containerName="main"
Aug 05 17:54:35.114549 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.114528    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"
Aug 05 17:54:35.117282 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.117262    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"kube-root-ca.crt\""
Aug 05 17:54:35.117282 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.117272    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v1-kserve-self-signed-certs\""
Aug 05 17:54:35.117472 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.117260    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"openshift-service-ca.crt\""
Aug 05 17:54:35.117472 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.117388    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"default-dockercfg-h76gq\""
Aug 05 17:54:35.122816 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.122794    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"]
Aug 05 17:54:35.185660 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.185583    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-dshm\") pod \"delw-v1-kserve-6cf8cbb847-kdqmm\" (UID: \"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"
Aug 05 17:54:35.185660 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.185640    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-kdqmm\" (UID: \"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"
Aug 05 17:54:35.185841 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.185731    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n728\" (UniqueName: \"kubernetes.io/projected/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-kube-api-access-4n728\") pod \"delw-v1-kserve-6cf8cbb847-kdqmm\" (UID: \"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"
Aug 05 17:54:35.185841 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.185766    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-kdqmm\" (UID: \"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"
Aug 05 17:54:35.185841 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.185802    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-kdqmm\" (UID: \"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"
Aug 05 17:54:35.185841 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.185824    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-home\") pod \"delw-v1-kserve-6cf8cbb847-kdqmm\" (UID: \"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"
Aug 05 17:54:35.186033 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.185879    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/190862fe-5e08-4934-a4f8-02ae51ef1619-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:54:35.186033 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.185894    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:54:35.186033 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.185904    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fq2x\" (UniqueName: \"kubernetes.io/projected/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-kube-api-access-7fq2x\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:54:35.186033 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.185913    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:54:35.186033 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.185920    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:54:35.186033 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.185930    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhzw8\" (UniqueName: \"kubernetes.io/projected/190862fe-5e08-4934-a4f8-02ae51ef1619-kube-api-access-lhzw8\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:54:35.186033 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.185937    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/190862fe-5e08-4934-a4f8-02ae51ef1619-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:54:35.286861 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.286822    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n728\" (UniqueName: \"kubernetes.io/projected/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-kube-api-access-4n728\") pod \"delw-v1-kserve-6cf8cbb847-kdqmm\" (UID: \"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"
Aug 05 17:54:35.287086 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.286878    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-kdqmm\" (UID: \"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"
Aug 05 17:54:35.287086 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.286911    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-kdqmm\" (UID: \"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"
Aug 05 17:54:35.287086 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.286936    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-home\") pod \"delw-v1-kserve-6cf8cbb847-kdqmm\" (UID: \"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"
Aug 05 17:54:35.287086 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.287058    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-dshm\") pod \"delw-v1-kserve-6cf8cbb847-kdqmm\" (UID: \"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"
Aug 05 17:54:35.287308 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.287178    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-kdqmm\" (UID: \"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"
Aug 05 17:54:35.287388 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.287368    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-home\") pod \"delw-v1-kserve-6cf8cbb847-kdqmm\" (UID: \"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"
Aug 05 17:54:35.287446 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.287427    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-kdqmm\" (UID: \"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"
Aug 05 17:54:35.287487 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.287473    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-kdqmm\" (UID: \"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"
Aug 05 17:54:35.290341 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.290314    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-kdqmm\" (UID: \"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"
Aug 05 17:54:35.290491 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.290468    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-dshm\") pod \"delw-v1-kserve-6cf8cbb847-kdqmm\" (UID: \"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"
Aug 05 17:54:35.295261 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.295241    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n728\" (UniqueName: \"kubernetes.io/projected/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-kube-api-access-4n728\") pod \"delw-v1-kserve-6cf8cbb847-kdqmm\" (UID: \"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"
Aug 05 17:54:35.403072 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.403033    2561 generic.go:356] "Generic (PLEG): container finished" podID="190862fe-5e08-4934-a4f8-02ae51ef1619" containerID="27daf3e701b9c128472d05eea2de0b008f12163d588f9f62d6494f1977960a93" exitCode=0
Aug 05 17:54:35.403271 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.403073    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx" event={"ID":"190862fe-5e08-4934-a4f8-02ae51ef1619","Type":"ContainerDied","Data":"27daf3e701b9c128472d05eea2de0b008f12163d588f9f62d6494f1977960a93"}
Aug 05 17:54:35.403271 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.403106    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"
Aug 05 17:54:35.403271 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.403145    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx" event={"ID":"190862fe-5e08-4934-a4f8-02ae51ef1619","Type":"ContainerDied","Data":"e2becd0c2d4332adf3c84ea318b986da98bd9fa8ed935b879b82c450ecf70265"}
Aug 05 17:54:35.403271 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.403175    2561 scope.go:117] "RemoveContainer" containerID="27daf3e701b9c128472d05eea2de0b008f12163d588f9f62d6494f1977960a93"
Aug 05 17:54:35.405037 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.404830    2561 generic.go:356] "Generic (PLEG): container finished" podID="8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb" containerID="b902c880aea167935043c9277d9595cd5a62bab30554d71d01cd143177ec9015" exitCode=0
Aug 05 17:54:35.405037 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.404865    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn" event={"ID":"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb","Type":"ContainerDied","Data":"b902c880aea167935043c9277d9595cd5a62bab30554d71d01cd143177ec9015"}
Aug 05 17:54:35.405037 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.404898    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"
Aug 05 17:54:35.405037 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.404901    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn" event={"ID":"8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb","Type":"ContainerDied","Data":"57c5747cf5ddfd24606cde0f381dc7657fc7e0f1aac2002186159be8ac06960e"}
Aug 05 17:54:35.415865 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.415835    2561 scope.go:117] "RemoveContainer" containerID="27daf3e701b9c128472d05eea2de0b008f12163d588f9f62d6494f1977960a93"
Aug 05 17:54:35.416217 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:54:35.416197    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27daf3e701b9c128472d05eea2de0b008f12163d588f9f62d6494f1977960a93\": container with ID starting with 27daf3e701b9c128472d05eea2de0b008f12163d588f9f62d6494f1977960a93 not found: ID does not exist" containerID="27daf3e701b9c128472d05eea2de0b008f12163d588f9f62d6494f1977960a93"
Aug 05 17:54:35.416367 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.416342    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27daf3e701b9c128472d05eea2de0b008f12163d588f9f62d6494f1977960a93"} err="failed to get container status \"27daf3e701b9c128472d05eea2de0b008f12163d588f9f62d6494f1977960a93\": rpc error: code = NotFound desc = could not find container \"27daf3e701b9c128472d05eea2de0b008f12163d588f9f62d6494f1977960a93\": container with ID starting with 27daf3e701b9c128472d05eea2de0b008f12163d588f9f62d6494f1977960a93 not found: ID does not exist"
Aug 05 17:54:35.416471 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.416458    2561 scope.go:117] "RemoveContainer" containerID="b902c880aea167935043c9277d9595cd5a62bab30554d71d01cd143177ec9015"
Aug 05 17:54:35.425371 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.425333    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"
Aug 05 17:54:35.427365 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.427316    2561 scope.go:117] "RemoveContainer" containerID="b902c880aea167935043c9277d9595cd5a62bab30554d71d01cd143177ec9015"
Aug 05 17:54:35.427637 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:54:35.427612    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b902c880aea167935043c9277d9595cd5a62bab30554d71d01cd143177ec9015\": container with ID starting with b902c880aea167935043c9277d9595cd5a62bab30554d71d01cd143177ec9015 not found: ID does not exist" containerID="b902c880aea167935043c9277d9595cd5a62bab30554d71d01cd143177ec9015"
Aug 05 17:54:35.427754 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.427645    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b902c880aea167935043c9277d9595cd5a62bab30554d71d01cd143177ec9015"} err="failed to get container status \"b902c880aea167935043c9277d9595cd5a62bab30554d71d01cd143177ec9015\": rpc error: code = NotFound desc = could not find container \"b902c880aea167935043c9277d9595cd5a62bab30554d71d01cd143177ec9015\": container with ID starting with b902c880aea167935043c9277d9595cd5a62bab30554d71d01cd143177ec9015 not found: ID does not exist"
Aug 05 17:54:35.437683 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.437617    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"]
Aug 05 17:54:35.441546 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.441520    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx"]
Aug 05 17:54:35.453089 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.453027    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"]
Aug 05 17:54:35.454816 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.454786    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-ws2cn"]
Aug 05 17:54:35.576720 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.576686    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"]
Aug 05 17:54:35.577610 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:54:35.577580    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f2be8c9_e2c2_46f1_ba6e_8496c7fad249.slice/crio-b759c00fd614c590b72cb2b302763da99231083ef89e788ec41e8d8247d9ecc2 WatchSource:0}: Error finding container b759c00fd614c590b72cb2b302763da99231083ef89e788ec41e8d8247d9ecc2: Status 404 returned error can't find the container with id b759c00fd614c590b72cb2b302763da99231083ef89e788ec41e8d8247d9ecc2
Aug 05 17:54:35.929516 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:35.929462    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-526wx" podUID="190862fe-5e08-4934-a4f8-02ae51ef1619" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Aug 05 17:54:36.413683 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:36.413650    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm" event={"ID":"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249","Type":"ContainerStarted","Data":"a86537f2f1ea965b85f4cce9d8aa34ac630d5087f29c42ecd5adfe42562f017e"}
Aug 05 17:54:36.413894 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:36.413701    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm" event={"ID":"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249","Type":"ContainerStarted","Data":"b759c00fd614c590b72cb2b302763da99231083ef89e788ec41e8d8247d9ecc2"}
Aug 05 17:54:36.426057 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:36.426032    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"
Aug 05 17:54:36.431652 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:36.431591    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm" podStartSLOduration=1.431573932 podStartE2EDuration="1.431573932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:54:36.430115219 +0000 UTC m=+2532.174749546" watchObservedRunningTime="2026-08-05 17:54:36.431573932 +0000 UTC m=+2532.176208260"
Aug 05 17:54:36.729690 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:36.729608    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="190862fe-5e08-4934-a4f8-02ae51ef1619" path="/var/lib/kubelet/pods/190862fe-5e08-4934-a4f8-02ae51ef1619/volumes"
Aug 05 17:54:36.730036 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:36.730021    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb" path="/var/lib/kubelet/pods/8b2e3afc-07f6-4ead-8ffe-8f8f55b97fbb/volumes"
Aug 05 17:54:38.727660 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:38.727619    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9" podUID="053d0b97-e4db-4147-90ea-fd5e0f35fbcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Aug 05 17:54:44.203747 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:44.203706    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"]
Aug 05 17:54:44.209197 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:44.209163    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"
Aug 05 17:54:44.212239 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:44.212200    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v2-kserve-self-signed-certs\""
Aug 05 17:54:44.214697 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:44.214667    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"]
Aug 05 17:54:44.375120 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:44.375080    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64f784b6-eeb1-45b0-a239-2559caab8c5e-home\") pod \"delw-v2-kserve-5b84cc7ddf-nvkgg\" (UID: \"64f784b6-eeb1-45b0-a239-2559caab8c5e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"
Aug 05 17:54:44.375313 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:44.375145    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64f784b6-eeb1-45b0-a239-2559caab8c5e-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-nvkgg\" (UID: \"64f784b6-eeb1-45b0-a239-2559caab8c5e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"
Aug 05 17:54:44.375313 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:44.375205    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64f784b6-eeb1-45b0-a239-2559caab8c5e-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-nvkgg\" (UID: \"64f784b6-eeb1-45b0-a239-2559caab8c5e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"
Aug 05 17:54:44.375313 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:44.375289    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64f784b6-eeb1-45b0-a239-2559caab8c5e-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-nvkgg\" (UID: \"64f784b6-eeb1-45b0-a239-2559caab8c5e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"
Aug 05 17:54:44.375466 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:44.375333    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64f784b6-eeb1-45b0-a239-2559caab8c5e-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-nvkgg\" (UID: \"64f784b6-eeb1-45b0-a239-2559caab8c5e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"
Aug 05 17:54:44.375466 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:44.375352    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6f2q\" (UniqueName: \"kubernetes.io/projected/64f784b6-eeb1-45b0-a239-2559caab8c5e-kube-api-access-l6f2q\") pod \"delw-v2-kserve-5b84cc7ddf-nvkgg\" (UID: \"64f784b6-eeb1-45b0-a239-2559caab8c5e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"
Aug 05 17:54:44.476396 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:44.476314    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64f784b6-eeb1-45b0-a239-2559caab8c5e-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-nvkgg\" (UID: \"64f784b6-eeb1-45b0-a239-2559caab8c5e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"
Aug 05 17:54:44.476396 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:44.476348    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6f2q\" (UniqueName: \"kubernetes.io/projected/64f784b6-eeb1-45b0-a239-2559caab8c5e-kube-api-access-l6f2q\") pod \"delw-v2-kserve-5b84cc7ddf-nvkgg\" (UID: \"64f784b6-eeb1-45b0-a239-2559caab8c5e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"
Aug 05 17:54:44.476396 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:44.476374    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64f784b6-eeb1-45b0-a239-2559caab8c5e-home\") pod \"delw-v2-kserve-5b84cc7ddf-nvkgg\" (UID: \"64f784b6-eeb1-45b0-a239-2559caab8c5e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"
Aug 05 17:54:44.476653 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:44.476410    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64f784b6-eeb1-45b0-a239-2559caab8c5e-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-nvkgg\" (UID: \"64f784b6-eeb1-45b0-a239-2559caab8c5e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"
Aug 05 17:54:44.476653 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:44.476459    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64f784b6-eeb1-45b0-a239-2559caab8c5e-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-nvkgg\" (UID: \"64f784b6-eeb1-45b0-a239-2559caab8c5e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"
Aug 05 17:54:44.476769 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:44.476641    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64f784b6-eeb1-45b0-a239-2559caab8c5e-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-nvkgg\" (UID: \"64f784b6-eeb1-45b0-a239-2559caab8c5e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"
Aug 05 17:54:44.476832 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:44.476788    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64f784b6-eeb1-45b0-a239-2559caab8c5e-home\") pod \"delw-v2-kserve-5b84cc7ddf-nvkgg\" (UID: \"64f784b6-eeb1-45b0-a239-2559caab8c5e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"
Aug 05 17:54:44.476890 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:44.476841    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64f784b6-eeb1-45b0-a239-2559caab8c5e-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-nvkgg\" (UID: \"64f784b6-eeb1-45b0-a239-2559caab8c5e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"
Aug 05 17:54:44.477017 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:44.476937    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64f784b6-eeb1-45b0-a239-2559caab8c5e-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-nvkgg\" (UID: \"64f784b6-eeb1-45b0-a239-2559caab8c5e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"
Aug 05 17:54:44.478773 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:44.478754    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64f784b6-eeb1-45b0-a239-2559caab8c5e-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-nvkgg\" (UID: \"64f784b6-eeb1-45b0-a239-2559caab8c5e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"
Aug 05 17:54:44.478964 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:44.478932    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64f784b6-eeb1-45b0-a239-2559caab8c5e-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-nvkgg\" (UID: \"64f784b6-eeb1-45b0-a239-2559caab8c5e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"
Aug 05 17:54:44.494700 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:44.494668    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6f2q\" (UniqueName: \"kubernetes.io/projected/64f784b6-eeb1-45b0-a239-2559caab8c5e-kube-api-access-l6f2q\") pod \"delw-v2-kserve-5b84cc7ddf-nvkgg\" (UID: \"64f784b6-eeb1-45b0-a239-2559caab8c5e\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"
Aug 05 17:54:44.521713 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:44.521680    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"
Aug 05 17:54:44.684197 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:44.684142    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"]
Aug 05 17:54:44.686605 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:54:44.686500    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f784b6_eeb1_45b0_a239_2559caab8c5e.slice/crio-f3fc7c0dd6f37009fd743fd67fcfce059f9661bee6387683180649a80324a466 WatchSource:0}: Error finding container f3fc7c0dd6f37009fd743fd67fcfce059f9661bee6387683180649a80324a466: Status 404 returned error can't find the container with id f3fc7c0dd6f37009fd743fd67fcfce059f9661bee6387683180649a80324a466
Aug 05 17:54:45.425693 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:45.425656    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"
Aug 05 17:54:45.444546 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:45.444518    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"
Aug 05 17:54:45.474180 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:45.474137    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg" event={"ID":"64f784b6-eeb1-45b0-a239-2559caab8c5e","Type":"ContainerStarted","Data":"b3071406ab770604068d31066216e72fabfede71fee6071e8aa3c364edc7ba78"}
Aug 05 17:54:45.474372 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:45.474209    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg" event={"ID":"64f784b6-eeb1-45b0-a239-2559caab8c5e","Type":"ContainerStarted","Data":"f3fc7c0dd6f37009fd743fd67fcfce059f9661bee6387683180649a80324a466"}
Aug 05 17:54:45.490116 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:45.490086    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"
Aug 05 17:54:45.492352 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:45.492306    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg" podStartSLOduration=1.492291936 podStartE2EDuration="1.492291936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:54:45.491473533 +0000 UTC m=+2541.236107861" watchObservedRunningTime="2026-08-05 17:54:45.492291936 +0000 UTC m=+2541.236926261"
Aug 05 17:54:45.522710 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:45.522682    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"
Aug 05 17:54:48.728177 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:48.728129    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9" podUID="053d0b97-e4db-4147-90ea-fd5e0f35fbcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Aug 05 17:54:54.522572 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:54.522543    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"
Aug 05 17:54:54.535747 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:54.535712    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"
Aug 05 17:54:55.533802 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:55.533765    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"
Aug 05 17:54:58.727890 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:54:58.727851    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9" podUID="053d0b97-e4db-4147-90ea-fd5e0f35fbcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Aug 05 17:55:08.728336 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:08.728295    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9" podUID="053d0b97-e4db-4147-90ea-fd5e0f35fbcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Aug 05 17:55:18.727985 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:18.727928    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9" podUID="053d0b97-e4db-4147-90ea-fd5e0f35fbcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Aug 05 17:55:21.187892 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:21.187849    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"]
Aug 05 17:55:28.728056 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:28.728012    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9" podUID="053d0b97-e4db-4147-90ea-fd5e0f35fbcf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/health\": dial tcp 10.133.0.87:8000: connect: connection refused"
Aug 05 17:55:36.207842 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:36.207795    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg" podUID="64f784b6-eeb1-45b0-a239-2559caab8c5e" containerName="main" containerID="cri-o://b3071406ab770604068d31066216e72fabfede71fee6071e8aa3c364edc7ba78" gracePeriod=45
Aug 05 17:55:36.463949 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:36.463876    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"
Aug 05 17:55:36.494214 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:36.494181    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64f784b6-eeb1-45b0-a239-2559caab8c5e-tmp-dir\") pod \"64f784b6-eeb1-45b0-a239-2559caab8c5e\" (UID: \"64f784b6-eeb1-45b0-a239-2559caab8c5e\") "
Aug 05 17:55:36.494410 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:36.494257    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64f784b6-eeb1-45b0-a239-2559caab8c5e-dshm\") pod \"64f784b6-eeb1-45b0-a239-2559caab8c5e\" (UID: \"64f784b6-eeb1-45b0-a239-2559caab8c5e\") "
Aug 05 17:55:36.494410 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:36.494333    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64f784b6-eeb1-45b0-a239-2559caab8c5e-home\") pod \"64f784b6-eeb1-45b0-a239-2559caab8c5e\" (UID: \"64f784b6-eeb1-45b0-a239-2559caab8c5e\") "
Aug 05 17:55:36.494410 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:36.494364    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64f784b6-eeb1-45b0-a239-2559caab8c5e-model-cache\") pod \"64f784b6-eeb1-45b0-a239-2559caab8c5e\" (UID: \"64f784b6-eeb1-45b0-a239-2559caab8c5e\") "
Aug 05 17:55:36.494410 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:36.494404    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64f784b6-eeb1-45b0-a239-2559caab8c5e-tls-certs\") pod \"64f784b6-eeb1-45b0-a239-2559caab8c5e\" (UID: \"64f784b6-eeb1-45b0-a239-2559caab8c5e\") "
Aug 05 17:55:36.494613 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:36.494440    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6f2q\" (UniqueName: \"kubernetes.io/projected/64f784b6-eeb1-45b0-a239-2559caab8c5e-kube-api-access-l6f2q\") pod \"64f784b6-eeb1-45b0-a239-2559caab8c5e\" (UID: \"64f784b6-eeb1-45b0-a239-2559caab8c5e\") "
Aug 05 17:55:36.494613 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:36.494578    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64f784b6-eeb1-45b0-a239-2559caab8c5e-home" (OuterVolumeSpecName: "home") pod "64f784b6-eeb1-45b0-a239-2559caab8c5e" (UID: "64f784b6-eeb1-45b0-a239-2559caab8c5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:55:36.495015 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:36.494742    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64f784b6-eeb1-45b0-a239-2559caab8c5e-model-cache" (OuterVolumeSpecName: "model-cache") pod "64f784b6-eeb1-45b0-a239-2559caab8c5e" (UID: "64f784b6-eeb1-45b0-a239-2559caab8c5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:55:36.495015 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:36.494764    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64f784b6-eeb1-45b0-a239-2559caab8c5e-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:55:36.495015 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:36.494891    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64f784b6-eeb1-45b0-a239-2559caab8c5e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "64f784b6-eeb1-45b0-a239-2559caab8c5e" (UID: "64f784b6-eeb1-45b0-a239-2559caab8c5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:55:36.497196 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:36.497155    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64f784b6-eeb1-45b0-a239-2559caab8c5e-dshm" (OuterVolumeSpecName: "dshm") pod "64f784b6-eeb1-45b0-a239-2559caab8c5e" (UID: "64f784b6-eeb1-45b0-a239-2559caab8c5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:55:36.497196 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:36.497174    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64f784b6-eeb1-45b0-a239-2559caab8c5e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "64f784b6-eeb1-45b0-a239-2559caab8c5e" (UID: "64f784b6-eeb1-45b0-a239-2559caab8c5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:55:36.497196 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:36.497181    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64f784b6-eeb1-45b0-a239-2559caab8c5e-kube-api-access-l6f2q" (OuterVolumeSpecName: "kube-api-access-l6f2q") pod "64f784b6-eeb1-45b0-a239-2559caab8c5e" (UID: "64f784b6-eeb1-45b0-a239-2559caab8c5e"). InnerVolumeSpecName "kube-api-access-l6f2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:55:36.596119 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:36.596082    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64f784b6-eeb1-45b0-a239-2559caab8c5e-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:55:36.596119 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:36.596111    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6f2q\" (UniqueName: \"kubernetes.io/projected/64f784b6-eeb1-45b0-a239-2559caab8c5e-kube-api-access-l6f2q\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:55:36.596119 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:36.596121    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64f784b6-eeb1-45b0-a239-2559caab8c5e-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:55:36.596119 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:36.596130    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64f784b6-eeb1-45b0-a239-2559caab8c5e-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:55:36.596397 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:36.596137    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64f784b6-eeb1-45b0-a239-2559caab8c5e-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:55:36.796325 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:36.796289    2561 generic.go:356] "Generic (PLEG): container finished" podID="64f784b6-eeb1-45b0-a239-2559caab8c5e" containerID="b3071406ab770604068d31066216e72fabfede71fee6071e8aa3c364edc7ba78" exitCode=0
Aug 05 17:55:36.796536 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:36.796345    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"
Aug 05 17:55:36.796536 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:36.796376    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg" event={"ID":"64f784b6-eeb1-45b0-a239-2559caab8c5e","Type":"ContainerDied","Data":"b3071406ab770604068d31066216e72fabfede71fee6071e8aa3c364edc7ba78"}
Aug 05 17:55:36.796536 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:36.796429    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg" event={"ID":"64f784b6-eeb1-45b0-a239-2559caab8c5e","Type":"ContainerDied","Data":"f3fc7c0dd6f37009fd743fd67fcfce059f9661bee6387683180649a80324a466"}
Aug 05 17:55:36.796536 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:36.796451    2561 scope.go:117] "RemoveContainer" containerID="b3071406ab770604068d31066216e72fabfede71fee6071e8aa3c364edc7ba78"
Aug 05 17:55:36.806151 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:36.806131    2561 scope.go:117] "RemoveContainer" containerID="b3071406ab770604068d31066216e72fabfede71fee6071e8aa3c364edc7ba78"
Aug 05 17:55:36.806483 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:55:36.806459    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3071406ab770604068d31066216e72fabfede71fee6071e8aa3c364edc7ba78\": container with ID starting with b3071406ab770604068d31066216e72fabfede71fee6071e8aa3c364edc7ba78 not found: ID does not exist" containerID="b3071406ab770604068d31066216e72fabfede71fee6071e8aa3c364edc7ba78"
Aug 05 17:55:36.806537 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:36.806493    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3071406ab770604068d31066216e72fabfede71fee6071e8aa3c364edc7ba78"} err="failed to get container status \"b3071406ab770604068d31066216e72fabfede71fee6071e8aa3c364edc7ba78\": rpc error: code = NotFound desc = could not find container \"b3071406ab770604068d31066216e72fabfede71fee6071e8aa3c364edc7ba78\": container with ID starting with b3071406ab770604068d31066216e72fabfede71fee6071e8aa3c364edc7ba78 not found: ID does not exist"
Aug 05 17:55:36.814279 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:36.814253    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"]
Aug 05 17:55:36.818555 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:36.818531    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-nvkgg"]
Aug 05 17:55:37.461431 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:37.461401    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"]
Aug 05 17:55:38.730258 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:38.730221    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64f784b6-eeb1-45b0-a239-2559caab8c5e" path="/var/lib/kubelet/pods/64f784b6-eeb1-45b0-a239-2559caab8c5e/volumes"
Aug 05 17:55:38.738315 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:38.738292    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:55:38.746062 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:38.746034    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:55:44.114365 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.114311    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"]
Aug 05 17:55:44.114916 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.114726    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64f784b6-eeb1-45b0-a239-2559caab8c5e" containerName="main"
Aug 05 17:55:44.114916 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.114739    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="64f784b6-eeb1-45b0-a239-2559caab8c5e" containerName="main"
Aug 05 17:55:44.114916 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.114807    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="64f784b6-eeb1-45b0-a239-2559caab8c5e" containerName="main"
Aug 05 17:55:44.119210 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.119191    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"
Aug 05 17:55:44.122926 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.122898    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"default-dockercfg-vx5w9\""
Aug 05 17:55:44.123098 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.122936    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"openshift-service-ca.crt\""
Aug 05 17:55:44.123098 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.122981    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v1-kserve-self-signed-certs\""
Aug 05 17:55:44.123098 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.123039    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"kube-root-ca.crt\""
Aug 05 17:55:44.127936 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.127326    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"]
Aug 05 17:55:44.266718 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.266683    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppw9r\" (UniqueName: \"kubernetes.io/projected/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-kube-api-access-ppw9r\") pod \"roll-v1-kserve-69cf8bcfc-s2fhr\" (UID: \"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"
Aug 05 17:55:44.266921 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.266739    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-s2fhr\" (UID: \"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"
Aug 05 17:55:44.266921 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.266852    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-s2fhr\" (UID: \"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"
Aug 05 17:55:44.266921 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.266895    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-s2fhr\" (UID: \"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"
Aug 05 17:55:44.267125 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.266982    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-home\") pod \"roll-v1-kserve-69cf8bcfc-s2fhr\" (UID: \"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"
Aug 05 17:55:44.267125 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.267026    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-dshm\") pod \"roll-v1-kserve-69cf8bcfc-s2fhr\" (UID: \"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"
Aug 05 17:55:44.368552 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.368460    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppw9r\" (UniqueName: \"kubernetes.io/projected/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-kube-api-access-ppw9r\") pod \"roll-v1-kserve-69cf8bcfc-s2fhr\" (UID: \"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"
Aug 05 17:55:44.368552 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.368534    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-s2fhr\" (UID: \"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"
Aug 05 17:55:44.368774 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.368591    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-s2fhr\" (UID: \"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"
Aug 05 17:55:44.368774 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.368617    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-s2fhr\" (UID: \"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"
Aug 05 17:55:44.368774 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.368759    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-home\") pod \"roll-v1-kserve-69cf8bcfc-s2fhr\" (UID: \"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"
Aug 05 17:55:44.368927 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.368809    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-dshm\") pod \"roll-v1-kserve-69cf8bcfc-s2fhr\" (UID: \"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"
Aug 05 17:55:44.369071 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.369051    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-s2fhr\" (UID: \"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"
Aug 05 17:55:44.369142 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.369104    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-s2fhr\" (UID: \"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"
Aug 05 17:55:44.369198 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.369167    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-home\") pod \"roll-v1-kserve-69cf8bcfc-s2fhr\" (UID: \"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"
Aug 05 17:55:44.371066 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.371042    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-dshm\") pod \"roll-v1-kserve-69cf8bcfc-s2fhr\" (UID: \"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"
Aug 05 17:55:44.371312 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.371296    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-s2fhr\" (UID: \"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"
Aug 05 17:55:44.376305 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.376281    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppw9r\" (UniqueName: \"kubernetes.io/projected/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-kube-api-access-ppw9r\") pod \"roll-v1-kserve-69cf8bcfc-s2fhr\" (UID: \"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"
Aug 05 17:55:44.431162 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.431120    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"
Aug 05 17:55:44.576574 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.576543    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"]
Aug 05 17:55:44.578354 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:55:44.578330    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a6c864_70c3_40fb_93ed_fb7e9f8eedf0.slice/crio-2366645a9382bc41bd0e4a0524c69265863de2d465f116954c1d17b5ad1874ec WatchSource:0}: Error finding container 2366645a9382bc41bd0e4a0524c69265863de2d465f116954c1d17b5ad1874ec: Status 404 returned error can't find the container with id 2366645a9382bc41bd0e4a0524c69265863de2d465f116954c1d17b5ad1874ec
Aug 05 17:55:44.856154 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.856111    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr" event={"ID":"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0","Type":"ContainerStarted","Data":"89e7912dcadaba31834b16784f0d5887de9312a24008f1f4843ff2fdc4873e17"}
Aug 05 17:55:44.856329 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.856200    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr" event={"ID":"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0","Type":"ContainerStarted","Data":"2366645a9382bc41bd0e4a0524c69265863de2d465f116954c1d17b5ad1874ec"}
Aug 05 17:55:44.879243 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:44.879145    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr" podStartSLOduration=0.879130755 podStartE2EDuration="879.130755ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:55:44.876283201 +0000 UTC m=+2600.620917725" watchObservedRunningTime="2026-08-05 17:55:44.879130755 +0000 UTC m=+2600.623765081"
Aug 05 17:55:45.431831 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:45.431798    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"
Aug 05 17:55:52.485032 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:52.484910    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm" podUID="3f2be8c9-e2c2-46f1-ba6e-8496c7fad249" containerName="main" containerID="cri-o://a86537f2f1ea965b85f4cce9d8aa34ac630d5087f29c42ecd5adfe42562f017e" gracePeriod=45
Aug 05 17:55:52.745970 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:52.745865    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"
Aug 05 17:55:52.846947 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:52.846907    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-tls-certs\") pod \"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249\" (UID: \"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249\") "
Aug 05 17:55:52.847231 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:52.847058    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4n728\" (UniqueName: \"kubernetes.io/projected/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-kube-api-access-4n728\") pod \"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249\" (UID: \"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249\") "
Aug 05 17:55:52.847231 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:52.847086    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-model-cache\") pod \"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249\" (UID: \"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249\") "
Aug 05 17:55:52.847231 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:52.847115    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-tmp-dir\") pod \"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249\" (UID: \"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249\") "
Aug 05 17:55:52.847231 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:52.847149    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-home\") pod \"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249\" (UID: \"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249\") "
Aug 05 17:55:52.847231 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:52.847196    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-dshm\") pod \"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249\" (UID: \"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249\") "
Aug 05 17:55:52.847520 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:52.847438    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-model-cache" (OuterVolumeSpecName: "model-cache") pod "3f2be8c9-e2c2-46f1-ba6e-8496c7fad249" (UID: "3f2be8c9-e2c2-46f1-ba6e-8496c7fad249"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:55:52.847520 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:52.847470    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3f2be8c9-e2c2-46f1-ba6e-8496c7fad249" (UID: "3f2be8c9-e2c2-46f1-ba6e-8496c7fad249"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:55:52.847520 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:52.847479    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-home" (OuterVolumeSpecName: "home") pod "3f2be8c9-e2c2-46f1-ba6e-8496c7fad249" (UID: "3f2be8c9-e2c2-46f1-ba6e-8496c7fad249"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:55:52.849356 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:52.849326    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-dshm" (OuterVolumeSpecName: "dshm") pod "3f2be8c9-e2c2-46f1-ba6e-8496c7fad249" (UID: "3f2be8c9-e2c2-46f1-ba6e-8496c7fad249"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:55:52.849621 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:52.849601    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f2be8c9-e2c2-46f1-ba6e-8496c7fad249" (UID: "3f2be8c9-e2c2-46f1-ba6e-8496c7fad249"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:55:52.849693 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:52.849619    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-kube-api-access-4n728" (OuterVolumeSpecName: "kube-api-access-4n728") pod "3f2be8c9-e2c2-46f1-ba6e-8496c7fad249" (UID: "3f2be8c9-e2c2-46f1-ba6e-8496c7fad249"). InnerVolumeSpecName "kube-api-access-4n728". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:55:52.906769 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:52.906734    2561 generic.go:356] "Generic (PLEG): container finished" podID="3f2be8c9-e2c2-46f1-ba6e-8496c7fad249" containerID="a86537f2f1ea965b85f4cce9d8aa34ac630d5087f29c42ecd5adfe42562f017e" exitCode=0
Aug 05 17:55:52.906947 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:52.906782    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"
Aug 05 17:55:52.906947 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:52.906819    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm" event={"ID":"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249","Type":"ContainerDied","Data":"a86537f2f1ea965b85f4cce9d8aa34ac630d5087f29c42ecd5adfe42562f017e"}
Aug 05 17:55:52.906947 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:52.906884    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm" event={"ID":"3f2be8c9-e2c2-46f1-ba6e-8496c7fad249","Type":"ContainerDied","Data":"b759c00fd614c590b72cb2b302763da99231083ef89e788ec41e8d8247d9ecc2"}
Aug 05 17:55:52.906947 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:52.906902    2561 scope.go:117] "RemoveContainer" containerID="a86537f2f1ea965b85f4cce9d8aa34ac630d5087f29c42ecd5adfe42562f017e"
Aug 05 17:55:52.917122 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:52.917103    2561 scope.go:117] "RemoveContainer" containerID="a86537f2f1ea965b85f4cce9d8aa34ac630d5087f29c42ecd5adfe42562f017e"
Aug 05 17:55:52.917372 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:55:52.917345    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a86537f2f1ea965b85f4cce9d8aa34ac630d5087f29c42ecd5adfe42562f017e\": container with ID starting with a86537f2f1ea965b85f4cce9d8aa34ac630d5087f29c42ecd5adfe42562f017e not found: ID does not exist" containerID="a86537f2f1ea965b85f4cce9d8aa34ac630d5087f29c42ecd5adfe42562f017e"
Aug 05 17:55:52.917437 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:52.917378    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a86537f2f1ea965b85f4cce9d8aa34ac630d5087f29c42ecd5adfe42562f017e"} err="failed to get container status \"a86537f2f1ea965b85f4cce9d8aa34ac630d5087f29c42ecd5adfe42562f017e\": rpc error: code = NotFound desc = could not find container \"a86537f2f1ea965b85f4cce9d8aa34ac630d5087f29c42ecd5adfe42562f017e\": container with ID starting with a86537f2f1ea965b85f4cce9d8aa34ac630d5087f29c42ecd5adfe42562f017e not found: ID does not exist"
Aug 05 17:55:52.931152 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:52.931121    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"]
Aug 05 17:55:52.934464 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:52.934439    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-kdqmm"]
Aug 05 17:55:52.948415 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:52.948388    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n728\" (UniqueName: \"kubernetes.io/projected/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-kube-api-access-4n728\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:55:52.948508 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:52.948417    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:55:52.948508 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:52.948460    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:55:52.948508 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:52.948472    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:55:52.948508 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:52.948484    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:55:52.948508 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:52.948496    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:55:54.432245 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:54.432195    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"
Aug 05 17:55:54.446032 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:54.445994    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"
Aug 05 17:55:54.733565 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:54.733470    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f2be8c9-e2c2-46f1-ba6e-8496c7fad249" path="/var/lib/kubelet/pods/3f2be8c9-e2c2-46f1-ba6e-8496c7fad249/volumes"
Aug 05 17:55:54.932357 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:54.932318    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"
Aug 05 17:55:57.498400 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:57.498365    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"]
Aug 05 17:55:57.498908 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:57.498889    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f2be8c9-e2c2-46f1-ba6e-8496c7fad249" containerName="main"
Aug 05 17:55:57.499015 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:57.498911    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f2be8c9-e2c2-46f1-ba6e-8496c7fad249" containerName="main"
Aug 05 17:55:57.499076 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:57.499043    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f2be8c9-e2c2-46f1-ba6e-8496c7fad249" containerName="main"
Aug 05 17:55:57.503472 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:57.503450    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"
Aug 05 17:55:57.505942 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:57.505922    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v2-kserve-self-signed-certs\""
Aug 05 17:55:57.514179 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:57.514153    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"]
Aug 05 17:55:57.593030 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:57.592999    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/769fc130-f45e-48a6-831e-961c254c26d0-dshm\") pod \"roll-v2-kserve-d4dbf9d67-99hx6\" (UID: \"769fc130-f45e-48a6-831e-961c254c26d0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"
Aug 05 17:55:57.593241 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:57.593044    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/769fc130-f45e-48a6-831e-961c254c26d0-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-99hx6\" (UID: \"769fc130-f45e-48a6-831e-961c254c26d0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"
Aug 05 17:55:57.593241 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:57.593115    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88mmm\" (UniqueName: \"kubernetes.io/projected/769fc130-f45e-48a6-831e-961c254c26d0-kube-api-access-88mmm\") pod \"roll-v2-kserve-d4dbf9d67-99hx6\" (UID: \"769fc130-f45e-48a6-831e-961c254c26d0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"
Aug 05 17:55:57.593241 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:57.593196    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/769fc130-f45e-48a6-831e-961c254c26d0-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-99hx6\" (UID: \"769fc130-f45e-48a6-831e-961c254c26d0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"
Aug 05 17:55:57.593241 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:57.593237    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/769fc130-f45e-48a6-831e-961c254c26d0-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-99hx6\" (UID: \"769fc130-f45e-48a6-831e-961c254c26d0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"
Aug 05 17:55:57.593419 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:57.593266    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/769fc130-f45e-48a6-831e-961c254c26d0-home\") pod \"roll-v2-kserve-d4dbf9d67-99hx6\" (UID: \"769fc130-f45e-48a6-831e-961c254c26d0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"
Aug 05 17:55:57.694093 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:57.694057    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/769fc130-f45e-48a6-831e-961c254c26d0-dshm\") pod \"roll-v2-kserve-d4dbf9d67-99hx6\" (UID: \"769fc130-f45e-48a6-831e-961c254c26d0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"
Aug 05 17:55:57.694295 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:57.694107    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/769fc130-f45e-48a6-831e-961c254c26d0-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-99hx6\" (UID: \"769fc130-f45e-48a6-831e-961c254c26d0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"
Aug 05 17:55:57.694295 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:57.694139    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88mmm\" (UniqueName: \"kubernetes.io/projected/769fc130-f45e-48a6-831e-961c254c26d0-kube-api-access-88mmm\") pod \"roll-v2-kserve-d4dbf9d67-99hx6\" (UID: \"769fc130-f45e-48a6-831e-961c254c26d0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"
Aug 05 17:55:57.694295 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:57.694170    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/769fc130-f45e-48a6-831e-961c254c26d0-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-99hx6\" (UID: \"769fc130-f45e-48a6-831e-961c254c26d0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"
Aug 05 17:55:57.694295 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:57.694192    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/769fc130-f45e-48a6-831e-961c254c26d0-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-99hx6\" (UID: \"769fc130-f45e-48a6-831e-961c254c26d0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"
Aug 05 17:55:57.694295 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:57.694218    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/769fc130-f45e-48a6-831e-961c254c26d0-home\") pod \"roll-v2-kserve-d4dbf9d67-99hx6\" (UID: \"769fc130-f45e-48a6-831e-961c254c26d0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"
Aug 05 17:55:57.694638 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:57.694613    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/769fc130-f45e-48a6-831e-961c254c26d0-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-99hx6\" (UID: \"769fc130-f45e-48a6-831e-961c254c26d0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"
Aug 05 17:55:57.694703 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:57.694638    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/769fc130-f45e-48a6-831e-961c254c26d0-home\") pod \"roll-v2-kserve-d4dbf9d67-99hx6\" (UID: \"769fc130-f45e-48a6-831e-961c254c26d0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"
Aug 05 17:55:57.694703 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:57.694624    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/769fc130-f45e-48a6-831e-961c254c26d0-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-99hx6\" (UID: \"769fc130-f45e-48a6-831e-961c254c26d0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"
Aug 05 17:55:57.696545 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:57.696504    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/769fc130-f45e-48a6-831e-961c254c26d0-dshm\") pod \"roll-v2-kserve-d4dbf9d67-99hx6\" (UID: \"769fc130-f45e-48a6-831e-961c254c26d0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"
Aug 05 17:55:57.696678 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:57.696645    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/769fc130-f45e-48a6-831e-961c254c26d0-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-99hx6\" (UID: \"769fc130-f45e-48a6-831e-961c254c26d0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"
Aug 05 17:55:57.707617 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:57.707596    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88mmm\" (UniqueName: \"kubernetes.io/projected/769fc130-f45e-48a6-831e-961c254c26d0-kube-api-access-88mmm\") pod \"roll-v2-kserve-d4dbf9d67-99hx6\" (UID: \"769fc130-f45e-48a6-831e-961c254c26d0\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"
Aug 05 17:55:57.815555 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:57.815517    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"
Aug 05 17:55:57.954903 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:57.954873    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"]
Aug 05 17:55:57.956534 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:55:57.956507    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod769fc130_f45e_48a6_831e_961c254c26d0.slice/crio-4d1ffc65add4a7ca6ba969c7807d3bad7e5f5fb308a71561a87f5084be78536f WatchSource:0}: Error finding container 4d1ffc65add4a7ca6ba969c7807d3bad7e5f5fb308a71561a87f5084be78536f: Status 404 returned error can't find the container with id 4d1ffc65add4a7ca6ba969c7807d3bad7e5f5fb308a71561a87f5084be78536f
Aug 05 17:55:58.944221 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:58.944186    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6" event={"ID":"769fc130-f45e-48a6-831e-961c254c26d0","Type":"ContainerStarted","Data":"acbe8a22ab442a555ea5b80bf2514ce1da381d4584a51844ebd30f7788316721"}
Aug 05 17:55:58.944590 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:58.944239    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6" event={"ID":"769fc130-f45e-48a6-831e-961c254c26d0","Type":"ContainerStarted","Data":"4d1ffc65add4a7ca6ba969c7807d3bad7e5f5fb308a71561a87f5084be78536f"}
Aug 05 17:55:58.962530 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:58.962475    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6" podStartSLOduration=1.9624582720000001 podStartE2EDuration="1.962458272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:55:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:55:58.960463277 +0000 UTC m=+2614.705097605" watchObservedRunningTime="2026-08-05 17:55:58.962458272 +0000 UTC m=+2614.707092598"
Aug 05 17:55:59.816197 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:55:59.816157    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"
Aug 05 17:56:07.815678 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:07.815637    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"
Aug 05 17:56:07.829604 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:07.829571    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"
Aug 05 17:56:08.021757 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:08.021716    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"
Aug 05 17:56:24.732900 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:24.732830    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"]
Aug 05 17:56:39.759110 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:39.759042    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9" podUID="053d0b97-e4db-4147-90ea-fd5e0f35fbcf" containerName="main" containerID="cri-o://2f51dcdcbf8148dddbc9397491f1f7a9961f9255d36939b6faf69dff85c60308" gracePeriod=45
Aug 05 17:56:40.424082 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:40.424058    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:56:40.471070 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:40.470979    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-kserve-provision-location\") pod \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") "
Aug 05 17:56:40.471070 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:40.471035    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-dshm\") pod \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") "
Aug 05 17:56:40.471070 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:40.471062    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-tmp-dir\") pod \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") "
Aug 05 17:56:40.471353 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:40.471119    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9x67\" (UniqueName: \"kubernetes.io/projected/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-kube-api-access-t9x67\") pod \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") "
Aug 05 17:56:40.471353 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:40.471182    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-tls-certs\") pod \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") "
Aug 05 17:56:40.471353 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:40.471207    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-model-cache\") pod \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") "
Aug 05 17:56:40.471353 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:40.471233    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-cabundle-cert\") pod \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") "
Aug 05 17:56:40.471353 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:40.471262    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-home\") pod \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\" (UID: \"053d0b97-e4db-4147-90ea-fd5e0f35fbcf\") "
Aug 05 17:56:40.472119 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:40.472025    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-home" (OuterVolumeSpecName: "home") pod "053d0b97-e4db-4147-90ea-fd5e0f35fbcf" (UID: "053d0b97-e4db-4147-90ea-fd5e0f35fbcf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:56:40.473781 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:40.472392    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "053d0b97-e4db-4147-90ea-fd5e0f35fbcf" (UID: "053d0b97-e4db-4147-90ea-fd5e0f35fbcf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:56:40.473781 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:40.472655    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-model-cache" (OuterVolumeSpecName: "model-cache") pod "053d0b97-e4db-4147-90ea-fd5e0f35fbcf" (UID: "053d0b97-e4db-4147-90ea-fd5e0f35fbcf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:56:40.474449 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:40.474415    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-kube-api-access-t9x67" (OuterVolumeSpecName: "kube-api-access-t9x67") pod "053d0b97-e4db-4147-90ea-fd5e0f35fbcf" (UID: "053d0b97-e4db-4147-90ea-fd5e0f35fbcf"). InnerVolumeSpecName "kube-api-access-t9x67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:56:40.474650 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:40.474624    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-dshm" (OuterVolumeSpecName: "dshm") pod "053d0b97-e4db-4147-90ea-fd5e0f35fbcf" (UID: "053d0b97-e4db-4147-90ea-fd5e0f35fbcf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:56:40.474975 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:40.474915    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "053d0b97-e4db-4147-90ea-fd5e0f35fbcf" (UID: "053d0b97-e4db-4147-90ea-fd5e0f35fbcf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:56:40.488667 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:40.488627    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "053d0b97-e4db-4147-90ea-fd5e0f35fbcf" (UID: "053d0b97-e4db-4147-90ea-fd5e0f35fbcf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:56:40.496632 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:40.496597    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "053d0b97-e4db-4147-90ea-fd5e0f35fbcf" (UID: "053d0b97-e4db-4147-90ea-fd5e0f35fbcf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:56:40.572678 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:40.572638    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:56:40.572678 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:40.572670    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:56:40.572678 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:40.572678    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-cabundle-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:56:40.572678 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:40.572687    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:56:40.572946 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:40.572696    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-kserve-provision-location\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:56:40.572946 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:40.572705    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:56:40.572946 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:40.572713    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:56:40.572946 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:40.572722    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9x67\" (UniqueName: \"kubernetes.io/projected/053d0b97-e4db-4147-90ea-fd5e0f35fbcf-kube-api-access-t9x67\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:56:41.199158 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:41.199121    2561 generic.go:356] "Generic (PLEG): container finished" podID="053d0b97-e4db-4147-90ea-fd5e0f35fbcf" containerID="2f51dcdcbf8148dddbc9397491f1f7a9961f9255d36939b6faf69dff85c60308" exitCode=0
Aug 05 17:56:41.199625 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:41.199174    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"
Aug 05 17:56:41.199625 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:41.199180    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9" event={"ID":"053d0b97-e4db-4147-90ea-fd5e0f35fbcf","Type":"ContainerDied","Data":"2f51dcdcbf8148dddbc9397491f1f7a9961f9255d36939b6faf69dff85c60308"}
Aug 05 17:56:41.199625 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:41.199247    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9" event={"ID":"053d0b97-e4db-4147-90ea-fd5e0f35fbcf","Type":"ContainerDied","Data":"458bd29a56ff1baa2ce2b73c203821f862d04c02649ee89dbf99e0a8666f33f2"}
Aug 05 17:56:41.199625 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:41.199272    2561 scope.go:117] "RemoveContainer" containerID="2f51dcdcbf8148dddbc9397491f1f7a9961f9255d36939b6faf69dff85c60308"
Aug 05 17:56:41.208650 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:41.208633    2561 scope.go:117] "RemoveContainer" containerID="f8c383fa940e986b5297dc8a1f4cde014797f50db7acbe9c9019e892c846ed30"
Aug 05 17:56:41.217324 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:41.217306    2561 scope.go:117] "RemoveContainer" containerID="2f51dcdcbf8148dddbc9397491f1f7a9961f9255d36939b6faf69dff85c60308"
Aug 05 17:56:41.217598 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:56:41.217580    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f51dcdcbf8148dddbc9397491f1f7a9961f9255d36939b6faf69dff85c60308\": container with ID starting with 2f51dcdcbf8148dddbc9397491f1f7a9961f9255d36939b6faf69dff85c60308 not found: ID does not exist" containerID="2f51dcdcbf8148dddbc9397491f1f7a9961f9255d36939b6faf69dff85c60308"
Aug 05 17:56:41.217674 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:41.217607    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f51dcdcbf8148dddbc9397491f1f7a9961f9255d36939b6faf69dff85c60308"} err="failed to get container status \"2f51dcdcbf8148dddbc9397491f1f7a9961f9255d36939b6faf69dff85c60308\": rpc error: code = NotFound desc = could not find container \"2f51dcdcbf8148dddbc9397491f1f7a9961f9255d36939b6faf69dff85c60308\": container with ID starting with 2f51dcdcbf8148dddbc9397491f1f7a9961f9255d36939b6faf69dff85c60308 not found: ID does not exist"
Aug 05 17:56:41.217674 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:41.217625    2561 scope.go:117] "RemoveContainer" containerID="f8c383fa940e986b5297dc8a1f4cde014797f50db7acbe9c9019e892c846ed30"
Aug 05 17:56:41.217674 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:41.217644    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"]
Aug 05 17:56:41.217858 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:56:41.217841    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8c383fa940e986b5297dc8a1f4cde014797f50db7acbe9c9019e892c846ed30\": container with ID starting with f8c383fa940e986b5297dc8a1f4cde014797f50db7acbe9c9019e892c846ed30 not found: ID does not exist" containerID="f8c383fa940e986b5297dc8a1f4cde014797f50db7acbe9c9019e892c846ed30"
Aug 05 17:56:41.217897 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:41.217865    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c383fa940e986b5297dc8a1f4cde014797f50db7acbe9c9019e892c846ed30"} err="failed to get container status \"f8c383fa940e986b5297dc8a1f4cde014797f50db7acbe9c9019e892c846ed30\": rpc error: code = NotFound desc = could not find container \"f8c383fa940e986b5297dc8a1f4cde014797f50db7acbe9c9019e892c846ed30\": container with ID starting with f8c383fa940e986b5297dc8a1f4cde014797f50db7acbe9c9019e892c846ed30 not found: ID does not exist"
Aug 05 17:56:41.223244 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:41.223222    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-bdbb5885c-282t9"]
Aug 05 17:56:42.730138 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:42.730093    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="053d0b97-e4db-4147-90ea-fd5e0f35fbcf" path="/var/lib/kubelet/pods/053d0b97-e4db-4147-90ea-fd5e0f35fbcf/volumes"
Aug 05 17:56:57.930898 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:57.930847    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"]
Aug 05 17:56:57.931713 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:57.931681    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="053d0b97-e4db-4147-90ea-fd5e0f35fbcf" containerName="storage-initializer"
Aug 05 17:56:57.931834 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:57.931716    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="053d0b97-e4db-4147-90ea-fd5e0f35fbcf" containerName="storage-initializer"
Aug 05 17:56:57.931834 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:57.931758    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="053d0b97-e4db-4147-90ea-fd5e0f35fbcf" containerName="main"
Aug 05 17:56:57.931834 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:57.931767    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="053d0b97-e4db-4147-90ea-fd5e0f35fbcf" containerName="main"
Aug 05 17:56:57.932026 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:57.931985    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="053d0b97-e4db-4147-90ea-fd5e0f35fbcf" containerName="main"
Aug 05 17:56:57.937862 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:57.937837    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:56:57.940583 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:57.940553    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Aug 05 17:56:57.940722 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:57.940590    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Aug 05 17:56:57.940826 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:57.940806    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Aug 05 17:56:57.940910 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:57.940829    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Aug 05 17:56:57.940910 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:57.940839    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Aug 05 17:56:57.941172 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:57.941152    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-cpwzs\""
Aug 05 17:56:57.943012 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:57.942987    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"]
Aug 05 17:56:58.028329 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:58.028295    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t8px\" (UniqueName: \"kubernetes.io/projected/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-kube-api-access-4t8px\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:56:58.028539 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:58.028356    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:56:58.028539 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:58.028388    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:56:58.028539 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:58.028422    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:56:58.028539 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:58.028501    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:56:58.028539 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:58.028531    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:56:58.028762 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:58.028554    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:56:58.028762 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:58.028603    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:56:58.129858 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:58.129819    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:56:58.130099 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:58.129867    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t8px\" (UniqueName: \"kubernetes.io/projected/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-kube-api-access-4t8px\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:56:58.130099 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:58.130059    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:56:58.130259 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:58.130108    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:56:58.130259 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:58.130152    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:56:58.130259 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:58.130226    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:56:58.130400 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:58.130270    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:56:58.130400 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:58.130276    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:56:58.130482 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:58.130392    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:56:58.130482 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:58.130369    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:56:58.130671 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:58.130651    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:56:58.130738 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:58.130674    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:56:58.130845 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:58.130826    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:56:58.132310 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:58.132286    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:56:58.132772 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:58.132756    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:56:58.142528 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:58.142496    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t8px\" (UniqueName: \"kubernetes.io/projected/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-kube-api-access-4t8px\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:56:58.251105 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:58.251013    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:56:58.398763 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:58.398735    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"]
Aug 05 17:56:58.399817 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:56:58.399790    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead6f2f1_1e85_49cc_b842_c7be92fa4e80.slice/crio-d3701a2de8a56fe8310692e0c76e14c00b80f6b8e5df8a67436082e71579378a WatchSource:0}: Error finding container d3701a2de8a56fe8310692e0c76e14c00b80f6b8e5df8a67436082e71579378a: Status 404 returned error can't find the container with id d3701a2de8a56fe8310692e0c76e14c00b80f6b8e5df8a67436082e71579378a
Aug 05 17:56:59.299256 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:59.299213    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5" event={"ID":"ead6f2f1-1e85-49cc-b842-c7be92fa4e80","Type":"ContainerStarted","Data":"95dfb6bca31f14038c8c1468aa32b7d7aef18e9e4171d9cc3087714766aaee07"}
Aug 05 17:56:59.299685 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:56:59.299276    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5" event={"ID":"ead6f2f1-1e85-49cc-b842-c7be92fa4e80","Type":"ContainerStarted","Data":"d3701a2de8a56fe8310692e0c76e14c00b80f6b8e5df8a67436082e71579378a"}
Aug 05 17:57:01.984516 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:01.984474    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"]
Aug 05 17:57:02.319384 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:02.319354    2561 generic.go:356] "Generic (PLEG): container finished" podID="ead6f2f1-1e85-49cc-b842-c7be92fa4e80" containerID="95dfb6bca31f14038c8c1468aa32b7d7aef18e9e4171d9cc3087714766aaee07" exitCode=0
Aug 05 17:57:02.319581 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:02.319425    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5" event={"ID":"ead6f2f1-1e85-49cc-b842-c7be92fa4e80","Type":"ContainerDied","Data":"95dfb6bca31f14038c8c1468aa32b7d7aef18e9e4171d9cc3087714766aaee07"}
Aug 05 17:57:02.591130 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:02.591037    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"]
Aug 05 17:57:03.327597 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:03.327564    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5" event={"ID":"ead6f2f1-1e85-49cc-b842-c7be92fa4e80","Type":"ContainerStarted","Data":"38252d4bd59d188d38dd89208f26490ef7d85769990ae5a8be74dbfb53b40858"}
Aug 05 17:57:03.352161 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:03.352109    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5" podStartSLOduration=3.403778127 podStartE2EDuration="6.352092211s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.948314084s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:57:03.349438005 +0000 UTC m=+2679.094072333" watchObservedRunningTime="2026-08-05 17:57:03.352092211 +0000 UTC m=+2679.096726537"
Aug 05 17:57:04.251759 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:04.251718    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:57:08.251644 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:08.251604    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:57:08.253742 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:08.253693    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5" podUID="ead6f2f1-1e85-49cc-b842-c7be92fa4e80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Aug 05 17:57:09.305651 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:09.305617    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"]
Aug 05 17:57:09.310220 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:09.310198    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"
Aug 05 17:57:09.313771 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:09.313740    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"openshift-service-ca.crt\""
Aug 05 17:57:09.313919 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:09.313801    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"default-dockercfg-v2plx\""
Aug 05 17:57:09.313919 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:09.313815    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"kube-root-ca.crt\""
Aug 05 17:57:09.313919 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:09.313850    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v1-kserve-self-signed-certs\""
Aug 05 17:57:09.320477 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:09.320453    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"]
Aug 05 17:57:09.338302 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:09.338268    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e97cae0-4e01-4944-b514-1418600db170-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-p57lm\" (UID: \"5e97cae0-4e01-4944-b514-1418600db170\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"
Aug 05 17:57:09.338454 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:09.338308    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e97cae0-4e01-4944-b514-1418600db170-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-p57lm\" (UID: \"5e97cae0-4e01-4944-b514-1418600db170\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"
Aug 05 17:57:09.338454 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:09.338338    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e97cae0-4e01-4944-b514-1418600db170-dshm\") pod \"owner-v1-kserve-75579ff5bc-p57lm\" (UID: \"5e97cae0-4e01-4944-b514-1418600db170\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"
Aug 05 17:57:09.338529 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:09.338471    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz27k\" (UniqueName: \"kubernetes.io/projected/5e97cae0-4e01-4944-b514-1418600db170-kube-api-access-hz27k\") pod \"owner-v1-kserve-75579ff5bc-p57lm\" (UID: \"5e97cae0-4e01-4944-b514-1418600db170\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"
Aug 05 17:57:09.338529 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:09.338521    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e97cae0-4e01-4944-b514-1418600db170-model-cache\") pod \"owner-v1-kserve-75579ff5bc-p57lm\" (UID: \"5e97cae0-4e01-4944-b514-1418600db170\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"
Aug 05 17:57:09.338610 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:09.338550    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e97cae0-4e01-4944-b514-1418600db170-home\") pod \"owner-v1-kserve-75579ff5bc-p57lm\" (UID: \"5e97cae0-4e01-4944-b514-1418600db170\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"
Aug 05 17:57:09.439180 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:09.439139    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e97cae0-4e01-4944-b514-1418600db170-dshm\") pod \"owner-v1-kserve-75579ff5bc-p57lm\" (UID: \"5e97cae0-4e01-4944-b514-1418600db170\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"
Aug 05 17:57:09.439392 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:09.439227    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz27k\" (UniqueName: \"kubernetes.io/projected/5e97cae0-4e01-4944-b514-1418600db170-kube-api-access-hz27k\") pod \"owner-v1-kserve-75579ff5bc-p57lm\" (UID: \"5e97cae0-4e01-4944-b514-1418600db170\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"
Aug 05 17:57:09.439392 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:09.439255    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e97cae0-4e01-4944-b514-1418600db170-model-cache\") pod \"owner-v1-kserve-75579ff5bc-p57lm\" (UID: \"5e97cae0-4e01-4944-b514-1418600db170\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"
Aug 05 17:57:09.439392 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:09.439278    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e97cae0-4e01-4944-b514-1418600db170-home\") pod \"owner-v1-kserve-75579ff5bc-p57lm\" (UID: \"5e97cae0-4e01-4944-b514-1418600db170\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"
Aug 05 17:57:09.439392 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:09.439343    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e97cae0-4e01-4944-b514-1418600db170-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-p57lm\" (UID: \"5e97cae0-4e01-4944-b514-1418600db170\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"
Aug 05 17:57:09.439392 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:09.439369    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e97cae0-4e01-4944-b514-1418600db170-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-p57lm\" (UID: \"5e97cae0-4e01-4944-b514-1418600db170\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"
Aug 05 17:57:09.439727 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:09.439703    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e97cae0-4e01-4944-b514-1418600db170-model-cache\") pod \"owner-v1-kserve-75579ff5bc-p57lm\" (UID: \"5e97cae0-4e01-4944-b514-1418600db170\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"
Aug 05 17:57:09.439837 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:09.439703    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e97cae0-4e01-4944-b514-1418600db170-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-p57lm\" (UID: \"5e97cae0-4e01-4944-b514-1418600db170\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"
Aug 05 17:57:09.439837 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:09.439744    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e97cae0-4e01-4944-b514-1418600db170-home\") pod \"owner-v1-kserve-75579ff5bc-p57lm\" (UID: \"5e97cae0-4e01-4944-b514-1418600db170\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"
Aug 05 17:57:09.441726 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:09.441695    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e97cae0-4e01-4944-b514-1418600db170-dshm\") pod \"owner-v1-kserve-75579ff5bc-p57lm\" (UID: \"5e97cae0-4e01-4944-b514-1418600db170\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"
Aug 05 17:57:09.442274 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:09.442251    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e97cae0-4e01-4944-b514-1418600db170-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-p57lm\" (UID: \"5e97cae0-4e01-4944-b514-1418600db170\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"
Aug 05 17:57:09.447831 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:09.447806    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz27k\" (UniqueName: \"kubernetes.io/projected/5e97cae0-4e01-4944-b514-1418600db170-kube-api-access-hz27k\") pod \"owner-v1-kserve-75579ff5bc-p57lm\" (UID: \"5e97cae0-4e01-4944-b514-1418600db170\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"
Aug 05 17:57:09.621847 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:09.621758    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"
Aug 05 17:57:09.794153 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:09.794113    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"]
Aug 05 17:57:09.796741 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:57:09.796710    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e97cae0_4e01_4944_b514_1418600db170.slice/crio-c299f6307c59438c0b7cca81bffa23791aa432ed5b882d3071b4ee69cbd3c39a WatchSource:0}: Error finding container c299f6307c59438c0b7cca81bffa23791aa432ed5b882d3071b4ee69cbd3c39a: Status 404 returned error can't find the container with id c299f6307c59438c0b7cca81bffa23791aa432ed5b882d3071b4ee69cbd3c39a
Aug 05 17:57:10.372140 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:10.372101    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm" event={"ID":"5e97cae0-4e01-4944-b514-1418600db170","Type":"ContainerStarted","Data":"df8483896a17afacd23664b1735b9ab316732de2e286cc895d5dc097538d2886"}
Aug 05 17:57:10.372541 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:10.372165    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm" event={"ID":"5e97cae0-4e01-4944-b514-1418600db170","Type":"ContainerStarted","Data":"c299f6307c59438c0b7cca81bffa23791aa432ed5b882d3071b4ee69cbd3c39a"}
Aug 05 17:57:10.393476 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:10.393401    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm" podStartSLOduration=1.393379597 podStartE2EDuration="1.393379597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:57:10.391652105 +0000 UTC m=+2686.136286432" watchObservedRunningTime="2026-08-05 17:57:10.393379597 +0000 UTC m=+2686.138013925"
Aug 05 17:57:10.622113 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:10.622078    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"
Aug 05 17:57:17.004009 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.003872    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr" podUID="52a6c864-70c3-40fb-93ed-fb7e9f8eedf0" containerName="main" containerID="cri-o://89e7912dcadaba31834b16784f0d5887de9312a24008f1f4843ff2fdc4873e17" gracePeriod=45
Aug 05 17:57:17.265758 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.265675    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"
Aug 05 17:57:17.325302 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.325252    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppw9r\" (UniqueName: \"kubernetes.io/projected/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-kube-api-access-ppw9r\") pod \"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0\" (UID: \"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0\") "
Aug 05 17:57:17.325500 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.325349    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-model-cache\") pod \"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0\" (UID: \"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0\") "
Aug 05 17:57:17.325500 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.325402    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-dshm\") pod \"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0\" (UID: \"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0\") "
Aug 05 17:57:17.325500 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.325433    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-home\") pod \"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0\" (UID: \"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0\") "
Aug 05 17:57:17.325500 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.325462    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-tls-certs\") pod \"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0\" (UID: \"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0\") "
Aug 05 17:57:17.325500 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.325498    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-tmp-dir\") pod \"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0\" (UID: \"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0\") "
Aug 05 17:57:17.325821 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.325609    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-model-cache" (OuterVolumeSpecName: "model-cache") pod "52a6c864-70c3-40fb-93ed-fb7e9f8eedf0" (UID: "52a6c864-70c3-40fb-93ed-fb7e9f8eedf0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:57:17.325886 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.325859    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:57:17.326065 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.326038    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-home" (OuterVolumeSpecName: "home") pod "52a6c864-70c3-40fb-93ed-fb7e9f8eedf0" (UID: "52a6c864-70c3-40fb-93ed-fb7e9f8eedf0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:57:17.326171 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.326152    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "52a6c864-70c3-40fb-93ed-fb7e9f8eedf0" (UID: "52a6c864-70c3-40fb-93ed-fb7e9f8eedf0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:57:17.328760 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.328731    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "52a6c864-70c3-40fb-93ed-fb7e9f8eedf0" (UID: "52a6c864-70c3-40fb-93ed-fb7e9f8eedf0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:57:17.329416 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.329390    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-kube-api-access-ppw9r" (OuterVolumeSpecName: "kube-api-access-ppw9r") pod "52a6c864-70c3-40fb-93ed-fb7e9f8eedf0" (UID: "52a6c864-70c3-40fb-93ed-fb7e9f8eedf0"). InnerVolumeSpecName "kube-api-access-ppw9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:57:17.329560 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.329540    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-dshm" (OuterVolumeSpecName: "dshm") pod "52a6c864-70c3-40fb-93ed-fb7e9f8eedf0" (UID: "52a6c864-70c3-40fb-93ed-fb7e9f8eedf0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:57:17.421884 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.421847    2561 generic.go:356] "Generic (PLEG): container finished" podID="52a6c864-70c3-40fb-93ed-fb7e9f8eedf0" containerID="89e7912dcadaba31834b16784f0d5887de9312a24008f1f4843ff2fdc4873e17" exitCode=0
Aug 05 17:57:17.422122 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.421897    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"
Aug 05 17:57:17.422122 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.421937    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr" event={"ID":"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0","Type":"ContainerDied","Data":"89e7912dcadaba31834b16784f0d5887de9312a24008f1f4843ff2fdc4873e17"}
Aug 05 17:57:17.422122 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.422015    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr" event={"ID":"52a6c864-70c3-40fb-93ed-fb7e9f8eedf0","Type":"ContainerDied","Data":"2366645a9382bc41bd0e4a0524c69265863de2d465f116954c1d17b5ad1874ec"}
Aug 05 17:57:17.422122 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.422050    2561 scope.go:117] "RemoveContainer" containerID="89e7912dcadaba31834b16784f0d5887de9312a24008f1f4843ff2fdc4873e17"
Aug 05 17:57:17.426784 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.426755    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:57:17.426912 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.426789    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:57:17.426912 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.426802    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:57:17.426912 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.426816    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:57:17.426912 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.426831    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppw9r\" (UniqueName: \"kubernetes.io/projected/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0-kube-api-access-ppw9r\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:57:17.431997 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.431981    2561 scope.go:117] "RemoveContainer" containerID="89e7912dcadaba31834b16784f0d5887de9312a24008f1f4843ff2fdc4873e17"
Aug 05 17:57:17.432296 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:57:17.432277    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89e7912dcadaba31834b16784f0d5887de9312a24008f1f4843ff2fdc4873e17\": container with ID starting with 89e7912dcadaba31834b16784f0d5887de9312a24008f1f4843ff2fdc4873e17 not found: ID does not exist" containerID="89e7912dcadaba31834b16784f0d5887de9312a24008f1f4843ff2fdc4873e17"
Aug 05 17:57:17.432362 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.432308    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89e7912dcadaba31834b16784f0d5887de9312a24008f1f4843ff2fdc4873e17"} err="failed to get container status \"89e7912dcadaba31834b16784f0d5887de9312a24008f1f4843ff2fdc4873e17\": rpc error: code = NotFound desc = could not find container \"89e7912dcadaba31834b16784f0d5887de9312a24008f1f4843ff2fdc4873e17\": container with ID starting with 89e7912dcadaba31834b16784f0d5887de9312a24008f1f4843ff2fdc4873e17 not found: ID does not exist"
Aug 05 17:57:17.443290 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.443261    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"]
Aug 05 17:57:17.447861 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.447839    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-s2fhr"]
Aug 05 17:57:17.612427 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.612385    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6" podUID="769fc130-f45e-48a6-831e-961c254c26d0" containerName="main" containerID="cri-o://acbe8a22ab442a555ea5b80bf2514ce1da381d4584a51844ebd30f7788316721" gracePeriod=45
Aug 05 17:57:17.879602 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.879575    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"
Aug 05 17:57:17.930639 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.930607    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/769fc130-f45e-48a6-831e-961c254c26d0-home\") pod \"769fc130-f45e-48a6-831e-961c254c26d0\" (UID: \"769fc130-f45e-48a6-831e-961c254c26d0\") "
Aug 05 17:57:17.930828 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.930657    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/769fc130-f45e-48a6-831e-961c254c26d0-model-cache\") pod \"769fc130-f45e-48a6-831e-961c254c26d0\" (UID: \"769fc130-f45e-48a6-831e-961c254c26d0\") "
Aug 05 17:57:17.930828 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.930695    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/769fc130-f45e-48a6-831e-961c254c26d0-tls-certs\") pod \"769fc130-f45e-48a6-831e-961c254c26d0\" (UID: \"769fc130-f45e-48a6-831e-961c254c26d0\") "
Aug 05 17:57:17.930828 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.930718    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/769fc130-f45e-48a6-831e-961c254c26d0-dshm\") pod \"769fc130-f45e-48a6-831e-961c254c26d0\" (UID: \"769fc130-f45e-48a6-831e-961c254c26d0\") "
Aug 05 17:57:17.930997 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.930861    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88mmm\" (UniqueName: \"kubernetes.io/projected/769fc130-f45e-48a6-831e-961c254c26d0-kube-api-access-88mmm\") pod \"769fc130-f45e-48a6-831e-961c254c26d0\" (UID: \"769fc130-f45e-48a6-831e-961c254c26d0\") "
Aug 05 17:57:17.930997 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.930904    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/769fc130-f45e-48a6-831e-961c254c26d0-tmp-dir\") pod \"769fc130-f45e-48a6-831e-961c254c26d0\" (UID: \"769fc130-f45e-48a6-831e-961c254c26d0\") "
Aug 05 17:57:17.930997 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.930915    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769fc130-f45e-48a6-831e-961c254c26d0-home" (OuterVolumeSpecName: "home") pod "769fc130-f45e-48a6-831e-961c254c26d0" (UID: "769fc130-f45e-48a6-831e-961c254c26d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:57:17.930997 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.930973    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769fc130-f45e-48a6-831e-961c254c26d0-model-cache" (OuterVolumeSpecName: "model-cache") pod "769fc130-f45e-48a6-831e-961c254c26d0" (UID: "769fc130-f45e-48a6-831e-961c254c26d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:57:17.931225 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.931163    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769fc130-f45e-48a6-831e-961c254c26d0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "769fc130-f45e-48a6-831e-961c254c26d0" (UID: "769fc130-f45e-48a6-831e-961c254c26d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:57:17.931281 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.931245    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/769fc130-f45e-48a6-831e-961c254c26d0-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:57:17.931281 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.931266    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/769fc130-f45e-48a6-831e-961c254c26d0-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:57:17.931281 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.931277    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/769fc130-f45e-48a6-831e-961c254c26d0-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:57:17.932876 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.932854    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/769fc130-f45e-48a6-831e-961c254c26d0-kube-api-access-88mmm" (OuterVolumeSpecName: "kube-api-access-88mmm") pod "769fc130-f45e-48a6-831e-961c254c26d0" (UID: "769fc130-f45e-48a6-831e-961c254c26d0"). InnerVolumeSpecName "kube-api-access-88mmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:57:17.932975 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.932857    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769fc130-f45e-48a6-831e-961c254c26d0-dshm" (OuterVolumeSpecName: "dshm") pod "769fc130-f45e-48a6-831e-961c254c26d0" (UID: "769fc130-f45e-48a6-831e-961c254c26d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:57:17.933139 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:17.933125    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/769fc130-f45e-48a6-831e-961c254c26d0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "769fc130-f45e-48a6-831e-961c254c26d0" (UID: "769fc130-f45e-48a6-831e-961c254c26d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:57:18.031734 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:18.031703    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/769fc130-f45e-48a6-831e-961c254c26d0-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:57:18.031734 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:18.031730    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/769fc130-f45e-48a6-831e-961c254c26d0-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:57:18.032136 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:18.031741    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88mmm\" (UniqueName: \"kubernetes.io/projected/769fc130-f45e-48a6-831e-961c254c26d0-kube-api-access-88mmm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:57:18.252412 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:18.252309    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5" podUID="ead6f2f1-1e85-49cc-b842-c7be92fa4e80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Aug 05 17:57:18.430162 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:18.430127    2561 generic.go:356] "Generic (PLEG): container finished" podID="769fc130-f45e-48a6-831e-961c254c26d0" containerID="acbe8a22ab442a555ea5b80bf2514ce1da381d4584a51844ebd30f7788316721" exitCode=0
Aug 05 17:57:18.430376 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:18.430172    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"
Aug 05 17:57:18.430376 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:18.430208    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6" event={"ID":"769fc130-f45e-48a6-831e-961c254c26d0","Type":"ContainerDied","Data":"acbe8a22ab442a555ea5b80bf2514ce1da381d4584a51844ebd30f7788316721"}
Aug 05 17:57:18.430376 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:18.430267    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6" event={"ID":"769fc130-f45e-48a6-831e-961c254c26d0","Type":"ContainerDied","Data":"4d1ffc65add4a7ca6ba969c7807d3bad7e5f5fb308a71561a87f5084be78536f"}
Aug 05 17:57:18.430376 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:18.430300    2561 scope.go:117] "RemoveContainer" containerID="acbe8a22ab442a555ea5b80bf2514ce1da381d4584a51844ebd30f7788316721"
Aug 05 17:57:18.439743 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:18.439727    2561 scope.go:117] "RemoveContainer" containerID="acbe8a22ab442a555ea5b80bf2514ce1da381d4584a51844ebd30f7788316721"
Aug 05 17:57:18.440016 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:57:18.439994    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acbe8a22ab442a555ea5b80bf2514ce1da381d4584a51844ebd30f7788316721\": container with ID starting with acbe8a22ab442a555ea5b80bf2514ce1da381d4584a51844ebd30f7788316721 not found: ID does not exist" containerID="acbe8a22ab442a555ea5b80bf2514ce1da381d4584a51844ebd30f7788316721"
Aug 05 17:57:18.440079 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:18.440025    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acbe8a22ab442a555ea5b80bf2514ce1da381d4584a51844ebd30f7788316721"} err="failed to get container status \"acbe8a22ab442a555ea5b80bf2514ce1da381d4584a51844ebd30f7788316721\": rpc error: code = NotFound desc = could not find container \"acbe8a22ab442a555ea5b80bf2514ce1da381d4584a51844ebd30f7788316721\": container with ID starting with acbe8a22ab442a555ea5b80bf2514ce1da381d4584a51844ebd30f7788316721 not found: ID does not exist"
Aug 05 17:57:18.453090 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:18.453062    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"]
Aug 05 17:57:18.458720 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:18.458687    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-99hx6"]
Aug 05 17:57:18.731113 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:18.731073    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52a6c864-70c3-40fb-93ed-fb7e9f8eedf0" path="/var/lib/kubelet/pods/52a6c864-70c3-40fb-93ed-fb7e9f8eedf0/volumes"
Aug 05 17:57:18.731701 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:18.731680    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="769fc130-f45e-48a6-831e-961c254c26d0" path="/var/lib/kubelet/pods/769fc130-f45e-48a6-831e-961c254c26d0/volumes"
Aug 05 17:57:19.622311 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:19.622277    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"
Aug 05 17:57:19.634728 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:19.634704    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"
Aug 05 17:57:20.456544 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:20.456513    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"
Aug 05 17:57:25.892061 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:25.892021    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"]
Aug 05 17:57:25.893361 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:25.893334    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52a6c864-70c3-40fb-93ed-fb7e9f8eedf0" containerName="main"
Aug 05 17:57:25.893518 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:25.893507    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="52a6c864-70c3-40fb-93ed-fb7e9f8eedf0" containerName="main"
Aug 05 17:57:25.893614 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:25.893603    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="769fc130-f45e-48a6-831e-961c254c26d0" containerName="main"
Aug 05 17:57:25.893713 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:25.893700    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="769fc130-f45e-48a6-831e-961c254c26d0" containerName="main"
Aug 05 17:57:25.893919 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:25.893903    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="769fc130-f45e-48a6-831e-961c254c26d0" containerName="main"
Aug 05 17:57:25.894079 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:25.894065    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="52a6c864-70c3-40fb-93ed-fb7e9f8eedf0" containerName="main"
Aug 05 17:57:25.899382 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:25.899360    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"
Aug 05 17:57:25.901936 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:25.901915    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v2-kserve-self-signed-certs\""
Aug 05 17:57:25.902785 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:25.902760    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"]
Aug 05 17:57:26.003241 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:26.003207    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af99eb09-1501-4e68-bd71-b4d501615b69-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-x7rjt\" (UID: \"af99eb09-1501-4e68-bd71-b4d501615b69\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"
Aug 05 17:57:26.003445 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:26.003261    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af99eb09-1501-4e68-bd71-b4d501615b69-model-cache\") pod \"owner-v2-kserve-5cf8959f95-x7rjt\" (UID: \"af99eb09-1501-4e68-bd71-b4d501615b69\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"
Aug 05 17:57:26.003445 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:26.003290    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vqcl\" (UniqueName: \"kubernetes.io/projected/af99eb09-1501-4e68-bd71-b4d501615b69-kube-api-access-8vqcl\") pod \"owner-v2-kserve-5cf8959f95-x7rjt\" (UID: \"af99eb09-1501-4e68-bd71-b4d501615b69\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"
Aug 05 17:57:26.003445 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:26.003309    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af99eb09-1501-4e68-bd71-b4d501615b69-dshm\") pod \"owner-v2-kserve-5cf8959f95-x7rjt\" (UID: \"af99eb09-1501-4e68-bd71-b4d501615b69\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"
Aug 05 17:57:26.003445 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:26.003400    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af99eb09-1501-4e68-bd71-b4d501615b69-home\") pod \"owner-v2-kserve-5cf8959f95-x7rjt\" (UID: \"af99eb09-1501-4e68-bd71-b4d501615b69\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"
Aug 05 17:57:26.003586 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:26.003450    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af99eb09-1501-4e68-bd71-b4d501615b69-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-x7rjt\" (UID: \"af99eb09-1501-4e68-bd71-b4d501615b69\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"
Aug 05 17:57:26.104288 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:26.104227    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af99eb09-1501-4e68-bd71-b4d501615b69-model-cache\") pod \"owner-v2-kserve-5cf8959f95-x7rjt\" (UID: \"af99eb09-1501-4e68-bd71-b4d501615b69\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"
Aug 05 17:57:26.104288 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:26.104287    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vqcl\" (UniqueName: \"kubernetes.io/projected/af99eb09-1501-4e68-bd71-b4d501615b69-kube-api-access-8vqcl\") pod \"owner-v2-kserve-5cf8959f95-x7rjt\" (UID: \"af99eb09-1501-4e68-bd71-b4d501615b69\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"
Aug 05 17:57:26.104573 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:26.104321    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af99eb09-1501-4e68-bd71-b4d501615b69-dshm\") pod \"owner-v2-kserve-5cf8959f95-x7rjt\" (UID: \"af99eb09-1501-4e68-bd71-b4d501615b69\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"
Aug 05 17:57:26.104573 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:26.104373    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af99eb09-1501-4e68-bd71-b4d501615b69-home\") pod \"owner-v2-kserve-5cf8959f95-x7rjt\" (UID: \"af99eb09-1501-4e68-bd71-b4d501615b69\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"
Aug 05 17:57:26.104573 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:26.104404    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af99eb09-1501-4e68-bd71-b4d501615b69-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-x7rjt\" (UID: \"af99eb09-1501-4e68-bd71-b4d501615b69\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"
Aug 05 17:57:26.104573 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:26.104513    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af99eb09-1501-4e68-bd71-b4d501615b69-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-x7rjt\" (UID: \"af99eb09-1501-4e68-bd71-b4d501615b69\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"
Aug 05 17:57:26.104876 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:26.104713    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af99eb09-1501-4e68-bd71-b4d501615b69-model-cache\") pod \"owner-v2-kserve-5cf8959f95-x7rjt\" (UID: \"af99eb09-1501-4e68-bd71-b4d501615b69\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"
Aug 05 17:57:26.105244 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:26.105218    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af99eb09-1501-4e68-bd71-b4d501615b69-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-x7rjt\" (UID: \"af99eb09-1501-4e68-bd71-b4d501615b69\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"
Aug 05 17:57:26.105557 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:26.105535    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af99eb09-1501-4e68-bd71-b4d501615b69-home\") pod \"owner-v2-kserve-5cf8959f95-x7rjt\" (UID: \"af99eb09-1501-4e68-bd71-b4d501615b69\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"
Aug 05 17:57:26.107528 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:26.107501    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af99eb09-1501-4e68-bd71-b4d501615b69-dshm\") pod \"owner-v2-kserve-5cf8959f95-x7rjt\" (UID: \"af99eb09-1501-4e68-bd71-b4d501615b69\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"
Aug 05 17:57:26.107832 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:26.107809    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af99eb09-1501-4e68-bd71-b4d501615b69-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-x7rjt\" (UID: \"af99eb09-1501-4e68-bd71-b4d501615b69\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"
Aug 05 17:57:26.115378 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:26.115351    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vqcl\" (UniqueName: \"kubernetes.io/projected/af99eb09-1501-4e68-bd71-b4d501615b69-kube-api-access-8vqcl\") pod \"owner-v2-kserve-5cf8959f95-x7rjt\" (UID: \"af99eb09-1501-4e68-bd71-b4d501615b69\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"
Aug 05 17:57:26.213638 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:26.213527    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"
Aug 05 17:57:26.358168 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:26.358141    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"]
Aug 05 17:57:26.359853 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:57:26.359812    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf99eb09_1501_4e68_bd71_b4d501615b69.slice/crio-28c38880bdf6025e620badb3a1ba07efcbce2cc0c9e68d91c5c670317cc212c5 WatchSource:0}: Error finding container 28c38880bdf6025e620badb3a1ba07efcbce2cc0c9e68d91c5c670317cc212c5: Status 404 returned error can't find the container with id 28c38880bdf6025e620badb3a1ba07efcbce2cc0c9e68d91c5c670317cc212c5
Aug 05 17:57:26.482530 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:26.482442    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt" event={"ID":"af99eb09-1501-4e68-bd71-b4d501615b69","Type":"ContainerStarted","Data":"9f96da5920849463fe48be3d506a398aab7d29f125f362bdfca47b13b1c4be01"}
Aug 05 17:57:26.482530 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:26.482499    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt" event={"ID":"af99eb09-1501-4e68-bd71-b4d501615b69","Type":"ContainerStarted","Data":"28c38880bdf6025e620badb3a1ba07efcbce2cc0c9e68d91c5c670317cc212c5"}
Aug 05 17:57:26.499905 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:26.499853    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt" podStartSLOduration=1.499835915 podStartE2EDuration="1.499835915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:57:26.498069355 +0000 UTC m=+2702.242703683" watchObservedRunningTime="2026-08-05 17:57:26.499835915 +0000 UTC m=+2702.244470242"
Aug 05 17:57:27.213919 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:27.213881    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"
Aug 05 17:57:28.251759 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:28.251712    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5" podUID="ead6f2f1-1e85-49cc-b842-c7be92fa4e80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Aug 05 17:57:36.213877 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:36.213826    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"
Aug 05 17:57:36.226648 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:36.226614    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"
Aug 05 17:57:36.554406 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:36.554370    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"
Aug 05 17:57:38.252388 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:38.252349    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5" podUID="ead6f2f1-1e85-49cc-b842-c7be92fa4e80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Aug 05 17:57:48.252590 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:48.252535    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5" podUID="ead6f2f1-1e85-49cc-b842-c7be92fa4e80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Aug 05 17:57:58.252069 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:57:58.252013    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5" podUID="ead6f2f1-1e85-49cc-b842-c7be92fa4e80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Aug 05 17:58:05.877791 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:05.877747    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"]
Aug 05 17:58:08.252446 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:08.252391    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5" podUID="ead6f2f1-1e85-49cc-b842-c7be92fa4e80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Aug 05 17:58:18.252430 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:18.252373    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5" podUID="ead6f2f1-1e85-49cc-b842-c7be92fa4e80" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": dial tcp 10.133.0.92:8000: connect: connection refused"
Aug 05 17:58:20.897017 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:20.896961    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm" podUID="5e97cae0-4e01-4944-b514-1418600db170" containerName="main" containerID="cri-o://df8483896a17afacd23664b1735b9ab316732de2e286cc895d5dc097538d2886" gracePeriod=45
Aug 05 17:58:21.156765 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:21.156700    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"
Aug 05 17:58:21.253142 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:21.253102    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e97cae0-4e01-4944-b514-1418600db170-model-cache\") pod \"5e97cae0-4e01-4944-b514-1418600db170\" (UID: \"5e97cae0-4e01-4944-b514-1418600db170\") "
Aug 05 17:58:21.253142 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:21.253146    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e97cae0-4e01-4944-b514-1418600db170-home\") pod \"5e97cae0-4e01-4944-b514-1418600db170\" (UID: \"5e97cae0-4e01-4944-b514-1418600db170\") "
Aug 05 17:58:21.253409 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:21.253186    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e97cae0-4e01-4944-b514-1418600db170-tmp-dir\") pod \"5e97cae0-4e01-4944-b514-1418600db170\" (UID: \"5e97cae0-4e01-4944-b514-1418600db170\") "
Aug 05 17:58:21.253409 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:21.253213    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e97cae0-4e01-4944-b514-1418600db170-dshm\") pod \"5e97cae0-4e01-4944-b514-1418600db170\" (UID: \"5e97cae0-4e01-4944-b514-1418600db170\") "
Aug 05 17:58:21.253409 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:21.253277    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hz27k\" (UniqueName: \"kubernetes.io/projected/5e97cae0-4e01-4944-b514-1418600db170-kube-api-access-hz27k\") pod \"5e97cae0-4e01-4944-b514-1418600db170\" (UID: \"5e97cae0-4e01-4944-b514-1418600db170\") "
Aug 05 17:58:21.253409 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:21.253304    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e97cae0-4e01-4944-b514-1418600db170-tls-certs\") pod \"5e97cae0-4e01-4944-b514-1418600db170\" (UID: \"5e97cae0-4e01-4944-b514-1418600db170\") "
Aug 05 17:58:21.253584 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:21.253431    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e97cae0-4e01-4944-b514-1418600db170-model-cache" (OuterVolumeSpecName: "model-cache") pod "5e97cae0-4e01-4944-b514-1418600db170" (UID: "5e97cae0-4e01-4944-b514-1418600db170"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:58:21.253584 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:21.253485    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e97cae0-4e01-4944-b514-1418600db170-home" (OuterVolumeSpecName: "home") pod "5e97cae0-4e01-4944-b514-1418600db170" (UID: "5e97cae0-4e01-4944-b514-1418600db170"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:58:21.253584 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:21.253495    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e97cae0-4e01-4944-b514-1418600db170-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5e97cae0-4e01-4944-b514-1418600db170" (UID: "5e97cae0-4e01-4944-b514-1418600db170"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:58:21.253735 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:21.253630    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e97cae0-4e01-4944-b514-1418600db170-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:58:21.253735 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:21.253646    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e97cae0-4e01-4944-b514-1418600db170-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:58:21.253735 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:21.253654    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e97cae0-4e01-4944-b514-1418600db170-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:58:21.255555 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:21.255532    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e97cae0-4e01-4944-b514-1418600db170-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5e97cae0-4e01-4944-b514-1418600db170" (UID: "5e97cae0-4e01-4944-b514-1418600db170"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:58:21.255653 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:21.255588    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e97cae0-4e01-4944-b514-1418600db170-kube-api-access-hz27k" (OuterVolumeSpecName: "kube-api-access-hz27k") pod "5e97cae0-4e01-4944-b514-1418600db170" (UID: "5e97cae0-4e01-4944-b514-1418600db170"). InnerVolumeSpecName "kube-api-access-hz27k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:58:21.255653 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:21.255611    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e97cae0-4e01-4944-b514-1418600db170-dshm" (OuterVolumeSpecName: "dshm") pod "5e97cae0-4e01-4944-b514-1418600db170" (UID: "5e97cae0-4e01-4944-b514-1418600db170"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:58:21.354968 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:21.354917    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e97cae0-4e01-4944-b514-1418600db170-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:58:21.354968 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:21.354945    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz27k\" (UniqueName: \"kubernetes.io/projected/5e97cae0-4e01-4944-b514-1418600db170-kube-api-access-hz27k\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:58:21.355242 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:21.354981    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e97cae0-4e01-4944-b514-1418600db170-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:58:21.825916 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:21.825878    2561 generic.go:356] "Generic (PLEG): container finished" podID="5e97cae0-4e01-4944-b514-1418600db170" containerID="df8483896a17afacd23664b1735b9ab316732de2e286cc895d5dc097538d2886" exitCode=0
Aug 05 17:58:21.826161 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:21.825919    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm" event={"ID":"5e97cae0-4e01-4944-b514-1418600db170","Type":"ContainerDied","Data":"df8483896a17afacd23664b1735b9ab316732de2e286cc895d5dc097538d2886"}
Aug 05 17:58:21.826161 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:21.826006    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm" event={"ID":"5e97cae0-4e01-4944-b514-1418600db170","Type":"ContainerDied","Data":"c299f6307c59438c0b7cca81bffa23791aa432ed5b882d3071b4ee69cbd3c39a"}
Aug 05 17:58:21.826161 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:21.825940    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"
Aug 05 17:58:21.826161 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:21.826051    2561 scope.go:117] "RemoveContainer" containerID="df8483896a17afacd23664b1735b9ab316732de2e286cc895d5dc097538d2886"
Aug 05 17:58:21.848799 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:21.848775    2561 scope.go:117] "RemoveContainer" containerID="df8483896a17afacd23664b1735b9ab316732de2e286cc895d5dc097538d2886"
Aug 05 17:58:21.849303 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:58:21.849274    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df8483896a17afacd23664b1735b9ab316732de2e286cc895d5dc097538d2886\": container with ID starting with df8483896a17afacd23664b1735b9ab316732de2e286cc895d5dc097538d2886 not found: ID does not exist" containerID="df8483896a17afacd23664b1735b9ab316732de2e286cc895d5dc097538d2886"
Aug 05 17:58:21.849474 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:21.849449    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8483896a17afacd23664b1735b9ab316732de2e286cc895d5dc097538d2886"} err="failed to get container status \"df8483896a17afacd23664b1735b9ab316732de2e286cc895d5dc097538d2886\": rpc error: code = NotFound desc = could not find container \"df8483896a17afacd23664b1735b9ab316732de2e286cc895d5dc097538d2886\": container with ID starting with df8483896a17afacd23664b1735b9ab316732de2e286cc895d5dc097538d2886 not found: ID does not exist"
Aug 05 17:58:21.863284 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:21.863249    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"]
Aug 05 17:58:21.867255 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:21.867219    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-p57lm"]
Aug 05 17:58:22.730724 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:22.730680    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e97cae0-4e01-4944-b514-1418600db170" path="/var/lib/kubelet/pods/5e97cae0-4e01-4944-b514-1418600db170/volumes"
Aug 05 17:58:27.759932 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.759897    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"]
Aug 05 17:58:27.760508 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.760481    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e97cae0-4e01-4944-b514-1418600db170" containerName="main"
Aug 05 17:58:27.760508 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.760506    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e97cae0-4e01-4944-b514-1418600db170" containerName="main"
Aug 05 17:58:27.760677 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.760622    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e97cae0-4e01-4944-b514-1418600db170" containerName="main"
Aug 05 17:58:27.765213 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.765193    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:27.767968 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.767922    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Aug 05 17:58:27.768100 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.767928    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Aug 05 17:58:27.768100 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.768018    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-zd5p9\""
Aug 05 17:58:27.768299 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.768285    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Aug 05 17:58:27.776495 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.776472    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"]
Aug 05 17:58:27.815159 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.815124    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daca7cad-7a28-47df-af89-c25576d556da-istio-podinfo\") pod \"router-gateway-1-openshift-default-65765d8947-vclkw\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:27.815337 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.815164    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daca7cad-7a28-47df-af89-c25576d556da-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-65765d8947-vclkw\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:27.815337 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.815245    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daca7cad-7a28-47df-af89-c25576d556da-credential-socket\") pod \"router-gateway-1-openshift-default-65765d8947-vclkw\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:27.815337 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.815299    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daca7cad-7a28-47df-af89-c25576d556da-istio-token\") pod \"router-gateway-1-openshift-default-65765d8947-vclkw\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:27.815337 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.815322    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daca7cad-7a28-47df-af89-c25576d556da-istio-data\") pod \"router-gateway-1-openshift-default-65765d8947-vclkw\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:27.815546 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.815361    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daca7cad-7a28-47df-af89-c25576d556da-workload-certs\") pod \"router-gateway-1-openshift-default-65765d8947-vclkw\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:27.815546 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.815378    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daca7cad-7a28-47df-af89-c25576d556da-istio-envoy\") pod \"router-gateway-1-openshift-default-65765d8947-vclkw\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:27.815546 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.815393    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwdxm\" (UniqueName: \"kubernetes.io/projected/daca7cad-7a28-47df-af89-c25576d556da-kube-api-access-vwdxm\") pod \"router-gateway-1-openshift-default-65765d8947-vclkw\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:27.815546 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.815469    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daca7cad-7a28-47df-af89-c25576d556da-workload-socket\") pod \"router-gateway-1-openshift-default-65765d8947-vclkw\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:27.916166 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.916129    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daca7cad-7a28-47df-af89-c25576d556da-workload-socket\") pod \"router-gateway-1-openshift-default-65765d8947-vclkw\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:27.916350 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.916272    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daca7cad-7a28-47df-af89-c25576d556da-istio-podinfo\") pod \"router-gateway-1-openshift-default-65765d8947-vclkw\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:27.916350 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.916306    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daca7cad-7a28-47df-af89-c25576d556da-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-65765d8947-vclkw\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:27.916535 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.916513    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daca7cad-7a28-47df-af89-c25576d556da-workload-socket\") pod \"router-gateway-1-openshift-default-65765d8947-vclkw\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:27.916607 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.916515    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daca7cad-7a28-47df-af89-c25576d556da-credential-socket\") pod \"router-gateway-1-openshift-default-65765d8947-vclkw\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:27.916607 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.916596    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daca7cad-7a28-47df-af89-c25576d556da-istio-token\") pod \"router-gateway-1-openshift-default-65765d8947-vclkw\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:27.916711 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.916619    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daca7cad-7a28-47df-af89-c25576d556da-istio-data\") pod \"router-gateway-1-openshift-default-65765d8947-vclkw\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:27.916711 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.916646    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daca7cad-7a28-47df-af89-c25576d556da-workload-certs\") pod \"router-gateway-1-openshift-default-65765d8947-vclkw\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:27.916711 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.916660    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daca7cad-7a28-47df-af89-c25576d556da-istio-envoy\") pod \"router-gateway-1-openshift-default-65765d8947-vclkw\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:27.916711 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.916682    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwdxm\" (UniqueName: \"kubernetes.io/projected/daca7cad-7a28-47df-af89-c25576d556da-kube-api-access-vwdxm\") pod \"router-gateway-1-openshift-default-65765d8947-vclkw\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:27.916920 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.916780    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daca7cad-7a28-47df-af89-c25576d556da-credential-socket\") pod \"router-gateway-1-openshift-default-65765d8947-vclkw\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:27.917027 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.916938    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daca7cad-7a28-47df-af89-c25576d556da-workload-certs\") pod \"router-gateway-1-openshift-default-65765d8947-vclkw\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:27.917146 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.917125    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daca7cad-7a28-47df-af89-c25576d556da-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-65765d8947-vclkw\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:27.917208 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.917191    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daca7cad-7a28-47df-af89-c25576d556da-istio-data\") pod \"router-gateway-1-openshift-default-65765d8947-vclkw\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:27.918934 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.918914    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daca7cad-7a28-47df-af89-c25576d556da-istio-envoy\") pod \"router-gateway-1-openshift-default-65765d8947-vclkw\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:27.919398 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.919383    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daca7cad-7a28-47df-af89-c25576d556da-istio-podinfo\") pod \"router-gateway-1-openshift-default-65765d8947-vclkw\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:27.923941 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.923913    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daca7cad-7a28-47df-af89-c25576d556da-istio-token\") pod \"router-gateway-1-openshift-default-65765d8947-vclkw\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:27.924274 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:27.924253    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwdxm\" (UniqueName: \"kubernetes.io/projected/daca7cad-7a28-47df-af89-c25576d556da-kube-api-access-vwdxm\") pod \"router-gateway-1-openshift-default-65765d8947-vclkw\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:28.076240 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:28.076206    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:28.219989 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:28.219941    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"]
Aug 05 17:58:28.221569 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:58:28.221539    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaca7cad_7a28_47df_af89_c25576d556da.slice/crio-d9381a817ee86a3f12421f71a20d954cd3711f25e8a5d5130f619e739d41bcd5 WatchSource:0}: Error finding container d9381a817ee86a3f12421f71a20d954cd3711f25e8a5d5130f619e739d41bcd5: Status 404 returned error can't find the container with id d9381a817ee86a3f12421f71a20d954cd3711f25e8a5d5130f619e739d41bcd5
Aug 05 17:58:28.227261 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:28.227214    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 17:58:28.227389 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:28.227308    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 17:58:28.227389 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:28.227341    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 17:58:28.262054 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:28.262025    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:58:28.271623 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:28.271598    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:58:28.868886 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:28.868852    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw" event={"ID":"daca7cad-7a28-47df-af89-c25576d556da","Type":"ContainerStarted","Data":"e51e30bf2365681f9d0744ef3ef91a112c85bdb543ef698713f7ab03e3beeb81"}
Aug 05 17:58:28.869402 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:28.868906    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw" event={"ID":"daca7cad-7a28-47df-af89-c25576d556da","Type":"ContainerStarted","Data":"d9381a817ee86a3f12421f71a20d954cd3711f25e8a5d5130f619e739d41bcd5"}
Aug 05 17:58:28.892183 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:28.892132    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw" podStartSLOduration=1.892115461 podStartE2EDuration="1.892115461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:58:28.890875341 +0000 UTC m=+2764.635509668" watchObservedRunningTime="2026-08-05 17:58:28.892115461 +0000 UTC m=+2764.636749788"
Aug 05 17:58:29.076740 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:29.076703    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:29.081983 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:29.081937    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:29.874482 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:29.874454    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:29.875468 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:29.875448    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:32.520669 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:32.520580    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"]
Aug 05 17:58:47.159850 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:47.159600    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"]
Aug 05 17:58:47.544640 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:47.544578    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt" podUID="af99eb09-1501-4e68-bd71-b4d501615b69" containerName="main" containerID="cri-o://9f96da5920849463fe48be3d506a398aab7d29f125f362bdfca47b13b1c4be01" gracePeriod=45
Aug 05 17:58:47.560010 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:47.559935    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt" podUID="af99eb09-1501-4e68-bd71-b4d501615b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": EOF"
Aug 05 17:58:47.831988 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:47.831939    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"
Aug 05 17:58:47.910976 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:47.910918    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af99eb09-1501-4e68-bd71-b4d501615b69-dshm\") pod \"af99eb09-1501-4e68-bd71-b4d501615b69\" (UID: \"af99eb09-1501-4e68-bd71-b4d501615b69\") "
Aug 05 17:58:47.911165 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:47.910998    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af99eb09-1501-4e68-bd71-b4d501615b69-tls-certs\") pod \"af99eb09-1501-4e68-bd71-b4d501615b69\" (UID: \"af99eb09-1501-4e68-bd71-b4d501615b69\") "
Aug 05 17:58:47.911165 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:47.911021    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af99eb09-1501-4e68-bd71-b4d501615b69-tmp-dir\") pod \"af99eb09-1501-4e68-bd71-b4d501615b69\" (UID: \"af99eb09-1501-4e68-bd71-b4d501615b69\") "
Aug 05 17:58:47.911165 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:47.911069    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af99eb09-1501-4e68-bd71-b4d501615b69-model-cache\") pod \"af99eb09-1501-4e68-bd71-b4d501615b69\" (UID: \"af99eb09-1501-4e68-bd71-b4d501615b69\") "
Aug 05 17:58:47.911324 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:47.911212    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vqcl\" (UniqueName: \"kubernetes.io/projected/af99eb09-1501-4e68-bd71-b4d501615b69-kube-api-access-8vqcl\") pod \"af99eb09-1501-4e68-bd71-b4d501615b69\" (UID: \"af99eb09-1501-4e68-bd71-b4d501615b69\") "
Aug 05 17:58:47.911324 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:47.911259    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af99eb09-1501-4e68-bd71-b4d501615b69-home\") pod \"af99eb09-1501-4e68-bd71-b4d501615b69\" (UID: \"af99eb09-1501-4e68-bd71-b4d501615b69\") "
Aug 05 17:58:47.911324 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:47.911287    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af99eb09-1501-4e68-bd71-b4d501615b69-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "af99eb09-1501-4e68-bd71-b4d501615b69" (UID: "af99eb09-1501-4e68-bd71-b4d501615b69"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:58:47.911487 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:47.911332    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af99eb09-1501-4e68-bd71-b4d501615b69-model-cache" (OuterVolumeSpecName: "model-cache") pod "af99eb09-1501-4e68-bd71-b4d501615b69" (UID: "af99eb09-1501-4e68-bd71-b4d501615b69"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:58:47.911645 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:47.911606    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af99eb09-1501-4e68-bd71-b4d501615b69-home" (OuterVolumeSpecName: "home") pod "af99eb09-1501-4e68-bd71-b4d501615b69" (UID: "af99eb09-1501-4e68-bd71-b4d501615b69"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:58:47.911743 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:47.911676    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af99eb09-1501-4e68-bd71-b4d501615b69-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:58:47.911743 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:47.911699    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af99eb09-1501-4e68-bd71-b4d501615b69-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:58:47.913220 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:47.913198    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af99eb09-1501-4e68-bd71-b4d501615b69-dshm" (OuterVolumeSpecName: "dshm") pod "af99eb09-1501-4e68-bd71-b4d501615b69" (UID: "af99eb09-1501-4e68-bd71-b4d501615b69"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:58:47.913220 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:47.913212    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af99eb09-1501-4e68-bd71-b4d501615b69-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "af99eb09-1501-4e68-bd71-b4d501615b69" (UID: "af99eb09-1501-4e68-bd71-b4d501615b69"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:58:47.913354 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:47.913238    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af99eb09-1501-4e68-bd71-b4d501615b69-kube-api-access-8vqcl" (OuterVolumeSpecName: "kube-api-access-8vqcl") pod "af99eb09-1501-4e68-bd71-b4d501615b69" (UID: "af99eb09-1501-4e68-bd71-b4d501615b69"). InnerVolumeSpecName "kube-api-access-8vqcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:58:47.982690 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:47.982658    2561 generic.go:356] "Generic (PLEG): container finished" podID="af99eb09-1501-4e68-bd71-b4d501615b69" containerID="9f96da5920849463fe48be3d506a398aab7d29f125f362bdfca47b13b1c4be01" exitCode=0
Aug 05 17:58:47.982881 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:47.982705    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"
Aug 05 17:58:47.982881 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:47.982746    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt" event={"ID":"af99eb09-1501-4e68-bd71-b4d501615b69","Type":"ContainerDied","Data":"9f96da5920849463fe48be3d506a398aab7d29f125f362bdfca47b13b1c4be01"}
Aug 05 17:58:47.982881 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:47.982803    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt" event={"ID":"af99eb09-1501-4e68-bd71-b4d501615b69","Type":"ContainerDied","Data":"28c38880bdf6025e620badb3a1ba07efcbce2cc0c9e68d91c5c670317cc212c5"}
Aug 05 17:58:47.982881 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:47.982827    2561 scope.go:117] "RemoveContainer" containerID="9f96da5920849463fe48be3d506a398aab7d29f125f362bdfca47b13b1c4be01"
Aug 05 17:58:47.992772 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:47.992753    2561 scope.go:117] "RemoveContainer" containerID="9f96da5920849463fe48be3d506a398aab7d29f125f362bdfca47b13b1c4be01"
Aug 05 17:58:47.993086 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:58:47.993063    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f96da5920849463fe48be3d506a398aab7d29f125f362bdfca47b13b1c4be01\": container with ID starting with 9f96da5920849463fe48be3d506a398aab7d29f125f362bdfca47b13b1c4be01 not found: ID does not exist" containerID="9f96da5920849463fe48be3d506a398aab7d29f125f362bdfca47b13b1c4be01"
Aug 05 17:58:47.993171 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:47.993095    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f96da5920849463fe48be3d506a398aab7d29f125f362bdfca47b13b1c4be01"} err="failed to get container status \"9f96da5920849463fe48be3d506a398aab7d29f125f362bdfca47b13b1c4be01\": rpc error: code = NotFound desc = could not find container \"9f96da5920849463fe48be3d506a398aab7d29f125f362bdfca47b13b1c4be01\": container with ID starting with 9f96da5920849463fe48be3d506a398aab7d29f125f362bdfca47b13b1c4be01 not found: ID does not exist"
Aug 05 17:58:48.003817 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:48.003789    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"]
Aug 05 17:58:48.009353 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:48.009329    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-x7rjt"]
Aug 05 17:58:48.012180 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:48.012150    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af99eb09-1501-4e68-bd71-b4d501615b69-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:58:48.012180 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:48.012172    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af99eb09-1501-4e68-bd71-b4d501615b69-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:58:48.012180 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:48.012182    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vqcl\" (UniqueName: \"kubernetes.io/projected/af99eb09-1501-4e68-bd71-b4d501615b69-kube-api-access-8vqcl\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:58:48.012406 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:48.012190    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af99eb09-1501-4e68-bd71-b4d501615b69-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:58:48.731087 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:48.731054    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af99eb09-1501-4e68-bd71-b4d501615b69" path="/var/lib/kubelet/pods/af99eb09-1501-4e68-bd71-b4d501615b69/volumes"
Aug 05 17:58:55.026915 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:55.026823    2561 generic.go:356] "Generic (PLEG): container finished" podID="daca7cad-7a28-47df-af89-c25576d556da" containerID="e51e30bf2365681f9d0744ef3ef91a112c85bdb543ef698713f7ab03e3beeb81" exitCode=0
Aug 05 17:58:55.026915 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:55.026904    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw" event={"ID":"daca7cad-7a28-47df-af89-c25576d556da","Type":"ContainerDied","Data":"e51e30bf2365681f9d0744ef3ef91a112c85bdb543ef698713f7ab03e3beeb81"}
Aug 05 17:58:55.027396 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:55.027377    2561 scope.go:117] "RemoveContainer" containerID="e51e30bf2365681f9d0744ef3ef91a112c85bdb543ef698713f7ab03e3beeb81"
Aug 05 17:58:55.028335 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:55.028307    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 17:58:55.028391 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:55.028379    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 17:58:55.028426 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:55.028412    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 17:58:56.035158 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:56.035118    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw" event={"ID":"daca7cad-7a28-47df-af89-c25576d556da","Type":"ContainerStarted","Data":"8c73ec4f63c064cc3572f55c8e0b7d99d8835726f5fbc7b98f3f833ab7c15fa7"}
Aug 05 17:58:56.076771 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:56.076734    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:58:56.077808 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:56.077777    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw" podUID="daca7cad-7a28-47df-af89-c25576d556da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Aug 05 17:58:57.077484 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:57.077446    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw" podUID="daca7cad-7a28-47df-af89-c25576d556da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Aug 05 17:58:58.076814 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:58.076770    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw" podUID="daca7cad-7a28-47df-af89-c25576d556da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Aug 05 17:58:59.077148 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:59.077108    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw" podUID="daca7cad-7a28-47df-af89-c25576d556da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Aug 05 17:58:59.875185 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:58:59.875148    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:59:00.077309 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:00.077264    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw" podUID="daca7cad-7a28-47df-af89-c25576d556da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Aug 05 17:59:01.080913 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:01.080882    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:59:01.081798 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:01.081781    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 17:59:02.195667 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:02.195570    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5" podUID="ead6f2f1-1e85-49cc-b842-c7be92fa4e80" containerName="main" containerID="cri-o://38252d4bd59d188d38dd89208f26490ef7d85769990ae5a8be74dbfb53b40858" gracePeriod=45
Aug 05 17:59:02.775820 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:02.775786    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:59:02.860174 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:02.860074    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-model-cache\") pod \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") "
Aug 05 17:59:02.860174 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:02.860148    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-dshm\") pod \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") "
Aug 05 17:59:02.860412 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:02.860194    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-cabundle-cert\") pod \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") "
Aug 05 17:59:02.860412 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:02.860228    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-kserve-provision-location\") pod \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") "
Aug 05 17:59:02.860412 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:02.860256    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-tls-certs\") pod \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") "
Aug 05 17:59:02.860412 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:02.860284    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4t8px\" (UniqueName: \"kubernetes.io/projected/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-kube-api-access-4t8px\") pod \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") "
Aug 05 17:59:02.860412 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:02.860317    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-tmp-dir\") pod \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") "
Aug 05 17:59:02.860412 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:02.860377    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-home\") pod \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\" (UID: \"ead6f2f1-1e85-49cc-b842-c7be92fa4e80\") "
Aug 05 17:59:02.860728 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:02.860396    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-model-cache" (OuterVolumeSpecName: "model-cache") pod "ead6f2f1-1e85-49cc-b842-c7be92fa4e80" (UID: "ead6f2f1-1e85-49cc-b842-c7be92fa4e80"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:59:02.860728 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:02.860685    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ead6f2f1-1e85-49cc-b842-c7be92fa4e80" (UID: "ead6f2f1-1e85-49cc-b842-c7be92fa4e80"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 17:59:02.860839 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:02.860775    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:59:02.860839 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:02.860794    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-cabundle-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:59:02.860839 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:02.860823    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-home" (OuterVolumeSpecName: "home") pod "ead6f2f1-1e85-49cc-b842-c7be92fa4e80" (UID: "ead6f2f1-1e85-49cc-b842-c7be92fa4e80"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:59:02.861034 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:02.860922    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ead6f2f1-1e85-49cc-b842-c7be92fa4e80" (UID: "ead6f2f1-1e85-49cc-b842-c7be92fa4e80"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:59:02.862680 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:02.862656    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-dshm" (OuterVolumeSpecName: "dshm") pod "ead6f2f1-1e85-49cc-b842-c7be92fa4e80" (UID: "ead6f2f1-1e85-49cc-b842-c7be92fa4e80"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:59:02.862839 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:02.862819    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-kube-api-access-4t8px" (OuterVolumeSpecName: "kube-api-access-4t8px") pod "ead6f2f1-1e85-49cc-b842-c7be92fa4e80" (UID: "ead6f2f1-1e85-49cc-b842-c7be92fa4e80"). InnerVolumeSpecName "kube-api-access-4t8px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:59:02.862909 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:02.862859    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ead6f2f1-1e85-49cc-b842-c7be92fa4e80" (UID: "ead6f2f1-1e85-49cc-b842-c7be92fa4e80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:59:02.883918 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:02.883873    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ead6f2f1-1e85-49cc-b842-c7be92fa4e80" (UID: "ead6f2f1-1e85-49cc-b842-c7be92fa4e80"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:59:02.961199 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:02.961168    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:59:02.961199 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:02.961194    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-kserve-provision-location\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:59:02.961199 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:02.961205    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:59:02.961438 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:02.961213    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t8px\" (UniqueName: \"kubernetes.io/projected/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-kube-api-access-4t8px\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:59:02.961438 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:02.961221    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:59:02.961438 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:02.961229    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ead6f2f1-1e85-49cc-b842-c7be92fa4e80-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:59:03.083831 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:03.083789    2561 generic.go:356] "Generic (PLEG): container finished" podID="ead6f2f1-1e85-49cc-b842-c7be92fa4e80" containerID="38252d4bd59d188d38dd89208f26490ef7d85769990ae5a8be74dbfb53b40858" exitCode=0
Aug 05 17:59:03.084033 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:03.083852    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"
Aug 05 17:59:03.084033 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:03.083868    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5" event={"ID":"ead6f2f1-1e85-49cc-b842-c7be92fa4e80","Type":"ContainerDied","Data":"38252d4bd59d188d38dd89208f26490ef7d85769990ae5a8be74dbfb53b40858"}
Aug 05 17:59:03.084033 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:03.083927    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5" event={"ID":"ead6f2f1-1e85-49cc-b842-c7be92fa4e80","Type":"ContainerDied","Data":"d3701a2de8a56fe8310692e0c76e14c00b80f6b8e5df8a67436082e71579378a"}
Aug 05 17:59:03.084033 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:03.083962    2561 scope.go:117] "RemoveContainer" containerID="38252d4bd59d188d38dd89208f26490ef7d85769990ae5a8be74dbfb53b40858"
Aug 05 17:59:03.093422 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:03.093395    2561 scope.go:117] "RemoveContainer" containerID="95dfb6bca31f14038c8c1468aa32b7d7aef18e9e4171d9cc3087714766aaee07"
Aug 05 17:59:03.102169 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:03.102148    2561 scope.go:117] "RemoveContainer" containerID="38252d4bd59d188d38dd89208f26490ef7d85769990ae5a8be74dbfb53b40858"
Aug 05 17:59:03.102470 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:59:03.102447    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38252d4bd59d188d38dd89208f26490ef7d85769990ae5a8be74dbfb53b40858\": container with ID starting with 38252d4bd59d188d38dd89208f26490ef7d85769990ae5a8be74dbfb53b40858 not found: ID does not exist" containerID="38252d4bd59d188d38dd89208f26490ef7d85769990ae5a8be74dbfb53b40858"
Aug 05 17:59:03.102548 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:03.102478    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38252d4bd59d188d38dd89208f26490ef7d85769990ae5a8be74dbfb53b40858"} err="failed to get container status \"38252d4bd59d188d38dd89208f26490ef7d85769990ae5a8be74dbfb53b40858\": rpc error: code = NotFound desc = could not find container \"38252d4bd59d188d38dd89208f26490ef7d85769990ae5a8be74dbfb53b40858\": container with ID starting with 38252d4bd59d188d38dd89208f26490ef7d85769990ae5a8be74dbfb53b40858 not found: ID does not exist"
Aug 05 17:59:03.102548 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:03.102497    2561 scope.go:117] "RemoveContainer" containerID="95dfb6bca31f14038c8c1468aa32b7d7aef18e9e4171d9cc3087714766aaee07"
Aug 05 17:59:03.102764 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:59:03.102747    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95dfb6bca31f14038c8c1468aa32b7d7aef18e9e4171d9cc3087714766aaee07\": container with ID starting with 95dfb6bca31f14038c8c1468aa32b7d7aef18e9e4171d9cc3087714766aaee07 not found: ID does not exist" containerID="95dfb6bca31f14038c8c1468aa32b7d7aef18e9e4171d9cc3087714766aaee07"
Aug 05 17:59:03.102803 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:03.102773    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95dfb6bca31f14038c8c1468aa32b7d7aef18e9e4171d9cc3087714766aaee07"} err="failed to get container status \"95dfb6bca31f14038c8c1468aa32b7d7aef18e9e4171d9cc3087714766aaee07\": rpc error: code = NotFound desc = could not find container \"95dfb6bca31f14038c8c1468aa32b7d7aef18e9e4171d9cc3087714766aaee07\": container with ID starting with 95dfb6bca31f14038c8c1468aa32b7d7aef18e9e4171d9cc3087714766aaee07 not found: ID does not exist"
Aug 05 17:59:03.108836 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:03.108810    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"]
Aug 05 17:59:03.112490 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:03.112429    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-8b8df646b-jqbs5"]
Aug 05 17:59:04.729783 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:04.729746    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ead6f2f1-1e85-49cc-b842-c7be92fa4e80" path="/var/lib/kubelet/pods/ead6f2f1-1e85-49cc-b842-c7be92fa4e80/volumes"
Aug 05 17:59:21.600840 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.600801    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"]
Aug 05 17:59:21.601392 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.601287    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ead6f2f1-1e85-49cc-b842-c7be92fa4e80" containerName="main"
Aug 05 17:59:21.601392 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.601302    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ead6f2f1-1e85-49cc-b842-c7be92fa4e80" containerName="main"
Aug 05 17:59:21.601392 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.601311    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af99eb09-1501-4e68-bd71-b4d501615b69" containerName="main"
Aug 05 17:59:21.601392 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.601319    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="af99eb09-1501-4e68-bd71-b4d501615b69" containerName="main"
Aug 05 17:59:21.601392 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.601326    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ead6f2f1-1e85-49cc-b842-c7be92fa4e80" containerName="storage-initializer"
Aug 05 17:59:21.601392 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.601332    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ead6f2f1-1e85-49cc-b842-c7be92fa4e80" containerName="storage-initializer"
Aug 05 17:59:21.601392 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.601386    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="af99eb09-1501-4e68-bd71-b4d501615b69" containerName="main"
Aug 05 17:59:21.601392 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.601394    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ead6f2f1-1e85-49cc-b842-c7be92fa4e80" containerName="main"
Aug 05 17:59:21.606404 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.606336    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"
Aug 05 17:59:21.606800 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.606769    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0f879b5-5909-41f3-8ee2-56698ae3731a-model-cache\") pod \"e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9\" (UID: \"b0f879b5-5909-41f3-8ee2-56698ae3731a\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"
Aug 05 17:59:21.606928 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.606861    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0f879b5-5909-41f3-8ee2-56698ae3731a-tls-certs\") pod \"e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9\" (UID: \"b0f879b5-5909-41f3-8ee2-56698ae3731a\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"
Aug 05 17:59:21.606928 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.606894    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grgl7\" (UniqueName: \"kubernetes.io/projected/b0f879b5-5909-41f3-8ee2-56698ae3731a-kube-api-access-grgl7\") pod \"e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9\" (UID: \"b0f879b5-5909-41f3-8ee2-56698ae3731a\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"
Aug 05 17:59:21.606928 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.606926    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0f879b5-5909-41f3-8ee2-56698ae3731a-home\") pod \"e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9\" (UID: \"b0f879b5-5909-41f3-8ee2-56698ae3731a\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"
Aug 05 17:59:21.607111 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.606970    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0f879b5-5909-41f3-8ee2-56698ae3731a-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9\" (UID: \"b0f879b5-5909-41f3-8ee2-56698ae3731a\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"
Aug 05 17:59:21.607111 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.607054    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0f879b5-5909-41f3-8ee2-56698ae3731a-dshm\") pod \"e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9\" (UID: \"b0f879b5-5909-41f3-8ee2-56698ae3731a\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"
Aug 05 17:59:21.607186 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.607122    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0f879b5-5909-41f3-8ee2-56698ae3731a-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9\" (UID: \"b0f879b5-5909-41f3-8ee2-56698ae3731a\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"
Aug 05 17:59:21.608974 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.608942    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Aug 05 17:59:21.608974 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.608964    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Aug 05 17:59:21.609144 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.608981    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Aug 05 17:59:21.609144 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.609017    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"seaweedfs-s3-creds\""
Aug 05 17:59:21.609144 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.609047    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-jxr2m\""
Aug 05 17:59:21.616667 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.616641    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"]
Aug 05 17:59:21.707705 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.707665    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0f879b5-5909-41f3-8ee2-56698ae3731a-tls-certs\") pod \"e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9\" (UID: \"b0f879b5-5909-41f3-8ee2-56698ae3731a\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"
Aug 05 17:59:21.707705 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.707705    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grgl7\" (UniqueName: \"kubernetes.io/projected/b0f879b5-5909-41f3-8ee2-56698ae3731a-kube-api-access-grgl7\") pod \"e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9\" (UID: \"b0f879b5-5909-41f3-8ee2-56698ae3731a\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"
Aug 05 17:59:21.707940 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.707731    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0f879b5-5909-41f3-8ee2-56698ae3731a-home\") pod \"e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9\" (UID: \"b0f879b5-5909-41f3-8ee2-56698ae3731a\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"
Aug 05 17:59:21.707940 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.707841    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0f879b5-5909-41f3-8ee2-56698ae3731a-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9\" (UID: \"b0f879b5-5909-41f3-8ee2-56698ae3731a\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"
Aug 05 17:59:21.707940 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.707904    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0f879b5-5909-41f3-8ee2-56698ae3731a-dshm\") pod \"e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9\" (UID: \"b0f879b5-5909-41f3-8ee2-56698ae3731a\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"
Aug 05 17:59:21.708104 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.707985    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0f879b5-5909-41f3-8ee2-56698ae3731a-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9\" (UID: \"b0f879b5-5909-41f3-8ee2-56698ae3731a\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"
Aug 05 17:59:21.708104 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.708027    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0f879b5-5909-41f3-8ee2-56698ae3731a-model-cache\") pod \"e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9\" (UID: \"b0f879b5-5909-41f3-8ee2-56698ae3731a\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"
Aug 05 17:59:21.708193 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.708163    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0f879b5-5909-41f3-8ee2-56698ae3731a-home\") pod \"e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9\" (UID: \"b0f879b5-5909-41f3-8ee2-56698ae3731a\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"
Aug 05 17:59:21.708232 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.708193    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0f879b5-5909-41f3-8ee2-56698ae3731a-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9\" (UID: \"b0f879b5-5909-41f3-8ee2-56698ae3731a\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"
Aug 05 17:59:21.708382 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.708365    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0f879b5-5909-41f3-8ee2-56698ae3731a-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9\" (UID: \"b0f879b5-5909-41f3-8ee2-56698ae3731a\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"
Aug 05 17:59:21.708442 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.708384    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0f879b5-5909-41f3-8ee2-56698ae3731a-model-cache\") pod \"e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9\" (UID: \"b0f879b5-5909-41f3-8ee2-56698ae3731a\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"
Aug 05 17:59:21.710238 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.710213    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0f879b5-5909-41f3-8ee2-56698ae3731a-dshm\") pod \"e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9\" (UID: \"b0f879b5-5909-41f3-8ee2-56698ae3731a\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"
Aug 05 17:59:21.710504 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.710488    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0f879b5-5909-41f3-8ee2-56698ae3731a-tls-certs\") pod \"e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9\" (UID: \"b0f879b5-5909-41f3-8ee2-56698ae3731a\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"
Aug 05 17:59:21.715126 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.715102    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grgl7\" (UniqueName: \"kubernetes.io/projected/b0f879b5-5909-41f3-8ee2-56698ae3731a-kube-api-access-grgl7\") pod \"e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9\" (UID: \"b0f879b5-5909-41f3-8ee2-56698ae3731a\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"
Aug 05 17:59:21.918625 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:21.918526    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"
Aug 05 17:59:22.073592 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:22.073561    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"]
Aug 05 17:59:22.075352 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:59:22.075318    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f879b5_5909_41f3_8ee2_56698ae3731a.slice/crio-973133a93b9003f83c2a2a826ce2112f7d0a91a761385cdc050bcf84137cd3c5 WatchSource:0}: Error finding container 973133a93b9003f83c2a2a826ce2112f7d0a91a761385cdc050bcf84137cd3c5: Status 404 returned error can't find the container with id 973133a93b9003f83c2a2a826ce2112f7d0a91a761385cdc050bcf84137cd3c5
Aug 05 17:59:22.208024 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:22.207887    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9" event={"ID":"b0f879b5-5909-41f3-8ee2-56698ae3731a","Type":"ContainerStarted","Data":"77db875219c333b8a70e091ea6fea067282c4be647a89828cb1f725cac9eb8db"}
Aug 05 17:59:22.208024 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:22.207993    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9" event={"ID":"b0f879b5-5909-41f3-8ee2-56698ae3731a","Type":"ContainerStarted","Data":"973133a93b9003f83c2a2a826ce2112f7d0a91a761385cdc050bcf84137cd3c5"}
Aug 05 17:59:24.438195 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:24.438149    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"]
Aug 05 17:59:24.438728 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:24.438532    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9" podUID="b0f879b5-5909-41f3-8ee2-56698ae3731a" containerName="storage-initializer" containerID="cri-o://77db875219c333b8a70e091ea6fea067282c4be647a89828cb1f725cac9eb8db" gracePeriod=60
Aug 05 17:59:27.239446 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:27.239403    2561 generic.go:356] "Generic (PLEG): container finished" podID="b0f879b5-5909-41f3-8ee2-56698ae3731a" containerID="77db875219c333b8a70e091ea6fea067282c4be647a89828cb1f725cac9eb8db" exitCode=0
Aug 05 17:59:27.239824 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:27.239471    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9" event={"ID":"b0f879b5-5909-41f3-8ee2-56698ae3731a","Type":"ContainerDied","Data":"77db875219c333b8a70e091ea6fea067282c4be647a89828cb1f725cac9eb8db"}
Aug 05 17:59:27.306551 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:27.306519    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"
Aug 05 17:59:27.378733 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:27.378641    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0f879b5-5909-41f3-8ee2-56698ae3731a-model-cache\") pod \"b0f879b5-5909-41f3-8ee2-56698ae3731a\" (UID: \"b0f879b5-5909-41f3-8ee2-56698ae3731a\") "
Aug 05 17:59:27.378733 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:27.378682    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grgl7\" (UniqueName: \"kubernetes.io/projected/b0f879b5-5909-41f3-8ee2-56698ae3731a-kube-api-access-grgl7\") pod \"b0f879b5-5909-41f3-8ee2-56698ae3731a\" (UID: \"b0f879b5-5909-41f3-8ee2-56698ae3731a\") "
Aug 05 17:59:27.378733 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:27.378722    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0f879b5-5909-41f3-8ee2-56698ae3731a-tls-certs\") pod \"b0f879b5-5909-41f3-8ee2-56698ae3731a\" (UID: \"b0f879b5-5909-41f3-8ee2-56698ae3731a\") "
Aug 05 17:59:27.379049 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:27.378745    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0f879b5-5909-41f3-8ee2-56698ae3731a-home\") pod \"b0f879b5-5909-41f3-8ee2-56698ae3731a\" (UID: \"b0f879b5-5909-41f3-8ee2-56698ae3731a\") "
Aug 05 17:59:27.379049 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:27.378809    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0f879b5-5909-41f3-8ee2-56698ae3731a-dshm\") pod \"b0f879b5-5909-41f3-8ee2-56698ae3731a\" (UID: \"b0f879b5-5909-41f3-8ee2-56698ae3731a\") "
Aug 05 17:59:27.379049 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:27.378856    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0f879b5-5909-41f3-8ee2-56698ae3731a-kserve-provision-location\") pod \"b0f879b5-5909-41f3-8ee2-56698ae3731a\" (UID: \"b0f879b5-5909-41f3-8ee2-56698ae3731a\") "
Aug 05 17:59:27.379049 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:27.378884    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0f879b5-5909-41f3-8ee2-56698ae3731a-tmp-dir\") pod \"b0f879b5-5909-41f3-8ee2-56698ae3731a\" (UID: \"b0f879b5-5909-41f3-8ee2-56698ae3731a\") "
Aug 05 17:59:27.379049 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:27.378984    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f879b5-5909-41f3-8ee2-56698ae3731a-model-cache" (OuterVolumeSpecName: "model-cache") pod "b0f879b5-5909-41f3-8ee2-56698ae3731a" (UID: "b0f879b5-5909-41f3-8ee2-56698ae3731a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:59:27.379324 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:27.379062    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f879b5-5909-41f3-8ee2-56698ae3731a-home" (OuterVolumeSpecName: "home") pod "b0f879b5-5909-41f3-8ee2-56698ae3731a" (UID: "b0f879b5-5909-41f3-8ee2-56698ae3731a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:59:27.379324 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:27.379245    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f879b5-5909-41f3-8ee2-56698ae3731a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b0f879b5-5909-41f3-8ee2-56698ae3731a" (UID: "b0f879b5-5909-41f3-8ee2-56698ae3731a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:59:27.379324 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:27.379290    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0f879b5-5909-41f3-8ee2-56698ae3731a-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:59:27.379324 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:27.379306    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0f879b5-5909-41f3-8ee2-56698ae3731a-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:59:27.381160 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:27.381138    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f879b5-5909-41f3-8ee2-56698ae3731a-dshm" (OuterVolumeSpecName: "dshm") pod "b0f879b5-5909-41f3-8ee2-56698ae3731a" (UID: "b0f879b5-5909-41f3-8ee2-56698ae3731a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:59:27.381412 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:27.381383    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0f879b5-5909-41f3-8ee2-56698ae3731a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b0f879b5-5909-41f3-8ee2-56698ae3731a" (UID: "b0f879b5-5909-41f3-8ee2-56698ae3731a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 17:59:27.381514 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:27.381462    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0f879b5-5909-41f3-8ee2-56698ae3731a-kube-api-access-grgl7" (OuterVolumeSpecName: "kube-api-access-grgl7") pod "b0f879b5-5909-41f3-8ee2-56698ae3731a" (UID: "b0f879b5-5909-41f3-8ee2-56698ae3731a"). InnerVolumeSpecName "kube-api-access-grgl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:59:27.458175 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:27.458124    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0f879b5-5909-41f3-8ee2-56698ae3731a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b0f879b5-5909-41f3-8ee2-56698ae3731a" (UID: "b0f879b5-5909-41f3-8ee2-56698ae3731a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:59:27.480277 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:27.480237    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0f879b5-5909-41f3-8ee2-56698ae3731a-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:59:27.480277 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:27.480269    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0f879b5-5909-41f3-8ee2-56698ae3731a-kserve-provision-location\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:59:27.480277 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:27.480283    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0f879b5-5909-41f3-8ee2-56698ae3731a-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:59:27.480560 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:27.480295    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grgl7\" (UniqueName: \"kubernetes.io/projected/b0f879b5-5909-41f3-8ee2-56698ae3731a-kube-api-access-grgl7\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:59:27.480560 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:27.480303    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0f879b5-5909-41f3-8ee2-56698ae3731a-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:59:28.246584 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:28.246554    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"
Aug 05 17:59:28.247030 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:28.246589    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9" event={"ID":"b0f879b5-5909-41f3-8ee2-56698ae3731a","Type":"ContainerDied","Data":"973133a93b9003f83c2a2a826ce2112f7d0a91a761385cdc050bcf84137cd3c5"}
Aug 05 17:59:28.247030 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:28.246652    2561 scope.go:117] "RemoveContainer" containerID="77db875219c333b8a70e091ea6fea067282c4be647a89828cb1f725cac9eb8db"
Aug 05 17:59:28.283485 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:28.283452    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"]
Aug 05 17:59:28.286866 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:28.286838    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6bdc48b47c-lhtp9"]
Aug 05 17:59:28.730207 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:28.730169    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0f879b5-5909-41f3-8ee2-56698ae3731a" path="/var/lib/kubelet/pods/b0f879b5-5909-41f3-8ee2-56698ae3731a/volumes"
Aug 05 17:59:45.722590 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:45.722557    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6785764b98-2vv5m"]
Aug 05 17:59:45.723013 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:45.722930    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0f879b5-5909-41f3-8ee2-56698ae3731a" containerName="storage-initializer"
Aug 05 17:59:45.723013 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:45.722943    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0f879b5-5909-41f3-8ee2-56698ae3731a" containerName="storage-initializer"
Aug 05 17:59:45.723013 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:45.723014    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0f879b5-5909-41f3-8ee2-56698ae3731a" containerName="storage-initializer"
Aug 05 17:59:45.727597 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:45.727571    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6785764b98-2vv5m"
Aug 05 17:59:45.730273 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:45.730205    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"openshift-service-ca.crt\""
Aug 05 17:59:45.730273 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:45.730211    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"seaweedfs-s3-creds\""
Aug 05 17:59:45.731169 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:45.731153    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"default-dockercfg-fqd9v\""
Aug 05 17:59:45.731246 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:45.731224    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"kube-root-ca.crt\""
Aug 05 17:59:45.737890 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:45.737868    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6785764b98-2vv5m"]
Aug 05 17:59:45.842459 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:45.842424    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f2e5c29-5d7f-41a4-9a01-f6a16381376a-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-6785764b98-2vv5m\" (UID: \"8f2e5c29-5d7f-41a4-9a01-f6a16381376a\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6785764b98-2vv5m"
Aug 05 17:59:45.842640 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:45.842620    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdmcn\" (UniqueName: \"kubernetes.io/projected/8f2e5c29-5d7f-41a4-9a01-f6a16381376a-kube-api-access-tdmcn\") pod \"e2e-wk-cfg-implicit-svc-kserve-6785764b98-2vv5m\" (UID: \"8f2e5c29-5d7f-41a4-9a01-f6a16381376a\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6785764b98-2vv5m"
Aug 05 17:59:45.943609 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:45.943573    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdmcn\" (UniqueName: \"kubernetes.io/projected/8f2e5c29-5d7f-41a4-9a01-f6a16381376a-kube-api-access-tdmcn\") pod \"e2e-wk-cfg-implicit-svc-kserve-6785764b98-2vv5m\" (UID: \"8f2e5c29-5d7f-41a4-9a01-f6a16381376a\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6785764b98-2vv5m"
Aug 05 17:59:45.943806 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:45.943653    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f2e5c29-5d7f-41a4-9a01-f6a16381376a-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-6785764b98-2vv5m\" (UID: \"8f2e5c29-5d7f-41a4-9a01-f6a16381376a\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6785764b98-2vv5m"
Aug 05 17:59:45.944071 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:45.944054    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f2e5c29-5d7f-41a4-9a01-f6a16381376a-kserve-provision-location\") pod \"e2e-wk-cfg-implicit-svc-kserve-6785764b98-2vv5m\" (UID: \"8f2e5c29-5d7f-41a4-9a01-f6a16381376a\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6785764b98-2vv5m"
Aug 05 17:59:45.952009 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:45.951979    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdmcn\" (UniqueName: \"kubernetes.io/projected/8f2e5c29-5d7f-41a4-9a01-f6a16381376a-kube-api-access-tdmcn\") pod \"e2e-wk-cfg-implicit-svc-kserve-6785764b98-2vv5m\" (UID: \"8f2e5c29-5d7f-41a4-9a01-f6a16381376a\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6785764b98-2vv5m"
Aug 05 17:59:46.039506 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:46.039476    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6785764b98-2vv5m"
Aug 05 17:59:46.176985 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:46.176890    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6785764b98-2vv5m"]
Aug 05 17:59:46.346362 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:46.346261    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6785764b98-2vv5m" event={"ID":"8f2e5c29-5d7f-41a4-9a01-f6a16381376a","Type":"ContainerStarted","Data":"440883457f7ca35d29065022a9f800a53851f57e675b082984cbba51ca82c8e7"}
Aug 05 17:59:46.346362 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:46.346342    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6785764b98-2vv5m" event={"ID":"8f2e5c29-5d7f-41a4-9a01-f6a16381376a","Type":"ContainerStarted","Data":"9a82d98268a4e878e71a1384ee3951d213ac2296e6fe29ad70d896f1274c4151"}
Aug 05 17:59:48.177477 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:48.177441    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6785764b98-2vv5m"]
Aug 05 17:59:48.360682 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:48.360608    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6785764b98-2vv5m" podUID="8f2e5c29-5d7f-41a4-9a01-f6a16381376a" containerName="storage-initializer" containerID="cri-o://440883457f7ca35d29065022a9f800a53851f57e675b082984cbba51ca82c8e7" gracePeriod=30
Aug 05 17:59:50.714333 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:50.714307    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6785764b98-2vv5m"
Aug 05 17:59:50.794275 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:50.794240    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdmcn\" (UniqueName: \"kubernetes.io/projected/8f2e5c29-5d7f-41a4-9a01-f6a16381376a-kube-api-access-tdmcn\") pod \"8f2e5c29-5d7f-41a4-9a01-f6a16381376a\" (UID: \"8f2e5c29-5d7f-41a4-9a01-f6a16381376a\") "
Aug 05 17:59:50.794443 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:50.794321    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f2e5c29-5d7f-41a4-9a01-f6a16381376a-kserve-provision-location\") pod \"8f2e5c29-5d7f-41a4-9a01-f6a16381376a\" (UID: \"8f2e5c29-5d7f-41a4-9a01-f6a16381376a\") "
Aug 05 17:59:50.796637 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:50.796608    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f2e5c29-5d7f-41a4-9a01-f6a16381376a-kube-api-access-tdmcn" (OuterVolumeSpecName: "kube-api-access-tdmcn") pod "8f2e5c29-5d7f-41a4-9a01-f6a16381376a" (UID: "8f2e5c29-5d7f-41a4-9a01-f6a16381376a"). InnerVolumeSpecName "kube-api-access-tdmcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 17:59:50.877819 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:50.877780    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2e5c29-5d7f-41a4-9a01-f6a16381376a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8f2e5c29-5d7f-41a4-9a01-f6a16381376a" (UID: "8f2e5c29-5d7f-41a4-9a01-f6a16381376a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 17:59:50.895070 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:50.895040    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdmcn\" (UniqueName: \"kubernetes.io/projected/8f2e5c29-5d7f-41a4-9a01-f6a16381376a-kube-api-access-tdmcn\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:59:50.895070 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:50.895067    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f2e5c29-5d7f-41a4-9a01-f6a16381376a-kserve-provision-location\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 17:59:51.377858 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:51.377819    2561 generic.go:356] "Generic (PLEG): container finished" podID="8f2e5c29-5d7f-41a4-9a01-f6a16381376a" containerID="440883457f7ca35d29065022a9f800a53851f57e675b082984cbba51ca82c8e7" exitCode=0
Aug 05 17:59:51.378120 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:51.377863    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6785764b98-2vv5m"
Aug 05 17:59:51.378120 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:51.377910    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6785764b98-2vv5m" event={"ID":"8f2e5c29-5d7f-41a4-9a01-f6a16381376a","Type":"ContainerDied","Data":"440883457f7ca35d29065022a9f800a53851f57e675b082984cbba51ca82c8e7"}
Aug 05 17:59:51.378120 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:51.377992    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6785764b98-2vv5m" event={"ID":"8f2e5c29-5d7f-41a4-9a01-f6a16381376a","Type":"ContainerDied","Data":"9a82d98268a4e878e71a1384ee3951d213ac2296e6fe29ad70d896f1274c4151"}
Aug 05 17:59:51.378120 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:51.378029    2561 scope.go:117] "RemoveContainer" containerID="440883457f7ca35d29065022a9f800a53851f57e675b082984cbba51ca82c8e7"
Aug 05 17:59:51.387462 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:51.387439    2561 scope.go:117] "RemoveContainer" containerID="440883457f7ca35d29065022a9f800a53851f57e675b082984cbba51ca82c8e7"
Aug 05 17:59:51.387726 ip-10-0-135-151 kubenswrapper[2561]: E0805 17:59:51.387704    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"440883457f7ca35d29065022a9f800a53851f57e675b082984cbba51ca82c8e7\": container with ID starting with 440883457f7ca35d29065022a9f800a53851f57e675b082984cbba51ca82c8e7 not found: ID does not exist" containerID="440883457f7ca35d29065022a9f800a53851f57e675b082984cbba51ca82c8e7"
Aug 05 17:59:51.387767 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:51.387735    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"440883457f7ca35d29065022a9f800a53851f57e675b082984cbba51ca82c8e7"} err="failed to get container status \"440883457f7ca35d29065022a9f800a53851f57e675b082984cbba51ca82c8e7\": rpc error: code = NotFound desc = could not find container \"440883457f7ca35d29065022a9f800a53851f57e675b082984cbba51ca82c8e7\": container with ID starting with 440883457f7ca35d29065022a9f800a53851f57e675b082984cbba51ca82c8e7 not found: ID does not exist"
Aug 05 17:59:51.416885 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:51.416850    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6785764b98-2vv5m"]
Aug 05 17:59:51.421924 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:51.421893    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-6785764b98-2vv5m"]
Aug 05 17:59:52.730316 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:52.730279    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f2e5c29-5d7f-41a4-9a01-f6a16381376a" path="/var/lib/kubelet/pods/8f2e5c29-5d7f-41a4-9a01-f6a16381376a/volumes"
Aug 05 17:59:57.621866 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.621825    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"]
Aug 05 17:59:57.622326 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.622260    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f2e5c29-5d7f-41a4-9a01-f6a16381376a" containerName="storage-initializer"
Aug 05 17:59:57.622326 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.622274    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f2e5c29-5d7f-41a4-9a01-f6a16381376a" containerName="storage-initializer"
Aug 05 17:59:57.622407 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.622342    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f2e5c29-5d7f-41a4-9a01-f6a16381376a" containerName="storage-initializer"
Aug 05 17:59:57.626947 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.626930    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 17:59:57.629501 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.629476    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Aug 05 17:59:57.629730 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.629714    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Aug 05 17:59:57.629816 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.629718    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Aug 05 17:59:57.629816 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.629720    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Aug 05 17:59:57.629934 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.629820    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Aug 05 17:59:57.629934 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.629868    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-jbxpc\""
Aug 05 17:59:57.635717 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.635689    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"]
Aug 05 17:59:57.760772 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.760727    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6f15972-a264-42f2-8e0a-b0f9b6638e57-home\") pod \"stop-feature-test-kserve-5b9dc4dbc9-jhlxd\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 17:59:57.760991 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.760829    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6f15972-a264-42f2-8e0a-b0f9b6638e57-tls-certs\") pod \"stop-feature-test-kserve-5b9dc4dbc9-jhlxd\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 17:59:57.760991 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.760860    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxqgm\" (UniqueName: \"kubernetes.io/projected/b6f15972-a264-42f2-8e0a-b0f9b6638e57-kube-api-access-zxqgm\") pod \"stop-feature-test-kserve-5b9dc4dbc9-jhlxd\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 17:59:57.760991 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.760894    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6f15972-a264-42f2-8e0a-b0f9b6638e57-dshm\") pod \"stop-feature-test-kserve-5b9dc4dbc9-jhlxd\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 17:59:57.760991 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.760920    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6f15972-a264-42f2-8e0a-b0f9b6638e57-model-cache\") pod \"stop-feature-test-kserve-5b9dc4dbc9-jhlxd\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 17:59:57.761199 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.760997    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6f15972-a264-42f2-8e0a-b0f9b6638e57-tmp-dir\") pod \"stop-feature-test-kserve-5b9dc4dbc9-jhlxd\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 17:59:57.761199 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.761043    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6f15972-a264-42f2-8e0a-b0f9b6638e57-kserve-provision-location\") pod \"stop-feature-test-kserve-5b9dc4dbc9-jhlxd\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 17:59:57.761199 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.761119    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6f15972-a264-42f2-8e0a-b0f9b6638e57-cabundle-cert\") pod \"stop-feature-test-kserve-5b9dc4dbc9-jhlxd\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 17:59:57.856696 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.856659    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg"]
Aug 05 17:59:57.860721 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.860696    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg"
Aug 05 17:59:57.861944 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.861919    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6f15972-a264-42f2-8e0a-b0f9b6638e57-cabundle-cert\") pod \"stop-feature-test-kserve-5b9dc4dbc9-jhlxd\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 17:59:57.862124 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.861977    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6f15972-a264-42f2-8e0a-b0f9b6638e57-home\") pod \"stop-feature-test-kserve-5b9dc4dbc9-jhlxd\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 17:59:57.862124 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.862071    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6f15972-a264-42f2-8e0a-b0f9b6638e57-tls-certs\") pod \"stop-feature-test-kserve-5b9dc4dbc9-jhlxd\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 17:59:57.862124 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.862100    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxqgm\" (UniqueName: \"kubernetes.io/projected/b6f15972-a264-42f2-8e0a-b0f9b6638e57-kube-api-access-zxqgm\") pod \"stop-feature-test-kserve-5b9dc4dbc9-jhlxd\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 17:59:57.862289 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.862165    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6f15972-a264-42f2-8e0a-b0f9b6638e57-dshm\") pod \"stop-feature-test-kserve-5b9dc4dbc9-jhlxd\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 17:59:57.862289 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.862216    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6f15972-a264-42f2-8e0a-b0f9b6638e57-model-cache\") pod \"stop-feature-test-kserve-5b9dc4dbc9-jhlxd\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 17:59:57.862289 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.862269    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6f15972-a264-42f2-8e0a-b0f9b6638e57-tmp-dir\") pod \"stop-feature-test-kserve-5b9dc4dbc9-jhlxd\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 17:59:57.862446 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.862351    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6f15972-a264-42f2-8e0a-b0f9b6638e57-kserve-provision-location\") pod \"stop-feature-test-kserve-5b9dc4dbc9-jhlxd\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 17:59:57.862446 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.862437    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6f15972-a264-42f2-8e0a-b0f9b6638e57-home\") pod \"stop-feature-test-kserve-5b9dc4dbc9-jhlxd\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 17:59:57.862672 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.862649    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6f15972-a264-42f2-8e0a-b0f9b6638e57-kserve-provision-location\") pod \"stop-feature-test-kserve-5b9dc4dbc9-jhlxd\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 17:59:57.862827 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.862797    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6f15972-a264-42f2-8e0a-b0f9b6638e57-cabundle-cert\") pod \"stop-feature-test-kserve-5b9dc4dbc9-jhlxd\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 17:59:57.863066 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.863048    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-lg8q4\""
Aug 05 17:59:57.863769 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.863739    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6f15972-a264-42f2-8e0a-b0f9b6638e57-tmp-dir\") pod \"stop-feature-test-kserve-5b9dc4dbc9-jhlxd\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 17:59:57.864076 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.864049    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6f15972-a264-42f2-8e0a-b0f9b6638e57-model-cache\") pod \"stop-feature-test-kserve-5b9dc4dbc9-jhlxd\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 17:59:57.865218 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.865195    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6f15972-a264-42f2-8e0a-b0f9b6638e57-dshm\") pod \"stop-feature-test-kserve-5b9dc4dbc9-jhlxd\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 17:59:57.865887 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.865861    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6f15972-a264-42f2-8e0a-b0f9b6638e57-tls-certs\") pod \"stop-feature-test-kserve-5b9dc4dbc9-jhlxd\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 17:59:57.871262 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.871238    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg"]
Aug 05 17:59:57.873116 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.873051    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxqgm\" (UniqueName: \"kubernetes.io/projected/b6f15972-a264-42f2-8e0a-b0f9b6638e57-kube-api-access-zxqgm\") pod \"stop-feature-test-kserve-5b9dc4dbc9-jhlxd\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 17:59:57.937971 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.937928    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 17:59:57.963725 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.963679    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv85p\" (UniqueName: \"kubernetes.io/projected/70cdd9f1-c0d2-4487-a160-d5f07ed97258-kube-api-access-qv85p\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg\" (UID: \"70cdd9f1-c0d2-4487-a160-d5f07ed97258\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg"
Aug 05 17:59:57.963985 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:57.963859    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70cdd9f1-c0d2-4487-a160-d5f07ed97258-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg\" (UID: \"70cdd9f1-c0d2-4487-a160-d5f07ed97258\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg"
Aug 05 17:59:58.064663 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:58.064627    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70cdd9f1-c0d2-4487-a160-d5f07ed97258-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg\" (UID: \"70cdd9f1-c0d2-4487-a160-d5f07ed97258\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg"
Aug 05 17:59:58.064853 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:58.064777    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv85p\" (UniqueName: \"kubernetes.io/projected/70cdd9f1-c0d2-4487-a160-d5f07ed97258-kube-api-access-qv85p\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg\" (UID: \"70cdd9f1-c0d2-4487-a160-d5f07ed97258\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg"
Aug 05 17:59:58.067299 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:58.067256    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70cdd9f1-c0d2-4487-a160-d5f07ed97258-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg\" (UID: \"70cdd9f1-c0d2-4487-a160-d5f07ed97258\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg"
Aug 05 17:59:58.075513 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:58.075462    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv85p\" (UniqueName: \"kubernetes.io/projected/70cdd9f1-c0d2-4487-a160-d5f07ed97258-kube-api-access-qv85p\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg\" (UID: \"70cdd9f1-c0d2-4487-a160-d5f07ed97258\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg"
Aug 05 17:59:58.082231 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:58.082210    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"]
Aug 05 17:59:58.083990 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:59:58.083964    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f15972_a264_42f2_8e0a_b0f9b6638e57.slice/crio-2e45968f067d0c5a7cd5fe25e34c288888ae83623e88677327156c8dcd47cd0a WatchSource:0}: Error finding container 2e45968f067d0c5a7cd5fe25e34c288888ae83623e88677327156c8dcd47cd0a: Status 404 returned error can't find the container with id 2e45968f067d0c5a7cd5fe25e34c288888ae83623e88677327156c8dcd47cd0a
Aug 05 17:59:58.191780 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:58.191744    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg"
Aug 05 17:59:58.344555 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:58.344521    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg"]
Aug 05 17:59:58.347551 ip-10-0-135-151 kubenswrapper[2561]: W0805 17:59:58.347516    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70cdd9f1_c0d2_4487_a160_d5f07ed97258.slice/crio-b05ab54e1d148f46e21d1be173c099c798da225cb0c42b9d24f4ffe2df8a1b6e WatchSource:0}: Error finding container b05ab54e1d148f46e21d1be173c099c798da225cb0c42b9d24f4ffe2df8a1b6e: Status 404 returned error can't find the container with id b05ab54e1d148f46e21d1be173c099c798da225cb0c42b9d24f4ffe2df8a1b6e
Aug 05 17:59:58.424721 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:58.424682    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg" event={"ID":"70cdd9f1-c0d2-4487-a160-d5f07ed97258","Type":"ContainerStarted","Data":"25b129e35b6a27b6ad8936ea6ade0bf6da527eec8a6377f116293f2311998103"}
Aug 05 17:59:58.424881 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:58.424765    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg" event={"ID":"70cdd9f1-c0d2-4487-a160-d5f07ed97258","Type":"ContainerStarted","Data":"b05ab54e1d148f46e21d1be173c099c798da225cb0c42b9d24f4ffe2df8a1b6e"}
Aug 05 17:59:58.424881 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:58.424799    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg"
Aug 05 17:59:58.426603 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:58.426575    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd" event={"ID":"b6f15972-a264-42f2-8e0a-b0f9b6638e57","Type":"ContainerStarted","Data":"e89469f8597c54282c4fd9bc05ce3a6e261d6df1ee43a9135c6b79097be4887d"}
Aug 05 17:59:58.426704 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:58.426635    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd" event={"ID":"b6f15972-a264-42f2-8e0a-b0f9b6638e57","Type":"ContainerStarted","Data":"2e45968f067d0c5a7cd5fe25e34c288888ae83623e88677327156c8dcd47cd0a"}
Aug 05 17:59:58.442531 ip-10-0-135-151 kubenswrapper[2561]: I0805 17:59:58.442475    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg" podStartSLOduration=1.442458744 podStartE2EDuration="1.442458744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 17:59:58.439764719 +0000 UTC m=+2854.184399045" watchObservedRunningTime="2026-08-05 17:59:58.442458744 +0000 UTC m=+2854.187093072"
Aug 05 18:00:01.446172 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:00:01.446128    2561 generic.go:356] "Generic (PLEG): container finished" podID="b6f15972-a264-42f2-8e0a-b0f9b6638e57" containerID="e89469f8597c54282c4fd9bc05ce3a6e261d6df1ee43a9135c6b79097be4887d" exitCode=0
Aug 05 18:00:01.446632 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:00:01.446203    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd" event={"ID":"b6f15972-a264-42f2-8e0a-b0f9b6638e57","Type":"ContainerDied","Data":"e89469f8597c54282c4fd9bc05ce3a6e261d6df1ee43a9135c6b79097be4887d"}
Aug 05 18:00:02.455139 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:00:02.455080    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd" event={"ID":"b6f15972-a264-42f2-8e0a-b0f9b6638e57","Type":"ContainerStarted","Data":"4f62c89b474be3ec123cabcf2b648722c8a2ec4992be5eeec164f95b0c9701e0"}
Aug 05 18:00:02.478920 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:00:02.478865    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd" podStartSLOduration=3.087228481 podStartE2EDuration="5.478848741s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.39162026s" isStatefulPod=false podCreationTimestamp="2026-08-05 17:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:00:02.474556963 +0000 UTC m=+2858.219191309" watchObservedRunningTime="2026-08-05 18:00:02.478848741 +0000 UTC m=+2858.223483068"
Aug 05 18:00:02.938450 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:00:02.938417    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 18:00:07.939043 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:00:07.939001    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 18:00:07.940693 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:00:07.940657    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd" podUID="b6f15972-a264-42f2-8e0a-b0f9b6638e57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused"
Aug 05 18:00:17.938704 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:00:17.938609    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd" podUID="b6f15972-a264-42f2-8e0a-b0f9b6638e57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused"
Aug 05 18:00:27.939057 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:00:27.939013    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd" podUID="b6f15972-a264-42f2-8e0a-b0f9b6638e57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused"
Aug 05 18:00:29.433996 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:00:29.433964    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg"
Aug 05 18:00:37.938820 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:00:37.938774    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd" podUID="b6f15972-a264-42f2-8e0a-b0f9b6638e57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused"
Aug 05 18:00:47.938542 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:00:47.938498    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd" podUID="b6f15972-a264-42f2-8e0a-b0f9b6638e57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused"
Aug 05 18:00:57.938582 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:00:57.938528    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd" podUID="b6f15972-a264-42f2-8e0a-b0f9b6638e57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused"
Aug 05 18:01:07.938480 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:07.938433    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd" podUID="b6f15972-a264-42f2-8e0a-b0f9b6638e57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused"
Aug 05 18:01:16.838540 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:16.838454    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"]
Aug 05 18:01:16.839031 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:16.838773    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw" podUID="daca7cad-7a28-47df-af89-c25576d556da" containerName="istio-proxy" containerID="cri-o://8c73ec4f63c064cc3572f55c8e0b7d99d8835726f5fbc7b98f3f833ab7c15fa7" gracePeriod=30
Aug 05 18:01:17.938727 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:17.938680    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd" podUID="b6f15972-a264-42f2-8e0a-b0f9b6638e57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused"
Aug 05 18:01:22.116691 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.116621    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 18:01:22.237217 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.237180    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daca7cad-7a28-47df-af89-c25576d556da-credential-socket\") pod \"daca7cad-7a28-47df-af89-c25576d556da\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") "
Aug 05 18:01:22.237217 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.237219    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daca7cad-7a28-47df-af89-c25576d556da-istio-envoy\") pod \"daca7cad-7a28-47df-af89-c25576d556da\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") "
Aug 05 18:01:22.237486 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.237250    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daca7cad-7a28-47df-af89-c25576d556da-istiod-ca-cert\") pod \"daca7cad-7a28-47df-af89-c25576d556da\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") "
Aug 05 18:01:22.237486 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.237284    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daca7cad-7a28-47df-af89-c25576d556da-istio-token\") pod \"daca7cad-7a28-47df-af89-c25576d556da\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") "
Aug 05 18:01:22.237486 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.237313    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daca7cad-7a28-47df-af89-c25576d556da-istio-data\") pod \"daca7cad-7a28-47df-af89-c25576d556da\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") "
Aug 05 18:01:22.237486 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.237448    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daca7cad-7a28-47df-af89-c25576d556da-istio-podinfo\") pod \"daca7cad-7a28-47df-af89-c25576d556da\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") "
Aug 05 18:01:22.237695 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.237515    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwdxm\" (UniqueName: \"kubernetes.io/projected/daca7cad-7a28-47df-af89-c25576d556da-kube-api-access-vwdxm\") pod \"daca7cad-7a28-47df-af89-c25576d556da\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") "
Aug 05 18:01:22.237695 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.237591    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daca7cad-7a28-47df-af89-c25576d556da-workload-socket\") pod \"daca7cad-7a28-47df-af89-c25576d556da\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") "
Aug 05 18:01:22.237695 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.237621    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daca7cad-7a28-47df-af89-c25576d556da-istio-data" (OuterVolumeSpecName: "istio-data") pod "daca7cad-7a28-47df-af89-c25576d556da" (UID: "daca7cad-7a28-47df-af89-c25576d556da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:01:22.237695 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.237648    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daca7cad-7a28-47df-af89-c25576d556da-workload-certs\") pod \"daca7cad-7a28-47df-af89-c25576d556da\" (UID: \"daca7cad-7a28-47df-af89-c25576d556da\") "
Aug 05 18:01:22.237695 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.237644    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daca7cad-7a28-47df-af89-c25576d556da-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "daca7cad-7a28-47df-af89-c25576d556da" (UID: "daca7cad-7a28-47df-af89-c25576d556da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 18:01:22.237695 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.237652    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daca7cad-7a28-47df-af89-c25576d556da-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "daca7cad-7a28-47df-af89-c25576d556da" (UID: "daca7cad-7a28-47df-af89-c25576d556da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:01:22.238046 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.237822    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daca7cad-7a28-47df-af89-c25576d556da-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "daca7cad-7a28-47df-af89-c25576d556da" (UID: "daca7cad-7a28-47df-af89-c25576d556da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:01:22.238046 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.237989    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daca7cad-7a28-47df-af89-c25576d556da-credential-socket\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:01:22.238046 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.238017    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daca7cad-7a28-47df-af89-c25576d556da-istiod-ca-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:01:22.238046 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.238031    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daca7cad-7a28-47df-af89-c25576d556da-istio-data\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:01:22.238046 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.238044    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daca7cad-7a28-47df-af89-c25576d556da-workload-socket\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:01:22.238352 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.238080    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daca7cad-7a28-47df-af89-c25576d556da-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "daca7cad-7a28-47df-af89-c25576d556da" (UID: "daca7cad-7a28-47df-af89-c25576d556da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:01:22.239748 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.239721    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daca7cad-7a28-47df-af89-c25576d556da-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "daca7cad-7a28-47df-af89-c25576d556da" (UID: "daca7cad-7a28-47df-af89-c25576d556da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:01:22.240012 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.239993    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daca7cad-7a28-47df-af89-c25576d556da-istio-token" (OuterVolumeSpecName: "istio-token") pod "daca7cad-7a28-47df-af89-c25576d556da" (UID: "daca7cad-7a28-47df-af89-c25576d556da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 18:01:22.240100 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.240017    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daca7cad-7a28-47df-af89-c25576d556da-kube-api-access-vwdxm" (OuterVolumeSpecName: "kube-api-access-vwdxm") pod "daca7cad-7a28-47df-af89-c25576d556da" (UID: "daca7cad-7a28-47df-af89-c25576d556da"). InnerVolumeSpecName "kube-api-access-vwdxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 18:01:22.240242 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.240226    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/daca7cad-7a28-47df-af89-c25576d556da-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "daca7cad-7a28-47df-af89-c25576d556da" (UID: "daca7cad-7a28-47df-af89-c25576d556da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Aug 05 18:01:22.338639 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.338596    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daca7cad-7a28-47df-af89-c25576d556da-istio-podinfo\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:01:22.338639 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.338633    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwdxm\" (UniqueName: \"kubernetes.io/projected/daca7cad-7a28-47df-af89-c25576d556da-kube-api-access-vwdxm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:01:22.338639 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.338646    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daca7cad-7a28-47df-af89-c25576d556da-workload-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:01:22.338894 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.338661    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daca7cad-7a28-47df-af89-c25576d556da-istio-envoy\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:01:22.338894 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.338672    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daca7cad-7a28-47df-af89-c25576d556da-istio-token\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:01:22.925192 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.925138    2561 generic.go:356] "Generic (PLEG): container finished" podID="daca7cad-7a28-47df-af89-c25576d556da" containerID="8c73ec4f63c064cc3572f55c8e0b7d99d8835726f5fbc7b98f3f833ab7c15fa7" exitCode=0
Aug 05 18:01:22.925475 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.925267    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"
Aug 05 18:01:22.925475 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.925287    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw" event={"ID":"daca7cad-7a28-47df-af89-c25576d556da","Type":"ContainerDied","Data":"8c73ec4f63c064cc3572f55c8e0b7d99d8835726f5fbc7b98f3f833ab7c15fa7"}
Aug 05 18:01:22.925475 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.925358    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw" event={"ID":"daca7cad-7a28-47df-af89-c25576d556da","Type":"ContainerDied","Data":"d9381a817ee86a3f12421f71a20d954cd3711f25e8a5d5130f619e739d41bcd5"}
Aug 05 18:01:22.925475 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.925395    2561 scope.go:117] "RemoveContainer" containerID="8c73ec4f63c064cc3572f55c8e0b7d99d8835726f5fbc7b98f3f833ab7c15fa7"
Aug 05 18:01:22.948903 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.948875    2561 scope.go:117] "RemoveContainer" containerID="e51e30bf2365681f9d0744ef3ef91a112c85bdb543ef698713f7ab03e3beeb81"
Aug 05 18:01:22.950282 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.950206    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"]
Aug 05 18:01:22.956940 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.956910    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-65765d8947-vclkw"]
Aug 05 18:01:22.967787 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.967666    2561 scope.go:117] "RemoveContainer" containerID="8c73ec4f63c064cc3572f55c8e0b7d99d8835726f5fbc7b98f3f833ab7c15fa7"
Aug 05 18:01:22.970513 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:01:22.970346    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c73ec4f63c064cc3572f55c8e0b7d99d8835726f5fbc7b98f3f833ab7c15fa7\": container with ID starting with 8c73ec4f63c064cc3572f55c8e0b7d99d8835726f5fbc7b98f3f833ab7c15fa7 not found: ID does not exist" containerID="8c73ec4f63c064cc3572f55c8e0b7d99d8835726f5fbc7b98f3f833ab7c15fa7"
Aug 05 18:01:22.970513 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.970389    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c73ec4f63c064cc3572f55c8e0b7d99d8835726f5fbc7b98f3f833ab7c15fa7"} err="failed to get container status \"8c73ec4f63c064cc3572f55c8e0b7d99d8835726f5fbc7b98f3f833ab7c15fa7\": rpc error: code = NotFound desc = could not find container \"8c73ec4f63c064cc3572f55c8e0b7d99d8835726f5fbc7b98f3f833ab7c15fa7\": container with ID starting with 8c73ec4f63c064cc3572f55c8e0b7d99d8835726f5fbc7b98f3f833ab7c15fa7 not found: ID does not exist"
Aug 05 18:01:22.970513 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.970421    2561 scope.go:117] "RemoveContainer" containerID="e51e30bf2365681f9d0744ef3ef91a112c85bdb543ef698713f7ab03e3beeb81"
Aug 05 18:01:22.971048 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:01:22.970947    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e51e30bf2365681f9d0744ef3ef91a112c85bdb543ef698713f7ab03e3beeb81\": container with ID starting with e51e30bf2365681f9d0744ef3ef91a112c85bdb543ef698713f7ab03e3beeb81 not found: ID does not exist" containerID="e51e30bf2365681f9d0744ef3ef91a112c85bdb543ef698713f7ab03e3beeb81"
Aug 05 18:01:22.971048 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:22.971006    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e51e30bf2365681f9d0744ef3ef91a112c85bdb543ef698713f7ab03e3beeb81"} err="failed to get container status \"e51e30bf2365681f9d0744ef3ef91a112c85bdb543ef698713f7ab03e3beeb81\": rpc error: code = NotFound desc = could not find container \"e51e30bf2365681f9d0744ef3ef91a112c85bdb543ef698713f7ab03e3beeb81\": container with ID starting with e51e30bf2365681f9d0744ef3ef91a112c85bdb543ef698713f7ab03e3beeb81 not found: ID does not exist"
Aug 05 18:01:24.730338 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:24.730304    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daca7cad-7a28-47df-af89-c25576d556da" path="/var/lib/kubelet/pods/daca7cad-7a28-47df-af89-c25576d556da/volumes"
Aug 05 18:01:27.939278 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:27.939227    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd" podUID="b6f15972-a264-42f2-8e0a-b0f9b6638e57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused"
Aug 05 18:01:37.938701 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:37.938647    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd" podUID="b6f15972-a264-42f2-8e0a-b0f9b6638e57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused"
Aug 05 18:01:47.940012 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:47.939879    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd" podUID="b6f15972-a264-42f2-8e0a-b0f9b6638e57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused"
Aug 05 18:01:57.948186 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:57.948149    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 18:01:57.956459 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:57.956430    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 18:01:59.384262 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:59.384220    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"]
Aug 05 18:01:59.475875 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:01:59.475837    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg"]
Aug 05 18:02:09.433515 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:09.433472    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg" podUID="70cdd9f1-c0d2-4487-a160-d5f07ed97258" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 05 18:02:14.408034 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:14.407989    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd" podUID="b6f15972-a264-42f2-8e0a-b0f9b6638e57" containerName="main" containerID="cri-o://4f62c89b474be3ec123cabcf2b648722c8a2ec4992be5eeec164f95b0c9701e0" gracePeriod=45
Aug 05 18:02:14.497325 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:14.497250    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg" podUID="70cdd9f1-c0d2-4487-a160-d5f07ed97258" containerName="main" containerID="cri-o://25b129e35b6a27b6ad8936ea6ade0bf6da527eec8a6377f116293f2311998103" gracePeriod=45
Aug 05 18:02:14.753455 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:14.753428    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg"
Aug 05 18:02:14.919457 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:14.919371    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qv85p\" (UniqueName: \"kubernetes.io/projected/70cdd9f1-c0d2-4487-a160-d5f07ed97258-kube-api-access-qv85p\") pod \"70cdd9f1-c0d2-4487-a160-d5f07ed97258\" (UID: \"70cdd9f1-c0d2-4487-a160-d5f07ed97258\") "
Aug 05 18:02:14.919629 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:14.919499    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70cdd9f1-c0d2-4487-a160-d5f07ed97258-tls-certs\") pod \"70cdd9f1-c0d2-4487-a160-d5f07ed97258\" (UID: \"70cdd9f1-c0d2-4487-a160-d5f07ed97258\") "
Aug 05 18:02:14.921709 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:14.921677    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70cdd9f1-c0d2-4487-a160-d5f07ed97258-kube-api-access-qv85p" (OuterVolumeSpecName: "kube-api-access-qv85p") pod "70cdd9f1-c0d2-4487-a160-d5f07ed97258" (UID: "70cdd9f1-c0d2-4487-a160-d5f07ed97258"). InnerVolumeSpecName "kube-api-access-qv85p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 18:02:14.921837 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:14.921762    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70cdd9f1-c0d2-4487-a160-d5f07ed97258-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "70cdd9f1-c0d2-4487-a160-d5f07ed97258" (UID: "70cdd9f1-c0d2-4487-a160-d5f07ed97258"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 18:02:14.948790 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:14.948760    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd" podUID="b6f15972-a264-42f2-8e0a-b0f9b6638e57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": dial tcp 10.133.0.98:8000: connect: connection refused"
Aug 05 18:02:15.021056 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.021015    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70cdd9f1-c0d2-4487-a160-d5f07ed97258-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:02:15.021056 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.021044    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv85p\" (UniqueName: \"kubernetes.io/projected/70cdd9f1-c0d2-4487-a160-d5f07ed97258-kube-api-access-qv85p\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:02:15.148485 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.148459    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 18:02:15.222817 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.222717    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6f15972-a264-42f2-8e0a-b0f9b6638e57-cabundle-cert\") pod \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") "
Aug 05 18:02:15.222817 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.222775    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6f15972-a264-42f2-8e0a-b0f9b6638e57-tls-certs\") pod \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") "
Aug 05 18:02:15.223114 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.222884    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6f15972-a264-42f2-8e0a-b0f9b6638e57-dshm\") pod \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") "
Aug 05 18:02:15.223114 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.222942    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6f15972-a264-42f2-8e0a-b0f9b6638e57-home\") pod \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") "
Aug 05 18:02:15.223114 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.222990    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxqgm\" (UniqueName: \"kubernetes.io/projected/b6f15972-a264-42f2-8e0a-b0f9b6638e57-kube-api-access-zxqgm\") pod \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") "
Aug 05 18:02:15.223114 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.223016    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6f15972-a264-42f2-8e0a-b0f9b6638e57-tmp-dir\") pod \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") "
Aug 05 18:02:15.223114 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.223057    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6f15972-a264-42f2-8e0a-b0f9b6638e57-kserve-provision-location\") pod \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") "
Aug 05 18:02:15.223114 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.223094    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6f15972-a264-42f2-8e0a-b0f9b6638e57-model-cache\") pod \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\" (UID: \"b6f15972-a264-42f2-8e0a-b0f9b6638e57\") "
Aug 05 18:02:15.223507 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.223113    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6f15972-a264-42f2-8e0a-b0f9b6638e57-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b6f15972-a264-42f2-8e0a-b0f9b6638e57" (UID: "b6f15972-a264-42f2-8e0a-b0f9b6638e57"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 18:02:15.223507 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.223404    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6f15972-a264-42f2-8e0a-b0f9b6638e57-cabundle-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:02:15.223507 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.223438    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f15972-a264-42f2-8e0a-b0f9b6638e57-model-cache" (OuterVolumeSpecName: "model-cache") pod "b6f15972-a264-42f2-8e0a-b0f9b6638e57" (UID: "b6f15972-a264-42f2-8e0a-b0f9b6638e57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:02:15.223889 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.223861    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f15972-a264-42f2-8e0a-b0f9b6638e57-home" (OuterVolumeSpecName: "home") pod "b6f15972-a264-42f2-8e0a-b0f9b6638e57" (UID: "b6f15972-a264-42f2-8e0a-b0f9b6638e57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:02:15.225115 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.225090    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6f15972-a264-42f2-8e0a-b0f9b6638e57-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b6f15972-a264-42f2-8e0a-b0f9b6638e57" (UID: "b6f15972-a264-42f2-8e0a-b0f9b6638e57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 18:02:15.225238 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.225211    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f15972-a264-42f2-8e0a-b0f9b6638e57-kube-api-access-zxqgm" (OuterVolumeSpecName: "kube-api-access-zxqgm") pod "b6f15972-a264-42f2-8e0a-b0f9b6638e57" (UID: "b6f15972-a264-42f2-8e0a-b0f9b6638e57"). InnerVolumeSpecName "kube-api-access-zxqgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 18:02:15.225388 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.225364    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f15972-a264-42f2-8e0a-b0f9b6638e57-dshm" (OuterVolumeSpecName: "dshm") pod "b6f15972-a264-42f2-8e0a-b0f9b6638e57" (UID: "b6f15972-a264-42f2-8e0a-b0f9b6638e57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:02:15.233154 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.233129    2561 generic.go:356] "Generic (PLEG): container finished" podID="70cdd9f1-c0d2-4487-a160-d5f07ed97258" containerID="25b129e35b6a27b6ad8936ea6ade0bf6da527eec8a6377f116293f2311998103" exitCode=0
Aug 05 18:02:15.233286 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.233168    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg"
Aug 05 18:02:15.233286 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.233210    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg" event={"ID":"70cdd9f1-c0d2-4487-a160-d5f07ed97258","Type":"ContainerDied","Data":"25b129e35b6a27b6ad8936ea6ade0bf6da527eec8a6377f116293f2311998103"}
Aug 05 18:02:15.233286 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.233270    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg" event={"ID":"70cdd9f1-c0d2-4487-a160-d5f07ed97258","Type":"ContainerDied","Data":"b05ab54e1d148f46e21d1be173c099c798da225cb0c42b9d24f4ffe2df8a1b6e"}
Aug 05 18:02:15.233448 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.233302    2561 scope.go:117] "RemoveContainer" containerID="25b129e35b6a27b6ad8936ea6ade0bf6da527eec8a6377f116293f2311998103"
Aug 05 18:02:15.234398 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.234364    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f15972-a264-42f2-8e0a-b0f9b6638e57-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b6f15972-a264-42f2-8e0a-b0f9b6638e57" (UID: "b6f15972-a264-42f2-8e0a-b0f9b6638e57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:02:15.235157 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.235132    2561 generic.go:356] "Generic (PLEG): container finished" podID="b6f15972-a264-42f2-8e0a-b0f9b6638e57" containerID="4f62c89b474be3ec123cabcf2b648722c8a2ec4992be5eeec164f95b0c9701e0" exitCode=0
Aug 05 18:02:15.235290 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.235202    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"
Aug 05 18:02:15.235290 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.235201    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd" event={"ID":"b6f15972-a264-42f2-8e0a-b0f9b6638e57","Type":"ContainerDied","Data":"4f62c89b474be3ec123cabcf2b648722c8a2ec4992be5eeec164f95b0c9701e0"}
Aug 05 18:02:15.235405 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.235316    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd" event={"ID":"b6f15972-a264-42f2-8e0a-b0f9b6638e57","Type":"ContainerDied","Data":"2e45968f067d0c5a7cd5fe25e34c288888ae83623e88677327156c8dcd47cd0a"}
Aug 05 18:02:15.243436 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.243418    2561 scope.go:117] "RemoveContainer" containerID="25b129e35b6a27b6ad8936ea6ade0bf6da527eec8a6377f116293f2311998103"
Aug 05 18:02:15.243695 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:02:15.243672    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25b129e35b6a27b6ad8936ea6ade0bf6da527eec8a6377f116293f2311998103\": container with ID starting with 25b129e35b6a27b6ad8936ea6ade0bf6da527eec8a6377f116293f2311998103 not found: ID does not exist" containerID="25b129e35b6a27b6ad8936ea6ade0bf6da527eec8a6377f116293f2311998103"
Aug 05 18:02:15.243766 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.243706    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25b129e35b6a27b6ad8936ea6ade0bf6da527eec8a6377f116293f2311998103"} err="failed to get container status \"25b129e35b6a27b6ad8936ea6ade0bf6da527eec8a6377f116293f2311998103\": rpc error: code = NotFound desc = could not find container \"25b129e35b6a27b6ad8936ea6ade0bf6da527eec8a6377f116293f2311998103\": container with ID starting with 25b129e35b6a27b6ad8936ea6ade0bf6da527eec8a6377f116293f2311998103 not found: ID does not exist"
Aug 05 18:02:15.243766 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.243743    2561 scope.go:117] "RemoveContainer" containerID="4f62c89b474be3ec123cabcf2b648722c8a2ec4992be5eeec164f95b0c9701e0"
Aug 05 18:02:15.249391 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.249367    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f15972-a264-42f2-8e0a-b0f9b6638e57-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b6f15972-a264-42f2-8e0a-b0f9b6638e57" (UID: "b6f15972-a264-42f2-8e0a-b0f9b6638e57"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:02:15.252049 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.252024    2561 scope.go:117] "RemoveContainer" containerID="e89469f8597c54282c4fd9bc05ce3a6e261d6df1ee43a9135c6b79097be4887d"
Aug 05 18:02:15.255586 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.255563    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg"]
Aug 05 18:02:15.259366 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.259343    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-g4jgg"]
Aug 05 18:02:15.261146 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.261122    2561 scope.go:117] "RemoveContainer" containerID="4f62c89b474be3ec123cabcf2b648722c8a2ec4992be5eeec164f95b0c9701e0"
Aug 05 18:02:15.261415 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:02:15.261393    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f62c89b474be3ec123cabcf2b648722c8a2ec4992be5eeec164f95b0c9701e0\": container with ID starting with 4f62c89b474be3ec123cabcf2b648722c8a2ec4992be5eeec164f95b0c9701e0 not found: ID does not exist" containerID="4f62c89b474be3ec123cabcf2b648722c8a2ec4992be5eeec164f95b0c9701e0"
Aug 05 18:02:15.261504 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.261420    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f62c89b474be3ec123cabcf2b648722c8a2ec4992be5eeec164f95b0c9701e0"} err="failed to get container status \"4f62c89b474be3ec123cabcf2b648722c8a2ec4992be5eeec164f95b0c9701e0\": rpc error: code = NotFound desc = could not find container \"4f62c89b474be3ec123cabcf2b648722c8a2ec4992be5eeec164f95b0c9701e0\": container with ID starting with 4f62c89b474be3ec123cabcf2b648722c8a2ec4992be5eeec164f95b0c9701e0 not found: ID does not exist"
Aug 05 18:02:15.261504 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.261441    2561 scope.go:117] "RemoveContainer" containerID="e89469f8597c54282c4fd9bc05ce3a6e261d6df1ee43a9135c6b79097be4887d"
Aug 05 18:02:15.261640 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:02:15.261624    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e89469f8597c54282c4fd9bc05ce3a6e261d6df1ee43a9135c6b79097be4887d\": container with ID starting with e89469f8597c54282c4fd9bc05ce3a6e261d6df1ee43a9135c6b79097be4887d not found: ID does not exist" containerID="e89469f8597c54282c4fd9bc05ce3a6e261d6df1ee43a9135c6b79097be4887d"
Aug 05 18:02:15.261684 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.261642    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e89469f8597c54282c4fd9bc05ce3a6e261d6df1ee43a9135c6b79097be4887d"} err="failed to get container status \"e89469f8597c54282c4fd9bc05ce3a6e261d6df1ee43a9135c6b79097be4887d\": rpc error: code = NotFound desc = could not find container \"e89469f8597c54282c4fd9bc05ce3a6e261d6df1ee43a9135c6b79097be4887d\": container with ID starting with e89469f8597c54282c4fd9bc05ce3a6e261d6df1ee43a9135c6b79097be4887d not found: ID does not exist"
Aug 05 18:02:15.324512 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.324464    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6f15972-a264-42f2-8e0a-b0f9b6638e57-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:02:15.324512 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.324509    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxqgm\" (UniqueName: \"kubernetes.io/projected/b6f15972-a264-42f2-8e0a-b0f9b6638e57-kube-api-access-zxqgm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:02:15.324512 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.324520    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6f15972-a264-42f2-8e0a-b0f9b6638e57-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:02:15.324762 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.324532    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6f15972-a264-42f2-8e0a-b0f9b6638e57-kserve-provision-location\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:02:15.324762 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.324541    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6f15972-a264-42f2-8e0a-b0f9b6638e57-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:02:15.324762 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.324549    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6f15972-a264-42f2-8e0a-b0f9b6638e57-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:02:15.324762 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.324555    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6f15972-a264-42f2-8e0a-b0f9b6638e57-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:02:15.559124 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.559094    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"]
Aug 05 18:02:15.562299 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:15.562274    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-jhlxd"]
Aug 05 18:02:16.730849 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:16.730815    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70cdd9f1-c0d2-4487-a160-d5f07ed97258" path="/var/lib/kubelet/pods/70cdd9f1-c0d2-4487-a160-d5f07ed97258/volumes"
Aug 05 18:02:16.731287 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:16.731268    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6f15972-a264-42f2-8e0a-b0f9b6638e57" path="/var/lib/kubelet/pods/b6f15972-a264-42f2-8e0a-b0f9b6638e57/volumes"
Aug 05 18:02:25.413506 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.413465    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"]
Aug 05 18:02:25.414087 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.414067    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daca7cad-7a28-47df-af89-c25576d556da" containerName="istio-proxy"
Aug 05 18:02:25.414087 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.414089    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="daca7cad-7a28-47df-af89-c25576d556da" containerName="istio-proxy"
Aug 05 18:02:25.414243 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.414108    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daca7cad-7a28-47df-af89-c25576d556da" containerName="istio-proxy"
Aug 05 18:02:25.414243 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.414116    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="daca7cad-7a28-47df-af89-c25576d556da" containerName="istio-proxy"
Aug 05 18:02:25.414243 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.414127    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6f15972-a264-42f2-8e0a-b0f9b6638e57" containerName="main"
Aug 05 18:02:25.414243 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.414135    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6f15972-a264-42f2-8e0a-b0f9b6638e57" containerName="main"
Aug 05 18:02:25.414243 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.414152    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6f15972-a264-42f2-8e0a-b0f9b6638e57" containerName="storage-initializer"
Aug 05 18:02:25.414243 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.414160    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6f15972-a264-42f2-8e0a-b0f9b6638e57" containerName="storage-initializer"
Aug 05 18:02:25.414243 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.414174    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70cdd9f1-c0d2-4487-a160-d5f07ed97258" containerName="main"
Aug 05 18:02:25.414243 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.414182    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="70cdd9f1-c0d2-4487-a160-d5f07ed97258" containerName="main"
Aug 05 18:02:25.414643 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.414277    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="daca7cad-7a28-47df-af89-c25576d556da" containerName="istio-proxy"
Aug 05 18:02:25.414643 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.414291    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6f15972-a264-42f2-8e0a-b0f9b6638e57" containerName="main"
Aug 05 18:02:25.414643 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.414300    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="70cdd9f1-c0d2-4487-a160-d5f07ed97258" containerName="main"
Aug 05 18:02:25.414643 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.414463    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="daca7cad-7a28-47df-af89-c25576d556da" containerName="istio-proxy"
Aug 05 18:02:25.419472 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.419444    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:02:25.422348 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.422195    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Aug 05 18:02:25.422348 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.422215    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-jbxpc\""
Aug 05 18:02:25.422348 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.422247    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Aug 05 18:02:25.422348 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.422299    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Aug 05 18:02:25.422348 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.422202    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Aug 05 18:02:25.422718 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.422563    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Aug 05 18:02:25.429447 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.429423    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"]
Aug 05 18:02:25.514631 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.514580    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0aee777-d8bd-41e9-a506-37edeffc0712-kserve-provision-location\") pod \"stop-feature-test-kserve-5b9dc4dbc9-mgqcd\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:02:25.514631 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.514643    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0aee777-d8bd-41e9-a506-37edeffc0712-dshm\") pod \"stop-feature-test-kserve-5b9dc4dbc9-mgqcd\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:02:25.514871 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.514682    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0aee777-d8bd-41e9-a506-37edeffc0712-cabundle-cert\") pod \"stop-feature-test-kserve-5b9dc4dbc9-mgqcd\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:02:25.514871 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.514702    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m56s7\" (UniqueName: \"kubernetes.io/projected/e0aee777-d8bd-41e9-a506-37edeffc0712-kube-api-access-m56s7\") pod \"stop-feature-test-kserve-5b9dc4dbc9-mgqcd\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:02:25.514871 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.514729    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0aee777-d8bd-41e9-a506-37edeffc0712-tls-certs\") pod \"stop-feature-test-kserve-5b9dc4dbc9-mgqcd\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:02:25.514871 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.514752    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0aee777-d8bd-41e9-a506-37edeffc0712-home\") pod \"stop-feature-test-kserve-5b9dc4dbc9-mgqcd\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:02:25.514871 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.514798    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0aee777-d8bd-41e9-a506-37edeffc0712-tmp-dir\") pod \"stop-feature-test-kserve-5b9dc4dbc9-mgqcd\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:02:25.515079 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.514866    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0aee777-d8bd-41e9-a506-37edeffc0712-model-cache\") pod \"stop-feature-test-kserve-5b9dc4dbc9-mgqcd\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:02:25.615606 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.615569    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0aee777-d8bd-41e9-a506-37edeffc0712-model-cache\") pod \"stop-feature-test-kserve-5b9dc4dbc9-mgqcd\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:02:25.615606 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.615611    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0aee777-d8bd-41e9-a506-37edeffc0712-kserve-provision-location\") pod \"stop-feature-test-kserve-5b9dc4dbc9-mgqcd\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:02:25.615866 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.615641    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0aee777-d8bd-41e9-a506-37edeffc0712-dshm\") pod \"stop-feature-test-kserve-5b9dc4dbc9-mgqcd\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:02:25.615866 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.615670    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0aee777-d8bd-41e9-a506-37edeffc0712-cabundle-cert\") pod \"stop-feature-test-kserve-5b9dc4dbc9-mgqcd\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:02:25.615866 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.615685    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m56s7\" (UniqueName: \"kubernetes.io/projected/e0aee777-d8bd-41e9-a506-37edeffc0712-kube-api-access-m56s7\") pod \"stop-feature-test-kserve-5b9dc4dbc9-mgqcd\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:02:25.615866 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.615702    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0aee777-d8bd-41e9-a506-37edeffc0712-tls-certs\") pod \"stop-feature-test-kserve-5b9dc4dbc9-mgqcd\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:02:25.615866 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.615722    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0aee777-d8bd-41e9-a506-37edeffc0712-home\") pod \"stop-feature-test-kserve-5b9dc4dbc9-mgqcd\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:02:25.615866 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.615763    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0aee777-d8bd-41e9-a506-37edeffc0712-tmp-dir\") pod \"stop-feature-test-kserve-5b9dc4dbc9-mgqcd\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:02:25.616220 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.616082    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0aee777-d8bd-41e9-a506-37edeffc0712-kserve-provision-location\") pod \"stop-feature-test-kserve-5b9dc4dbc9-mgqcd\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:02:25.616576 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.616548    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0aee777-d8bd-41e9-a506-37edeffc0712-cabundle-cert\") pod \"stop-feature-test-kserve-5b9dc4dbc9-mgqcd\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:02:25.616696 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.616625    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0aee777-d8bd-41e9-a506-37edeffc0712-model-cache\") pod \"stop-feature-test-kserve-5b9dc4dbc9-mgqcd\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:02:25.616696 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.616648    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0aee777-d8bd-41e9-a506-37edeffc0712-home\") pod \"stop-feature-test-kserve-5b9dc4dbc9-mgqcd\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:02:25.616696 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.616690    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0aee777-d8bd-41e9-a506-37edeffc0712-tmp-dir\") pod \"stop-feature-test-kserve-5b9dc4dbc9-mgqcd\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:02:25.618617 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.618589    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0aee777-d8bd-41e9-a506-37edeffc0712-dshm\") pod \"stop-feature-test-kserve-5b9dc4dbc9-mgqcd\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:02:25.618799 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.618781    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0aee777-d8bd-41e9-a506-37edeffc0712-tls-certs\") pod \"stop-feature-test-kserve-5b9dc4dbc9-mgqcd\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:02:25.626104 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.626080    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m56s7\" (UniqueName: \"kubernetes.io/projected/e0aee777-d8bd-41e9-a506-37edeffc0712-kube-api-access-m56s7\") pod \"stop-feature-test-kserve-5b9dc4dbc9-mgqcd\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:02:25.732066 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.731975    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:02:25.875675 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:25.875649    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"]
Aug 05 18:02:25.877075 ip-10-0-135-151 kubenswrapper[2561]: W0805 18:02:25.877049    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0aee777_d8bd_41e9_a506_37edeffc0712.slice/crio-b0e204b89bb22e825a4ba4ed4c01f923cb4a31635ea08e1085200bebbb873429 WatchSource:0}: Error finding container b0e204b89bb22e825a4ba4ed4c01f923cb4a31635ea08e1085200bebbb873429: Status 404 returned error can't find the container with id b0e204b89bb22e825a4ba4ed4c01f923cb4a31635ea08e1085200bebbb873429
Aug 05 18:02:26.297850 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:26.297811    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd" event={"ID":"e0aee777-d8bd-41e9-a506-37edeffc0712","Type":"ContainerStarted","Data":"454d5f6cc62060d477aae80bae3418d544ff9b8f759ba31944b52cbefc77d817"}
Aug 05 18:02:26.298107 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:26.297877    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd" event={"ID":"e0aee777-d8bd-41e9-a506-37edeffc0712","Type":"ContainerStarted","Data":"b0e204b89bb22e825a4ba4ed4c01f923cb4a31635ea08e1085200bebbb873429"}
Aug 05 18:02:29.315863 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:29.315831    2561 generic.go:356] "Generic (PLEG): container finished" podID="e0aee777-d8bd-41e9-a506-37edeffc0712" containerID="454d5f6cc62060d477aae80bae3418d544ff9b8f759ba31944b52cbefc77d817" exitCode=0
Aug 05 18:02:29.316282 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:29.315910    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd" event={"ID":"e0aee777-d8bd-41e9-a506-37edeffc0712","Type":"ContainerDied","Data":"454d5f6cc62060d477aae80bae3418d544ff9b8f759ba31944b52cbefc77d817"}
Aug 05 18:02:30.323715 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:30.323683    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd" event={"ID":"e0aee777-d8bd-41e9-a506-37edeffc0712","Type":"ContainerStarted","Data":"865cc572d1735ca145e1b3552a7f24a740d9011dd0bae5948c158752d9a800df"}
Aug 05 18:02:30.346258 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:30.346208    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd" podStartSLOduration=2.9648451380000003 podStartE2EDuration="5.34619331s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.381348172s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:02:30.343214221 +0000 UTC m=+3006.087848545" watchObservedRunningTime="2026-08-05 18:02:30.34619331 +0000 UTC m=+3006.090827639"
Aug 05 18:02:30.736857 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:30.736771    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:02:35.732619 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:35.732577    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:02:35.734332 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:35.734298    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd" podUID="e0aee777-d8bd-41e9-a506-37edeffc0712" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": dial tcp 10.133.0.100:8000: connect: connection refused"
Aug 05 18:02:45.732681 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:45.732638    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd" podUID="e0aee777-d8bd-41e9-a506-37edeffc0712" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": dial tcp 10.133.0.100:8000: connect: connection refused"
Aug 05 18:02:55.733121 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:02:55.733070    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd" podUID="e0aee777-d8bd-41e9-a506-37edeffc0712" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": dial tcp 10.133.0.100:8000: connect: connection refused"
Aug 05 18:03:05.732604 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:05.732561    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd" podUID="e0aee777-d8bd-41e9-a506-37edeffc0712" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": dial tcp 10.133.0.100:8000: connect: connection refused"
Aug 05 18:03:15.733008 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:15.732944    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd" podUID="e0aee777-d8bd-41e9-a506-37edeffc0712" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": dial tcp 10.133.0.100:8000: connect: connection refused"
Aug 05 18:03:25.732914 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:25.732790    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd" podUID="e0aee777-d8bd-41e9-a506-37edeffc0712" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": dial tcp 10.133.0.100:8000: connect: connection refused"
Aug 05 18:03:35.733142 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:35.733098    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd" podUID="e0aee777-d8bd-41e9-a506-37edeffc0712" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": dial tcp 10.133.0.100:8000: connect: connection refused"
Aug 05 18:03:45.733060 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:45.733012    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd" podUID="e0aee777-d8bd-41e9-a506-37edeffc0712" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": dial tcp 10.133.0.100:8000: connect: connection refused"
Aug 05 18:03:53.210389 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.210356    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"]
Aug 05 18:03:53.213801 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.213781    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:03:53.216255 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.216228    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Aug 05 18:03:53.216397 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.216364    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Aug 05 18:03:53.216530 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.216501    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Aug 05 18:03:53.216659 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.216501    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Aug 05 18:03:53.216659 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.216631    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-ftz95\""
Aug 05 18:03:53.216870 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.216856    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Aug 05 18:03:53.226564 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.226535    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"]
Aug 05 18:03:53.321013 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.320947    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-65b75744d-jktxz\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:03:53.321013 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.321018    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc8vc\" (UniqueName: \"kubernetes.io/projected/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-kube-api-access-sc8vc\") pod \"custom-route-timeout-test-kserve-65b75744d-jktxz\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:03:53.321264 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.321113    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-home\") pod \"custom-route-timeout-test-kserve-65b75744d-jktxz\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:03:53.321264 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.321160    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-cabundle-cert\") pod \"custom-route-timeout-test-kserve-65b75744d-jktxz\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:03:53.321264 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.321229    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-model-cache\") pod \"custom-route-timeout-test-kserve-65b75744d-jktxz\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:03:53.321413 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.321281    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-tmp-dir\") pod \"custom-route-timeout-test-kserve-65b75744d-jktxz\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:03:53.321413 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.321310    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-tls-certs\") pod \"custom-route-timeout-test-kserve-65b75744d-jktxz\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:03:53.321510 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.321415    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-dshm\") pod \"custom-route-timeout-test-kserve-65b75744d-jktxz\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:03:53.422332 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.422286    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-tmp-dir\") pod \"custom-route-timeout-test-kserve-65b75744d-jktxz\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:03:53.422545 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.422359    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-tls-certs\") pod \"custom-route-timeout-test-kserve-65b75744d-jktxz\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:03:53.422545 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.422442    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-dshm\") pod \"custom-route-timeout-test-kserve-65b75744d-jktxz\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:03:53.422545 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.422475    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-65b75744d-jktxz\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:03:53.422731 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.422613    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc8vc\" (UniqueName: \"kubernetes.io/projected/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-kube-api-access-sc8vc\") pod \"custom-route-timeout-test-kserve-65b75744d-jktxz\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:03:53.422731 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.422695    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-home\") pod \"custom-route-timeout-test-kserve-65b75744d-jktxz\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:03:53.422841 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.422751    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-cabundle-cert\") pod \"custom-route-timeout-test-kserve-65b75744d-jktxz\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:03:53.422841 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.422784    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-model-cache\") pod \"custom-route-timeout-test-kserve-65b75744d-jktxz\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:03:53.422841 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.422791    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-tmp-dir\") pod \"custom-route-timeout-test-kserve-65b75744d-jktxz\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:03:53.423098 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.423036    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-65b75744d-jktxz\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:03:53.423164 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.423142    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-home\") pod \"custom-route-timeout-test-kserve-65b75744d-jktxz\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:03:53.423515 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.423468    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-model-cache\") pod \"custom-route-timeout-test-kserve-65b75744d-jktxz\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:03:53.423657 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.423597    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-cabundle-cert\") pod \"custom-route-timeout-test-kserve-65b75744d-jktxz\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:03:53.425184 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.425153    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-dshm\") pod \"custom-route-timeout-test-kserve-65b75744d-jktxz\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:03:53.425310 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.425284    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-tls-certs\") pod \"custom-route-timeout-test-kserve-65b75744d-jktxz\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:03:53.430407 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.430380    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc8vc\" (UniqueName: \"kubernetes.io/projected/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-kube-api-access-sc8vc\") pod \"custom-route-timeout-test-kserve-65b75744d-jktxz\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:03:53.526182 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.526083    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:03:53.687660 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.684683    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"]
Aug 05 18:03:53.800203 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.800157    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz" event={"ID":"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b","Type":"ContainerStarted","Data":"b75ed9a236fe08ce2ec953c6a6998f8c63b162d0580df010dae7a1c76cb75e28"}
Aug 05 18:03:53.800422 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:53.800233    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz" event={"ID":"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b","Type":"ContainerStarted","Data":"515ab147c90c9065627bbbc7ca4b74fa3e8695c845b9d3f6dfe9d8a5052ce5b1"}
Aug 05 18:03:55.732989 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:55.732931    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd" podUID="e0aee777-d8bd-41e9-a506-37edeffc0712" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": dial tcp 10.133.0.100:8000: connect: connection refused"
Aug 05 18:03:56.820160 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:56.820125    2561 generic.go:356] "Generic (PLEG): container finished" podID="a4c7d8a3-38b3-46cf-8ef6-9af552bb172b" containerID="b75ed9a236fe08ce2ec953c6a6998f8c63b162d0580df010dae7a1c76cb75e28" exitCode=0
Aug 05 18:03:56.820654 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:56.820214    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz" event={"ID":"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b","Type":"ContainerDied","Data":"b75ed9a236fe08ce2ec953c6a6998f8c63b162d0580df010dae7a1c76cb75e28"}
Aug 05 18:03:57.828782 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:57.828741    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz" event={"ID":"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b","Type":"ContainerStarted","Data":"8c9e294135f1ae4211d0501a778c270d88c4cd9dc0fa308f65c41e9bdb99aae0"}
Aug 05 18:03:57.852108 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:57.852042    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz" podStartSLOduration=2.377984912 podStartE2EDuration="4.852020784s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.474035872s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:03:57.850614891 +0000 UTC m=+3093.595249249" watchObservedRunningTime="2026-08-05 18:03:57.852020784 +0000 UTC m=+3093.596655116"
Aug 05 18:03:58.526689 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:03:58.526648    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:04:03.526603 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:03.526555    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:04:03.528350 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:03.528314    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz" podUID="a4c7d8a3-38b3-46cf-8ef6-9af552bb172b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused"
Aug 05 18:04:05.732482 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:05.732433    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd" podUID="e0aee777-d8bd-41e9-a506-37edeffc0712" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": dial tcp 10.133.0.100:8000: connect: connection refused"
Aug 05 18:04:13.526867 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:13.526814    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz" podUID="a4c7d8a3-38b3-46cf-8ef6-9af552bb172b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused"
Aug 05 18:04:15.732825 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:15.732774    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd" podUID="e0aee777-d8bd-41e9-a506-37edeffc0712" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": dial tcp 10.133.0.100:8000: connect: connection refused"
Aug 05 18:04:23.527371 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:23.527325    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz" podUID="a4c7d8a3-38b3-46cf-8ef6-9af552bb172b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused"
Aug 05 18:04:25.743422 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:25.743380    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:04:25.751729 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:25.751690    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:04:27.464898 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:27.464859    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"]
Aug 05 18:04:33.527425 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:33.527385    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz" podUID="a4c7d8a3-38b3-46cf-8ef6-9af552bb172b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused"
Aug 05 18:04:42.497534 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:42.497466    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd" podUID="e0aee777-d8bd-41e9-a506-37edeffc0712" containerName="main" containerID="cri-o://865cc572d1735ca145e1b3552a7f24a740d9011dd0bae5948c158752d9a800df" gracePeriod=45
Aug 05 18:04:42.744252 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:42.744210    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd" podUID="e0aee777-d8bd-41e9-a506-37edeffc0712" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/health\": dial tcp 10.133.0.100:8000: connect: connection refused"
Aug 05 18:04:43.264848 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:43.264817    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:04:43.288121 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:43.288091    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0aee777-d8bd-41e9-a506-37edeffc0712-tls-certs\") pod \"e0aee777-d8bd-41e9-a506-37edeffc0712\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") "
Aug 05 18:04:43.288357 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:43.288132    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0aee777-d8bd-41e9-a506-37edeffc0712-home\") pod \"e0aee777-d8bd-41e9-a506-37edeffc0712\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") "
Aug 05 18:04:43.288357 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:43.288196    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0aee777-d8bd-41e9-a506-37edeffc0712-dshm\") pod \"e0aee777-d8bd-41e9-a506-37edeffc0712\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") "
Aug 05 18:04:43.288357 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:43.288220    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0aee777-d8bd-41e9-a506-37edeffc0712-tmp-dir\") pod \"e0aee777-d8bd-41e9-a506-37edeffc0712\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") "
Aug 05 18:04:43.288357 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:43.288278    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0aee777-d8bd-41e9-a506-37edeffc0712-cabundle-cert\") pod \"e0aee777-d8bd-41e9-a506-37edeffc0712\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") "
Aug 05 18:04:43.288357 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:43.288310    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0aee777-d8bd-41e9-a506-37edeffc0712-model-cache\") pod \"e0aee777-d8bd-41e9-a506-37edeffc0712\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") "
Aug 05 18:04:43.288357 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:43.288349    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0aee777-d8bd-41e9-a506-37edeffc0712-kserve-provision-location\") pod \"e0aee777-d8bd-41e9-a506-37edeffc0712\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") "
Aug 05 18:04:43.288692 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:43.288382    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m56s7\" (UniqueName: \"kubernetes.io/projected/e0aee777-d8bd-41e9-a506-37edeffc0712-kube-api-access-m56s7\") pod \"e0aee777-d8bd-41e9-a506-37edeffc0712\" (UID: \"e0aee777-d8bd-41e9-a506-37edeffc0712\") "
Aug 05 18:04:43.291366 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:43.288812    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0aee777-d8bd-41e9-a506-37edeffc0712-model-cache" (OuterVolumeSpecName: "model-cache") pod "e0aee777-d8bd-41e9-a506-37edeffc0712" (UID: "e0aee777-d8bd-41e9-a506-37edeffc0712"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:04:43.291366 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:43.288899    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0aee777-d8bd-41e9-a506-37edeffc0712-home" (OuterVolumeSpecName: "home") pod "e0aee777-d8bd-41e9-a506-37edeffc0712" (UID: "e0aee777-d8bd-41e9-a506-37edeffc0712"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:04:43.291366 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:43.289151    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0aee777-d8bd-41e9-a506-37edeffc0712-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e0aee777-d8bd-41e9-a506-37edeffc0712" (UID: "e0aee777-d8bd-41e9-a506-37edeffc0712"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 18:04:43.291366 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:43.290634    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0aee777-d8bd-41e9-a506-37edeffc0712-dshm" (OuterVolumeSpecName: "dshm") pod "e0aee777-d8bd-41e9-a506-37edeffc0712" (UID: "e0aee777-d8bd-41e9-a506-37edeffc0712"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:04:43.291675 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:43.291422    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0aee777-d8bd-41e9-a506-37edeffc0712-kube-api-access-m56s7" (OuterVolumeSpecName: "kube-api-access-m56s7") pod "e0aee777-d8bd-41e9-a506-37edeffc0712" (UID: "e0aee777-d8bd-41e9-a506-37edeffc0712"). InnerVolumeSpecName "kube-api-access-m56s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 18:04:43.291938 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:43.291912    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0aee777-d8bd-41e9-a506-37edeffc0712-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e0aee777-d8bd-41e9-a506-37edeffc0712" (UID: "e0aee777-d8bd-41e9-a506-37edeffc0712"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 18:04:43.308866 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:43.308822    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0aee777-d8bd-41e9-a506-37edeffc0712-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e0aee777-d8bd-41e9-a506-37edeffc0712" (UID: "e0aee777-d8bd-41e9-a506-37edeffc0712"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:04:43.320405 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:43.320372    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0aee777-d8bd-41e9-a506-37edeffc0712-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e0aee777-d8bd-41e9-a506-37edeffc0712" (UID: "e0aee777-d8bd-41e9-a506-37edeffc0712"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:04:43.389453 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:43.389419    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0aee777-d8bd-41e9-a506-37edeffc0712-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:04:43.389453 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:43.389451    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0aee777-d8bd-41e9-a506-37edeffc0712-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:04:43.389453 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:43.389459    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0aee777-d8bd-41e9-a506-37edeffc0712-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:04:43.389684 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:43.389487    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0aee777-d8bd-41e9-a506-37edeffc0712-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:04:43.389684 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:43.389500    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0aee777-d8bd-41e9-a506-37edeffc0712-cabundle-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:04:43.389684 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:43.389514    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0aee777-d8bd-41e9-a506-37edeffc0712-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:04:43.389684 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:43.389524    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0aee777-d8bd-41e9-a506-37edeffc0712-kserve-provision-location\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:04:43.389684 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:43.389533    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m56s7\" (UniqueName: \"kubernetes.io/projected/e0aee777-d8bd-41e9-a506-37edeffc0712-kube-api-access-m56s7\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:04:43.526758 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:43.526712    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz" podUID="a4c7d8a3-38b3-46cf-8ef6-9af552bb172b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused"
Aug 05 18:04:44.111782 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:44.111744    2561 generic.go:356] "Generic (PLEG): container finished" podID="e0aee777-d8bd-41e9-a506-37edeffc0712" containerID="865cc572d1735ca145e1b3552a7f24a740d9011dd0bae5948c158752d9a800df" exitCode=0
Aug 05 18:04:44.112052 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:44.111803    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"
Aug 05 18:04:44.112052 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:44.111842    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd" event={"ID":"e0aee777-d8bd-41e9-a506-37edeffc0712","Type":"ContainerDied","Data":"865cc572d1735ca145e1b3552a7f24a740d9011dd0bae5948c158752d9a800df"}
Aug 05 18:04:44.112052 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:44.111916    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd" event={"ID":"e0aee777-d8bd-41e9-a506-37edeffc0712","Type":"ContainerDied","Data":"b0e204b89bb22e825a4ba4ed4c01f923cb4a31635ea08e1085200bebbb873429"}
Aug 05 18:04:44.112052 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:44.111972    2561 scope.go:117] "RemoveContainer" containerID="865cc572d1735ca145e1b3552a7f24a740d9011dd0bae5948c158752d9a800df"
Aug 05 18:04:44.122483 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:44.122467    2561 scope.go:117] "RemoveContainer" containerID="454d5f6cc62060d477aae80bae3418d544ff9b8f759ba31944b52cbefc77d817"
Aug 05 18:04:44.130585 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:44.130565    2561 scope.go:117] "RemoveContainer" containerID="865cc572d1735ca145e1b3552a7f24a740d9011dd0bae5948c158752d9a800df"
Aug 05 18:04:44.130882 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:04:44.130861    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"865cc572d1735ca145e1b3552a7f24a740d9011dd0bae5948c158752d9a800df\": container with ID starting with 865cc572d1735ca145e1b3552a7f24a740d9011dd0bae5948c158752d9a800df not found: ID does not exist" containerID="865cc572d1735ca145e1b3552a7f24a740d9011dd0bae5948c158752d9a800df"
Aug 05 18:04:44.130936 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:44.130893    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"865cc572d1735ca145e1b3552a7f24a740d9011dd0bae5948c158752d9a800df"} err="failed to get container status \"865cc572d1735ca145e1b3552a7f24a740d9011dd0bae5948c158752d9a800df\": rpc error: code = NotFound desc = could not find container \"865cc572d1735ca145e1b3552a7f24a740d9011dd0bae5948c158752d9a800df\": container with ID starting with 865cc572d1735ca145e1b3552a7f24a740d9011dd0bae5948c158752d9a800df not found: ID does not exist"
Aug 05 18:04:44.130936 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:44.130911    2561 scope.go:117] "RemoveContainer" containerID="454d5f6cc62060d477aae80bae3418d544ff9b8f759ba31944b52cbefc77d817"
Aug 05 18:04:44.131210 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:04:44.131190    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"454d5f6cc62060d477aae80bae3418d544ff9b8f759ba31944b52cbefc77d817\": container with ID starting with 454d5f6cc62060d477aae80bae3418d544ff9b8f759ba31944b52cbefc77d817 not found: ID does not exist" containerID="454d5f6cc62060d477aae80bae3418d544ff9b8f759ba31944b52cbefc77d817"
Aug 05 18:04:44.131256 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:44.131218    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454d5f6cc62060d477aae80bae3418d544ff9b8f759ba31944b52cbefc77d817"} err="failed to get container status \"454d5f6cc62060d477aae80bae3418d544ff9b8f759ba31944b52cbefc77d817\": rpc error: code = NotFound desc = could not find container \"454d5f6cc62060d477aae80bae3418d544ff9b8f759ba31944b52cbefc77d817\": container with ID starting with 454d5f6cc62060d477aae80bae3418d544ff9b8f759ba31944b52cbefc77d817 not found: ID does not exist"
Aug 05 18:04:44.138190 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:44.138165    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"]
Aug 05 18:04:44.141010 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:44.140986    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5b9dc4dbc9-mgqcd"]
Aug 05 18:04:44.731065 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:44.731025    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0aee777-d8bd-41e9-a506-37edeffc0712" path="/var/lib/kubelet/pods/e0aee777-d8bd-41e9-a506-37edeffc0712/volumes"
Aug 05 18:04:53.527526 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:53.527424    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz" podUID="a4c7d8a3-38b3-46cf-8ef6-9af552bb172b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused"
Aug 05 18:04:54.815910 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.815866    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"]
Aug 05 18:04:54.816467 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.816432    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0aee777-d8bd-41e9-a506-37edeffc0712" containerName="storage-initializer"
Aug 05 18:04:54.816467 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.816459    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0aee777-d8bd-41e9-a506-37edeffc0712" containerName="storage-initializer"
Aug 05 18:04:54.816467 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.816470    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0aee777-d8bd-41e9-a506-37edeffc0712" containerName="main"
Aug 05 18:04:54.816707 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.816476    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0aee777-d8bd-41e9-a506-37edeffc0712" containerName="main"
Aug 05 18:04:54.816707 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.816557    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0aee777-d8bd-41e9-a506-37edeffc0712" containerName="main"
Aug 05 18:04:54.821423 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.821400    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:04:54.824098 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.824074    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Aug 05 18:04:54.824098 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.824090    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Aug 05 18:04:54.824306 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.824111    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Aug 05 18:04:54.824306 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.824155    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Aug 05 18:04:54.824306 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.824074    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-dmdqj\""
Aug 05 18:04:54.824306 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.824208    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Aug 05 18:04:54.830526 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.830502    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"]
Aug 05 18:04:54.896066 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.896025    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99f51fdd-d13c-48b8-8d41-977b7dd5356a-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-c56d9cccf-plsrc\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:04:54.896066 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.896070    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99f51fdd-d13c-48b8-8d41-977b7dd5356a-tmp-dir\") pod \"lora-single-adapter-test-kserve-c56d9cccf-plsrc\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:04:54.896281 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.896087    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99f51fdd-d13c-48b8-8d41-977b7dd5356a-tls-certs\") pod \"lora-single-adapter-test-kserve-c56d9cccf-plsrc\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:04:54.896281 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.896164    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7grw\" (UniqueName: \"kubernetes.io/projected/99f51fdd-d13c-48b8-8d41-977b7dd5356a-kube-api-access-h7grw\") pod \"lora-single-adapter-test-kserve-c56d9cccf-plsrc\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:04:54.896281 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.896250    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99f51fdd-d13c-48b8-8d41-977b7dd5356a-home\") pod \"lora-single-adapter-test-kserve-c56d9cccf-plsrc\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:04:54.896395 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.896280    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99f51fdd-d13c-48b8-8d41-977b7dd5356a-dshm\") pod \"lora-single-adapter-test-kserve-c56d9cccf-plsrc\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:04:54.896395 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.896311    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99f51fdd-d13c-48b8-8d41-977b7dd5356a-model-cache\") pod \"lora-single-adapter-test-kserve-c56d9cccf-plsrc\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:04:54.896395 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.896385    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99f51fdd-d13c-48b8-8d41-977b7dd5356a-cabundle-cert\") pod \"lora-single-adapter-test-kserve-c56d9cccf-plsrc\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:04:54.997060 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.997023    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99f51fdd-d13c-48b8-8d41-977b7dd5356a-cabundle-cert\") pod \"lora-single-adapter-test-kserve-c56d9cccf-plsrc\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:04:54.997238 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.997081    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99f51fdd-d13c-48b8-8d41-977b7dd5356a-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-c56d9cccf-plsrc\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:04:54.997238 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.997101    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99f51fdd-d13c-48b8-8d41-977b7dd5356a-tmp-dir\") pod \"lora-single-adapter-test-kserve-c56d9cccf-plsrc\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:04:54.997343 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.997229    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99f51fdd-d13c-48b8-8d41-977b7dd5356a-tls-certs\") pod \"lora-single-adapter-test-kserve-c56d9cccf-plsrc\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:04:54.997343 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.997288    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7grw\" (UniqueName: \"kubernetes.io/projected/99f51fdd-d13c-48b8-8d41-977b7dd5356a-kube-api-access-h7grw\") pod \"lora-single-adapter-test-kserve-c56d9cccf-plsrc\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:04:54.997455 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.997374    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99f51fdd-d13c-48b8-8d41-977b7dd5356a-home\") pod \"lora-single-adapter-test-kserve-c56d9cccf-plsrc\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:04:54.997455 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.997400    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99f51fdd-d13c-48b8-8d41-977b7dd5356a-dshm\") pod \"lora-single-adapter-test-kserve-c56d9cccf-plsrc\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:04:54.997562 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.997461    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99f51fdd-d13c-48b8-8d41-977b7dd5356a-tmp-dir\") pod \"lora-single-adapter-test-kserve-c56d9cccf-plsrc\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:04:54.997621 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.997575    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99f51fdd-d13c-48b8-8d41-977b7dd5356a-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-c56d9cccf-plsrc\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:04:54.997742 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.997719    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99f51fdd-d13c-48b8-8d41-977b7dd5356a-home\") pod \"lora-single-adapter-test-kserve-c56d9cccf-plsrc\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:04:54.997826 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.997786    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99f51fdd-d13c-48b8-8d41-977b7dd5356a-model-cache\") pod \"lora-single-adapter-test-kserve-c56d9cccf-plsrc\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:04:54.997887 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.997866    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99f51fdd-d13c-48b8-8d41-977b7dd5356a-cabundle-cert\") pod \"lora-single-adapter-test-kserve-c56d9cccf-plsrc\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:04:54.998204 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.998179    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99f51fdd-d13c-48b8-8d41-977b7dd5356a-model-cache\") pod \"lora-single-adapter-test-kserve-c56d9cccf-plsrc\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:04:54.999730 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:54.999710    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99f51fdd-d13c-48b8-8d41-977b7dd5356a-dshm\") pod \"lora-single-adapter-test-kserve-c56d9cccf-plsrc\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:04:55.000051 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:55.000029    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99f51fdd-d13c-48b8-8d41-977b7dd5356a-tls-certs\") pod \"lora-single-adapter-test-kserve-c56d9cccf-plsrc\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:04:55.004884 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:55.004859    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7grw\" (UniqueName: \"kubernetes.io/projected/99f51fdd-d13c-48b8-8d41-977b7dd5356a-kube-api-access-h7grw\") pod \"lora-single-adapter-test-kserve-c56d9cccf-plsrc\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:04:55.133116 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:55.133018    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:04:55.300672 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:55.300639    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"]
Aug 05 18:04:55.302887 ip-10-0-135-151 kubenswrapper[2561]: W0805 18:04:55.302861    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f51fdd_d13c_48b8_8d41_977b7dd5356a.slice/crio-6910b895ee9afb7cceaf234c92e3a6494d2022af4c526753b486cb16634139eb WatchSource:0}: Error finding container 6910b895ee9afb7cceaf234c92e3a6494d2022af4c526753b486cb16634139eb: Status 404 returned error can't find the container with id 6910b895ee9afb7cceaf234c92e3a6494d2022af4c526753b486cb16634139eb
Aug 05 18:04:56.180670 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:56.180632    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc" event={"ID":"99f51fdd-d13c-48b8-8d41-977b7dd5356a","Type":"ContainerStarted","Data":"ef7e5abdba1b4ffda7d929bac2cee07dfa98c02c13dfa1e9997f382c00f6b9d2"}
Aug 05 18:04:56.181192 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:56.180700    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc" event={"ID":"99f51fdd-d13c-48b8-8d41-977b7dd5356a","Type":"ContainerStarted","Data":"6910b895ee9afb7cceaf234c92e3a6494d2022af4c526753b486cb16634139eb"}
Aug 05 18:04:59.201546 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:59.201508    2561 generic.go:356] "Generic (PLEG): container finished" podID="99f51fdd-d13c-48b8-8d41-977b7dd5356a" containerID="ef7e5abdba1b4ffda7d929bac2cee07dfa98c02c13dfa1e9997f382c00f6b9d2" exitCode=0
Aug 05 18:04:59.202096 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:04:59.201593    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc" event={"ID":"99f51fdd-d13c-48b8-8d41-977b7dd5356a","Type":"ContainerDied","Data":"ef7e5abdba1b4ffda7d929bac2cee07dfa98c02c13dfa1e9997f382c00f6b9d2"}
Aug 05 18:05:00.210791 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:05:00.210749    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc" event={"ID":"99f51fdd-d13c-48b8-8d41-977b7dd5356a","Type":"ContainerStarted","Data":"ec3206e2c3eb1cfb1c74008652896510355e999d2c8bce057a8aecab71da986c"}
Aug 05 18:05:00.233654 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:05:00.233590    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc" podStartSLOduration=3.127442023 podStartE2EDuration="6.23356772s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.106125697s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:05:00.232045605 +0000 UTC m=+3155.976679934" watchObservedRunningTime="2026-08-05 18:05:00.23356772 +0000 UTC m=+3155.978202051"
Aug 05 18:05:01.134014 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:05:01.133973    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:05:03.526782 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:05:03.526722    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz" podUID="a4c7d8a3-38b3-46cf-8ef6-9af552bb172b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused"
Aug 05 18:05:05.134212 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:05:05.134174    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:05:05.135995 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:05:05.135938    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc" podUID="99f51fdd-d13c-48b8-8d41-977b7dd5356a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/health\": dial tcp 10.133.0.102:8000: connect: connection refused"
Aug 05 18:05:13.527141 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:05:13.527088    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz" podUID="a4c7d8a3-38b3-46cf-8ef6-9af552bb172b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused"
Aug 05 18:05:15.134198 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:05:15.134136    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc" podUID="99f51fdd-d13c-48b8-8d41-977b7dd5356a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/health\": dial tcp 10.133.0.102:8000: connect: connection refused"
Aug 05 18:05:23.527024 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:05:23.526976    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz" podUID="a4c7d8a3-38b3-46cf-8ef6-9af552bb172b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused"
Aug 05 18:05:25.133783 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:05:25.133736    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc" podUID="99f51fdd-d13c-48b8-8d41-977b7dd5356a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/health\": dial tcp 10.133.0.102:8000: connect: connection refused"
Aug 05 18:05:33.526776 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:05:33.526734    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz" podUID="a4c7d8a3-38b3-46cf-8ef6-9af552bb172b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused"
Aug 05 18:05:35.134286 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:05:35.134239    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc" podUID="99f51fdd-d13c-48b8-8d41-977b7dd5356a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/health\": dial tcp 10.133.0.102:8000: connect: connection refused"
Aug 05 18:05:43.526563 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:05:43.526511    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz" podUID="a4c7d8a3-38b3-46cf-8ef6-9af552bb172b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/health\": dial tcp 10.133.0.101:8000: connect: connection refused"
Aug 05 18:05:45.134427 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:05:45.134376    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc" podUID="99f51fdd-d13c-48b8-8d41-977b7dd5356a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/health\": dial tcp 10.133.0.102:8000: connect: connection refused"
Aug 05 18:05:53.537586 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:05:53.537550    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:05:53.545841 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:05:53.545808    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:05:55.134580 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:05:55.134527    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc" podUID="99f51fdd-d13c-48b8-8d41-977b7dd5356a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/health\": dial tcp 10.133.0.102:8000: connect: connection refused"
Aug 05 18:05:58.455229 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:05:58.455192    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"]
Aug 05 18:06:05.133575 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:05.133528    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc" podUID="99f51fdd-d13c-48b8-8d41-977b7dd5356a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/health\": dial tcp 10.133.0.102:8000: connect: connection refused"
Aug 05 18:06:13.483043 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:13.482923    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz" podUID="a4c7d8a3-38b3-46cf-8ef6-9af552bb172b" containerName="main" containerID="cri-o://8c9e294135f1ae4211d0501a778c270d88c4cd9dc0fa308f65c41e9bdb99aae0" gracePeriod=45
Aug 05 18:06:14.453603 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.453569    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:06:14.550746 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.550712    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-model-cache\") pod \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") "
Aug 05 18:06:14.551290 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.550785    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-home\") pod \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") "
Aug 05 18:06:14.551290 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.550811    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-tls-certs\") pod \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") "
Aug 05 18:06:14.551290 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.550859    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-tmp-dir\") pod \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") "
Aug 05 18:06:14.551290 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.550927    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-cabundle-cert\") pod \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") "
Aug 05 18:06:14.551290 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.550978    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-kserve-provision-location\") pod \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") "
Aug 05 18:06:14.551290 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.551004    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-dshm\") pod \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") "
Aug 05 18:06:14.551290 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.551039    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sc8vc\" (UniqueName: \"kubernetes.io/projected/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-kube-api-access-sc8vc\") pod \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\" (UID: \"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b\") "
Aug 05 18:06:14.551290 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.551069    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-model-cache" (OuterVolumeSpecName: "model-cache") pod "a4c7d8a3-38b3-46cf-8ef6-9af552bb172b" (UID: "a4c7d8a3-38b3-46cf-8ef6-9af552bb172b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:06:14.551290 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.551287    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:06:14.551907 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.551357    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a4c7d8a3-38b3-46cf-8ef6-9af552bb172b" (UID: "a4c7d8a3-38b3-46cf-8ef6-9af552bb172b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 18:06:14.551907 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.551625    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-home" (OuterVolumeSpecName: "home") pod "a4c7d8a3-38b3-46cf-8ef6-9af552bb172b" (UID: "a4c7d8a3-38b3-46cf-8ef6-9af552bb172b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:06:14.553672 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.553623    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-dshm" (OuterVolumeSpecName: "dshm") pod "a4c7d8a3-38b3-46cf-8ef6-9af552bb172b" (UID: "a4c7d8a3-38b3-46cf-8ef6-9af552bb172b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:06:14.553789 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.553670    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a4c7d8a3-38b3-46cf-8ef6-9af552bb172b" (UID: "a4c7d8a3-38b3-46cf-8ef6-9af552bb172b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 18:06:14.553789 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.553736    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-kube-api-access-sc8vc" (OuterVolumeSpecName: "kube-api-access-sc8vc") pod "a4c7d8a3-38b3-46cf-8ef6-9af552bb172b" (UID: "a4c7d8a3-38b3-46cf-8ef6-9af552bb172b"). InnerVolumeSpecName "kube-api-access-sc8vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 18:06:14.564099 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.564063    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a4c7d8a3-38b3-46cf-8ef6-9af552bb172b" (UID: "a4c7d8a3-38b3-46cf-8ef6-9af552bb172b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:06:14.573346 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.573317    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a4c7d8a3-38b3-46cf-8ef6-9af552bb172b" (UID: "a4c7d8a3-38b3-46cf-8ef6-9af552bb172b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:06:14.651713 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.651674    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:06:14.651713 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.651707    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-cabundle-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:06:14.651713 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.651717    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-kserve-provision-location\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:06:14.651945 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.651725    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:06:14.651945 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.651734    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc8vc\" (UniqueName: \"kubernetes.io/projected/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-kube-api-access-sc8vc\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:06:14.651945 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.651742    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:06:14.651945 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.651753    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:06:14.698856 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.698813    2561 generic.go:356] "Generic (PLEG): container finished" podID="a4c7d8a3-38b3-46cf-8ef6-9af552bb172b" containerID="8c9e294135f1ae4211d0501a778c270d88c4cd9dc0fa308f65c41e9bdb99aae0" exitCode=0
Aug 05 18:06:14.699091 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.698887    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"
Aug 05 18:06:14.699091 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.698899    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz" event={"ID":"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b","Type":"ContainerDied","Data":"8c9e294135f1ae4211d0501a778c270d88c4cd9dc0fa308f65c41e9bdb99aae0"}
Aug 05 18:06:14.699091 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.698990    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz" event={"ID":"a4c7d8a3-38b3-46cf-8ef6-9af552bb172b","Type":"ContainerDied","Data":"515ab147c90c9065627bbbc7ca4b74fa3e8695c845b9d3f6dfe9d8a5052ce5b1"}
Aug 05 18:06:14.699091 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.699026    2561 scope.go:117] "RemoveContainer" containerID="8c9e294135f1ae4211d0501a778c270d88c4cd9dc0fa308f65c41e9bdb99aae0"
Aug 05 18:06:14.712004 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.711979    2561 scope.go:117] "RemoveContainer" containerID="b75ed9a236fe08ce2ec953c6a6998f8c63b162d0580df010dae7a1c76cb75e28"
Aug 05 18:06:14.724765 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.724646    2561 scope.go:117] "RemoveContainer" containerID="8c9e294135f1ae4211d0501a778c270d88c4cd9dc0fa308f65c41e9bdb99aae0"
Aug 05 18:06:14.725041 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:06:14.724987    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c9e294135f1ae4211d0501a778c270d88c4cd9dc0fa308f65c41e9bdb99aae0\": container with ID starting with 8c9e294135f1ae4211d0501a778c270d88c4cd9dc0fa308f65c41e9bdb99aae0 not found: ID does not exist" containerID="8c9e294135f1ae4211d0501a778c270d88c4cd9dc0fa308f65c41e9bdb99aae0"
Aug 05 18:06:14.725041 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.725022    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c9e294135f1ae4211d0501a778c270d88c4cd9dc0fa308f65c41e9bdb99aae0"} err="failed to get container status \"8c9e294135f1ae4211d0501a778c270d88c4cd9dc0fa308f65c41e9bdb99aae0\": rpc error: code = NotFound desc = could not find container \"8c9e294135f1ae4211d0501a778c270d88c4cd9dc0fa308f65c41e9bdb99aae0\": container with ID starting with 8c9e294135f1ae4211d0501a778c270d88c4cd9dc0fa308f65c41e9bdb99aae0 not found: ID does not exist"
Aug 05 18:06:14.725208 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.725049    2561 scope.go:117] "RemoveContainer" containerID="b75ed9a236fe08ce2ec953c6a6998f8c63b162d0580df010dae7a1c76cb75e28"
Aug 05 18:06:14.725384 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:06:14.725358    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b75ed9a236fe08ce2ec953c6a6998f8c63b162d0580df010dae7a1c76cb75e28\": container with ID starting with b75ed9a236fe08ce2ec953c6a6998f8c63b162d0580df010dae7a1c76cb75e28 not found: ID does not exist" containerID="b75ed9a236fe08ce2ec953c6a6998f8c63b162d0580df010dae7a1c76cb75e28"
Aug 05 18:06:14.725499 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.725392    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b75ed9a236fe08ce2ec953c6a6998f8c63b162d0580df010dae7a1c76cb75e28"} err="failed to get container status \"b75ed9a236fe08ce2ec953c6a6998f8c63b162d0580df010dae7a1c76cb75e28\": rpc error: code = NotFound desc = could not find container \"b75ed9a236fe08ce2ec953c6a6998f8c63b162d0580df010dae7a1c76cb75e28\": container with ID starting with b75ed9a236fe08ce2ec953c6a6998f8c63b162d0580df010dae7a1c76cb75e28 not found: ID does not exist"
Aug 05 18:06:14.732068 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.732043    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"]
Aug 05 18:06:14.732172 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:14.732084    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-65b75744d-jktxz"]
Aug 05 18:06:15.133897 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:15.133830    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc" podUID="99f51fdd-d13c-48b8-8d41-977b7dd5356a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/health\": dial tcp 10.133.0.102:8000: connect: connection refused"
Aug 05 18:06:16.730209 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:16.730106    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4c7d8a3-38b3-46cf-8ef6-9af552bb172b" path="/var/lib/kubelet/pods/a4c7d8a3-38b3-46cf-8ef6-9af552bb172b/volumes"
Aug 05 18:06:18.715863 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.715825    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"]
Aug 05 18:06:18.716351 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.716292    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4c7d8a3-38b3-46cf-8ef6-9af552bb172b" containerName="storage-initializer"
Aug 05 18:06:18.716351 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.716307    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4c7d8a3-38b3-46cf-8ef6-9af552bb172b" containerName="storage-initializer"
Aug 05 18:06:18.716351 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.716332    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4c7d8a3-38b3-46cf-8ef6-9af552bb172b" containerName="main"
Aug 05 18:06:18.716351 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.716338    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4c7d8a3-38b3-46cf-8ef6-9af552bb172b" containerName="main"
Aug 05 18:06:18.716586 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.716400    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4c7d8a3-38b3-46cf-8ef6-9af552bb172b" containerName="main"
Aug 05 18:06:18.721040 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.721020    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:18.724259 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.724233    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-t6fpc\""
Aug 05 18:06:18.724554 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.724536    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Aug 05 18:06:18.724660 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.724604    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Aug 05 18:06:18.724992 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.724974    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Aug 05 18:06:18.737994 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.737945    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"]
Aug 05 18:06:18.791617 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.791583    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db64709d-92b9-4ec7-a5cb-f4408befd5ea-istio-data\") pod \"router-gateway-1-openshift-default-6cc7987576-64thm\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:18.791799 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.791631    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db64709d-92b9-4ec7-a5cb-f4408befd5ea-workload-certs\") pod \"router-gateway-1-openshift-default-6cc7987576-64thm\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:18.791799 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.791707    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db64709d-92b9-4ec7-a5cb-f4408befd5ea-istio-podinfo\") pod \"router-gateway-1-openshift-default-6cc7987576-64thm\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:18.791877 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.791809    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db64709d-92b9-4ec7-a5cb-f4408befd5ea-credential-socket\") pod \"router-gateway-1-openshift-default-6cc7987576-64thm\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:18.791922 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.791869    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db64709d-92b9-4ec7-a5cb-f4408befd5ea-istio-envoy\") pod \"router-gateway-1-openshift-default-6cc7987576-64thm\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:18.791922 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.791911    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tq25\" (UniqueName: \"kubernetes.io/projected/db64709d-92b9-4ec7-a5cb-f4408befd5ea-kube-api-access-2tq25\") pod \"router-gateway-1-openshift-default-6cc7987576-64thm\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:18.792023 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.791996    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db64709d-92b9-4ec7-a5cb-f4408befd5ea-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6cc7987576-64thm\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:18.792117 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.792100    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db64709d-92b9-4ec7-a5cb-f4408befd5ea-istio-token\") pod \"router-gateway-1-openshift-default-6cc7987576-64thm\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:18.792166 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.792151    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db64709d-92b9-4ec7-a5cb-f4408befd5ea-workload-socket\") pod \"router-gateway-1-openshift-default-6cc7987576-64thm\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:18.893495 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.893455    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db64709d-92b9-4ec7-a5cb-f4408befd5ea-istio-podinfo\") pod \"router-gateway-1-openshift-default-6cc7987576-64thm\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:18.893704 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.893508    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db64709d-92b9-4ec7-a5cb-f4408befd5ea-credential-socket\") pod \"router-gateway-1-openshift-default-6cc7987576-64thm\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:18.893704 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.893542    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db64709d-92b9-4ec7-a5cb-f4408befd5ea-istio-envoy\") pod \"router-gateway-1-openshift-default-6cc7987576-64thm\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:18.893704 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.893576    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tq25\" (UniqueName: \"kubernetes.io/projected/db64709d-92b9-4ec7-a5cb-f4408befd5ea-kube-api-access-2tq25\") pod \"router-gateway-1-openshift-default-6cc7987576-64thm\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:18.893704 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.893626    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db64709d-92b9-4ec7-a5cb-f4408befd5ea-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6cc7987576-64thm\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:18.893704 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.893653    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db64709d-92b9-4ec7-a5cb-f4408befd5ea-istio-token\") pod \"router-gateway-1-openshift-default-6cc7987576-64thm\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:18.893704 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.893683    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db64709d-92b9-4ec7-a5cb-f4408befd5ea-workload-socket\") pod \"router-gateway-1-openshift-default-6cc7987576-64thm\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:18.894080 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.893724    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db64709d-92b9-4ec7-a5cb-f4408befd5ea-istio-data\") pod \"router-gateway-1-openshift-default-6cc7987576-64thm\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:18.894080 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.893759    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db64709d-92b9-4ec7-a5cb-f4408befd5ea-workload-certs\") pod \"router-gateway-1-openshift-default-6cc7987576-64thm\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:18.894201 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.894145    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db64709d-92b9-4ec7-a5cb-f4408befd5ea-credential-socket\") pod \"router-gateway-1-openshift-default-6cc7987576-64thm\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:18.894539 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.894330    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db64709d-92b9-4ec7-a5cb-f4408befd5ea-istio-data\") pod \"router-gateway-1-openshift-default-6cc7987576-64thm\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:18.894539 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.894363    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db64709d-92b9-4ec7-a5cb-f4408befd5ea-workload-certs\") pod \"router-gateway-1-openshift-default-6cc7987576-64thm\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:18.894539 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.894419    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db64709d-92b9-4ec7-a5cb-f4408befd5ea-workload-socket\") pod \"router-gateway-1-openshift-default-6cc7987576-64thm\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:18.894707 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.894618    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db64709d-92b9-4ec7-a5cb-f4408befd5ea-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6cc7987576-64thm\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:18.896060 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.896037    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db64709d-92b9-4ec7-a5cb-f4408befd5ea-istio-envoy\") pod \"router-gateway-1-openshift-default-6cc7987576-64thm\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:18.896264 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.896248    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db64709d-92b9-4ec7-a5cb-f4408befd5ea-istio-podinfo\") pod \"router-gateway-1-openshift-default-6cc7987576-64thm\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:18.901762 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.901731    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db64709d-92b9-4ec7-a5cb-f4408befd5ea-istio-token\") pod \"router-gateway-1-openshift-default-6cc7987576-64thm\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:18.901881 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:18.901862    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tq25\" (UniqueName: \"kubernetes.io/projected/db64709d-92b9-4ec7-a5cb-f4408befd5ea-kube-api-access-2tq25\") pod \"router-gateway-1-openshift-default-6cc7987576-64thm\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:19.033218 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:19.033182    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:19.179158 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:19.179130    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"]
Aug 05 18:06:19.182878 ip-10-0-135-151 kubenswrapper[2561]: W0805 18:06:19.182849    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb64709d_92b9_4ec7_a5cb_f4408befd5ea.slice/crio-205888ea231efa210865fafeda9bf679366d7b64044ba0f194919cc0882e3a34 WatchSource:0}: Error finding container 205888ea231efa210865fafeda9bf679366d7b64044ba0f194919cc0882e3a34: Status 404 returned error can't find the container with id 205888ea231efa210865fafeda9bf679366d7b64044ba0f194919cc0882e3a34
Aug 05 18:06:19.185756 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:19.185716    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 18:06:19.185839 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:19.185803    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 18:06:19.185921 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:19.185844    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 18:06:19.733546 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:19.733511    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm" event={"ID":"db64709d-92b9-4ec7-a5cb-f4408befd5ea","Type":"ContainerStarted","Data":"c32f42ddcec9737f256099a0b59ae80d4c6839182dc024cc92b7b3f21146546c"}
Aug 05 18:06:19.733979 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:19.733574    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm" event={"ID":"db64709d-92b9-4ec7-a5cb-f4408befd5ea","Type":"ContainerStarted","Data":"205888ea231efa210865fafeda9bf679366d7b64044ba0f194919cc0882e3a34"}
Aug 05 18:06:19.754747 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:19.754688    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm" podStartSLOduration=1.754671207 podStartE2EDuration="1.754671207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:06:19.752970674 +0000 UTC m=+3235.497605075" watchObservedRunningTime="2026-08-05 18:06:19.754671207 +0000 UTC m=+3235.499305535"
Aug 05 18:06:20.033474 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:20.033441    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:20.039000 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:20.038971    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:20.739016 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:20.738984    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:20.740070 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:20.740050    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:25.157661 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:25.157626    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:06:25.176007 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:25.175972    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:06:30.808082 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:30.808035    2561 generic.go:356] "Generic (PLEG): container finished" podID="db64709d-92b9-4ec7-a5cb-f4408befd5ea" containerID="c32f42ddcec9737f256099a0b59ae80d4c6839182dc024cc92b7b3f21146546c" exitCode=0
Aug 05 18:06:30.808575 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:30.808089    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm" event={"ID":"db64709d-92b9-4ec7-a5cb-f4408befd5ea","Type":"ContainerDied","Data":"c32f42ddcec9737f256099a0b59ae80d4c6839182dc024cc92b7b3f21146546c"}
Aug 05 18:06:30.808575 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:30.808554    2561 scope.go:117] "RemoveContainer" containerID="c32f42ddcec9737f256099a0b59ae80d4c6839182dc024cc92b7b3f21146546c"
Aug 05 18:06:30.809697 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:30.809666    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 18:06:30.809811 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:30.809750    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 18:06:30.809811 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:30.809800    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 18:06:31.815339 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:31.815306    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm" event={"ID":"db64709d-92b9-4ec7-a5cb-f4408befd5ea","Type":"ContainerStarted","Data":"5033a7b3a83429d0d0debb32a66b95bd5981837799adc156bf2df252318cc64b"}
Aug 05 18:06:32.034059 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:32.033999    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:32.035342 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:32.035307    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm" podUID="db64709d-92b9-4ec7-a5cb-f4408befd5ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Aug 05 18:06:33.034155 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:33.034115    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm" podUID="db64709d-92b9-4ec7-a5cb-f4408befd5ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Aug 05 18:06:34.034251 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:34.034200    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm" podUID="db64709d-92b9-4ec7-a5cb-f4408befd5ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Aug 05 18:06:34.822099 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:34.822034    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"]
Aug 05 18:06:34.832178 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:34.832146    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:06:34.834755 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:34.834734    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Aug 05 18:06:34.835085 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:34.835068    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Aug 05 18:06:34.835327 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:34.835313    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Aug 05 18:06:34.835654 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:34.835639    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-d4wpz\""
Aug 05 18:06:34.839537 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:34.839517    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"]
Aug 05 18:06:34.954506 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:34.954466    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn768\" (UniqueName: \"kubernetes.io/projected/d0b046be-679d-4550-b677-486a1f64e610-kube-api-access-jn768\") pod \"router-with-refs-test-kserve-676b64f468-b5jfp\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:06:34.954816 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:34.954792    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0b046be-679d-4550-b677-486a1f64e610-dshm\") pod \"router-with-refs-test-kserve-676b64f468-b5jfp\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:06:34.954947 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:34.954933    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0b046be-679d-4550-b677-486a1f64e610-cabundle-cert\") pod \"router-with-refs-test-kserve-676b64f468-b5jfp\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:06:34.955105 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:34.955089    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0b046be-679d-4550-b677-486a1f64e610-home\") pod \"router-with-refs-test-kserve-676b64f468-b5jfp\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:06:34.955238 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:34.955222    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0b046be-679d-4550-b677-486a1f64e610-kserve-provision-location\") pod \"router-with-refs-test-kserve-676b64f468-b5jfp\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:06:34.955366 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:34.955352    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0b046be-679d-4550-b677-486a1f64e610-tls-certs\") pod \"router-with-refs-test-kserve-676b64f468-b5jfp\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:06:34.955483 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:34.955470    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0b046be-679d-4550-b677-486a1f64e610-tmp-dir\") pod \"router-with-refs-test-kserve-676b64f468-b5jfp\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:06:34.955588 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:34.955574    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0b046be-679d-4550-b677-486a1f64e610-model-cache\") pod \"router-with-refs-test-kserve-676b64f468-b5jfp\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:06:35.033760 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:35.033716    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm" podUID="db64709d-92b9-4ec7-a5cb-f4408befd5ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Aug 05 18:06:35.056872 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:35.056833    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0b046be-679d-4550-b677-486a1f64e610-tls-certs\") pod \"router-with-refs-test-kserve-676b64f468-b5jfp\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:06:35.057488 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:35.057464    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0b046be-679d-4550-b677-486a1f64e610-tmp-dir\") pod \"router-with-refs-test-kserve-676b64f468-b5jfp\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:06:35.057665 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:35.057647    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0b046be-679d-4550-b677-486a1f64e610-model-cache\") pod \"router-with-refs-test-kserve-676b64f468-b5jfp\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:06:35.057895 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:35.057876    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn768\" (UniqueName: \"kubernetes.io/projected/d0b046be-679d-4550-b677-486a1f64e610-kube-api-access-jn768\") pod \"router-with-refs-test-kserve-676b64f468-b5jfp\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:06:35.058430 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:35.058393    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0b046be-679d-4550-b677-486a1f64e610-tmp-dir\") pod \"router-with-refs-test-kserve-676b64f468-b5jfp\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:06:35.058602 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:35.058075    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0b046be-679d-4550-b677-486a1f64e610-model-cache\") pod \"router-with-refs-test-kserve-676b64f468-b5jfp\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:06:35.058730 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:35.058443    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0b046be-679d-4550-b677-486a1f64e610-dshm\") pod \"router-with-refs-test-kserve-676b64f468-b5jfp\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:06:35.058897 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:35.058874    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0b046be-679d-4550-b677-486a1f64e610-cabundle-cert\") pod \"router-with-refs-test-kserve-676b64f468-b5jfp\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:06:35.060573 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:35.060556    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0b046be-679d-4550-b677-486a1f64e610-home\") pod \"router-with-refs-test-kserve-676b64f468-b5jfp\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:06:35.061177 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:35.060472    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0b046be-679d-4550-b677-486a1f64e610-cabundle-cert\") pod \"router-with-refs-test-kserve-676b64f468-b5jfp\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:06:35.061340 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:35.060403    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0b046be-679d-4550-b677-486a1f64e610-tls-certs\") pod \"router-with-refs-test-kserve-676b64f468-b5jfp\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:06:35.061828 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:35.061042    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0b046be-679d-4550-b677-486a1f64e610-home\") pod \"router-with-refs-test-kserve-676b64f468-b5jfp\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:06:35.061998 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:35.061775    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0b046be-679d-4550-b677-486a1f64e610-kserve-provision-location\") pod \"router-with-refs-test-kserve-676b64f468-b5jfp\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:06:35.062162 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:35.061140    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0b046be-679d-4550-b677-486a1f64e610-kserve-provision-location\") pod \"router-with-refs-test-kserve-676b64f468-b5jfp\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:06:35.067226 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:35.067184    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0b046be-679d-4550-b677-486a1f64e610-dshm\") pod \"router-with-refs-test-kserve-676b64f468-b5jfp\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:06:35.070122 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:35.070099    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn768\" (UniqueName: \"kubernetes.io/projected/d0b046be-679d-4550-b677-486a1f64e610-kube-api-access-jn768\") pod \"router-with-refs-test-kserve-676b64f468-b5jfp\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:06:35.161243 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:35.161154    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:06:35.350978 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:35.345093    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"]
Aug 05 18:06:35.739975 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:35.739857    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:35.847125 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:35.847082    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp" event={"ID":"d0b046be-679d-4550-b677-486a1f64e610","Type":"ContainerStarted","Data":"ebc7fb1a88ae37c0b24a726471da9cf7017013c93dedee5027c9fa292edd7359"}
Aug 05 18:06:35.847339 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:35.847156    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp" event={"ID":"d0b046be-679d-4550-b677-486a1f64e610","Type":"ContainerStarted","Data":"a71e0cb878581766fcf32f2401e72c2e41b7a029e9f6fcad23f861dd12fe2e72"}
Aug 05 18:06:36.034401 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:36.034356    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm" podUID="db64709d-92b9-4ec7-a5cb-f4408befd5ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused"
Aug 05 18:06:37.040223 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:37.040154    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:37.041293 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:37.041268    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:06:38.868477 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:38.868444    2561 generic.go:356] "Generic (PLEG): container finished" podID="d0b046be-679d-4550-b677-486a1f64e610" containerID="ebc7fb1a88ae37c0b24a726471da9cf7017013c93dedee5027c9fa292edd7359" exitCode=0
Aug 05 18:06:38.868885 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:38.868519    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp" event={"ID":"d0b046be-679d-4550-b677-486a1f64e610","Type":"ContainerDied","Data":"ebc7fb1a88ae37c0b24a726471da9cf7017013c93dedee5027c9fa292edd7359"}
Aug 05 18:06:39.878292 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:39.878252    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp" event={"ID":"d0b046be-679d-4550-b677-486a1f64e610","Type":"ContainerStarted","Data":"21ee5617ee726851452263a0ae583728a523c3b7c756ca96ded07d9331edb315"}
Aug 05 18:06:39.901322 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:39.901264    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp" podStartSLOduration=3.396326191 podStartE2EDuration="5.901247462s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.504921271s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:06:39.89860044 +0000 UTC m=+3255.643234769" watchObservedRunningTime="2026-08-05 18:06:39.901247462 +0000 UTC m=+3255.645881789"
Aug 05 18:06:40.162527 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:40.162428    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:06:45.162654 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:45.162619    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:06:45.164241 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:45.164203    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp" podUID="d0b046be-679d-4550-b677-486a1f64e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/health\": dial tcp 10.133.0.104:8000: connect: connection refused"
Aug 05 18:06:55.162477 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:06:55.162427    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp" podUID="d0b046be-679d-4550-b677-486a1f64e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/health\": dial tcp 10.133.0.104:8000: connect: connection refused"
Aug 05 18:07:04.262558 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:04.262524    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-c56d9cccf-plsrc_99f51fdd-d13c-48b8-8d41-977b7dd5356a/storage-initializer/0.log"
Aug 05 18:07:04.273065 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:04.273035    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-c56d9cccf-plsrc_99f51fdd-d13c-48b8-8d41-977b7dd5356a/main/0.log"
Aug 05 18:07:05.162806 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:05.162760    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp" podUID="d0b046be-679d-4550-b677-486a1f64e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/health\": dial tcp 10.133.0.104:8000: connect: connection refused"
Aug 05 18:07:05.594040 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:05.593977    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"]
Aug 05 18:07:15.163178 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:15.163130    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp" podUID="d0b046be-679d-4550-b677-486a1f64e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/health\": dial tcp 10.133.0.104:8000: connect: connection refused"
Aug 05 18:07:20.617865 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:20.617803    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc" podUID="99f51fdd-d13c-48b8-8d41-977b7dd5356a" containerName="main" containerID="cri-o://ec3206e2c3eb1cfb1c74008652896510355e999d2c8bce057a8aecab71da986c" gracePeriod=45
Aug 05 18:07:21.142492 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:21.142453    2561 generic.go:356] "Generic (PLEG): container finished" podID="99f51fdd-d13c-48b8-8d41-977b7dd5356a" containerID="ec3206e2c3eb1cfb1c74008652896510355e999d2c8bce057a8aecab71da986c" exitCode=0
Aug 05 18:07:21.142651 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:21.142519    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc" event={"ID":"99f51fdd-d13c-48b8-8d41-977b7dd5356a","Type":"ContainerDied","Data":"ec3206e2c3eb1cfb1c74008652896510355e999d2c8bce057a8aecab71da986c"}
Aug 05 18:07:21.159563 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:21.159522    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc" podUID="99f51fdd-d13c-48b8-8d41-977b7dd5356a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.102:8000/health\": dial tcp 10.133.0.102:8000: connect: connection refused"
Aug 05 18:07:21.304026 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:21.303998    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:07:21.433448 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:21.433362    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7grw\" (UniqueName: \"kubernetes.io/projected/99f51fdd-d13c-48b8-8d41-977b7dd5356a-kube-api-access-h7grw\") pod \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") "
Aug 05 18:07:21.433448 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:21.433396    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99f51fdd-d13c-48b8-8d41-977b7dd5356a-home\") pod \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") "
Aug 05 18:07:21.433692 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:21.433466    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99f51fdd-d13c-48b8-8d41-977b7dd5356a-tmp-dir\") pod \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") "
Aug 05 18:07:21.433692 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:21.433528    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99f51fdd-d13c-48b8-8d41-977b7dd5356a-cabundle-cert\") pod \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") "
Aug 05 18:07:21.433692 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:21.433572    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99f51fdd-d13c-48b8-8d41-977b7dd5356a-dshm\") pod \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") "
Aug 05 18:07:21.433859 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:21.433692    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99f51fdd-d13c-48b8-8d41-977b7dd5356a-kserve-provision-location\") pod \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") "
Aug 05 18:07:21.433859 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:21.433738    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99f51fdd-d13c-48b8-8d41-977b7dd5356a-tls-certs\") pod \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") "
Aug 05 18:07:21.433859 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:21.433771    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99f51fdd-d13c-48b8-8d41-977b7dd5356a-model-cache\") pod \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\" (UID: \"99f51fdd-d13c-48b8-8d41-977b7dd5356a\") "
Aug 05 18:07:21.434059 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:21.433864    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f51fdd-d13c-48b8-8d41-977b7dd5356a-home" (OuterVolumeSpecName: "home") pod "99f51fdd-d13c-48b8-8d41-977b7dd5356a" (UID: "99f51fdd-d13c-48b8-8d41-977b7dd5356a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:07:21.434059 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:21.433919    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99f51fdd-d13c-48b8-8d41-977b7dd5356a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "99f51fdd-d13c-48b8-8d41-977b7dd5356a" (UID: "99f51fdd-d13c-48b8-8d41-977b7dd5356a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 18:07:21.434165 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:21.434141    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f51fdd-d13c-48b8-8d41-977b7dd5356a-model-cache" (OuterVolumeSpecName: "model-cache") pod "99f51fdd-d13c-48b8-8d41-977b7dd5356a" (UID: "99f51fdd-d13c-48b8-8d41-977b7dd5356a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:07:21.434387 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:21.434258    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99f51fdd-d13c-48b8-8d41-977b7dd5356a-cabundle-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:07:21.434387 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:21.434283    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99f51fdd-d13c-48b8-8d41-977b7dd5356a-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:07:21.434387 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:21.434297    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99f51fdd-d13c-48b8-8d41-977b7dd5356a-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:07:21.436791 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:21.436765    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f51fdd-d13c-48b8-8d41-977b7dd5356a-dshm" (OuterVolumeSpecName: "dshm") pod "99f51fdd-d13c-48b8-8d41-977b7dd5356a" (UID: "99f51fdd-d13c-48b8-8d41-977b7dd5356a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:07:21.436791 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:21.436785    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99f51fdd-d13c-48b8-8d41-977b7dd5356a-kube-api-access-h7grw" (OuterVolumeSpecName: "kube-api-access-h7grw") pod "99f51fdd-d13c-48b8-8d41-977b7dd5356a" (UID: "99f51fdd-d13c-48b8-8d41-977b7dd5356a"). InnerVolumeSpecName "kube-api-access-h7grw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 18:07:21.436991 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:21.436755    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99f51fdd-d13c-48b8-8d41-977b7dd5356a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "99f51fdd-d13c-48b8-8d41-977b7dd5356a" (UID: "99f51fdd-d13c-48b8-8d41-977b7dd5356a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 18:07:21.444596 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:21.444570    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f51fdd-d13c-48b8-8d41-977b7dd5356a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "99f51fdd-d13c-48b8-8d41-977b7dd5356a" (UID: "99f51fdd-d13c-48b8-8d41-977b7dd5356a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:07:21.457237 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:21.457209    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f51fdd-d13c-48b8-8d41-977b7dd5356a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "99f51fdd-d13c-48b8-8d41-977b7dd5356a" (UID: "99f51fdd-d13c-48b8-8d41-977b7dd5356a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:07:21.535343 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:21.535309    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99f51fdd-d13c-48b8-8d41-977b7dd5356a-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:07:21.535343 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:21.535339    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99f51fdd-d13c-48b8-8d41-977b7dd5356a-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:07:21.535343 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:21.535348    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99f51fdd-d13c-48b8-8d41-977b7dd5356a-kserve-provision-location\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:07:21.535583 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:21.535359    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99f51fdd-d13c-48b8-8d41-977b7dd5356a-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:07:21.535583 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:21.535369    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7grw\" (UniqueName: \"kubernetes.io/projected/99f51fdd-d13c-48b8-8d41-977b7dd5356a-kube-api-access-h7grw\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:07:22.150064 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:22.150028    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc" event={"ID":"99f51fdd-d13c-48b8-8d41-977b7dd5356a","Type":"ContainerDied","Data":"6910b895ee9afb7cceaf234c92e3a6494d2022af4c526753b486cb16634139eb"}
Aug 05 18:07:22.150498 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:22.150092    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"
Aug 05 18:07:22.150498 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:22.150125    2561 scope.go:117] "RemoveContainer" containerID="ec3206e2c3eb1cfb1c74008652896510355e999d2c8bce057a8aecab71da986c"
Aug 05 18:07:22.159922 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:22.159901    2561 scope.go:117] "RemoveContainer" containerID="ef7e5abdba1b4ffda7d929bac2cee07dfa98c02c13dfa1e9997f382c00f6b9d2"
Aug 05 18:07:22.177059 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:22.176944    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"]
Aug 05 18:07:22.180625 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:22.180601    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-c56d9cccf-plsrc"]
Aug 05 18:07:22.730470 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:22.730436    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99f51fdd-d13c-48b8-8d41-977b7dd5356a" path="/var/lib/kubelet/pods/99f51fdd-d13c-48b8-8d41-977b7dd5356a/volumes"
Aug 05 18:07:25.162628 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:25.162589    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp" podUID="d0b046be-679d-4550-b677-486a1f64e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/health\": dial tcp 10.133.0.104:8000: connect: connection refused"
Aug 05 18:07:30.923367 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:30.923328    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"]
Aug 05 18:07:30.923894 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:30.923874    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99f51fdd-d13c-48b8-8d41-977b7dd5356a" containerName="storage-initializer"
Aug 05 18:07:30.923999 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:30.923897    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="99f51fdd-d13c-48b8-8d41-977b7dd5356a" containerName="storage-initializer"
Aug 05 18:07:30.923999 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:30.923915    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99f51fdd-d13c-48b8-8d41-977b7dd5356a" containerName="main"
Aug 05 18:07:30.923999 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:30.923923    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="99f51fdd-d13c-48b8-8d41-977b7dd5356a" containerName="main"
Aug 05 18:07:30.924170 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:30.924032    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="99f51fdd-d13c-48b8-8d41-977b7dd5356a" containerName="main"
Aug 05 18:07:30.927337 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:30.927315    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:07:30.930306 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:30.929995    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Aug 05 18:07:30.930306 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:30.930023    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Aug 05 18:07:30.930306 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:30.930066    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Aug 05 18:07:30.930306 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:30.930124    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-5wwqz\""
Aug 05 18:07:30.930306 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:30.930000    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Aug 05 18:07:30.932307 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:30.930892    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Aug 05 18:07:30.939062 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:30.939039    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"]
Aug 05 18:07:31.024117 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:31.024072    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6129ebd7-73bf-4a8f-a208-6047e0a486ca-home\") pod \"lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:07:31.024117 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:31.024111    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6129ebd7-73bf-4a8f-a208-6047e0a486ca-tls-certs\") pod \"lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:07:31.024408 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:31.024136    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6129ebd7-73bf-4a8f-a208-6047e0a486ca-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:07:31.024408 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:31.024204    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjdqm\" (UniqueName: \"kubernetes.io/projected/6129ebd7-73bf-4a8f-a208-6047e0a486ca-kube-api-access-hjdqm\") pod \"lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:07:31.024408 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:31.024284    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6129ebd7-73bf-4a8f-a208-6047e0a486ca-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:07:31.024408 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:31.024320    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6129ebd7-73bf-4a8f-a208-6047e0a486ca-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:07:31.024408 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:31.024365    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6129ebd7-73bf-4a8f-a208-6047e0a486ca-model-cache\") pod \"lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:07:31.024708 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:31.024418    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6129ebd7-73bf-4a8f-a208-6047e0a486ca-dshm\") pod \"lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:07:31.125377 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:31.125332    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6129ebd7-73bf-4a8f-a208-6047e0a486ca-tls-certs\") pod \"lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:07:31.125580 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:31.125397    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6129ebd7-73bf-4a8f-a208-6047e0a486ca-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:07:31.125580 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:31.125445    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjdqm\" (UniqueName: \"kubernetes.io/projected/6129ebd7-73bf-4a8f-a208-6047e0a486ca-kube-api-access-hjdqm\") pod \"lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:07:31.125580 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:31.125477    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6129ebd7-73bf-4a8f-a208-6047e0a486ca-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:07:31.125580 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:31.125519    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6129ebd7-73bf-4a8f-a208-6047e0a486ca-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:07:31.125832 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:31.125586    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6129ebd7-73bf-4a8f-a208-6047e0a486ca-model-cache\") pod \"lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:07:31.125832 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:31.125653    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6129ebd7-73bf-4a8f-a208-6047e0a486ca-dshm\") pod \"lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:07:31.125832 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:31.125691    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6129ebd7-73bf-4a8f-a208-6047e0a486ca-home\") pod \"lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:07:31.126047 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:31.125996    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6129ebd7-73bf-4a8f-a208-6047e0a486ca-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:07:31.126107 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:31.126039    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6129ebd7-73bf-4a8f-a208-6047e0a486ca-model-cache\") pod \"lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:07:31.126107 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:31.126092    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6129ebd7-73bf-4a8f-a208-6047e0a486ca-home\") pod \"lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:07:31.126486 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:31.126458    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6129ebd7-73bf-4a8f-a208-6047e0a486ca-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:07:31.126675 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:31.126605    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6129ebd7-73bf-4a8f-a208-6047e0a486ca-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:07:31.128394 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:31.128361    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6129ebd7-73bf-4a8f-a208-6047e0a486ca-dshm\") pod \"lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:07:31.128514 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:31.128368    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6129ebd7-73bf-4a8f-a208-6047e0a486ca-tls-certs\") pod \"lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:07:31.132815 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:31.132794    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjdqm\" (UniqueName: \"kubernetes.io/projected/6129ebd7-73bf-4a8f-a208-6047e0a486ca-kube-api-access-hjdqm\") pod \"lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:07:31.241632 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:31.241540    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:07:31.388807 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:31.388777    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"]
Aug 05 18:07:31.390491 ip-10-0-135-151 kubenswrapper[2561]: W0805 18:07:31.390463    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6129ebd7_73bf_4a8f_a208_6047e0a486ca.slice/crio-c4e264115a6b434c5554e315db4cc78261953ff1a4f24adfbc2d97dd87adb4eb WatchSource:0}: Error finding container c4e264115a6b434c5554e315db4cc78261953ff1a4f24adfbc2d97dd87adb4eb: Status 404 returned error can't find the container with id c4e264115a6b434c5554e315db4cc78261953ff1a4f24adfbc2d97dd87adb4eb
Aug 05 18:07:32.218595 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:32.218545    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5" event={"ID":"6129ebd7-73bf-4a8f-a208-6047e0a486ca","Type":"ContainerStarted","Data":"957aed06403aaa7b08d3e2e03bd5bb24604866bf8dc295ac680b3a37930dff6f"}
Aug 05 18:07:32.219071 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:32.218624    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5" event={"ID":"6129ebd7-73bf-4a8f-a208-6047e0a486ca","Type":"ContainerStarted","Data":"c4e264115a6b434c5554e315db4cc78261953ff1a4f24adfbc2d97dd87adb4eb"}
Aug 05 18:07:35.163298 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:35.163187    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp" podUID="d0b046be-679d-4550-b677-486a1f64e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/health\": dial tcp 10.133.0.104:8000: connect: connection refused"
Aug 05 18:07:35.237258 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:35.237222    2561 generic.go:356] "Generic (PLEG): container finished" podID="6129ebd7-73bf-4a8f-a208-6047e0a486ca" containerID="957aed06403aaa7b08d3e2e03bd5bb24604866bf8dc295ac680b3a37930dff6f" exitCode=0
Aug 05 18:07:35.237455 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:35.237292    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5" event={"ID":"6129ebd7-73bf-4a8f-a208-6047e0a486ca","Type":"ContainerDied","Data":"957aed06403aaa7b08d3e2e03bd5bb24604866bf8dc295ac680b3a37930dff6f"}
Aug 05 18:07:36.249347 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:36.249312    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5" event={"ID":"6129ebd7-73bf-4a8f-a208-6047e0a486ca","Type":"ContainerStarted","Data":"3aa87fb534115727d25820e99cdbce31b4528b5f9862ea317a990f0505100416"}
Aug 05 18:07:36.273967 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:36.273888    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5" podStartSLOduration=2.907737101 podStartE2EDuration="6.2738663s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.366129199s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:07:36.27150279 +0000 UTC m=+3312.016137123" watchObservedRunningTime="2026-08-05 18:07:36.2738663 +0000 UTC m=+3312.018500628"
Aug 05 18:07:37.241699 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:37.241649    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:07:41.242082 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:41.242032    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:07:41.243673 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:41.243633    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5" podUID="6129ebd7-73bf-4a8f-a208-6047e0a486ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/health\": dial tcp 10.133.0.105:8000: connect: connection refused"
Aug 05 18:07:45.163376 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:45.163325    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp" podUID="d0b046be-679d-4550-b677-486a1f64e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/health\": dial tcp 10.133.0.104:8000: connect: connection refused"
Aug 05 18:07:51.242760 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:51.242665    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5" podUID="6129ebd7-73bf-4a8f-a208-6047e0a486ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/health\": dial tcp 10.133.0.105:8000: connect: connection refused"
Aug 05 18:07:55.162719 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:07:55.162659    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp" podUID="d0b046be-679d-4550-b677-486a1f64e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/health\": dial tcp 10.133.0.104:8000: connect: connection refused"
Aug 05 18:08:01.242549 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:08:01.242494    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5" podUID="6129ebd7-73bf-4a8f-a208-6047e0a486ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/health\": dial tcp 10.133.0.105:8000: connect: connection refused"
Aug 05 18:08:05.162762 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:08:05.162713    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp" podUID="d0b046be-679d-4550-b677-486a1f64e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/health\": dial tcp 10.133.0.104:8000: connect: connection refused"
Aug 05 18:08:11.242995 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:08:11.242907    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5" podUID="6129ebd7-73bf-4a8f-a208-6047e0a486ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/health\": dial tcp 10.133.0.105:8000: connect: connection refused"
Aug 05 18:08:15.163505 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:08:15.163457    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp" podUID="d0b046be-679d-4550-b677-486a1f64e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/health\": dial tcp 10.133.0.104:8000: connect: connection refused"
Aug 05 18:08:21.242620 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:08:21.242557    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5" podUID="6129ebd7-73bf-4a8f-a208-6047e0a486ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/health\": dial tcp 10.133.0.105:8000: connect: connection refused"
Aug 05 18:08:25.163044 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:08:25.162996    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp" podUID="d0b046be-679d-4550-b677-486a1f64e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/health\": dial tcp 10.133.0.104:8000: connect: connection refused"
Aug 05 18:08:31.243138 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:08:31.243071    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5" podUID="6129ebd7-73bf-4a8f-a208-6047e0a486ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/health\": dial tcp 10.133.0.105:8000: connect: connection refused"
Aug 05 18:08:35.163378 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:08:35.163326    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp" podUID="d0b046be-679d-4550-b677-486a1f64e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/health\": dial tcp 10.133.0.104:8000: connect: connection refused"
Aug 05 18:08:41.242565 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:08:41.242514    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5" podUID="6129ebd7-73bf-4a8f-a208-6047e0a486ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/health\": dial tcp 10.133.0.105:8000: connect: connection refused"
Aug 05 18:08:45.178932 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:08:45.178896    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:08:45.191121 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:08:45.191093    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:08:49.811554 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:08:49.811508    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"]
Aug 05 18:08:51.242451 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:08:51.242401    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5" podUID="6129ebd7-73bf-4a8f-a208-6047e0a486ca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/health\": dial tcp 10.133.0.105:8000: connect: connection refused"
Aug 05 18:09:01.252605 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:01.252568    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:09:01.261290 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:01.261261    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:09:04.846208 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:04.846129    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp" podUID="d0b046be-679d-4550-b677-486a1f64e610" containerName="main" containerID="cri-o://21ee5617ee726851452263a0ae583728a523c3b7c756ca96ded07d9331edb315" gracePeriod=45
Aug 05 18:09:05.179813 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:05.179718    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp" podUID="d0b046be-679d-4550-b677-486a1f64e610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.104:8000/health\": dial tcp 10.133.0.104:8000: connect: connection refused"
Aug 05 18:09:05.831477 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:05.831417    2561 generic.go:356] "Generic (PLEG): container finished" podID="d0b046be-679d-4550-b677-486a1f64e610" containerID="21ee5617ee726851452263a0ae583728a523c3b7c756ca96ded07d9331edb315" exitCode=0
Aug 05 18:09:05.831650 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:05.831624    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp" event={"ID":"d0b046be-679d-4550-b677-486a1f64e610","Type":"ContainerDied","Data":"21ee5617ee726851452263a0ae583728a523c3b7c756ca96ded07d9331edb315"}
Aug 05 18:09:05.907625 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:05.907597    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:09:05.957332 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:05.957235    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0b046be-679d-4550-b677-486a1f64e610-tls-certs\") pod \"d0b046be-679d-4550-b677-486a1f64e610\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") "
Aug 05 18:09:05.957332 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:05.957318    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jn768\" (UniqueName: \"kubernetes.io/projected/d0b046be-679d-4550-b677-486a1f64e610-kube-api-access-jn768\") pod \"d0b046be-679d-4550-b677-486a1f64e610\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") "
Aug 05 18:09:05.957575 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:05.957448    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0b046be-679d-4550-b677-486a1f64e610-dshm\") pod \"d0b046be-679d-4550-b677-486a1f64e610\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") "
Aug 05 18:09:05.957575 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:05.957495    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0b046be-679d-4550-b677-486a1f64e610-home\") pod \"d0b046be-679d-4550-b677-486a1f64e610\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") "
Aug 05 18:09:05.957575 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:05.957538    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0b046be-679d-4550-b677-486a1f64e610-model-cache\") pod \"d0b046be-679d-4550-b677-486a1f64e610\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") "
Aug 05 18:09:05.957748 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:05.957592    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0b046be-679d-4550-b677-486a1f64e610-cabundle-cert\") pod \"d0b046be-679d-4550-b677-486a1f64e610\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") "
Aug 05 18:09:05.957748 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:05.957661    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0b046be-679d-4550-b677-486a1f64e610-kserve-provision-location\") pod \"d0b046be-679d-4550-b677-486a1f64e610\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") "
Aug 05 18:09:05.957748 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:05.957690    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0b046be-679d-4550-b677-486a1f64e610-tmp-dir\") pod \"d0b046be-679d-4550-b677-486a1f64e610\" (UID: \"d0b046be-679d-4550-b677-486a1f64e610\") "
Aug 05 18:09:05.958163 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:05.958130    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0b046be-679d-4550-b677-486a1f64e610-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d0b046be-679d-4550-b677-486a1f64e610" (UID: "d0b046be-679d-4550-b677-486a1f64e610"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 18:09:05.958386 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:05.958359    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b046be-679d-4550-b677-486a1f64e610-model-cache" (OuterVolumeSpecName: "model-cache") pod "d0b046be-679d-4550-b677-486a1f64e610" (UID: "d0b046be-679d-4550-b677-486a1f64e610"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:09:05.958386 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:05.958370    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b046be-679d-4550-b677-486a1f64e610-home" (OuterVolumeSpecName: "home") pod "d0b046be-679d-4550-b677-486a1f64e610" (UID: "d0b046be-679d-4550-b677-486a1f64e610"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:09:05.961052 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:05.961011    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b046be-679d-4550-b677-486a1f64e610-dshm" (OuterVolumeSpecName: "dshm") pod "d0b046be-679d-4550-b677-486a1f64e610" (UID: "d0b046be-679d-4550-b677-486a1f64e610"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:09:05.961494 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:05.961441    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0b046be-679d-4550-b677-486a1f64e610-kube-api-access-jn768" (OuterVolumeSpecName: "kube-api-access-jn768") pod "d0b046be-679d-4550-b677-486a1f64e610" (UID: "d0b046be-679d-4550-b677-486a1f64e610"). InnerVolumeSpecName "kube-api-access-jn768". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 18:09:05.963133 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:05.963111    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0b046be-679d-4550-b677-486a1f64e610-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d0b046be-679d-4550-b677-486a1f64e610" (UID: "d0b046be-679d-4550-b677-486a1f64e610"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 18:09:05.977931 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:05.977887    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b046be-679d-4550-b677-486a1f64e610-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d0b046be-679d-4550-b677-486a1f64e610" (UID: "d0b046be-679d-4550-b677-486a1f64e610"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:09:05.980372 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:05.980349    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0b046be-679d-4550-b677-486a1f64e610-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d0b046be-679d-4550-b677-486a1f64e610" (UID: "d0b046be-679d-4550-b677-486a1f64e610"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:09:06.059084 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:06.059052    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0b046be-679d-4550-b677-486a1f64e610-cabundle-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:09:06.059084 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:06.059082    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0b046be-679d-4550-b677-486a1f64e610-kserve-provision-location\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:09:06.059084 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:06.059092    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0b046be-679d-4550-b677-486a1f64e610-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:09:06.059334 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:06.059102    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0b046be-679d-4550-b677-486a1f64e610-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:09:06.059334 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:06.059112    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn768\" (UniqueName: \"kubernetes.io/projected/d0b046be-679d-4550-b677-486a1f64e610-kube-api-access-jn768\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:09:06.059334 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:06.059120    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0b046be-679d-4550-b677-486a1f64e610-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:09:06.059334 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:06.059127    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0b046be-679d-4550-b677-486a1f64e610-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:09:06.059334 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:06.059136    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0b046be-679d-4550-b677-486a1f64e610-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:09:06.839447 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:06.839415    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"
Aug 05 18:09:06.839662 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:06.839448    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp" event={"ID":"d0b046be-679d-4550-b677-486a1f64e610","Type":"ContainerDied","Data":"a71e0cb878581766fcf32f2401e72c2e41b7a029e9f6fcad23f861dd12fe2e72"}
Aug 05 18:09:06.839662 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:06.839523    2561 scope.go:117] "RemoveContainer" containerID="21ee5617ee726851452263a0ae583728a523c3b7c756ca96ded07d9331edb315"
Aug 05 18:09:06.851844 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:06.851232    2561 scope.go:117] "RemoveContainer" containerID="ebc7fb1a88ae37c0b24a726471da9cf7017013c93dedee5027c9fa292edd7359"
Aug 05 18:09:06.861754 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:06.861718    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"]
Aug 05 18:09:06.865900 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:06.865865    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-676b64f468-b5jfp"]
Aug 05 18:09:08.731348 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:08.731305    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0b046be-679d-4550-b677-486a1f64e610" path="/var/lib/kubelet/pods/d0b046be-679d-4550-b677-486a1f64e610/volumes"
Aug 05 18:09:14.810470 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:14.810428    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"]
Aug 05 18:09:14.811010 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:14.810675    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm" podUID="db64709d-92b9-4ec7-a5cb-f4408befd5ea" containerName="istio-proxy" containerID="cri-o://5033a7b3a83429d0d0debb32a66b95bd5981837799adc156bf2df252318cc64b" gracePeriod=30
Aug 05 18:09:19.932721 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:19.932684    2561 generic.go:356] "Generic (PLEG): container finished" podID="db64709d-92b9-4ec7-a5cb-f4408befd5ea" containerID="5033a7b3a83429d0d0debb32a66b95bd5981837799adc156bf2df252318cc64b" exitCode=0
Aug 05 18:09:19.933195 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:19.932773    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm" event={"ID":"db64709d-92b9-4ec7-a5cb-f4408befd5ea","Type":"ContainerDied","Data":"5033a7b3a83429d0d0debb32a66b95bd5981837799adc156bf2df252318cc64b"}
Aug 05 18:09:19.933195 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:19.932861    2561 scope.go:117] "RemoveContainer" containerID="c32f42ddcec9737f256099a0b59ae80d4c6839182dc024cc92b7b3f21146546c"
Aug 05 18:09:20.098714 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.098683    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:09:20.195175 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.195143    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db64709d-92b9-4ec7-a5cb-f4408befd5ea-istio-envoy\") pod \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") "
Aug 05 18:09:20.195175 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.195177    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db64709d-92b9-4ec7-a5cb-f4408befd5ea-workload-socket\") pod \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") "
Aug 05 18:09:20.195435 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.195196    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db64709d-92b9-4ec7-a5cb-f4408befd5ea-istio-token\") pod \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") "
Aug 05 18:09:20.195435 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.195250    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db64709d-92b9-4ec7-a5cb-f4408befd5ea-credential-socket\") pod \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") "
Aug 05 18:09:20.195435 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.195278    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db64709d-92b9-4ec7-a5cb-f4408befd5ea-istiod-ca-cert\") pod \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") "
Aug 05 18:09:20.195435 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.195320    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db64709d-92b9-4ec7-a5cb-f4408befd5ea-istio-data\") pod \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") "
Aug 05 18:09:20.195435 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.195343    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db64709d-92b9-4ec7-a5cb-f4408befd5ea-istio-podinfo\") pod \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") "
Aug 05 18:09:20.195435 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.195380    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db64709d-92b9-4ec7-a5cb-f4408befd5ea-workload-certs\") pod \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") "
Aug 05 18:09:20.195435 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.195401    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tq25\" (UniqueName: \"kubernetes.io/projected/db64709d-92b9-4ec7-a5cb-f4408befd5ea-kube-api-access-2tq25\") pod \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\" (UID: \"db64709d-92b9-4ec7-a5cb-f4408befd5ea\") "
Aug 05 18:09:20.195828 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.195479    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db64709d-92b9-4ec7-a5cb-f4408befd5ea-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "db64709d-92b9-4ec7-a5cb-f4408befd5ea" (UID: "db64709d-92b9-4ec7-a5cb-f4408befd5ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:09:20.195828 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.195555    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db64709d-92b9-4ec7-a5cb-f4408befd5ea-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "db64709d-92b9-4ec7-a5cb-f4408befd5ea" (UID: "db64709d-92b9-4ec7-a5cb-f4408befd5ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:09:20.195828 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.195705    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db64709d-92b9-4ec7-a5cb-f4408befd5ea-istio-data" (OuterVolumeSpecName: "istio-data") pod "db64709d-92b9-4ec7-a5cb-f4408befd5ea" (UID: "db64709d-92b9-4ec7-a5cb-f4408befd5ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:09:20.195828 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.195720    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db64709d-92b9-4ec7-a5cb-f4408befd5ea-credential-socket\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:09:20.195828 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.195741    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db64709d-92b9-4ec7-a5cb-f4408befd5ea-workload-socket\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:09:20.196137 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.195882    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db64709d-92b9-4ec7-a5cb-f4408befd5ea-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "db64709d-92b9-4ec7-a5cb-f4408befd5ea" (UID: "db64709d-92b9-4ec7-a5cb-f4408befd5ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:09:20.196137 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.195977    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db64709d-92b9-4ec7-a5cb-f4408befd5ea-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "db64709d-92b9-4ec7-a5cb-f4408befd5ea" (UID: "db64709d-92b9-4ec7-a5cb-f4408befd5ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 18:09:20.197810 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.197770    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db64709d-92b9-4ec7-a5cb-f4408befd5ea-istio-token" (OuterVolumeSpecName: "istio-token") pod "db64709d-92b9-4ec7-a5cb-f4408befd5ea" (UID: "db64709d-92b9-4ec7-a5cb-f4408befd5ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 18:09:20.198083 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.198034    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db64709d-92b9-4ec7-a5cb-f4408befd5ea-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "db64709d-92b9-4ec7-a5cb-f4408befd5ea" (UID: "db64709d-92b9-4ec7-a5cb-f4408befd5ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:09:20.198179 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.198080    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db64709d-92b9-4ec7-a5cb-f4408befd5ea-kube-api-access-2tq25" (OuterVolumeSpecName: "kube-api-access-2tq25") pod "db64709d-92b9-4ec7-a5cb-f4408befd5ea" (UID: "db64709d-92b9-4ec7-a5cb-f4408befd5ea"). InnerVolumeSpecName "kube-api-access-2tq25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 18:09:20.198179 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.198125    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db64709d-92b9-4ec7-a5cb-f4408befd5ea-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "db64709d-92b9-4ec7-a5cb-f4408befd5ea" (UID: "db64709d-92b9-4ec7-a5cb-f4408befd5ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Aug 05 18:09:20.296368 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.296325    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db64709d-92b9-4ec7-a5cb-f4408befd5ea-istiod-ca-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:09:20.296368 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.296369    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db64709d-92b9-4ec7-a5cb-f4408befd5ea-istio-data\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:09:20.296589 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.296384    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db64709d-92b9-4ec7-a5cb-f4408befd5ea-istio-podinfo\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:09:20.296589 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.296397    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db64709d-92b9-4ec7-a5cb-f4408befd5ea-workload-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:09:20.296589 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.296412    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tq25\" (UniqueName: \"kubernetes.io/projected/db64709d-92b9-4ec7-a5cb-f4408befd5ea-kube-api-access-2tq25\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:09:20.296589 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.296425    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db64709d-92b9-4ec7-a5cb-f4408befd5ea-istio-envoy\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:09:20.296589 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.296438    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db64709d-92b9-4ec7-a5cb-f4408befd5ea-istio-token\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:09:20.941590 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.941557    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm" event={"ID":"db64709d-92b9-4ec7-a5cb-f4408befd5ea","Type":"ContainerDied","Data":"205888ea231efa210865fafeda9bf679366d7b64044ba0f194919cc0882e3a34"}
Aug 05 18:09:20.942109 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.941610    2561 scope.go:117] "RemoveContainer" containerID="5033a7b3a83429d0d0debb32a66b95bd5981837799adc156bf2df252318cc64b"
Aug 05 18:09:20.942109 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.941561    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"
Aug 05 18:09:20.960688 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.960652    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"]
Aug 05 18:09:20.965679 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:20.965653    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-6cc7987576-64thm"]
Aug 05 18:09:22.730132 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:22.730099    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db64709d-92b9-4ec7-a5cb-f4408befd5ea" path="/var/lib/kubelet/pods/db64709d-92b9-4ec7-a5cb-f4408befd5ea/volumes"
Aug 05 18:09:23.072474 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.072433    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"]
Aug 05 18:09:23.072812 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.072799    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0b046be-679d-4550-b677-486a1f64e610" containerName="storage-initializer"
Aug 05 18:09:23.072869 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.072813    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0b046be-679d-4550-b677-486a1f64e610" containerName="storage-initializer"
Aug 05 18:09:23.072869 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.072828    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db64709d-92b9-4ec7-a5cb-f4408befd5ea" containerName="istio-proxy"
Aug 05 18:09:23.072869 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.072834    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="db64709d-92b9-4ec7-a5cb-f4408befd5ea" containerName="istio-proxy"
Aug 05 18:09:23.072869 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.072841    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0b046be-679d-4550-b677-486a1f64e610" containerName="main"
Aug 05 18:09:23.072869 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.072847    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0b046be-679d-4550-b677-486a1f64e610" containerName="main"
Aug 05 18:09:23.073100 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.072901    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0b046be-679d-4550-b677-486a1f64e610" containerName="main"
Aug 05 18:09:23.073100 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.072910    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="db64709d-92b9-4ec7-a5cb-f4408befd5ea" containerName="istio-proxy"
Aug 05 18:09:23.073100 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.072916    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="db64709d-92b9-4ec7-a5cb-f4408befd5ea" containerName="istio-proxy"
Aug 05 18:09:23.073100 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.073016    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db64709d-92b9-4ec7-a5cb-f4408befd5ea" containerName="istio-proxy"
Aug 05 18:09:23.073100 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.073023    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="db64709d-92b9-4ec7-a5cb-f4408befd5ea" containerName="istio-proxy"
Aug 05 18:09:23.075903 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.075881    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:09:23.078678 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.078651    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Aug 05 18:09:23.078817 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.078653    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-bsf2m\""
Aug 05 18:09:23.078817 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.078653    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Aug 05 18:09:23.078817 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.078721    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Aug 05 18:09:23.079511 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.079486    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Aug 05 18:09:23.079511 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.079498    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-mwxt8\""
Aug 05 18:09:23.079748 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.079728    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Aug 05 18:09:23.089280 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.089251    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"]
Aug 05 18:09:23.233079 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.233035    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d7a0e6-a2c5-417c-af31-061f926af0ed-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:09:23.233269 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.233086    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7d7a0e6-a2c5-417c-af31-061f926af0ed-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:09:23.233269 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.233161    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d7a0e6-a2c5-417c-af31-061f926af0ed-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:09:23.233269 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.233224    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gppk\" (UniqueName: \"kubernetes.io/projected/c7d7a0e6-a2c5-417c-af31-061f926af0ed-kube-api-access-2gppk\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:09:23.233435 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.233285    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d7a0e6-a2c5-417c-af31-061f926af0ed-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:09:23.233435 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.233329    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7d7a0e6-a2c5-417c-af31-061f926af0ed-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:09:23.233435 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.233346    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d7a0e6-a2c5-417c-af31-061f926af0ed-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:09:23.233435 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.233390    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d7a0e6-a2c5-417c-af31-061f926af0ed-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:09:23.334437 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.334352    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7d7a0e6-a2c5-417c-af31-061f926af0ed-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:09:23.334437 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.334392    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d7a0e6-a2c5-417c-af31-061f926af0ed-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:09:23.334437 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.334429    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d7a0e6-a2c5-417c-af31-061f926af0ed-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:09:23.334736 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.334472    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d7a0e6-a2c5-417c-af31-061f926af0ed-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:09:23.334736 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.334495    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7d7a0e6-a2c5-417c-af31-061f926af0ed-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:09:23.334736 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.334528    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d7a0e6-a2c5-417c-af31-061f926af0ed-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:09:23.334736 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.334570    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gppk\" (UniqueName: \"kubernetes.io/projected/c7d7a0e6-a2c5-417c-af31-061f926af0ed-kube-api-access-2gppk\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:09:23.334736 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.334622    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d7a0e6-a2c5-417c-af31-061f926af0ed-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:09:23.335036 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.334979    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d7a0e6-a2c5-417c-af31-061f926af0ed-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:09:23.335036 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.335009    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7d7a0e6-a2c5-417c-af31-061f926af0ed-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:09:23.335259 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.335233    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7d7a0e6-a2c5-417c-af31-061f926af0ed-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:09:23.335259 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.335247    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d7a0e6-a2c5-417c-af31-061f926af0ed-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:09:23.335510 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.335486    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d7a0e6-a2c5-417c-af31-061f926af0ed-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:09:23.337330 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.337301    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d7a0e6-a2c5-417c-af31-061f926af0ed-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:09:23.337430 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.337365    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d7a0e6-a2c5-417c-af31-061f926af0ed-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:09:23.342807 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.342778    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gppk\" (UniqueName: \"kubernetes.io/projected/c7d7a0e6-a2c5-417c-af31-061f926af0ed-kube-api-access-2gppk\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:09:23.386901 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.386866    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:09:23.552340 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.552311    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"]
Aug 05 18:09:23.553718 ip-10-0-135-151 kubenswrapper[2561]: W0805 18:09:23.553692    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d7a0e6_a2c5_417c_af31_061f926af0ed.slice/crio-ddbceb0781cd9b0db484a4dada3760412fb760248657908319e9fe77f028bb33 WatchSource:0}: Error finding container ddbceb0781cd9b0db484a4dada3760412fb760248657908319e9fe77f028bb33: Status 404 returned error can't find the container with id ddbceb0781cd9b0db484a4dada3760412fb760248657908319e9fe77f028bb33
Aug 05 18:09:23.967165 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.967048    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs" event={"ID":"c7d7a0e6-a2c5-417c-af31-061f926af0ed","Type":"ContainerStarted","Data":"8167ad5dae6a8d3148a666353367e0cc2305ac851487ed90e9de16583ec9853d"}
Aug 05 18:09:23.967165 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.967113    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs" event={"ID":"c7d7a0e6-a2c5-417c-af31-061f926af0ed","Type":"ContainerStarted","Data":"ddbceb0781cd9b0db484a4dada3760412fb760248657908319e9fe77f028bb33"}
Aug 05 18:09:23.967650 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:23.967263    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:09:24.978298 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:24.978254    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs" event={"ID":"c7d7a0e6-a2c5-417c-af31-061f926af0ed","Type":"ContainerStarted","Data":"5d2bbd6d3671a4905f8477f07a73479a96b8a3e6408ff03d1ace72b9753e29fb"}
Aug 05 18:09:27.000229 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:27.000134    2561 generic.go:356] "Generic (PLEG): container finished" podID="c7d7a0e6-a2c5-417c-af31-061f926af0ed" containerID="5d2bbd6d3671a4905f8477f07a73479a96b8a3e6408ff03d1ace72b9753e29fb" exitCode=0
Aug 05 18:09:27.000229 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:27.000193    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs" event={"ID":"c7d7a0e6-a2c5-417c-af31-061f926af0ed","Type":"ContainerDied","Data":"5d2bbd6d3671a4905f8477f07a73479a96b8a3e6408ff03d1ace72b9753e29fb"}
Aug 05 18:09:28.014116 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:28.014056    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs" event={"ID":"c7d7a0e6-a2c5-417c-af31-061f926af0ed","Type":"ContainerStarted","Data":"962597c2bd2e3ce554319332643223808f661158832e4e3e363289959bf6539f"}
Aug 05 18:09:28.040286 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:28.040224    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs" podStartSLOduration=2.539811489 podStartE2EDuration="5.040203148s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.500391659s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:09:28.038564118 +0000 UTC m=+3423.783198440" watchObservedRunningTime="2026-08-05 18:09:28.040203148 +0000 UTC m=+3423.784837476"
Aug 05 18:09:33.389804 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:33.389765    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:09:33.392871 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:33.392789    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:09:33.393202 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:33.393165    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs" podUID="c7d7a0e6-a2c5-417c-af31-061f926af0ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8001/health\": dial tcp 10.133.0.106:8001: connect: connection refused"
Aug 05 18:09:33.416708 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:33.416675    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:09:33.849239 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:33.849205    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5_6129ebd7-73bf-4a8f-a208-6047e0a486ca/storage-initializer/0.log"
Aug 05 18:09:33.860697 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:33.860661    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5_6129ebd7-73bf-4a8f-a208-6047e0a486ca/main/0.log"
Aug 05 18:09:40.192536 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:40.192494    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"]
Aug 05 18:09:43.387538 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:43.387493    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs" podUID="c7d7a0e6-a2c5-417c-af31-061f926af0ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8001/health\": dial tcp 10.133.0.106:8001: connect: connection refused"
Aug 05 18:09:53.388013 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:53.387929    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs" podUID="c7d7a0e6-a2c5-417c-af31-061f926af0ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8001/health\": dial tcp 10.133.0.106:8001: connect: connection refused"
Aug 05 18:09:55.216164 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:55.216092    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5" podUID="6129ebd7-73bf-4a8f-a208-6047e0a486ca" containerName="main" containerID="cri-o://3aa87fb534115727d25820e99cdbce31b4528b5f9862ea317a990f0505100416" gracePeriod=45
Aug 05 18:09:55.878253 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:55.878228    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:09:55.965468 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:55.965383    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6129ebd7-73bf-4a8f-a208-6047e0a486ca-home\") pod \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") "
Aug 05 18:09:55.965468 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:55.965425    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6129ebd7-73bf-4a8f-a208-6047e0a486ca-cabundle-cert\") pod \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") "
Aug 05 18:09:55.965468 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:55.965466    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6129ebd7-73bf-4a8f-a208-6047e0a486ca-model-cache\") pod \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") "
Aug 05 18:09:55.965769 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:55.965484    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6129ebd7-73bf-4a8f-a208-6047e0a486ca-dshm\") pod \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") "
Aug 05 18:09:55.965769 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:55.965552    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6129ebd7-73bf-4a8f-a208-6047e0a486ca-tmp-dir\") pod \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") "
Aug 05 18:09:55.965769 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:55.965577    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjdqm\" (UniqueName: \"kubernetes.io/projected/6129ebd7-73bf-4a8f-a208-6047e0a486ca-kube-api-access-hjdqm\") pod \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") "
Aug 05 18:09:55.965769 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:55.965637    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6129ebd7-73bf-4a8f-a208-6047e0a486ca-tls-certs\") pod \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") "
Aug 05 18:09:55.965769 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:55.965662    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6129ebd7-73bf-4a8f-a208-6047e0a486ca-kserve-provision-location\") pod \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\" (UID: \"6129ebd7-73bf-4a8f-a208-6047e0a486ca\") "
Aug 05 18:09:55.966076 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:55.965787    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6129ebd7-73bf-4a8f-a208-6047e0a486ca-model-cache" (OuterVolumeSpecName: "model-cache") pod "6129ebd7-73bf-4a8f-a208-6047e0a486ca" (UID: "6129ebd7-73bf-4a8f-a208-6047e0a486ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:09:55.966076 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:55.965820    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6129ebd7-73bf-4a8f-a208-6047e0a486ca-home" (OuterVolumeSpecName: "home") pod "6129ebd7-73bf-4a8f-a208-6047e0a486ca" (UID: "6129ebd7-73bf-4a8f-a208-6047e0a486ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:09:55.966076 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:55.965860    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6129ebd7-73bf-4a8f-a208-6047e0a486ca-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6129ebd7-73bf-4a8f-a208-6047e0a486ca" (UID: "6129ebd7-73bf-4a8f-a208-6047e0a486ca"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 18:09:55.966076 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:55.966005    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6129ebd7-73bf-4a8f-a208-6047e0a486ca-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:09:55.966076 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:55.966024    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6129ebd7-73bf-4a8f-a208-6047e0a486ca-cabundle-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:09:55.966076 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:55.966037    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6129ebd7-73bf-4a8f-a208-6047e0a486ca-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:09:55.968215 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:55.968174    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6129ebd7-73bf-4a8f-a208-6047e0a486ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6129ebd7-73bf-4a8f-a208-6047e0a486ca" (UID: "6129ebd7-73bf-4a8f-a208-6047e0a486ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 18:09:55.968340 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:55.968249    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6129ebd7-73bf-4a8f-a208-6047e0a486ca-dshm" (OuterVolumeSpecName: "dshm") pod "6129ebd7-73bf-4a8f-a208-6047e0a486ca" (UID: "6129ebd7-73bf-4a8f-a208-6047e0a486ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:09:55.968534 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:55.968506    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6129ebd7-73bf-4a8f-a208-6047e0a486ca-kube-api-access-hjdqm" (OuterVolumeSpecName: "kube-api-access-hjdqm") pod "6129ebd7-73bf-4a8f-a208-6047e0a486ca" (UID: "6129ebd7-73bf-4a8f-a208-6047e0a486ca"). InnerVolumeSpecName "kube-api-access-hjdqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 18:09:55.986996 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:55.986932    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6129ebd7-73bf-4a8f-a208-6047e0a486ca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6129ebd7-73bf-4a8f-a208-6047e0a486ca" (UID: "6129ebd7-73bf-4a8f-a208-6047e0a486ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:09:55.998849 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:55.998813    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6129ebd7-73bf-4a8f-a208-6047e0a486ca-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6129ebd7-73bf-4a8f-a208-6047e0a486ca" (UID: "6129ebd7-73bf-4a8f-a208-6047e0a486ca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:09:56.066551 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:56.066521    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6129ebd7-73bf-4a8f-a208-6047e0a486ca-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:09:56.066551 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:56.066549    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6129ebd7-73bf-4a8f-a208-6047e0a486ca-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:09:56.066551 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:56.066560    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjdqm\" (UniqueName: \"kubernetes.io/projected/6129ebd7-73bf-4a8f-a208-6047e0a486ca-kube-api-access-hjdqm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:09:56.066799 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:56.066570    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6129ebd7-73bf-4a8f-a208-6047e0a486ca-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:09:56.066799 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:56.066578    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6129ebd7-73bf-4a8f-a208-6047e0a486ca-kserve-provision-location\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:09:56.180678 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:56.180639    2561 generic.go:356] "Generic (PLEG): container finished" podID="6129ebd7-73bf-4a8f-a208-6047e0a486ca" containerID="3aa87fb534115727d25820e99cdbce31b4528b5f9862ea317a990f0505100416" exitCode=0
Aug 05 18:09:56.180851 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:56.180685    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"
Aug 05 18:09:56.180851 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:56.180720    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5" event={"ID":"6129ebd7-73bf-4a8f-a208-6047e0a486ca","Type":"ContainerDied","Data":"3aa87fb534115727d25820e99cdbce31b4528b5f9862ea317a990f0505100416"}
Aug 05 18:09:56.180851 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:56.180780    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5" event={"ID":"6129ebd7-73bf-4a8f-a208-6047e0a486ca","Type":"ContainerDied","Data":"c4e264115a6b434c5554e315db4cc78261953ff1a4f24adfbc2d97dd87adb4eb"}
Aug 05 18:09:56.180851 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:56.180804    2561 scope.go:117] "RemoveContainer" containerID="3aa87fb534115727d25820e99cdbce31b4528b5f9862ea317a990f0505100416"
Aug 05 18:09:56.197807 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:56.197770    2561 scope.go:117] "RemoveContainer" containerID="957aed06403aaa7b08d3e2e03bd5bb24604866bf8dc295ac680b3a37930dff6f"
Aug 05 18:09:56.207276 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:56.207246    2561 scope.go:117] "RemoveContainer" containerID="3aa87fb534115727d25820e99cdbce31b4528b5f9862ea317a990f0505100416"
Aug 05 18:09:56.207774 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:09:56.207746    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aa87fb534115727d25820e99cdbce31b4528b5f9862ea317a990f0505100416\": container with ID starting with 3aa87fb534115727d25820e99cdbce31b4528b5f9862ea317a990f0505100416 not found: ID does not exist" containerID="3aa87fb534115727d25820e99cdbce31b4528b5f9862ea317a990f0505100416"
Aug 05 18:09:56.207858 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:56.207785    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aa87fb534115727d25820e99cdbce31b4528b5f9862ea317a990f0505100416"} err="failed to get container status \"3aa87fb534115727d25820e99cdbce31b4528b5f9862ea317a990f0505100416\": rpc error: code = NotFound desc = could not find container \"3aa87fb534115727d25820e99cdbce31b4528b5f9862ea317a990f0505100416\": container with ID starting with 3aa87fb534115727d25820e99cdbce31b4528b5f9862ea317a990f0505100416 not found: ID does not exist"
Aug 05 18:09:56.207858 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:56.207810    2561 scope.go:117] "RemoveContainer" containerID="957aed06403aaa7b08d3e2e03bd5bb24604866bf8dc295ac680b3a37930dff6f"
Aug 05 18:09:56.208179 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:09:56.208153    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"957aed06403aaa7b08d3e2e03bd5bb24604866bf8dc295ac680b3a37930dff6f\": container with ID starting with 957aed06403aaa7b08d3e2e03bd5bb24604866bf8dc295ac680b3a37930dff6f not found: ID does not exist" containerID="957aed06403aaa7b08d3e2e03bd5bb24604866bf8dc295ac680b3a37930dff6f"
Aug 05 18:09:56.208282 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:56.208187    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"957aed06403aaa7b08d3e2e03bd5bb24604866bf8dc295ac680b3a37930dff6f"} err="failed to get container status \"957aed06403aaa7b08d3e2e03bd5bb24604866bf8dc295ac680b3a37930dff6f\": rpc error: code = NotFound desc = could not find container \"957aed06403aaa7b08d3e2e03bd5bb24604866bf8dc295ac680b3a37930dff6f\": container with ID starting with 957aed06403aaa7b08d3e2e03bd5bb24604866bf8dc295ac680b3a37930dff6f not found: ID does not exist"
Aug 05 18:09:56.209849 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:56.209816    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"]
Aug 05 18:09:56.212071 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:56.212051    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-57dc4bd888-qzwx5"]
Aug 05 18:09:56.731108 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:09:56.731074    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6129ebd7-73bf-4a8f-a208-6047e0a486ca" path="/var/lib/kubelet/pods/6129ebd7-73bf-4a8f-a208-6047e0a486ca/volumes"
Aug 05 18:10:03.388137 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:03.388093    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs" podUID="c7d7a0e6-a2c5-417c-af31-061f926af0ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8001/health\": dial tcp 10.133.0.106:8001: connect: connection refused"
Aug 05 18:10:08.096479 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.096437    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"]
Aug 05 18:10:08.097088 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.097069    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6129ebd7-73bf-4a8f-a208-6047e0a486ca" containerName="storage-initializer"
Aug 05 18:10:08.097143 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.097092    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6129ebd7-73bf-4a8f-a208-6047e0a486ca" containerName="storage-initializer"
Aug 05 18:10:08.097143 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.097109    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6129ebd7-73bf-4a8f-a208-6047e0a486ca" containerName="main"
Aug 05 18:10:08.097143 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.097118    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6129ebd7-73bf-4a8f-a208-6047e0a486ca" containerName="main"
Aug 05 18:10:08.097238 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.097219    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6129ebd7-73bf-4a8f-a208-6047e0a486ca" containerName="main"
Aug 05 18:10:08.102807 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.102781    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:10:08.106262 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.106234    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Aug 05 18:10:08.107507 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.107481    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Aug 05 18:10:08.107735 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.107717    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-mlptb\""
Aug 05 18:10:08.107971 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.107937    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Aug 05 18:10:08.108746 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.108194    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Aug 05 18:10:08.108746 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.108311    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Aug 05 18:10:08.110132 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.110105    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"]
Aug 05 18:10:08.171624 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.171590    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0027bdda-0419-4432-9da4-41ba49aa3aee-tmp-dir\") pod \"tls-verification-test-kserve-84698fc874-jgbsv\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:10:08.171624 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.171627    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0027bdda-0419-4432-9da4-41ba49aa3aee-tls-certs\") pod \"tls-verification-test-kserve-84698fc874-jgbsv\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:10:08.171857 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.171651    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0027bdda-0419-4432-9da4-41ba49aa3aee-cabundle-cert\") pod \"tls-verification-test-kserve-84698fc874-jgbsv\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:10:08.171857 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.171669    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0027bdda-0419-4432-9da4-41ba49aa3aee-model-cache\") pod \"tls-verification-test-kserve-84698fc874-jgbsv\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:10:08.171857 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.171715    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0027bdda-0419-4432-9da4-41ba49aa3aee-dshm\") pod \"tls-verification-test-kserve-84698fc874-jgbsv\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:10:08.171857 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.171774    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0027bdda-0419-4432-9da4-41ba49aa3aee-kserve-provision-location\") pod \"tls-verification-test-kserve-84698fc874-jgbsv\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:10:08.171857 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.171796    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlwg8\" (UniqueName: \"kubernetes.io/projected/0027bdda-0419-4432-9da4-41ba49aa3aee-kube-api-access-wlwg8\") pod \"tls-verification-test-kserve-84698fc874-jgbsv\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:10:08.171857 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.171844    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0027bdda-0419-4432-9da4-41ba49aa3aee-home\") pod \"tls-verification-test-kserve-84698fc874-jgbsv\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:10:08.273122 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.273086    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0027bdda-0419-4432-9da4-41ba49aa3aee-tmp-dir\") pod \"tls-verification-test-kserve-84698fc874-jgbsv\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:10:08.273122 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.273121    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0027bdda-0419-4432-9da4-41ba49aa3aee-tls-certs\") pod \"tls-verification-test-kserve-84698fc874-jgbsv\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:10:08.273381 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.273154    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0027bdda-0419-4432-9da4-41ba49aa3aee-cabundle-cert\") pod \"tls-verification-test-kserve-84698fc874-jgbsv\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:10:08.273381 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.273173    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0027bdda-0419-4432-9da4-41ba49aa3aee-model-cache\") pod \"tls-verification-test-kserve-84698fc874-jgbsv\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:10:08.273381 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.273301    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0027bdda-0419-4432-9da4-41ba49aa3aee-dshm\") pod \"tls-verification-test-kserve-84698fc874-jgbsv\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:10:08.273381 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.273371    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0027bdda-0419-4432-9da4-41ba49aa3aee-kserve-provision-location\") pod \"tls-verification-test-kserve-84698fc874-jgbsv\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:10:08.273687 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.273407    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlwg8\" (UniqueName: \"kubernetes.io/projected/0027bdda-0419-4432-9da4-41ba49aa3aee-kube-api-access-wlwg8\") pod \"tls-verification-test-kserve-84698fc874-jgbsv\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:10:08.273687 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.273458    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0027bdda-0419-4432-9da4-41ba49aa3aee-home\") pod \"tls-verification-test-kserve-84698fc874-jgbsv\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:10:08.273687 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.273588    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0027bdda-0419-4432-9da4-41ba49aa3aee-tmp-dir\") pod \"tls-verification-test-kserve-84698fc874-jgbsv\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:10:08.273687 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.273605    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0027bdda-0419-4432-9da4-41ba49aa3aee-model-cache\") pod \"tls-verification-test-kserve-84698fc874-jgbsv\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:10:08.273905 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.273789    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0027bdda-0419-4432-9da4-41ba49aa3aee-home\") pod \"tls-verification-test-kserve-84698fc874-jgbsv\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:10:08.273905 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.273849    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0027bdda-0419-4432-9da4-41ba49aa3aee-kserve-provision-location\") pod \"tls-verification-test-kserve-84698fc874-jgbsv\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:10:08.274108 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.274087    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0027bdda-0419-4432-9da4-41ba49aa3aee-cabundle-cert\") pod \"tls-verification-test-kserve-84698fc874-jgbsv\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:10:08.276046 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.276024    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0027bdda-0419-4432-9da4-41ba49aa3aee-dshm\") pod \"tls-verification-test-kserve-84698fc874-jgbsv\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:10:08.276463 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.276442    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0027bdda-0419-4432-9da4-41ba49aa3aee-tls-certs\") pod \"tls-verification-test-kserve-84698fc874-jgbsv\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:10:08.282600 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.282553    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlwg8\" (UniqueName: \"kubernetes.io/projected/0027bdda-0419-4432-9da4-41ba49aa3aee-kube-api-access-wlwg8\") pod \"tls-verification-test-kserve-84698fc874-jgbsv\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:10:08.417604 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.417502    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:10:08.570014 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:08.569975    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"]
Aug 05 18:10:08.571892 ip-10-0-135-151 kubenswrapper[2561]: W0805 18:10:08.571856    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0027bdda_0419_4432_9da4_41ba49aa3aee.slice/crio-0692ab558a011d3e9861209973bc2cec5498124764df52a18682faed8eb36708 WatchSource:0}: Error finding container 0692ab558a011d3e9861209973bc2cec5498124764df52a18682faed8eb36708: Status 404 returned error can't find the container with id 0692ab558a011d3e9861209973bc2cec5498124764df52a18682faed8eb36708
Aug 05 18:10:09.255982 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:09.255917    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv" event={"ID":"0027bdda-0419-4432-9da4-41ba49aa3aee","Type":"ContainerStarted","Data":"0ef4b6e0b5a6c12a1da625179b3e83c4ca132187dff30c017917586718926212"}
Aug 05 18:10:09.256368 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:09.256046    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv" event={"ID":"0027bdda-0419-4432-9da4-41ba49aa3aee","Type":"ContainerStarted","Data":"0692ab558a011d3e9861209973bc2cec5498124764df52a18682faed8eb36708"}
Aug 05 18:10:11.271903 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:11.271870    2561 generic.go:356] "Generic (PLEG): container finished" podID="0027bdda-0419-4432-9da4-41ba49aa3aee" containerID="0ef4b6e0b5a6c12a1da625179b3e83c4ca132187dff30c017917586718926212" exitCode=0
Aug 05 18:10:11.272310 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:11.271966    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv" event={"ID":"0027bdda-0419-4432-9da4-41ba49aa3aee","Type":"ContainerDied","Data":"0ef4b6e0b5a6c12a1da625179b3e83c4ca132187dff30c017917586718926212"}
Aug 05 18:10:12.286267 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:12.286231    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv" event={"ID":"0027bdda-0419-4432-9da4-41ba49aa3aee","Type":"ContainerStarted","Data":"ca0187b5eb227bea52b89c0a84da5dea74fa272a86347cbcb31084935ccfa0cd"}
Aug 05 18:10:12.309692 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:12.309627    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv" podStartSLOduration=1.880358277 podStartE2EDuration="4.309606991s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.429248714s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:10:12.3070598 +0000 UTC m=+3468.051694147" watchObservedRunningTime="2026-08-05 18:10:12.309606991 +0000 UTC m=+3468.054241319"
Aug 05 18:10:12.418650 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:12.418606    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:10:13.388256 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:13.388200    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs" podUID="c7d7a0e6-a2c5-417c-af31-061f926af0ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8001/health\": dial tcp 10.133.0.106:8001: connect: connection refused"
Aug 05 18:10:18.418151 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:18.418101    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:10:18.419754 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:18.419719    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv" podUID="0027bdda-0419-4432-9da4-41ba49aa3aee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/health\": dial tcp 10.133.0.107:8000: connect: connection refused"
Aug 05 18:10:23.388414 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:23.388357    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs" podUID="c7d7a0e6-a2c5-417c-af31-061f926af0ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8001/health\": dial tcp 10.133.0.106:8001: connect: connection refused"
Aug 05 18:10:28.419028 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:28.418979    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv" podUID="0027bdda-0419-4432-9da4-41ba49aa3aee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/health\": dial tcp 10.133.0.107:8000: connect: connection refused"
Aug 05 18:10:33.388395 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:33.388351    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs" podUID="c7d7a0e6-a2c5-417c-af31-061f926af0ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8001/health\": dial tcp 10.133.0.106:8001: connect: connection refused"
Aug 05 18:10:38.417968 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:38.417913    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv" podUID="0027bdda-0419-4432-9da4-41ba49aa3aee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/health\": dial tcp 10.133.0.107:8000: connect: connection refused"
Aug 05 18:10:43.387494 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:43.387442    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs" podUID="c7d7a0e6-a2c5-417c-af31-061f926af0ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8001/health\": dial tcp 10.133.0.106:8001: connect: connection refused"
Aug 05 18:10:48.418592 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:48.418474    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv" podUID="0027bdda-0419-4432-9da4-41ba49aa3aee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/health\": dial tcp 10.133.0.107:8000: connect: connection refused"
Aug 05 18:10:53.387616 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:53.387566    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs" podUID="c7d7a0e6-a2c5-417c-af31-061f926af0ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8001/health\": dial tcp 10.133.0.106:8001: connect: connection refused"
Aug 05 18:10:58.418398 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:10:58.418355    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv" podUID="0027bdda-0419-4432-9da4-41ba49aa3aee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/health\": dial tcp 10.133.0.107:8000: connect: connection refused"
Aug 05 18:11:03.388190 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:03.388128    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs" podUID="c7d7a0e6-a2c5-417c-af31-061f926af0ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8001/health\": dial tcp 10.133.0.106:8001: connect: connection refused"
Aug 05 18:11:08.418706 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:08.418644    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv" podUID="0027bdda-0419-4432-9da4-41ba49aa3aee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/health\": dial tcp 10.133.0.107:8000: connect: connection refused"
Aug 05 18:11:13.388069 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:13.388008    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs" podUID="c7d7a0e6-a2c5-417c-af31-061f926af0ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.106:8001/health\": dial tcp 10.133.0.106:8001: connect: connection refused"
Aug 05 18:11:18.418388 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:18.418331    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv" podUID="0027bdda-0419-4432-9da4-41ba49aa3aee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/health\": dial tcp 10.133.0.107:8000: connect: connection refused"
Aug 05 18:11:23.397903 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:23.397869    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:11:23.416918 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:23.416894    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:11:28.418259 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:28.418206    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv" podUID="0027bdda-0419-4432-9da4-41ba49aa3aee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/health\": dial tcp 10.133.0.107:8000: connect: connection refused"
Aug 05 18:11:35.471329 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:35.471294    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"]
Aug 05 18:11:38.418203 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:38.418161    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv" podUID="0027bdda-0419-4432-9da4-41ba49aa3aee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/health\": dial tcp 10.133.0.107:8000: connect: connection refused"
Aug 05 18:11:48.418597 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:48.418546    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv" podUID="0027bdda-0419-4432-9da4-41ba49aa3aee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/health\": dial tcp 10.133.0.107:8000: connect: connection refused"
Aug 05 18:11:50.494590 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:50.494528    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs" podUID="c7d7a0e6-a2c5-417c-af31-061f926af0ed" containerName="main" containerID="cri-o://962597c2bd2e3ce554319332643223808f661158832e4e3e363289959bf6539f" gracePeriod=45
Aug 05 18:11:51.411624 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.411564    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs" podUID="c7d7a0e6-a2c5-417c-af31-061f926af0ed" containerName="llm-d-routing-sidecar" containerID="cri-o://8167ad5dae6a8d3148a666353367e0cc2305ac851487ed90e9de16583ec9853d" gracePeriod=45
Aug 05 18:11:51.665771 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.665746    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:11:51.778706 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.778674    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d7a0e6-a2c5-417c-af31-061f926af0ed-tmp-dir\") pod \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") "
Aug 05 18:11:51.778706 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.778708    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gppk\" (UniqueName: \"kubernetes.io/projected/c7d7a0e6-a2c5-417c-af31-061f926af0ed-kube-api-access-2gppk\") pod \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") "
Aug 05 18:11:51.778945 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.778737    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7d7a0e6-a2c5-417c-af31-061f926af0ed-cabundle-cert\") pod \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") "
Aug 05 18:11:51.778945 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.778874    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d7a0e6-a2c5-417c-af31-061f926af0ed-home\") pod \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") "
Aug 05 18:11:51.778945 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.778923    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d7a0e6-a2c5-417c-af31-061f926af0ed-dshm\") pod \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") "
Aug 05 18:11:51.779139 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.778975    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7d7a0e6-a2c5-417c-af31-061f926af0ed-kserve-provision-location\") pod \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") "
Aug 05 18:11:51.779139 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.779028    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d7a0e6-a2c5-417c-af31-061f926af0ed-model-cache\") pod \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") "
Aug 05 18:11:51.779139 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.779071    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d7a0e6-a2c5-417c-af31-061f926af0ed-tls-certs\") pod \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\" (UID: \"c7d7a0e6-a2c5-417c-af31-061f926af0ed\") "
Aug 05 18:11:51.779303 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.779149    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7d7a0e6-a2c5-417c-af31-061f926af0ed-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c7d7a0e6-a2c5-417c-af31-061f926af0ed" (UID: "c7d7a0e6-a2c5-417c-af31-061f926af0ed"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 18:11:51.779518 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.779472    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7d7a0e6-a2c5-417c-af31-061f926af0ed-cabundle-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:11:51.779518 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.779487    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d7a0e6-a2c5-417c-af31-061f926af0ed-model-cache" (OuterVolumeSpecName: "model-cache") pod "c7d7a0e6-a2c5-417c-af31-061f926af0ed" (UID: "c7d7a0e6-a2c5-417c-af31-061f926af0ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:11:51.779728 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.779621    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d7a0e6-a2c5-417c-af31-061f926af0ed-home" (OuterVolumeSpecName: "home") pod "c7d7a0e6-a2c5-417c-af31-061f926af0ed" (UID: "c7d7a0e6-a2c5-417c-af31-061f926af0ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:11:51.781536 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.781513    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7d7a0e6-a2c5-417c-af31-061f926af0ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c7d7a0e6-a2c5-417c-af31-061f926af0ed" (UID: "c7d7a0e6-a2c5-417c-af31-061f926af0ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 18:11:51.781666 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.781645    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d7a0e6-a2c5-417c-af31-061f926af0ed-dshm" (OuterVolumeSpecName: "dshm") pod "c7d7a0e6-a2c5-417c-af31-061f926af0ed" (UID: "c7d7a0e6-a2c5-417c-af31-061f926af0ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:11:51.781727 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.781688    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d7a0e6-a2c5-417c-af31-061f926af0ed-kube-api-access-2gppk" (OuterVolumeSpecName: "kube-api-access-2gppk") pod "c7d7a0e6-a2c5-417c-af31-061f926af0ed" (UID: "c7d7a0e6-a2c5-417c-af31-061f926af0ed"). InnerVolumeSpecName "kube-api-access-2gppk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 18:11:51.797611 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.797568    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d7a0e6-a2c5-417c-af31-061f926af0ed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c7d7a0e6-a2c5-417c-af31-061f926af0ed" (UID: "c7d7a0e6-a2c5-417c-af31-061f926af0ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:11:51.802392 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.802370    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d7a0e6-a2c5-417c-af31-061f926af0ed-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c7d7a0e6-a2c5-417c-af31-061f926af0ed" (UID: "c7d7a0e6-a2c5-417c-af31-061f926af0ed"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:11:51.880627 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.880593    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d7a0e6-a2c5-417c-af31-061f926af0ed-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:11:51.880627 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.880622    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d7a0e6-a2c5-417c-af31-061f926af0ed-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:11:51.880627 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.880632    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7d7a0e6-a2c5-417c-af31-061f926af0ed-kserve-provision-location\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:11:51.880869 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.880643    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d7a0e6-a2c5-417c-af31-061f926af0ed-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:11:51.880869 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.880652    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d7a0e6-a2c5-417c-af31-061f926af0ed-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:11:51.880869 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.880660    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d7a0e6-a2c5-417c-af31-061f926af0ed-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:11:51.880869 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.880667    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gppk\" (UniqueName: \"kubernetes.io/projected/c7d7a0e6-a2c5-417c-af31-061f926af0ed-kube-api-access-2gppk\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:11:51.899115 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.899023    2561 generic.go:356] "Generic (PLEG): container finished" podID="c7d7a0e6-a2c5-417c-af31-061f926af0ed" containerID="962597c2bd2e3ce554319332643223808f661158832e4e3e363289959bf6539f" exitCode=0
Aug 05 18:11:51.899115 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.899049    2561 generic.go:356] "Generic (PLEG): container finished" podID="c7d7a0e6-a2c5-417c-af31-061f926af0ed" containerID="8167ad5dae6a8d3148a666353367e0cc2305ac851487ed90e9de16583ec9853d" exitCode=0
Aug 05 18:11:51.899115 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.899098    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"
Aug 05 18:11:51.899115 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.899104    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs" event={"ID":"c7d7a0e6-a2c5-417c-af31-061f926af0ed","Type":"ContainerDied","Data":"962597c2bd2e3ce554319332643223808f661158832e4e3e363289959bf6539f"}
Aug 05 18:11:51.899429 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.899163    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs" event={"ID":"c7d7a0e6-a2c5-417c-af31-061f926af0ed","Type":"ContainerDied","Data":"8167ad5dae6a8d3148a666353367e0cc2305ac851487ed90e9de16583ec9853d"}
Aug 05 18:11:51.899429 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.899182    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs" event={"ID":"c7d7a0e6-a2c5-417c-af31-061f926af0ed","Type":"ContainerDied","Data":"ddbceb0781cd9b0db484a4dada3760412fb760248657908319e9fe77f028bb33"}
Aug 05 18:11:51.899429 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.899202    2561 scope.go:117] "RemoveContainer" containerID="962597c2bd2e3ce554319332643223808f661158832e4e3e363289959bf6539f"
Aug 05 18:11:51.909745 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.909726    2561 scope.go:117] "RemoveContainer" containerID="5d2bbd6d3671a4905f8477f07a73479a96b8a3e6408ff03d1ace72b9753e29fb"
Aug 05 18:11:51.921224 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.921177    2561 scope.go:117] "RemoveContainer" containerID="8167ad5dae6a8d3148a666353367e0cc2305ac851487ed90e9de16583ec9853d"
Aug 05 18:11:51.925218 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.925160    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"]
Aug 05 18:11:51.928854 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.928828    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-77d4d657dc2t5gs"]
Aug 05 18:11:51.930481 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.930461    2561 scope.go:117] "RemoveContainer" containerID="962597c2bd2e3ce554319332643223808f661158832e4e3e363289959bf6539f"
Aug 05 18:11:51.930751 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:11:51.930730    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"962597c2bd2e3ce554319332643223808f661158832e4e3e363289959bf6539f\": container with ID starting with 962597c2bd2e3ce554319332643223808f661158832e4e3e363289959bf6539f not found: ID does not exist" containerID="962597c2bd2e3ce554319332643223808f661158832e4e3e363289959bf6539f"
Aug 05 18:11:51.930839 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.930758    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"962597c2bd2e3ce554319332643223808f661158832e4e3e363289959bf6539f"} err="failed to get container status \"962597c2bd2e3ce554319332643223808f661158832e4e3e363289959bf6539f\": rpc error: code = NotFound desc = could not find container \"962597c2bd2e3ce554319332643223808f661158832e4e3e363289959bf6539f\": container with ID starting with 962597c2bd2e3ce554319332643223808f661158832e4e3e363289959bf6539f not found: ID does not exist"
Aug 05 18:11:51.930839 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.930779    2561 scope.go:117] "RemoveContainer" containerID="5d2bbd6d3671a4905f8477f07a73479a96b8a3e6408ff03d1ace72b9753e29fb"
Aug 05 18:11:51.931050 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:11:51.931022    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d2bbd6d3671a4905f8477f07a73479a96b8a3e6408ff03d1ace72b9753e29fb\": container with ID starting with 5d2bbd6d3671a4905f8477f07a73479a96b8a3e6408ff03d1ace72b9753e29fb not found: ID does not exist" containerID="5d2bbd6d3671a4905f8477f07a73479a96b8a3e6408ff03d1ace72b9753e29fb"
Aug 05 18:11:51.931134 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.931060    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d2bbd6d3671a4905f8477f07a73479a96b8a3e6408ff03d1ace72b9753e29fb"} err="failed to get container status \"5d2bbd6d3671a4905f8477f07a73479a96b8a3e6408ff03d1ace72b9753e29fb\": rpc error: code = NotFound desc = could not find container \"5d2bbd6d3671a4905f8477f07a73479a96b8a3e6408ff03d1ace72b9753e29fb\": container with ID starting with 5d2bbd6d3671a4905f8477f07a73479a96b8a3e6408ff03d1ace72b9753e29fb not found: ID does not exist"
Aug 05 18:11:51.931134 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.931085    2561 scope.go:117] "RemoveContainer" containerID="8167ad5dae6a8d3148a666353367e0cc2305ac851487ed90e9de16583ec9853d"
Aug 05 18:11:51.931326 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:11:51.931307    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8167ad5dae6a8d3148a666353367e0cc2305ac851487ed90e9de16583ec9853d\": container with ID starting with 8167ad5dae6a8d3148a666353367e0cc2305ac851487ed90e9de16583ec9853d not found: ID does not exist" containerID="8167ad5dae6a8d3148a666353367e0cc2305ac851487ed90e9de16583ec9853d"
Aug 05 18:11:51.931368 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.931332    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8167ad5dae6a8d3148a666353367e0cc2305ac851487ed90e9de16583ec9853d"} err="failed to get container status \"8167ad5dae6a8d3148a666353367e0cc2305ac851487ed90e9de16583ec9853d\": rpc error: code = NotFound desc = could not find container \"8167ad5dae6a8d3148a666353367e0cc2305ac851487ed90e9de16583ec9853d\": container with ID starting with 8167ad5dae6a8d3148a666353367e0cc2305ac851487ed90e9de16583ec9853d not found: ID does not exist"
Aug 05 18:11:51.931368 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.931347    2561 scope.go:117] "RemoveContainer" containerID="962597c2bd2e3ce554319332643223808f661158832e4e3e363289959bf6539f"
Aug 05 18:11:51.931619 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.931598    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"962597c2bd2e3ce554319332643223808f661158832e4e3e363289959bf6539f"} err="failed to get container status \"962597c2bd2e3ce554319332643223808f661158832e4e3e363289959bf6539f\": rpc error: code = NotFound desc = could not find container \"962597c2bd2e3ce554319332643223808f661158832e4e3e363289959bf6539f\": container with ID starting with 962597c2bd2e3ce554319332643223808f661158832e4e3e363289959bf6539f not found: ID does not exist"
Aug 05 18:11:51.931675 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.931627    2561 scope.go:117] "RemoveContainer" containerID="5d2bbd6d3671a4905f8477f07a73479a96b8a3e6408ff03d1ace72b9753e29fb"
Aug 05 18:11:51.931882 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.931863    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d2bbd6d3671a4905f8477f07a73479a96b8a3e6408ff03d1ace72b9753e29fb"} err="failed to get container status \"5d2bbd6d3671a4905f8477f07a73479a96b8a3e6408ff03d1ace72b9753e29fb\": rpc error: code = NotFound desc = could not find container \"5d2bbd6d3671a4905f8477f07a73479a96b8a3e6408ff03d1ace72b9753e29fb\": container with ID starting with 5d2bbd6d3671a4905f8477f07a73479a96b8a3e6408ff03d1ace72b9753e29fb not found: ID does not exist"
Aug 05 18:11:51.931922 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.931882    2561 scope.go:117] "RemoveContainer" containerID="8167ad5dae6a8d3148a666353367e0cc2305ac851487ed90e9de16583ec9853d"
Aug 05 18:11:51.932128 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:51.932110    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8167ad5dae6a8d3148a666353367e0cc2305ac851487ed90e9de16583ec9853d"} err="failed to get container status \"8167ad5dae6a8d3148a666353367e0cc2305ac851487ed90e9de16583ec9853d\": rpc error: code = NotFound desc = could not find container \"8167ad5dae6a8d3148a666353367e0cc2305ac851487ed90e9de16583ec9853d\": container with ID starting with 8167ad5dae6a8d3148a666353367e0cc2305ac851487ed90e9de16583ec9853d not found: ID does not exist"
Aug 05 18:11:52.729396 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:52.729360    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7d7a0e6-a2c5-417c-af31-061f926af0ed" path="/var/lib/kubelet/pods/c7d7a0e6-a2c5-417c-af31-061f926af0ed/volumes"
Aug 05 18:11:58.418309 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:11:58.418251    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv" podUID="0027bdda-0419-4432-9da4-41ba49aa3aee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/health\": dial tcp 10.133.0.107:8000: connect: connection refused"
Aug 05 18:12:08.427692 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:08.427659    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:12:08.436122 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:08.436097    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:12:09.358268 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.358235    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"]
Aug 05 18:12:09.358818 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.358796    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7d7a0e6-a2c5-417c-af31-061f926af0ed" containerName="llm-d-routing-sidecar"
Aug 05 18:12:09.358818 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.358817    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7d7a0e6-a2c5-417c-af31-061f926af0ed" containerName="llm-d-routing-sidecar"
Aug 05 18:12:09.359034 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.358842    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7d7a0e6-a2c5-417c-af31-061f926af0ed" containerName="main"
Aug 05 18:12:09.359034 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.358848    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7d7a0e6-a2c5-417c-af31-061f926af0ed" containerName="main"
Aug 05 18:12:09.359034 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.358855    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7d7a0e6-a2c5-417c-af31-061f926af0ed" containerName="storage-initializer"
Aug 05 18:12:09.359034 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.358860    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7d7a0e6-a2c5-417c-af31-061f926af0ed" containerName="storage-initializer"
Aug 05 18:12:09.359034 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.358927    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7d7a0e6-a2c5-417c-af31-061f926af0ed" containerName="main"
Aug 05 18:12:09.359034 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.358935    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7d7a0e6-a2c5-417c-af31-061f926af0ed" containerName="llm-d-routing-sidecar"
Aug 05 18:12:09.363148 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.363125    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:12:09.365937 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.365913    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-hfhrr\""
Aug 05 18:12:09.366083 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.365999    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Aug 05 18:12:09.366157 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.366080    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Aug 05 18:12:09.366715 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.366692    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-ppb4w\""
Aug 05 18:12:09.366867 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.366701    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Aug 05 18:12:09.366867 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.366708    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Aug 05 18:12:09.367060 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.366741    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Aug 05 18:12:09.376021 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.375996    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"]
Aug 05 18:12:09.439612 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.439583    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d96594b-ec30-4b1e-ac94-57bad258b6c4-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:12:09.440094 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.439623    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d96594b-ec30-4b1e-ac94-57bad258b6c4-dshm\") pod \"custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:12:09.440094 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.439644    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d96594b-ec30-4b1e-ac94-57bad258b6c4-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:12:09.440094 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.439693    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d96594b-ec30-4b1e-ac94-57bad258b6c4-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:12:09.440094 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.439737    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d96594b-ec30-4b1e-ac94-57bad258b6c4-home\") pod \"custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:12:09.440094 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.439770    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d96594b-ec30-4b1e-ac94-57bad258b6c4-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:12:09.440094 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.439824    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d96594b-ec30-4b1e-ac94-57bad258b6c4-model-cache\") pod \"custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:12:09.440094 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.439853    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pzr5\" (UniqueName: \"kubernetes.io/projected/8d96594b-ec30-4b1e-ac94-57bad258b6c4-kube-api-access-4pzr5\") pod \"custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:12:09.540530 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.540497    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d96594b-ec30-4b1e-ac94-57bad258b6c4-model-cache\") pod \"custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:12:09.540530 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.540541    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pzr5\" (UniqueName: \"kubernetes.io/projected/8d96594b-ec30-4b1e-ac94-57bad258b6c4-kube-api-access-4pzr5\") pod \"custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:12:09.540805 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.540613    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d96594b-ec30-4b1e-ac94-57bad258b6c4-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:12:09.540805 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.540657    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d96594b-ec30-4b1e-ac94-57bad258b6c4-dshm\") pod \"custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:12:09.540805 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.540692    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d96594b-ec30-4b1e-ac94-57bad258b6c4-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:12:09.540805 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.540720    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d96594b-ec30-4b1e-ac94-57bad258b6c4-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:12:09.540805 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.540751    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d96594b-ec30-4b1e-ac94-57bad258b6c4-home\") pod \"custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:12:09.540805 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.540781    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d96594b-ec30-4b1e-ac94-57bad258b6c4-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:12:09.541162 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.541021    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d96594b-ec30-4b1e-ac94-57bad258b6c4-model-cache\") pod \"custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:12:09.541162 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.541071    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d96594b-ec30-4b1e-ac94-57bad258b6c4-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:12:09.541277 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.541258    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d96594b-ec30-4b1e-ac94-57bad258b6c4-home\") pod \"custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:12:09.541325 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.541279    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d96594b-ec30-4b1e-ac94-57bad258b6c4-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:12:09.541394 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.541374    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d96594b-ec30-4b1e-ac94-57bad258b6c4-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:12:09.543041 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.543020    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d96594b-ec30-4b1e-ac94-57bad258b6c4-dshm\") pod \"custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:12:09.543283 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.543259    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d96594b-ec30-4b1e-ac94-57bad258b6c4-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:12:09.552243 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.552219    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pzr5\" (UniqueName: \"kubernetes.io/projected/8d96594b-ec30-4b1e-ac94-57bad258b6c4-kube-api-access-4pzr5\") pod \"custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:12:09.677144 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.677039    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:12:09.818087 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:09.818060    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"]
Aug 05 18:12:09.818983 ip-10-0-135-151 kubenswrapper[2561]: W0805 18:12:09.818943    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d96594b_ec30_4b1e_ac94_57bad258b6c4.slice/crio-11f5382e0f430c8475c4cd60b4c8fd0e7cf80526b100f9a638019ed5aff53aa0 WatchSource:0}: Error finding container 11f5382e0f430c8475c4cd60b4c8fd0e7cf80526b100f9a638019ed5aff53aa0: Status 404 returned error can't find the container with id 11f5382e0f430c8475c4cd60b4c8fd0e7cf80526b100f9a638019ed5aff53aa0
Aug 05 18:12:10.006660 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:10.006573    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp" event={"ID":"8d96594b-ec30-4b1e-ac94-57bad258b6c4","Type":"ContainerStarted","Data":"4353f862d9b280bf5dbcc8a7e3301c687d86eef3c7f23a505faaaad2c0c5630f"}
Aug 05 18:12:10.006660 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:10.006632    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp" event={"ID":"8d96594b-ec30-4b1e-ac94-57bad258b6c4","Type":"ContainerStarted","Data":"11f5382e0f430c8475c4cd60b4c8fd0e7cf80526b100f9a638019ed5aff53aa0"}
Aug 05 18:12:10.006887 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:10.006767    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:12:11.014364 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:11.014327    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp" event={"ID":"8d96594b-ec30-4b1e-ac94-57bad258b6c4","Type":"ContainerStarted","Data":"f4b235219518ee3fa504d181d687e6d13715b049eff8c3e297acd83bbac05b6d"}
Aug 05 18:12:13.028563 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:13.028527    2561 generic.go:356] "Generic (PLEG): container finished" podID="8d96594b-ec30-4b1e-ac94-57bad258b6c4" containerID="f4b235219518ee3fa504d181d687e6d13715b049eff8c3e297acd83bbac05b6d" exitCode=0
Aug 05 18:12:13.028563 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:13.028568    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp" event={"ID":"8d96594b-ec30-4b1e-ac94-57bad258b6c4","Type":"ContainerDied","Data":"f4b235219518ee3fa504d181d687e6d13715b049eff8c3e297acd83bbac05b6d"}
Aug 05 18:12:14.041458 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:14.041411    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp" event={"ID":"8d96594b-ec30-4b1e-ac94-57bad258b6c4","Type":"ContainerStarted","Data":"c59fa491c4a8135ba72deeeb020c1df361b493f46191c375983de250d039c4c9"}
Aug 05 18:12:14.066824 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:14.066764    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp" podStartSLOduration=2.817639409 podStartE2EDuration="5.066746504s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.249107095s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:12:14.064094716 +0000 UTC m=+3589.808729043" watchObservedRunningTime="2026-08-05 18:12:14.066746504 +0000 UTC m=+3589.811380831"
Aug 05 18:12:19.677568 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:19.677469    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:12:19.677568 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:19.677551    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:12:19.679050 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:19.679006    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp" podUID="8d96594b-ec30-4b1e-ac94-57bad258b6c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8001/health\": dial tcp 10.133.0.108:8001: connect: connection refused"
Aug 05 18:12:19.692125 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:19.692097    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:12:29.678025 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:29.677968    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp" podUID="8d96594b-ec30-4b1e-ac94-57bad258b6c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8001/health\": dial tcp 10.133.0.108:8001: connect: connection refused"
Aug 05 18:12:35.618808 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:35.618770    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"]
Aug 05 18:12:39.677804 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:39.677724    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp" podUID="8d96594b-ec30-4b1e-ac94-57bad258b6c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8001/health\": dial tcp 10.133.0.108:8001: connect: connection refused"
Aug 05 18:12:49.678148 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:49.678072    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp" podUID="8d96594b-ec30-4b1e-ac94-57bad258b6c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8001/health\": dial tcp 10.133.0.108:8001: connect: connection refused"
Aug 05 18:12:50.643146 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:50.643092    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv" podUID="0027bdda-0419-4432-9da4-41ba49aa3aee" containerName="main" containerID="cri-o://ca0187b5eb227bea52b89c0a84da5dea74fa272a86347cbcb31084935ccfa0cd" gracePeriod=45
Aug 05 18:12:51.275225 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:51.275187    2561 generic.go:356] "Generic (PLEG): container finished" podID="0027bdda-0419-4432-9da4-41ba49aa3aee" containerID="ca0187b5eb227bea52b89c0a84da5dea74fa272a86347cbcb31084935ccfa0cd" exitCode=0
Aug 05 18:12:51.275615 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:51.275256    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv" event={"ID":"0027bdda-0419-4432-9da4-41ba49aa3aee","Type":"ContainerDied","Data":"ca0187b5eb227bea52b89c0a84da5dea74fa272a86347cbcb31084935ccfa0cd"}
Aug 05 18:12:51.394160 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:51.394132    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:12:51.430514 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:51.430418    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0027bdda-0419-4432-9da4-41ba49aa3aee-tmp-dir\") pod \"0027bdda-0419-4432-9da4-41ba49aa3aee\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") "
Aug 05 18:12:51.430514 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:51.430453    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0027bdda-0419-4432-9da4-41ba49aa3aee-model-cache\") pod \"0027bdda-0419-4432-9da4-41ba49aa3aee\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") "
Aug 05 18:12:51.430514 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:51.430490    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0027bdda-0419-4432-9da4-41ba49aa3aee-tls-certs\") pod \"0027bdda-0419-4432-9da4-41ba49aa3aee\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") "
Aug 05 18:12:51.430827 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:51.430519    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0027bdda-0419-4432-9da4-41ba49aa3aee-dshm\") pod \"0027bdda-0419-4432-9da4-41ba49aa3aee\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") "
Aug 05 18:12:51.430827 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:51.430556    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlwg8\" (UniqueName: \"kubernetes.io/projected/0027bdda-0419-4432-9da4-41ba49aa3aee-kube-api-access-wlwg8\") pod \"0027bdda-0419-4432-9da4-41ba49aa3aee\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") "
Aug 05 18:12:51.430827 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:51.430606    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0027bdda-0419-4432-9da4-41ba49aa3aee-home\") pod \"0027bdda-0419-4432-9da4-41ba49aa3aee\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") "
Aug 05 18:12:51.430827 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:51.430671    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0027bdda-0419-4432-9da4-41ba49aa3aee-cabundle-cert\") pod \"0027bdda-0419-4432-9da4-41ba49aa3aee\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") "
Aug 05 18:12:51.430827 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:51.430702    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0027bdda-0419-4432-9da4-41ba49aa3aee-kserve-provision-location\") pod \"0027bdda-0419-4432-9da4-41ba49aa3aee\" (UID: \"0027bdda-0419-4432-9da4-41ba49aa3aee\") "
Aug 05 18:12:51.430827 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:51.430745    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0027bdda-0419-4432-9da4-41ba49aa3aee-model-cache" (OuterVolumeSpecName: "model-cache") pod "0027bdda-0419-4432-9da4-41ba49aa3aee" (UID: "0027bdda-0419-4432-9da4-41ba49aa3aee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:12:51.431262 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:51.430976    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0027bdda-0419-4432-9da4-41ba49aa3aee-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:12:51.431403 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:51.431377    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0027bdda-0419-4432-9da4-41ba49aa3aee-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0027bdda-0419-4432-9da4-41ba49aa3aee" (UID: "0027bdda-0419-4432-9da4-41ba49aa3aee"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 18:12:51.431472 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:51.431419    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0027bdda-0419-4432-9da4-41ba49aa3aee-home" (OuterVolumeSpecName: "home") pod "0027bdda-0419-4432-9da4-41ba49aa3aee" (UID: "0027bdda-0419-4432-9da4-41ba49aa3aee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:12:51.435236 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:51.435201    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0027bdda-0419-4432-9da4-41ba49aa3aee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0027bdda-0419-4432-9da4-41ba49aa3aee" (UID: "0027bdda-0419-4432-9da4-41ba49aa3aee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 18:12:51.435382 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:51.435338    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0027bdda-0419-4432-9da4-41ba49aa3aee-kube-api-access-wlwg8" (OuterVolumeSpecName: "kube-api-access-wlwg8") pod "0027bdda-0419-4432-9da4-41ba49aa3aee" (UID: "0027bdda-0419-4432-9da4-41ba49aa3aee"). InnerVolumeSpecName "kube-api-access-wlwg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 18:12:51.435715 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:51.435692    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0027bdda-0419-4432-9da4-41ba49aa3aee-dshm" (OuterVolumeSpecName: "dshm") pod "0027bdda-0419-4432-9da4-41ba49aa3aee" (UID: "0027bdda-0419-4432-9da4-41ba49aa3aee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:12:51.451407 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:51.451348    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0027bdda-0419-4432-9da4-41ba49aa3aee-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0027bdda-0419-4432-9da4-41ba49aa3aee" (UID: "0027bdda-0419-4432-9da4-41ba49aa3aee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:12:51.460773 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:51.460740    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0027bdda-0419-4432-9da4-41ba49aa3aee-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0027bdda-0419-4432-9da4-41ba49aa3aee" (UID: "0027bdda-0419-4432-9da4-41ba49aa3aee"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:12:51.532024 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:51.531976    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0027bdda-0419-4432-9da4-41ba49aa3aee-cabundle-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:12:51.532024 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:51.532011    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0027bdda-0419-4432-9da4-41ba49aa3aee-kserve-provision-location\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:12:51.532024 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:51.532021    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0027bdda-0419-4432-9da4-41ba49aa3aee-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:12:51.532024 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:51.532031    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0027bdda-0419-4432-9da4-41ba49aa3aee-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:12:51.532024 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:51.532039    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0027bdda-0419-4432-9da4-41ba49aa3aee-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:12:51.532436 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:51.532049    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlwg8\" (UniqueName: \"kubernetes.io/projected/0027bdda-0419-4432-9da4-41ba49aa3aee-kube-api-access-wlwg8\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:12:51.532436 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:51.532057    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0027bdda-0419-4432-9da4-41ba49aa3aee-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:12:52.282698 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:52.282660    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv" event={"ID":"0027bdda-0419-4432-9da4-41ba49aa3aee","Type":"ContainerDied","Data":"0692ab558a011d3e9861209973bc2cec5498124764df52a18682faed8eb36708"}
Aug 05 18:12:52.283254 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:52.282726    2561 scope.go:117] "RemoveContainer" containerID="ca0187b5eb227bea52b89c0a84da5dea74fa272a86347cbcb31084935ccfa0cd"
Aug 05 18:12:52.283254 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:52.282679    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"
Aug 05 18:12:52.294188 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:52.294167    2561 scope.go:117] "RemoveContainer" containerID="0ef4b6e0b5a6c12a1da625179b3e83c4ca132187dff30c017917586718926212"
Aug 05 18:12:52.306856 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:52.306830    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"]
Aug 05 18:12:52.309751 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:52.309727    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-84698fc874-jgbsv"]
Aug 05 18:12:52.731067 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:52.730966    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0027bdda-0419-4432-9da4-41ba49aa3aee" path="/var/lib/kubelet/pods/0027bdda-0419-4432-9da4-41ba49aa3aee/volumes"
Aug 05 18:12:59.677972 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:12:59.677891    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp" podUID="8d96594b-ec30-4b1e-ac94-57bad258b6c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8001/health\": dial tcp 10.133.0.108:8001: connect: connection refused"
Aug 05 18:13:03.520248 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.520201    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"]
Aug 05 18:13:03.520716 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.520585    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0027bdda-0419-4432-9da4-41ba49aa3aee" containerName="main"
Aug 05 18:13:03.520716 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.520595    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0027bdda-0419-4432-9da4-41ba49aa3aee" containerName="main"
Aug 05 18:13:03.520716 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.520608    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0027bdda-0419-4432-9da4-41ba49aa3aee" containerName="storage-initializer"
Aug 05 18:13:03.520716 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.520616    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0027bdda-0419-4432-9da4-41ba49aa3aee" containerName="storage-initializer"
Aug 05 18:13:03.520716 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.520695    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0027bdda-0419-4432-9da4-41ba49aa3aee" containerName="main"
Aug 05 18:13:03.525934 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.525912    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:13:03.528414 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.528376    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Aug 05 18:13:03.528414 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.528405    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Aug 05 18:13:03.528620 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.528432    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-n6n48\""
Aug 05 18:13:03.528620 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.528450    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Aug 05 18:13:03.528620 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.528473    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Aug 05 18:13:03.528620 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.528404    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Aug 05 18:13:03.535729 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.535700    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"]
Aug 05 18:13:03.640134 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.640097    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-dshm\") pod \"prestop-hook-test-kserve-6f5dbd7f74-j2qjq\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:13:03.640309 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.640167    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-model-cache\") pod \"prestop-hook-test-kserve-6f5dbd7f74-j2qjq\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:13:03.640309 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.640215    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkksp\" (UniqueName: \"kubernetes.io/projected/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-kube-api-access-jkksp\") pod \"prestop-hook-test-kserve-6f5dbd7f74-j2qjq\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:13:03.640434 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.640302    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-tmp-dir\") pod \"prestop-hook-test-kserve-6f5dbd7f74-j2qjq\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:13:03.640434 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.640365    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-tls-certs\") pod \"prestop-hook-test-kserve-6f5dbd7f74-j2qjq\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:13:03.640434 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.640395    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-kserve-provision-location\") pod \"prestop-hook-test-kserve-6f5dbd7f74-j2qjq\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:13:03.640535 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.640442    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-home\") pod \"prestop-hook-test-kserve-6f5dbd7f74-j2qjq\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:13:03.640535 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.640479    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-cabundle-cert\") pod \"prestop-hook-test-kserve-6f5dbd7f74-j2qjq\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:13:03.741124 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.741089    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-tls-certs\") pod \"prestop-hook-test-kserve-6f5dbd7f74-j2qjq\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:13:03.741124 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.741127    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-kserve-provision-location\") pod \"prestop-hook-test-kserve-6f5dbd7f74-j2qjq\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:13:03.741406 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.741155    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-home\") pod \"prestop-hook-test-kserve-6f5dbd7f74-j2qjq\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:13:03.741406 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.741185    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-cabundle-cert\") pod \"prestop-hook-test-kserve-6f5dbd7f74-j2qjq\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:13:03.741406 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.741230    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-dshm\") pod \"prestop-hook-test-kserve-6f5dbd7f74-j2qjq\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:13:03.741406 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.741274    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-model-cache\") pod \"prestop-hook-test-kserve-6f5dbd7f74-j2qjq\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:13:03.741406 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.741299    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkksp\" (UniqueName: \"kubernetes.io/projected/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-kube-api-access-jkksp\") pod \"prestop-hook-test-kserve-6f5dbd7f74-j2qjq\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:13:03.741406 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.741352    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-tmp-dir\") pod \"prestop-hook-test-kserve-6f5dbd7f74-j2qjq\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:13:03.741743 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.741597    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-kserve-provision-location\") pod \"prestop-hook-test-kserve-6f5dbd7f74-j2qjq\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:13:03.741743 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.741662    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-home\") pod \"prestop-hook-test-kserve-6f5dbd7f74-j2qjq\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:13:03.741861 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.741813    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-model-cache\") pod \"prestop-hook-test-kserve-6f5dbd7f74-j2qjq\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:13:03.741916 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.741884    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-tmp-dir\") pod \"prestop-hook-test-kserve-6f5dbd7f74-j2qjq\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:13:03.742207 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.742184    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-cabundle-cert\") pod \"prestop-hook-test-kserve-6f5dbd7f74-j2qjq\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:13:03.743694 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.743674    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-dshm\") pod \"prestop-hook-test-kserve-6f5dbd7f74-j2qjq\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:13:03.744111 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.744090    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-tls-certs\") pod \"prestop-hook-test-kserve-6f5dbd7f74-j2qjq\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:13:03.749249 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.749220    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkksp\" (UniqueName: \"kubernetes.io/projected/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-kube-api-access-jkksp\") pod \"prestop-hook-test-kserve-6f5dbd7f74-j2qjq\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:13:03.837262 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:03.837219    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:13:04.014630 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:04.014598    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"]
Aug 05 18:13:04.015667 ip-10-0-135-151 kubenswrapper[2561]: W0805 18:13:04.015638    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6bbcde_0de7_4bdd_8b7d_5d0706737739.slice/crio-3febd68460a767d67935d0a4c5519a2302d1efa41fbbff4cb057229aaa0b595d WatchSource:0}: Error finding container 3febd68460a767d67935d0a4c5519a2302d1efa41fbbff4cb057229aaa0b595d: Status 404 returned error can't find the container with id 3febd68460a767d67935d0a4c5519a2302d1efa41fbbff4cb057229aaa0b595d
Aug 05 18:13:04.357204 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:04.357170    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq" event={"ID":"ae6bbcde-0de7-4bdd-8b7d-5d0706737739","Type":"ContainerStarted","Data":"524ba269289dd0575810c3de875868faaab5f61f39591d1f38706677ccb6c5e4"}
Aug 05 18:13:04.357384 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:04.357225    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq" event={"ID":"ae6bbcde-0de7-4bdd-8b7d-5d0706737739","Type":"ContainerStarted","Data":"3febd68460a767d67935d0a4c5519a2302d1efa41fbbff4cb057229aaa0b595d"}
Aug 05 18:13:07.377830 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:07.377745    2561 generic.go:356] "Generic (PLEG): container finished" podID="ae6bbcde-0de7-4bdd-8b7d-5d0706737739" containerID="524ba269289dd0575810c3de875868faaab5f61f39591d1f38706677ccb6c5e4" exitCode=0
Aug 05 18:13:07.378421 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:07.377829    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq" event={"ID":"ae6bbcde-0de7-4bdd-8b7d-5d0706737739","Type":"ContainerDied","Data":"524ba269289dd0575810c3de875868faaab5f61f39591d1f38706677ccb6c5e4"}
Aug 05 18:13:08.386391 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:08.386356    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq" event={"ID":"ae6bbcde-0de7-4bdd-8b7d-5d0706737739","Type":"ContainerStarted","Data":"1450ee1888d88c0a1320dc520cac3ef0a4f79242a832bd9964b0750d8f17c464"}
Aug 05 18:13:08.411562 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:08.411501    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq" podStartSLOduration=2.96580516 podStartE2EDuration="5.411484416s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.445679256s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:13:08.408668336 +0000 UTC m=+3644.153302726" watchObservedRunningTime="2026-08-05 18:13:08.411484416 +0000 UTC m=+3644.156118744"
Aug 05 18:13:08.837825 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:08.837785    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:13:09.677971 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:09.677896    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp" podUID="8d96594b-ec30-4b1e-ac94-57bad258b6c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8001/health\": dial tcp 10.133.0.108:8001: connect: connection refused"
Aug 05 18:13:13.837784 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:13.837750    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:13:13.839686 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:13.839656    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq" podUID="ae6bbcde-0de7-4bdd-8b7d-5d0706737739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/health\": dial tcp 10.133.0.109:8000: connect: connection refused"
Aug 05 18:13:19.677653 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:19.677599    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp" podUID="8d96594b-ec30-4b1e-ac94-57bad258b6c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8001/health\": dial tcp 10.133.0.108:8001: connect: connection refused"
Aug 05 18:13:23.837972 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:23.837905    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq" podUID="ae6bbcde-0de7-4bdd-8b7d-5d0706737739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/health\": dial tcp 10.133.0.109:8000: connect: connection refused"
Aug 05 18:13:29.678208 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:29.678156    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp" podUID="8d96594b-ec30-4b1e-ac94-57bad258b6c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8001/health\": dial tcp 10.133.0.108:8001: connect: connection refused"
Aug 05 18:13:33.837909 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:33.837864    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq" podUID="ae6bbcde-0de7-4bdd-8b7d-5d0706737739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/health\": dial tcp 10.133.0.109:8000: connect: connection refused"
Aug 05 18:13:39.677972 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:39.677888    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp" podUID="8d96594b-ec30-4b1e-ac94-57bad258b6c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8001/health\": dial tcp 10.133.0.108:8001: connect: connection refused"
Aug 05 18:13:43.838164 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:43.838113    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq" podUID="ae6bbcde-0de7-4bdd-8b7d-5d0706737739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/health\": dial tcp 10.133.0.109:8000: connect: connection refused"
Aug 05 18:13:49.677920 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:49.677812    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp" podUID="8d96594b-ec30-4b1e-ac94-57bad258b6c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8001/health\": dial tcp 10.133.0.108:8001: connect: connection refused"
Aug 05 18:13:53.838408 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:53.838366    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq" podUID="ae6bbcde-0de7-4bdd-8b7d-5d0706737739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/health\": dial tcp 10.133.0.109:8000: connect: connection refused"
Aug 05 18:13:59.678406 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:13:59.678351    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp" podUID="8d96594b-ec30-4b1e-ac94-57bad258b6c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8001/health\": dial tcp 10.133.0.108:8001: connect: connection refused"
Aug 05 18:14:03.838133 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:03.838080    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq" podUID="ae6bbcde-0de7-4bdd-8b7d-5d0706737739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/health\": dial tcp 10.133.0.109:8000: connect: connection refused"
Aug 05 18:14:09.687990 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:09.687933    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:14:09.701498 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:09.701470    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:14:13.838558 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:13.838519    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq" podUID="ae6bbcde-0de7-4bdd-8b7d-5d0706737739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/health\": dial tcp 10.133.0.109:8000: connect: connection refused"
Aug 05 18:14:19.063864 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:19.063826    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"]
Aug 05 18:14:23.838361 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:23.838321    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq" podUID="ae6bbcde-0de7-4bdd-8b7d-5d0706737739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/health\": dial tcp 10.133.0.109:8000: connect: connection refused"
Aug 05 18:14:33.838337 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:33.838300    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq" podUID="ae6bbcde-0de7-4bdd-8b7d-5d0706737739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/health\": dial tcp 10.133.0.109:8000: connect: connection refused"
Aug 05 18:14:34.089134 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:34.089032    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp" podUID="8d96594b-ec30-4b1e-ac94-57bad258b6c4" containerName="main" containerID="cri-o://c59fa491c4a8135ba72deeeb020c1df361b493f46191c375983de250d039c4c9" gracePeriod=45
Aug 05 18:14:35.006131 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.006076    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp" podUID="8d96594b-ec30-4b1e-ac94-57bad258b6c4" containerName="llm-d-routing-sidecar" containerID="cri-o://4353f862d9b280bf5dbcc8a7e3301c687d86eef3c7f23a505faaaad2c0c5630f" gracePeriod=45
Aug 05 18:14:35.260141 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.260111    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:14:35.334217 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.334185    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d96594b-ec30-4b1e-ac94-57bad258b6c4-kserve-provision-location\") pod \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") "
Aug 05 18:14:35.334416 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.334235    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d96594b-ec30-4b1e-ac94-57bad258b6c4-tmp-dir\") pod \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") "
Aug 05 18:14:35.334416 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.334262    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d96594b-ec30-4b1e-ac94-57bad258b6c4-tls-certs\") pod \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") "
Aug 05 18:14:35.334416 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.334283    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d96594b-ec30-4b1e-ac94-57bad258b6c4-home\") pod \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") "
Aug 05 18:14:35.334592 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.334566    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d96594b-ec30-4b1e-ac94-57bad258b6c4-cabundle-cert\") pod \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") "
Aug 05 18:14:35.334667 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.334630    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d96594b-ec30-4b1e-ac94-57bad258b6c4-dshm\") pod \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") "
Aug 05 18:14:35.334725 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.334663    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d96594b-ec30-4b1e-ac94-57bad258b6c4-model-cache\") pod \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") "
Aug 05 18:14:35.335125 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.335011    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d96594b-ec30-4b1e-ac94-57bad258b6c4-home" (OuterVolumeSpecName: "home") pod "8d96594b-ec30-4b1e-ac94-57bad258b6c4" (UID: "8d96594b-ec30-4b1e-ac94-57bad258b6c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:14:35.335469 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.335309    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d96594b-ec30-4b1e-ac94-57bad258b6c4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8d96594b-ec30-4b1e-ac94-57bad258b6c4" (UID: "8d96594b-ec30-4b1e-ac94-57bad258b6c4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 18:14:35.335758 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.335670    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d96594b-ec30-4b1e-ac94-57bad258b6c4-model-cache" (OuterVolumeSpecName: "model-cache") pod "8d96594b-ec30-4b1e-ac94-57bad258b6c4" (UID: "8d96594b-ec30-4b1e-ac94-57bad258b6c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:14:35.337302 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.337270    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d96594b-ec30-4b1e-ac94-57bad258b6c4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8d96594b-ec30-4b1e-ac94-57bad258b6c4" (UID: "8d96594b-ec30-4b1e-ac94-57bad258b6c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 18:14:35.337722 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.337696    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d96594b-ec30-4b1e-ac94-57bad258b6c4-dshm" (OuterVolumeSpecName: "dshm") pod "8d96594b-ec30-4b1e-ac94-57bad258b6c4" (UID: "8d96594b-ec30-4b1e-ac94-57bad258b6c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:14:35.340001 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.339974    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pzr5\" (UniqueName: \"kubernetes.io/projected/8d96594b-ec30-4b1e-ac94-57bad258b6c4-kube-api-access-4pzr5\") pod \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\" (UID: \"8d96594b-ec30-4b1e-ac94-57bad258b6c4\") "
Aug 05 18:14:35.340407 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.340304    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d96594b-ec30-4b1e-ac94-57bad258b6c4-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:14:35.340407 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.340328    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d96594b-ec30-4b1e-ac94-57bad258b6c4-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:14:35.340407 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.340343    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d96594b-ec30-4b1e-ac94-57bad258b6c4-cabundle-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:14:35.340407 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.340357    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d96594b-ec30-4b1e-ac94-57bad258b6c4-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:14:35.340407 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.340370    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d96594b-ec30-4b1e-ac94-57bad258b6c4-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:14:35.342416 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.342376    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d96594b-ec30-4b1e-ac94-57bad258b6c4-kube-api-access-4pzr5" (OuterVolumeSpecName: "kube-api-access-4pzr5") pod "8d96594b-ec30-4b1e-ac94-57bad258b6c4" (UID: "8d96594b-ec30-4b1e-ac94-57bad258b6c4"). InnerVolumeSpecName "kube-api-access-4pzr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 18:14:35.352055 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.352027    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d96594b-ec30-4b1e-ac94-57bad258b6c4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8d96594b-ec30-4b1e-ac94-57bad258b6c4" (UID: "8d96594b-ec30-4b1e-ac94-57bad258b6c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:14:35.364115 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.364088    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d96594b-ec30-4b1e-ac94-57bad258b6c4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8d96594b-ec30-4b1e-ac94-57bad258b6c4" (UID: "8d96594b-ec30-4b1e-ac94-57bad258b6c4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:14:35.441625 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.441584    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d96594b-ec30-4b1e-ac94-57bad258b6c4-kserve-provision-location\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:14:35.441625 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.441617    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d96594b-ec30-4b1e-ac94-57bad258b6c4-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:14:35.441625 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.441628    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pzr5\" (UniqueName: \"kubernetes.io/projected/8d96594b-ec30-4b1e-ac94-57bad258b6c4-kube-api-access-4pzr5\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:14:35.969778 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.969733    2561 generic.go:356] "Generic (PLEG): container finished" podID="8d96594b-ec30-4b1e-ac94-57bad258b6c4" containerID="c59fa491c4a8135ba72deeeb020c1df361b493f46191c375983de250d039c4c9" exitCode=0
Aug 05 18:14:35.969778 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.969773    2561 generic.go:356] "Generic (PLEG): container finished" podID="8d96594b-ec30-4b1e-ac94-57bad258b6c4" containerID="4353f862d9b280bf5dbcc8a7e3301c687d86eef3c7f23a505faaaad2c0c5630f" exitCode=0
Aug 05 18:14:35.970012 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.969822    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp" event={"ID":"8d96594b-ec30-4b1e-ac94-57bad258b6c4","Type":"ContainerDied","Data":"c59fa491c4a8135ba72deeeb020c1df361b493f46191c375983de250d039c4c9"}
Aug 05 18:14:35.970012 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.969840    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"
Aug 05 18:14:35.970012 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.969889    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp" event={"ID":"8d96594b-ec30-4b1e-ac94-57bad258b6c4","Type":"ContainerDied","Data":"4353f862d9b280bf5dbcc8a7e3301c687d86eef3c7f23a505faaaad2c0c5630f"}
Aug 05 18:14:35.970012 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.969919    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp" event={"ID":"8d96594b-ec30-4b1e-ac94-57bad258b6c4","Type":"ContainerDied","Data":"11f5382e0f430c8475c4cd60b4c8fd0e7cf80526b100f9a638019ed5aff53aa0"}
Aug 05 18:14:35.970012 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.969970    2561 scope.go:117] "RemoveContainer" containerID="c59fa491c4a8135ba72deeeb020c1df361b493f46191c375983de250d039c4c9"
Aug 05 18:14:35.979802 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.979780    2561 scope.go:117] "RemoveContainer" containerID="f4b235219518ee3fa504d181d687e6d13715b049eff8c3e297acd83bbac05b6d"
Aug 05 18:14:35.988719 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.988705    2561 scope.go:117] "RemoveContainer" containerID="4353f862d9b280bf5dbcc8a7e3301c687d86eef3c7f23a505faaaad2c0c5630f"
Aug 05 18:14:35.993808 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.993780    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"]
Aug 05 18:14:35.997651 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:35.997628    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-556dbf4ff4-z8fbp"]
Aug 05 18:14:36.000921 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:36.000903    2561 scope.go:117] "RemoveContainer" containerID="c59fa491c4a8135ba72deeeb020c1df361b493f46191c375983de250d039c4c9"
Aug 05 18:14:36.001251 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:14:36.001222    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c59fa491c4a8135ba72deeeb020c1df361b493f46191c375983de250d039c4c9\": container with ID starting with c59fa491c4a8135ba72deeeb020c1df361b493f46191c375983de250d039c4c9 not found: ID does not exist" containerID="c59fa491c4a8135ba72deeeb020c1df361b493f46191c375983de250d039c4c9"
Aug 05 18:14:36.001328 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:36.001263    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c59fa491c4a8135ba72deeeb020c1df361b493f46191c375983de250d039c4c9"} err="failed to get container status \"c59fa491c4a8135ba72deeeb020c1df361b493f46191c375983de250d039c4c9\": rpc error: code = NotFound desc = could not find container \"c59fa491c4a8135ba72deeeb020c1df361b493f46191c375983de250d039c4c9\": container with ID starting with c59fa491c4a8135ba72deeeb020c1df361b493f46191c375983de250d039c4c9 not found: ID does not exist"
Aug 05 18:14:36.001328 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:36.001290    2561 scope.go:117] "RemoveContainer" containerID="f4b235219518ee3fa504d181d687e6d13715b049eff8c3e297acd83bbac05b6d"
Aug 05 18:14:36.001568 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:14:36.001545    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b235219518ee3fa504d181d687e6d13715b049eff8c3e297acd83bbac05b6d\": container with ID starting with f4b235219518ee3fa504d181d687e6d13715b049eff8c3e297acd83bbac05b6d not found: ID does not exist" containerID="f4b235219518ee3fa504d181d687e6d13715b049eff8c3e297acd83bbac05b6d"
Aug 05 18:14:36.001620 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:36.001580    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b235219518ee3fa504d181d687e6d13715b049eff8c3e297acd83bbac05b6d"} err="failed to get container status \"f4b235219518ee3fa504d181d687e6d13715b049eff8c3e297acd83bbac05b6d\": rpc error: code = NotFound desc = could not find container \"f4b235219518ee3fa504d181d687e6d13715b049eff8c3e297acd83bbac05b6d\": container with ID starting with f4b235219518ee3fa504d181d687e6d13715b049eff8c3e297acd83bbac05b6d not found: ID does not exist"
Aug 05 18:14:36.001620 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:36.001597    2561 scope.go:117] "RemoveContainer" containerID="4353f862d9b280bf5dbcc8a7e3301c687d86eef3c7f23a505faaaad2c0c5630f"
Aug 05 18:14:36.001845 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:14:36.001826    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4353f862d9b280bf5dbcc8a7e3301c687d86eef3c7f23a505faaaad2c0c5630f\": container with ID starting with 4353f862d9b280bf5dbcc8a7e3301c687d86eef3c7f23a505faaaad2c0c5630f not found: ID does not exist" containerID="4353f862d9b280bf5dbcc8a7e3301c687d86eef3c7f23a505faaaad2c0c5630f"
Aug 05 18:14:36.001921 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:36.001851    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4353f862d9b280bf5dbcc8a7e3301c687d86eef3c7f23a505faaaad2c0c5630f"} err="failed to get container status \"4353f862d9b280bf5dbcc8a7e3301c687d86eef3c7f23a505faaaad2c0c5630f\": rpc error: code = NotFound desc = could not find container \"4353f862d9b280bf5dbcc8a7e3301c687d86eef3c7f23a505faaaad2c0c5630f\": container with ID starting with 4353f862d9b280bf5dbcc8a7e3301c687d86eef3c7f23a505faaaad2c0c5630f not found: ID does not exist"
Aug 05 18:14:36.001921 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:36.001872    2561 scope.go:117] "RemoveContainer" containerID="c59fa491c4a8135ba72deeeb020c1df361b493f46191c375983de250d039c4c9"
Aug 05 18:14:36.002118 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:36.002100    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c59fa491c4a8135ba72deeeb020c1df361b493f46191c375983de250d039c4c9"} err="failed to get container status \"c59fa491c4a8135ba72deeeb020c1df361b493f46191c375983de250d039c4c9\": rpc error: code = NotFound desc = could not find container \"c59fa491c4a8135ba72deeeb020c1df361b493f46191c375983de250d039c4c9\": container with ID starting with c59fa491c4a8135ba72deeeb020c1df361b493f46191c375983de250d039c4c9 not found: ID does not exist"
Aug 05 18:14:36.002193 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:36.002120    2561 scope.go:117] "RemoveContainer" containerID="f4b235219518ee3fa504d181d687e6d13715b049eff8c3e297acd83bbac05b6d"
Aug 05 18:14:36.002309 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:36.002291    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b235219518ee3fa504d181d687e6d13715b049eff8c3e297acd83bbac05b6d"} err="failed to get container status \"f4b235219518ee3fa504d181d687e6d13715b049eff8c3e297acd83bbac05b6d\": rpc error: code = NotFound desc = could not find container \"f4b235219518ee3fa504d181d687e6d13715b049eff8c3e297acd83bbac05b6d\": container with ID starting with f4b235219518ee3fa504d181d687e6d13715b049eff8c3e297acd83bbac05b6d not found: ID does not exist"
Aug 05 18:14:36.002377 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:36.002309    2561 scope.go:117] "RemoveContainer" containerID="4353f862d9b280bf5dbcc8a7e3301c687d86eef3c7f23a505faaaad2c0c5630f"
Aug 05 18:14:36.002529 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:36.002512    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4353f862d9b280bf5dbcc8a7e3301c687d86eef3c7f23a505faaaad2c0c5630f"} err="failed to get container status \"4353f862d9b280bf5dbcc8a7e3301c687d86eef3c7f23a505faaaad2c0c5630f\": rpc error: code = NotFound desc = could not find container \"4353f862d9b280bf5dbcc8a7e3301c687d86eef3c7f23a505faaaad2c0c5630f\": container with ID starting with 4353f862d9b280bf5dbcc8a7e3301c687d86eef3c7f23a505faaaad2c0c5630f not found: ID does not exist"
Aug 05 18:14:36.730662 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:36.730628    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d96594b-ec30-4b1e-ac94-57bad258b6c4" path="/var/lib/kubelet/pods/8d96594b-ec30-4b1e-ac94-57bad258b6c4/volumes"
Aug 05 18:14:39.375782 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.375739    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"]
Aug 05 18:14:39.376400 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.376375    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d96594b-ec30-4b1e-ac94-57bad258b6c4" containerName="storage-initializer"
Aug 05 18:14:39.376473 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.376403    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d96594b-ec30-4b1e-ac94-57bad258b6c4" containerName="storage-initializer"
Aug 05 18:14:39.376473 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.376430    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d96594b-ec30-4b1e-ac94-57bad258b6c4" containerName="main"
Aug 05 18:14:39.376473 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.376439    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d96594b-ec30-4b1e-ac94-57bad258b6c4" containerName="main"
Aug 05 18:14:39.376473 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.376460    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d96594b-ec30-4b1e-ac94-57bad258b6c4" containerName="llm-d-routing-sidecar"
Aug 05 18:14:39.376473 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.376469    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d96594b-ec30-4b1e-ac94-57bad258b6c4" containerName="llm-d-routing-sidecar"
Aug 05 18:14:39.376683 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.376556    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d96594b-ec30-4b1e-ac94-57bad258b6c4" containerName="main"
Aug 05 18:14:39.376683 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.376569    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d96594b-ec30-4b1e-ac94-57bad258b6c4" containerName="llm-d-routing-sidecar"
Aug 05 18:14:39.382197 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.382172    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:39.384662 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.384637    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Aug 05 18:14:39.384824 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.384803    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Aug 05 18:14:39.384911 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.384674    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Aug 05 18:14:39.384911 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.384718    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-s8zqf\""
Aug 05 18:14:39.391106 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.390700    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"]
Aug 05 18:14:39.479369 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.479284    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9bd4a8-4d15-442b-a299-d0530b0be122-workload-certs\") pod \"router-gateway-2-openshift-default-6cb79f7856-xpkq5\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:39.479592 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.479408    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9bd4a8-4d15-442b-a299-d0530b0be122-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-6cb79f7856-xpkq5\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:39.479592 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.479453    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6kmx\" (UniqueName: \"kubernetes.io/projected/8d9bd4a8-4d15-442b-a299-d0530b0be122-kube-api-access-t6kmx\") pod \"router-gateway-2-openshift-default-6cb79f7856-xpkq5\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:39.479592 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.479509    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9bd4a8-4d15-442b-a299-d0530b0be122-istio-data\") pod \"router-gateway-2-openshift-default-6cb79f7856-xpkq5\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:39.479592 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.479552    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9bd4a8-4d15-442b-a299-d0530b0be122-credential-socket\") pod \"router-gateway-2-openshift-default-6cb79f7856-xpkq5\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:39.479592 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.479582    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9bd4a8-4d15-442b-a299-d0530b0be122-istio-envoy\") pod \"router-gateway-2-openshift-default-6cb79f7856-xpkq5\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:39.479890 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.479639    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9bd4a8-4d15-442b-a299-d0530b0be122-istio-podinfo\") pod \"router-gateway-2-openshift-default-6cb79f7856-xpkq5\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:39.479890 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.479695    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9bd4a8-4d15-442b-a299-d0530b0be122-workload-socket\") pod \"router-gateway-2-openshift-default-6cb79f7856-xpkq5\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:39.479890 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.479748    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9bd4a8-4d15-442b-a299-d0530b0be122-istio-token\") pod \"router-gateway-2-openshift-default-6cb79f7856-xpkq5\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:39.580375 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.580330    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9bd4a8-4d15-442b-a299-d0530b0be122-istio-data\") pod \"router-gateway-2-openshift-default-6cb79f7856-xpkq5\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:39.580375 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.580383    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9bd4a8-4d15-442b-a299-d0530b0be122-credential-socket\") pod \"router-gateway-2-openshift-default-6cb79f7856-xpkq5\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:39.580659 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.580414    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9bd4a8-4d15-442b-a299-d0530b0be122-istio-envoy\") pod \"router-gateway-2-openshift-default-6cb79f7856-xpkq5\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:39.580659 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.580534    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9bd4a8-4d15-442b-a299-d0530b0be122-istio-podinfo\") pod \"router-gateway-2-openshift-default-6cb79f7856-xpkq5\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:39.580659 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.580577    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9bd4a8-4d15-442b-a299-d0530b0be122-workload-socket\") pod \"router-gateway-2-openshift-default-6cb79f7856-xpkq5\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:39.580659 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.580629    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9bd4a8-4d15-442b-a299-d0530b0be122-istio-token\") pod \"router-gateway-2-openshift-default-6cb79f7856-xpkq5\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:39.580903 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.580675    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9bd4a8-4d15-442b-a299-d0530b0be122-workload-certs\") pod \"router-gateway-2-openshift-default-6cb79f7856-xpkq5\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:39.580903 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.580736    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9bd4a8-4d15-442b-a299-d0530b0be122-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-6cb79f7856-xpkq5\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:39.580903 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.580760    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6kmx\" (UniqueName: \"kubernetes.io/projected/8d9bd4a8-4d15-442b-a299-d0530b0be122-kube-api-access-t6kmx\") pod \"router-gateway-2-openshift-default-6cb79f7856-xpkq5\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:39.580903 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.580771    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9bd4a8-4d15-442b-a299-d0530b0be122-credential-socket\") pod \"router-gateway-2-openshift-default-6cb79f7856-xpkq5\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:39.580903 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.580832    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9bd4a8-4d15-442b-a299-d0530b0be122-istio-data\") pod \"router-gateway-2-openshift-default-6cb79f7856-xpkq5\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:39.581235 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.580998    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9bd4a8-4d15-442b-a299-d0530b0be122-workload-socket\") pod \"router-gateway-2-openshift-default-6cb79f7856-xpkq5\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:39.581327 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.581303    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9bd4a8-4d15-442b-a299-d0530b0be122-workload-certs\") pod \"router-gateway-2-openshift-default-6cb79f7856-xpkq5\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:39.581498 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.581478    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9bd4a8-4d15-442b-a299-d0530b0be122-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-6cb79f7856-xpkq5\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:39.582852 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.582830    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9bd4a8-4d15-442b-a299-d0530b0be122-istio-envoy\") pod \"router-gateway-2-openshift-default-6cb79f7856-xpkq5\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:39.583092 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.583072    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9bd4a8-4d15-442b-a299-d0530b0be122-istio-podinfo\") pod \"router-gateway-2-openshift-default-6cb79f7856-xpkq5\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:39.588910 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.588888    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6kmx\" (UniqueName: \"kubernetes.io/projected/8d9bd4a8-4d15-442b-a299-d0530b0be122-kube-api-access-t6kmx\") pod \"router-gateway-2-openshift-default-6cb79f7856-xpkq5\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:39.589063 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.588983    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9bd4a8-4d15-442b-a299-d0530b0be122-istio-token\") pod \"router-gateway-2-openshift-default-6cb79f7856-xpkq5\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:39.695056 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.694939    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:39.844616 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.844585    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"]
Aug 05 18:14:39.846560 ip-10-0-135-151 kubenswrapper[2561]: W0805 18:14:39.846519    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9bd4a8_4d15_442b_a299_d0530b0be122.slice/crio-07e7acf02e57c56df5cc26b01b8b89c303e7ae4e8f95c3018fb1f07b8ba99325 WatchSource:0}: Error finding container 07e7acf02e57c56df5cc26b01b8b89c303e7ae4e8f95c3018fb1f07b8ba99325: Status 404 returned error can't find the container with id 07e7acf02e57c56df5cc26b01b8b89c303e7ae4e8f95c3018fb1f07b8ba99325
Aug 05 18:14:39.849149 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.849113    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 18:14:39.849267 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.849200    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 18:14:39.849267 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.849250    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 18:14:39.996103 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.996002    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5" event={"ID":"8d9bd4a8-4d15-442b-a299-d0530b0be122","Type":"ContainerStarted","Data":"98395a4289af1b3f9b3182e9de622102a3df16cebeba62c1d11e8e13d2a2edae"}
Aug 05 18:14:39.996103 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:39.996067    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5" event={"ID":"8d9bd4a8-4d15-442b-a299-d0530b0be122","Type":"ContainerStarted","Data":"07e7acf02e57c56df5cc26b01b8b89c303e7ae4e8f95c3018fb1f07b8ba99325"}
Aug 05 18:14:40.015672 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:40.015605    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5" podStartSLOduration=1.015584442 podStartE2EDuration="1.015584442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:14:40.015199487 +0000 UTC m=+3735.759833818" watchObservedRunningTime="2026-08-05 18:14:40.015584442 +0000 UTC m=+3735.760218771"
Aug 05 18:14:40.696144 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:40.696099    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:40.701671 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:40.701646    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:41.001534 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:41.001437    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:41.002624 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:41.002603    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:43.838256 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:43.838217    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq" podUID="ae6bbcde-0de7-4bdd-8b7d-5d0706737739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/health\": dial tcp 10.133.0.109:8000: connect: connection refused"
Aug 05 18:14:52.078319 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:52.078285    2561 generic.go:356] "Generic (PLEG): container finished" podID="8d9bd4a8-4d15-442b-a299-d0530b0be122" containerID="98395a4289af1b3f9b3182e9de622102a3df16cebeba62c1d11e8e13d2a2edae" exitCode=0
Aug 05 18:14:52.078755 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:52.078355    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5" event={"ID":"8d9bd4a8-4d15-442b-a299-d0530b0be122","Type":"ContainerDied","Data":"98395a4289af1b3f9b3182e9de622102a3df16cebeba62c1d11e8e13d2a2edae"}
Aug 05 18:14:52.078800 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:52.078782    2561 scope.go:117] "RemoveContainer" containerID="98395a4289af1b3f9b3182e9de622102a3df16cebeba62c1d11e8e13d2a2edae"
Aug 05 18:14:52.079827 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:52.079793    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 18:14:52.079904 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:52.079868    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 18:14:52.079943 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:52.079907    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 05 18:14:53.085624 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:53.085587    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5" event={"ID":"8d9bd4a8-4d15-442b-a299-d0530b0be122","Type":"ContainerStarted","Data":"464fb709e163b176a47eafaf1e5c5c6dddd3e67f86f69bc2000ac7a1157b29b9"}
Aug 05 18:14:53.695918 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:53.695879    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:53.697020 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:53.696984    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5" podUID="8d9bd4a8-4d15-442b-a299-d0530b0be122" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Aug 05 18:14:53.838003 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:53.837941    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq" podUID="ae6bbcde-0de7-4bdd-8b7d-5d0706737739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/health\": dial tcp 10.133.0.109:8000: connect: connection refused"
Aug 05 18:14:54.696336 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:54.696289    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5" podUID="8d9bd4a8-4d15-442b-a299-d0530b0be122" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Aug 05 18:14:55.695453 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:55.695409    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5" podUID="8d9bd4a8-4d15-442b-a299-d0530b0be122" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Aug 05 18:14:56.001983 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:56.001880    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:56.695937 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:56.695886    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5" podUID="8d9bd4a8-4d15-442b-a299-d0530b0be122" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Aug 05 18:14:56.874497 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:56.874454    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"]
Aug 05 18:14:56.878567 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:56.878529    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:14:56.880974 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:56.880930    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Aug 05 18:14:56.881154 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:56.881137    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Aug 05 18:14:56.882238 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:56.882217    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-nc6z7\""
Aug 05 18:14:56.882356 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:56.882286    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-4xrgk\""
Aug 05 18:14:56.882356 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:56.882314    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Aug 05 18:14:56.895005 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:56.894978    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"]
Aug 05 18:14:56.940269 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:56.940238    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c99d3ec-3f22-4230-976f-e1507d64a94b-tls-certs\") pod \"router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:14:56.940471 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:56.940280    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c99d3ec-3f22-4230-976f-e1507d64a94b-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:14:56.940471 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:56.940301    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6nhj\" (UniqueName: \"kubernetes.io/projected/2c99d3ec-3f22-4230-976f-e1507d64a94b-kube-api-access-c6nhj\") pod \"router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:14:56.940471 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:56.940320    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c99d3ec-3f22-4230-976f-e1507d64a94b-dshm\") pod \"router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:14:56.940471 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:56.940361    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c99d3ec-3f22-4230-976f-e1507d64a94b-model-cache\") pod \"router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:14:56.940471 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:56.940389    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c99d3ec-3f22-4230-976f-e1507d64a94b-home\") pod \"router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:14:56.940471 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:56.940404    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c99d3ec-3f22-4230-976f-e1507d64a94b-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:14:56.940471 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:56.940439    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c99d3ec-3f22-4230-976f-e1507d64a94b-tmp-dir\") pod \"router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:14:57.041508 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:57.041477    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c99d3ec-3f22-4230-976f-e1507d64a94b-model-cache\") pod \"router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:14:57.042036 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:57.041517    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c99d3ec-3f22-4230-976f-e1507d64a94b-home\") pod \"router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:14:57.042036 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:57.041533    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c99d3ec-3f22-4230-976f-e1507d64a94b-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:14:57.042036 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:57.041564    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c99d3ec-3f22-4230-976f-e1507d64a94b-tmp-dir\") pod \"router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:14:57.042036 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:57.041597    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c99d3ec-3f22-4230-976f-e1507d64a94b-tls-certs\") pod \"router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:14:57.042036 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:57.041620    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c99d3ec-3f22-4230-976f-e1507d64a94b-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:14:57.042036 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:57.041637    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6nhj\" (UniqueName: \"kubernetes.io/projected/2c99d3ec-3f22-4230-976f-e1507d64a94b-kube-api-access-c6nhj\") pod \"router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:14:57.042036 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:57.041656    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c99d3ec-3f22-4230-976f-e1507d64a94b-dshm\") pod \"router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:14:57.042036 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:57.041940    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c99d3ec-3f22-4230-976f-e1507d64a94b-model-cache\") pod \"router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:14:57.042447 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:57.042058    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c99d3ec-3f22-4230-976f-e1507d64a94b-tmp-dir\") pod \"router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:14:57.042447 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:57.042081    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c99d3ec-3f22-4230-976f-e1507d64a94b-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:14:57.042447 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:57.042356    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c99d3ec-3f22-4230-976f-e1507d64a94b-home\") pod \"router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:14:57.042550 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:57.042539    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c99d3ec-3f22-4230-976f-e1507d64a94b-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:14:57.043921 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:57.043901    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c99d3ec-3f22-4230-976f-e1507d64a94b-dshm\") pod \"router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:14:57.044668 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:57.044652    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c99d3ec-3f22-4230-976f-e1507d64a94b-tls-certs\") pod \"router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:14:57.050368 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:57.050318    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6nhj\" (UniqueName: \"kubernetes.io/projected/2c99d3ec-3f22-4230-976f-e1507d64a94b-kube-api-access-c6nhj\") pod \"router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:14:57.189102 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:57.189064    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:14:57.398752 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:57.398716    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"]
Aug 05 18:14:57.401178 ip-10-0-135-151 kubenswrapper[2561]: W0805 18:14:57.401150    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c99d3ec_3f22_4230_976f_e1507d64a94b.slice/crio-c6101dbeed5a657169b12cde10557ca2785595723edb0dc23d4aa4c6c53ed27d WatchSource:0}: Error finding container c6101dbeed5a657169b12cde10557ca2785595723edb0dc23d4aa4c6c53ed27d: Status 404 returned error can't find the container with id c6101dbeed5a657169b12cde10557ca2785595723edb0dc23d4aa4c6c53ed27d
Aug 05 18:14:57.695851 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:57.695755    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5" podUID="8d9bd4a8-4d15-442b-a299-d0530b0be122" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Aug 05 18:14:58.118455 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:58.118404    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm" event={"ID":"2c99d3ec-3f22-4230-976f-e1507d64a94b","Type":"ContainerStarted","Data":"4932f548081561953fe86f25ca89772bb54a5c9c3633d6ba47d4e55037cc0520"}
Aug 05 18:14:58.119009 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:58.118480    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:14:58.119009 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:58.118499    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm" event={"ID":"2c99d3ec-3f22-4230-976f-e1507d64a94b","Type":"ContainerStarted","Data":"c6101dbeed5a657169b12cde10557ca2785595723edb0dc23d4aa4c6c53ed27d"}
Aug 05 18:14:58.700653 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:58.700617    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:58.701941 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:58.701881    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:14:59.128966 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:14:59.128912    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm" event={"ID":"2c99d3ec-3f22-4230-976f-e1507d64a94b","Type":"ContainerStarted","Data":"42ff87c3a640b1fa475c6bb736f9da92e86fad2d5581e0b3594a4f2b19b8a77f"}
Aug 05 18:15:01.141245 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:01.141202    2561 generic.go:356] "Generic (PLEG): container finished" podID="2c99d3ec-3f22-4230-976f-e1507d64a94b" containerID="42ff87c3a640b1fa475c6bb736f9da92e86fad2d5581e0b3594a4f2b19b8a77f" exitCode=0
Aug 05 18:15:01.141761 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:01.141254    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm" event={"ID":"2c99d3ec-3f22-4230-976f-e1507d64a94b","Type":"ContainerDied","Data":"42ff87c3a640b1fa475c6bb736f9da92e86fad2d5581e0b3594a4f2b19b8a77f"}
Aug 05 18:15:02.150092 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:02.149990    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm" event={"ID":"2c99d3ec-3f22-4230-976f-e1507d64a94b","Type":"ContainerStarted","Data":"7a9868560b37fdf568881ffe12f402c8da610ed45a96697ed4ca26313305a5c0"}
Aug 05 18:15:02.174515 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:02.174443    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm" podStartSLOduration=3.634863426 podStartE2EDuration="6.174424576s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.53956115s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:15:02.172522975 +0000 UTC m=+3757.917157323" watchObservedRunningTime="2026-08-05 18:15:02.174424576 +0000 UTC m=+3757.919058920"
Aug 05 18:15:03.849154 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:03.849119    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:15:03.857949 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:03.857925    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:15:07.190175 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:07.190134    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:15:07.190638 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:07.190204    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:15:07.191706 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:07.191668    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm" podUID="2c99d3ec-3f22-4230-976f-e1507d64a94b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8001/health\": dial tcp 10.133.0.111:8001: connect: connection refused"
Aug 05 18:15:07.203449 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:07.203417    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:15:08.651879 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:08.651839    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"]
Aug 05 18:15:08.675187 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:08.675155    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"]
Aug 05 18:15:08.678052 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:08.678028    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"
Aug 05 18:15:08.693686 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:08.693656    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"]
Aug 05 18:15:08.760943 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:08.760897    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab560477-018f-4e7e-b217-9d7314392009-tls-certs\") pod \"prestop-hook-test-kserve-6f5dbd7f74-hst2g\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"
Aug 05 18:15:08.761164 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:08.760945    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab560477-018f-4e7e-b217-9d7314392009-home\") pod \"prestop-hook-test-kserve-6f5dbd7f74-hst2g\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"
Aug 05 18:15:08.761164 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:08.760994    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab560477-018f-4e7e-b217-9d7314392009-dshm\") pod \"prestop-hook-test-kserve-6f5dbd7f74-hst2g\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"
Aug 05 18:15:08.761164 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:08.761014    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab560477-018f-4e7e-b217-9d7314392009-model-cache\") pod \"prestop-hook-test-kserve-6f5dbd7f74-hst2g\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"
Aug 05 18:15:08.761164 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:08.761036    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab560477-018f-4e7e-b217-9d7314392009-kserve-provision-location\") pod \"prestop-hook-test-kserve-6f5dbd7f74-hst2g\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"
Aug 05 18:15:08.761164 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:08.761065    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28knv\" (UniqueName: \"kubernetes.io/projected/ab560477-018f-4e7e-b217-9d7314392009-kube-api-access-28knv\") pod \"prestop-hook-test-kserve-6f5dbd7f74-hst2g\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"
Aug 05 18:15:08.761164 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:08.761151    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab560477-018f-4e7e-b217-9d7314392009-cabundle-cert\") pod \"prestop-hook-test-kserve-6f5dbd7f74-hst2g\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"
Aug 05 18:15:08.761522 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:08.761207    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab560477-018f-4e7e-b217-9d7314392009-tmp-dir\") pod \"prestop-hook-test-kserve-6f5dbd7f74-hst2g\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"
Aug 05 18:15:08.862510 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:08.862473    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab560477-018f-4e7e-b217-9d7314392009-tls-certs\") pod \"prestop-hook-test-kserve-6f5dbd7f74-hst2g\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"
Aug 05 18:15:08.862510 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:08.862517    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab560477-018f-4e7e-b217-9d7314392009-home\") pod \"prestop-hook-test-kserve-6f5dbd7f74-hst2g\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"
Aug 05 18:15:08.862778 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:08.862536    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab560477-018f-4e7e-b217-9d7314392009-dshm\") pod \"prestop-hook-test-kserve-6f5dbd7f74-hst2g\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"
Aug 05 18:15:08.862778 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:08.862551    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab560477-018f-4e7e-b217-9d7314392009-model-cache\") pod \"prestop-hook-test-kserve-6f5dbd7f74-hst2g\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"
Aug 05 18:15:08.862778 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:08.862569    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab560477-018f-4e7e-b217-9d7314392009-kserve-provision-location\") pod \"prestop-hook-test-kserve-6f5dbd7f74-hst2g\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"
Aug 05 18:15:08.862778 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:08.862593    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28knv\" (UniqueName: \"kubernetes.io/projected/ab560477-018f-4e7e-b217-9d7314392009-kube-api-access-28knv\") pod \"prestop-hook-test-kserve-6f5dbd7f74-hst2g\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"
Aug 05 18:15:08.862778 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:08.862684    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab560477-018f-4e7e-b217-9d7314392009-cabundle-cert\") pod \"prestop-hook-test-kserve-6f5dbd7f74-hst2g\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"
Aug 05 18:15:08.862778 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:08.862746    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab560477-018f-4e7e-b217-9d7314392009-tmp-dir\") pod \"prestop-hook-test-kserve-6f5dbd7f74-hst2g\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"
Aug 05 18:15:08.863130 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:08.862929    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab560477-018f-4e7e-b217-9d7314392009-home\") pod \"prestop-hook-test-kserve-6f5dbd7f74-hst2g\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"
Aug 05 18:15:08.863130 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:08.863022    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab560477-018f-4e7e-b217-9d7314392009-kserve-provision-location\") pod \"prestop-hook-test-kserve-6f5dbd7f74-hst2g\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"
Aug 05 18:15:08.863214 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:08.863153    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab560477-018f-4e7e-b217-9d7314392009-tmp-dir\") pod \"prestop-hook-test-kserve-6f5dbd7f74-hst2g\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"
Aug 05 18:15:08.863255 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:08.863228    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab560477-018f-4e7e-b217-9d7314392009-model-cache\") pod \"prestop-hook-test-kserve-6f5dbd7f74-hst2g\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"
Aug 05 18:15:08.863589 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:08.863572    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab560477-018f-4e7e-b217-9d7314392009-cabundle-cert\") pod \"prestop-hook-test-kserve-6f5dbd7f74-hst2g\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"
Aug 05 18:15:08.864964 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:08.864927    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab560477-018f-4e7e-b217-9d7314392009-dshm\") pod \"prestop-hook-test-kserve-6f5dbd7f74-hst2g\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"
Aug 05 18:15:08.865653 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:08.865633    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab560477-018f-4e7e-b217-9d7314392009-tls-certs\") pod \"prestop-hook-test-kserve-6f5dbd7f74-hst2g\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"
Aug 05 18:15:08.872011 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:08.871987    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28knv\" (UniqueName: \"kubernetes.io/projected/ab560477-018f-4e7e-b217-9d7314392009-kube-api-access-28knv\") pod \"prestop-hook-test-kserve-6f5dbd7f74-hst2g\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"
Aug 05 18:15:08.991735 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:08.991643    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"
Aug 05 18:15:09.146900 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:09.146869    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"]
Aug 05 18:15:09.150420 ip-10-0-135-151 kubenswrapper[2561]: W0805 18:15:09.150391    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab560477_018f_4e7e_b217_9d7314392009.slice/crio-59510074b02cfd79ef05bc836685082568c90cfac3a1b37abc2a2fce269bd702 WatchSource:0}: Error finding container 59510074b02cfd79ef05bc836685082568c90cfac3a1b37abc2a2fce269bd702: Status 404 returned error can't find the container with id 59510074b02cfd79ef05bc836685082568c90cfac3a1b37abc2a2fce269bd702
Aug 05 18:15:09.193977 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:09.193930    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g" event={"ID":"ab560477-018f-4e7e-b217-9d7314392009","Type":"ContainerStarted","Data":"59510074b02cfd79ef05bc836685082568c90cfac3a1b37abc2a2fce269bd702"}
Aug 05 18:15:10.200726 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:10.200688    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g" event={"ID":"ab560477-018f-4e7e-b217-9d7314392009","Type":"ContainerStarted","Data":"1d15063678313c711a443b9cacce6b8081e089bd53e925b82e2ecfa1c29a6cbb"}
Aug 05 18:15:12.216663 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:12.216624    2561 generic.go:356] "Generic (PLEG): container finished" podID="ab560477-018f-4e7e-b217-9d7314392009" containerID="1d15063678313c711a443b9cacce6b8081e089bd53e925b82e2ecfa1c29a6cbb" exitCode=0
Aug 05 18:15:12.217160 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:12.216698    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g" event={"ID":"ab560477-018f-4e7e-b217-9d7314392009","Type":"ContainerDied","Data":"1d15063678313c711a443b9cacce6b8081e089bd53e925b82e2ecfa1c29a6cbb"}
Aug 05 18:15:13.224695 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:13.224657    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g" event={"ID":"ab560477-018f-4e7e-b217-9d7314392009","Type":"ContainerStarted","Data":"1177f8d3c8b061cff704fe9c80685b9d35d6695f9f4802b020e2d8b8c914d90b"}
Aug 05 18:15:13.249305 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:13.249242    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g" podStartSLOduration=2.91174173 podStartE2EDuration="5.249225674s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.337483944s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:15:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:15:13.245837362 +0000 UTC m=+3768.990471709" watchObservedRunningTime="2026-08-05 18:15:13.249225674 +0000 UTC m=+3768.993860002"
Aug 05 18:15:13.992166 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:13.992123    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"
Aug 05 18:15:17.189921 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:17.189807    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm" podUID="2c99d3ec-3f22-4230-976f-e1507d64a94b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8001/health\": dial tcp 10.133.0.111:8001: connect: connection refused"
Aug 05 18:15:18.992121 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:18.992074    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"
Aug 05 18:15:18.993983 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:18.993931    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g" podUID="ab560477-018f-4e7e-b217-9d7314392009" containerName="main" probeResult="failure" output="Get \"https://10.133.0.112:8000/health\": dial tcp 10.133.0.112:8000: connect: connection refused"
Aug 05 18:15:23.675210 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:23.675145    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq" podUID="ae6bbcde-0de7-4bdd-8b7d-5d0706737739" containerName="main" containerID="cri-o://1450ee1888d88c0a1320dc520cac3ef0a4f79242a832bd9964b0750d8f17c464" gracePeriod=45
Aug 05 18:15:23.850293 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:23.850248    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq" podUID="ae6bbcde-0de7-4bdd-8b7d-5d0706737739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.109:8000/health\": dial tcp 10.133.0.109:8000: connect: connection refused"
Aug 05 18:15:24.299317 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:24.299280    2561 generic.go:356] "Generic (PLEG): container finished" podID="ae6bbcde-0de7-4bdd-8b7d-5d0706737739" containerID="1450ee1888d88c0a1320dc520cac3ef0a4f79242a832bd9964b0750d8f17c464" exitCode=0
Aug 05 18:15:24.299487 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:24.299360    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq" event={"ID":"ae6bbcde-0de7-4bdd-8b7d-5d0706737739","Type":"ContainerDied","Data":"1450ee1888d88c0a1320dc520cac3ef0a4f79242a832bd9964b0750d8f17c464"}
Aug 05 18:15:24.433140 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:24.433112    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:15:24.515490 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:24.515376    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-kserve-provision-location\") pod \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") "
Aug 05 18:15:24.515490 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:24.515436    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-dshm\") pod \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") "
Aug 05 18:15:24.515748 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:24.515495    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkksp\" (UniqueName: \"kubernetes.io/projected/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-kube-api-access-jkksp\") pod \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") "
Aug 05 18:15:24.515748 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:24.515535    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-model-cache\") pod \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") "
Aug 05 18:15:24.515748 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:24.515610    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-tls-certs\") pod \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") "
Aug 05 18:15:24.515748 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:24.515645    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-tmp-dir\") pod \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") "
Aug 05 18:15:24.515748 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:24.515692    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-home\") pod \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") "
Aug 05 18:15:24.515748 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:24.515716    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-cabundle-cert\") pod \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\" (UID: \"ae6bbcde-0de7-4bdd-8b7d-5d0706737739\") "
Aug 05 18:15:24.517797 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:24.516784    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae6bbcde-0de7-4bdd-8b7d-5d0706737739" (UID: "ae6bbcde-0de7-4bdd-8b7d-5d0706737739"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:15:24.517797 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:24.517258    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-home" (OuterVolumeSpecName: "home") pod "ae6bbcde-0de7-4bdd-8b7d-5d0706737739" (UID: "ae6bbcde-0de7-4bdd-8b7d-5d0706737739"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:15:24.517797 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:24.517417    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ae6bbcde-0de7-4bdd-8b7d-5d0706737739" (UID: "ae6bbcde-0de7-4bdd-8b7d-5d0706737739"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 18:15:24.520774 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:24.520746    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-kube-api-access-jkksp" (OuterVolumeSpecName: "kube-api-access-jkksp") pod "ae6bbcde-0de7-4bdd-8b7d-5d0706737739" (UID: "ae6bbcde-0de7-4bdd-8b7d-5d0706737739"). InnerVolumeSpecName "kube-api-access-jkksp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 18:15:24.521231 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:24.521195    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-dshm" (OuterVolumeSpecName: "dshm") pod "ae6bbcde-0de7-4bdd-8b7d-5d0706737739" (UID: "ae6bbcde-0de7-4bdd-8b7d-5d0706737739"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:15:24.525185 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:24.525156    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae6bbcde-0de7-4bdd-8b7d-5d0706737739" (UID: "ae6bbcde-0de7-4bdd-8b7d-5d0706737739"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 18:15:24.535873 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:24.535835    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae6bbcde-0de7-4bdd-8b7d-5d0706737739" (UID: "ae6bbcde-0de7-4bdd-8b7d-5d0706737739"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:15:24.544456 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:24.544421    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ae6bbcde-0de7-4bdd-8b7d-5d0706737739" (UID: "ae6bbcde-0de7-4bdd-8b7d-5d0706737739"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:15:24.616823 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:24.616783    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:15:24.616823 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:24.616825    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:15:24.617112 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:24.616839    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:15:24.617112 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:24.616853    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-cabundle-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:15:24.617112 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:24.616867    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-kserve-provision-location\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:15:24.617112 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:24.616882    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:15:24.617112 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:24.616897    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkksp\" (UniqueName: \"kubernetes.io/projected/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-kube-api-access-jkksp\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:15:24.617112 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:24.616911    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae6bbcde-0de7-4bdd-8b7d-5d0706737739-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:15:25.306931 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:25.306900    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"
Aug 05 18:15:25.307390 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:25.306899    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq" event={"ID":"ae6bbcde-0de7-4bdd-8b7d-5d0706737739","Type":"ContainerDied","Data":"3febd68460a767d67935d0a4c5519a2302d1efa41fbbff4cb057229aaa0b595d"}
Aug 05 18:15:25.307390 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:25.307029    2561 scope.go:117] "RemoveContainer" containerID="1450ee1888d88c0a1320dc520cac3ef0a4f79242a832bd9964b0750d8f17c464"
Aug 05 18:15:25.317466 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:25.317438    2561 scope.go:117] "RemoveContainer" containerID="524ba269289dd0575810c3de875868faaab5f61f39591d1f38706677ccb6c5e4"
Aug 05 18:15:25.329250 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:25.329203    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"]
Aug 05 18:15:25.332071 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:25.332044    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-j2qjq"]
Aug 05 18:15:26.731833 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:26.731796    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae6bbcde-0de7-4bdd-8b7d-5d0706737739" path="/var/lib/kubelet/pods/ae6bbcde-0de7-4bdd-8b7d-5d0706737739/volumes"
Aug 05 18:15:27.025449 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:27.025355    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"]
Aug 05 18:15:27.190325 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:27.190262    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm" podUID="2c99d3ec-3f22-4230-976f-e1507d64a94b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8001/health\": dial tcp 10.133.0.111:8001: connect: connection refused"
Aug 05 18:15:37.190086 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:37.190032    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm" podUID="2c99d3ec-3f22-4230-976f-e1507d64a94b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8001/health\": dial tcp 10.133.0.111:8001: connect: connection refused"
Aug 05 18:15:42.056662 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:42.056617    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g" podUID="ab560477-018f-4e7e-b217-9d7314392009" containerName="main" containerID="cri-o://1177f8d3c8b061cff704fe9c80685b9d35d6695f9f4802b020e2d8b8c914d90b" gracePeriod=45
Aug 05 18:15:47.190306 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:47.190249    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm" podUID="2c99d3ec-3f22-4230-976f-e1507d64a94b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8001/health\": dial tcp 10.133.0.111:8001: connect: connection refused"
Aug 05 18:15:57.190388 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:15:57.190337    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm" podUID="2c99d3ec-3f22-4230-976f-e1507d64a94b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8001/health\": dial tcp 10.133.0.111:8001: connect: connection refused"
Aug 05 18:16:07.190196 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:07.190146    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm" podUID="2c99d3ec-3f22-4230-976f-e1507d64a94b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8001/health\": dial tcp 10.133.0.111:8001: connect: connection refused"
Aug 05 18:16:17.189477 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:17.189426    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm" podUID="2c99d3ec-3f22-4230-976f-e1507d64a94b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8001/health\": dial tcp 10.133.0.111:8001: connect: connection refused"
Aug 05 18:16:27.189811 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.189761    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm" podUID="2c99d3ec-3f22-4230-976f-e1507d64a94b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8001/health\": dial tcp 10.133.0.111:8001: connect: connection refused"
Aug 05 18:16:27.710285 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.710254    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-6f5dbd7f74-hst2g_ab560477-018f-4e7e-b217-9d7314392009/main/0.log"
Aug 05 18:16:27.710741 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.710713    2561 generic.go:356] "Generic (PLEG): container finished" podID="ab560477-018f-4e7e-b217-9d7314392009" containerID="1177f8d3c8b061cff704fe9c80685b9d35d6695f9f4802b020e2d8b8c914d90b" exitCode=137
Aug 05 18:16:27.710878 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.710785    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g" event={"ID":"ab560477-018f-4e7e-b217-9d7314392009","Type":"ContainerDied","Data":"1177f8d3c8b061cff704fe9c80685b9d35d6695f9f4802b020e2d8b8c914d90b"}
Aug 05 18:16:27.710878 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.710843    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g" event={"ID":"ab560477-018f-4e7e-b217-9d7314392009","Type":"ContainerDied","Data":"59510074b02cfd79ef05bc836685082568c90cfac3a1b37abc2a2fce269bd702"}
Aug 05 18:16:27.710878 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.710861    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59510074b02cfd79ef05bc836685082568c90cfac3a1b37abc2a2fce269bd702"
Aug 05 18:16:27.721045 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.721018    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-6f5dbd7f74-hst2g_ab560477-018f-4e7e-b217-9d7314392009/main/0.log"
Aug 05 18:16:27.722640 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.722619    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"
Aug 05 18:16:27.835375 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.835282    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab560477-018f-4e7e-b217-9d7314392009-model-cache\") pod \"ab560477-018f-4e7e-b217-9d7314392009\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") "
Aug 05 18:16:27.835375 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.835348    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab560477-018f-4e7e-b217-9d7314392009-tls-certs\") pod \"ab560477-018f-4e7e-b217-9d7314392009\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") "
Aug 05 18:16:27.835598 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.835378    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab560477-018f-4e7e-b217-9d7314392009-tmp-dir\") pod \"ab560477-018f-4e7e-b217-9d7314392009\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") "
Aug 05 18:16:27.835598 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.835406    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab560477-018f-4e7e-b217-9d7314392009-cabundle-cert\") pod \"ab560477-018f-4e7e-b217-9d7314392009\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") "
Aug 05 18:16:27.835598 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.835514    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab560477-018f-4e7e-b217-9d7314392009-home\") pod \"ab560477-018f-4e7e-b217-9d7314392009\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") "
Aug 05 18:16:27.835757 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.835595    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab560477-018f-4e7e-b217-9d7314392009-model-cache" (OuterVolumeSpecName: "model-cache") pod "ab560477-018f-4e7e-b217-9d7314392009" (UID: "ab560477-018f-4e7e-b217-9d7314392009"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:16:27.835757 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.835599    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab560477-018f-4e7e-b217-9d7314392009-dshm\") pod \"ab560477-018f-4e7e-b217-9d7314392009\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") "
Aug 05 18:16:27.835757 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.835655    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab560477-018f-4e7e-b217-9d7314392009-kserve-provision-location\") pod \"ab560477-018f-4e7e-b217-9d7314392009\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") "
Aug 05 18:16:27.835757 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.835715    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28knv\" (UniqueName: \"kubernetes.io/projected/ab560477-018f-4e7e-b217-9d7314392009-kube-api-access-28knv\") pod \"ab560477-018f-4e7e-b217-9d7314392009\" (UID: \"ab560477-018f-4e7e-b217-9d7314392009\") "
Aug 05 18:16:27.835757 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.835726    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab560477-018f-4e7e-b217-9d7314392009-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ab560477-018f-4e7e-b217-9d7314392009" (UID: "ab560477-018f-4e7e-b217-9d7314392009"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:16:27.836040 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.836010    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab560477-018f-4e7e-b217-9d7314392009-home" (OuterVolumeSpecName: "home") pod "ab560477-018f-4e7e-b217-9d7314392009" (UID: "ab560477-018f-4e7e-b217-9d7314392009"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:16:27.836104 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.836068    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab560477-018f-4e7e-b217-9d7314392009-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:16:27.836104 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.836087    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab560477-018f-4e7e-b217-9d7314392009-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:16:27.836371 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.836345    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab560477-018f-4e7e-b217-9d7314392009-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ab560477-018f-4e7e-b217-9d7314392009" (UID: "ab560477-018f-4e7e-b217-9d7314392009"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 18:16:27.837649 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.837620    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab560477-018f-4e7e-b217-9d7314392009-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab560477-018f-4e7e-b217-9d7314392009" (UID: "ab560477-018f-4e7e-b217-9d7314392009"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 18:16:27.838384 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.838346    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab560477-018f-4e7e-b217-9d7314392009-dshm" (OuterVolumeSpecName: "dshm") pod "ab560477-018f-4e7e-b217-9d7314392009" (UID: "ab560477-018f-4e7e-b217-9d7314392009"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:16:27.838477 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.838457    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab560477-018f-4e7e-b217-9d7314392009-kube-api-access-28knv" (OuterVolumeSpecName: "kube-api-access-28knv") pod "ab560477-018f-4e7e-b217-9d7314392009" (UID: "ab560477-018f-4e7e-b217-9d7314392009"). InnerVolumeSpecName "kube-api-access-28knv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 18:16:27.859366 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.859325    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab560477-018f-4e7e-b217-9d7314392009-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ab560477-018f-4e7e-b217-9d7314392009" (UID: "ab560477-018f-4e7e-b217-9d7314392009"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:16:27.936860 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.936821    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab560477-018f-4e7e-b217-9d7314392009-cabundle-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:16:27.936860 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.936855    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab560477-018f-4e7e-b217-9d7314392009-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:16:27.936860 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.936868    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab560477-018f-4e7e-b217-9d7314392009-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:16:27.937108 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.936877    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab560477-018f-4e7e-b217-9d7314392009-kserve-provision-location\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:16:27.937108 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.936886    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28knv\" (UniqueName: \"kubernetes.io/projected/ab560477-018f-4e7e-b217-9d7314392009-kube-api-access-28knv\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:16:27.937108 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:27.936894    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab560477-018f-4e7e-b217-9d7314392009-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:16:28.716541 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:28.716506    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"
Aug 05 18:16:28.740969 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:28.740927    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"]
Aug 05 18:16:28.744408 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:28.744383    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6f5dbd7f74-hst2g"]
Aug 05 18:16:30.730278 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:30.730250    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab560477-018f-4e7e-b217-9d7314392009" path="/var/lib/kubelet/pods/ab560477-018f-4e7e-b217-9d7314392009/volumes"
Aug 05 18:16:37.189594 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:37.189540    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm" podUID="2c99d3ec-3f22-4230-976f-e1507d64a94b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8001/health\": dial tcp 10.133.0.111:8001: connect: connection refused"
Aug 05 18:16:46.742865 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:46.742666    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b"]
Aug 05 18:16:46.743430 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:46.743403    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab560477-018f-4e7e-b217-9d7314392009" containerName="main"
Aug 05 18:16:46.743568 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:46.743438    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab560477-018f-4e7e-b217-9d7314392009" containerName="main"
Aug 05 18:16:46.743568 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:46.743455    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae6bbcde-0de7-4bdd-8b7d-5d0706737739" containerName="storage-initializer"
Aug 05 18:16:46.743568 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:46.743460    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae6bbcde-0de7-4bdd-8b7d-5d0706737739" containerName="storage-initializer"
Aug 05 18:16:46.743568 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:46.743480    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae6bbcde-0de7-4bdd-8b7d-5d0706737739" containerName="main"
Aug 05 18:16:46.743568 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:46.743486    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae6bbcde-0de7-4bdd-8b7d-5d0706737739" containerName="main"
Aug 05 18:16:46.743568 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:46.743491    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab560477-018f-4e7e-b217-9d7314392009" containerName="storage-initializer"
Aug 05 18:16:46.743568 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:46.743497    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab560477-018f-4e7e-b217-9d7314392009" containerName="storage-initializer"
Aug 05 18:16:46.743568 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:46.743559    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae6bbcde-0de7-4bdd-8b7d-5d0706737739" containerName="main"
Aug 05 18:16:46.743568 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:46.743569    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab560477-018f-4e7e-b217-9d7314392009" containerName="main"
Aug 05 18:16:46.746232 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:46.746209    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b"]
Aug 05 18:16:46.746355 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:46.746311    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b"
Aug 05 18:16:46.750498 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:46.750471    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Aug 05 18:16:46.750639 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:46.750498    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-5r4pk\""
Aug 05 18:16:46.750714 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:46.750685    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Aug 05 18:16:46.750865 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:46.750842    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Aug 05 18:16:46.750865 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:46.750860    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-ls57f\""
Aug 05 18:16:46.813654 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:46.813618    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b\" (UID: \"c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b"
Aug 05 18:16:46.813861 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:46.813731    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr7fn\" (UniqueName: \"kubernetes.io/projected/c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d-kube-api-access-xr7fn\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b\" (UID: \"c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b"
Aug 05 18:16:46.914333 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:46.914296    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr7fn\" (UniqueName: \"kubernetes.io/projected/c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d-kube-api-access-xr7fn\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b\" (UID: \"c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b"
Aug 05 18:16:46.914547 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:46.914396    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b\" (UID: \"c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b"
Aug 05 18:16:46.917111 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:46.917079    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b\" (UID: \"c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b"
Aug 05 18:16:46.923210 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:46.923183    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr7fn\" (UniqueName: \"kubernetes.io/projected/c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d-kube-api-access-xr7fn\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b\" (UID: \"c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b"
Aug 05 18:16:47.058497 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:47.058464    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b"
Aug 05 18:16:47.189612 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:47.189576    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm" podUID="2c99d3ec-3f22-4230-976f-e1507d64a94b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.111:8001/health\": dial tcp 10.133.0.111:8001: connect: connection refused"
Aug 05 18:16:47.201717 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:47.201687    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b"]
Aug 05 18:16:47.202761 ip-10-0-135-151 kubenswrapper[2561]: W0805 18:16:47.202735    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a3b5fa_1cb8_47fa_839f_81e06c9a1d2d.slice/crio-ea3ad8bf0d92097dfd3a43b7012124aa820a016d89712b86188c60c5381466c5 WatchSource:0}: Error finding container ea3ad8bf0d92097dfd3a43b7012124aa820a016d89712b86188c60c5381466c5: Status 404 returned error can't find the container with id ea3ad8bf0d92097dfd3a43b7012124aa820a016d89712b86188c60c5381466c5
Aug 05 18:16:47.826874 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:47.826837    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b" event={"ID":"c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d","Type":"ContainerStarted","Data":"c2f40fb795bc66620c06e37795a827f80f104c53af7f1b78fb06e84d02ba1160"}
Aug 05 18:16:47.827374 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:47.826887    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b" event={"ID":"c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d","Type":"ContainerStarted","Data":"ea3ad8bf0d92097dfd3a43b7012124aa820a016d89712b86188c60c5381466c5"}
Aug 05 18:16:47.827374 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:47.826930    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b"
Aug 05 18:16:47.844230 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:47.844182    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b" podStartSLOduration=1.8441672009999999 podStartE2EDuration="1.844167201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:16:47.842384444 +0000 UTC m=+3863.587018773" watchObservedRunningTime="2026-08-05 18:16:47.844167201 +0000 UTC m=+3863.588801556"
Aug 05 18:16:57.199797 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:57.199760    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:16:57.211934 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:16:57.211907    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:17:11.056347 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:11.056309    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"]
Aug 05 18:17:18.833934 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:18.833901    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b"
Aug 05 18:17:24.235820 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:24.235783    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b"]
Aug 05 18:17:26.078654 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:26.078613    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm" podUID="2c99d3ec-3f22-4230-976f-e1507d64a94b" containerName="main" containerID="cri-o://7a9868560b37fdf568881ffe12f402c8da610ed45a96697ed4ca26313305a5c0" gracePeriod=45
Aug 05 18:17:26.696492 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:26.696456    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm" podUID="2c99d3ec-3f22-4230-976f-e1507d64a94b" containerName="llm-d-routing-sidecar" containerID="cri-o://4932f548081561953fe86f25ca89772bb54a5c9c3633d6ba47d4e55037cc0520" gracePeriod=45
Aug 05 18:17:26.950031 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:26.949998    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:17:27.050143 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.050110    2561 generic.go:356] "Generic (PLEG): container finished" podID="2c99d3ec-3f22-4230-976f-e1507d64a94b" containerID="7a9868560b37fdf568881ffe12f402c8da610ed45a96697ed4ca26313305a5c0" exitCode=0
Aug 05 18:17:27.050143 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.050136    2561 generic.go:356] "Generic (PLEG): container finished" podID="2c99d3ec-3f22-4230-976f-e1507d64a94b" containerID="4932f548081561953fe86f25ca89772bb54a5c9c3633d6ba47d4e55037cc0520" exitCode=0
Aug 05 18:17:27.050370 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.050164    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"
Aug 05 18:17:27.050370 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.050192    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm" event={"ID":"2c99d3ec-3f22-4230-976f-e1507d64a94b","Type":"ContainerDied","Data":"7a9868560b37fdf568881ffe12f402c8da610ed45a96697ed4ca26313305a5c0"}
Aug 05 18:17:27.050370 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.050232    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm" event={"ID":"2c99d3ec-3f22-4230-976f-e1507d64a94b","Type":"ContainerDied","Data":"4932f548081561953fe86f25ca89772bb54a5c9c3633d6ba47d4e55037cc0520"}
Aug 05 18:17:27.050370 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.050247    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm" event={"ID":"2c99d3ec-3f22-4230-976f-e1507d64a94b","Type":"ContainerDied","Data":"c6101dbeed5a657169b12cde10557ca2785595723edb0dc23d4aa4c6c53ed27d"}
Aug 05 18:17:27.050370 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.050267    2561 scope.go:117] "RemoveContainer" containerID="7a9868560b37fdf568881ffe12f402c8da610ed45a96697ed4ca26313305a5c0"
Aug 05 18:17:27.059061 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.059045    2561 scope.go:117] "RemoveContainer" containerID="42ff87c3a640b1fa475c6bb736f9da92e86fad2d5581e0b3594a4f2b19b8a77f"
Aug 05 18:17:27.066822 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.066806    2561 scope.go:117] "RemoveContainer" containerID="4932f548081561953fe86f25ca89772bb54a5c9c3633d6ba47d4e55037cc0520"
Aug 05 18:17:27.074216 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.074193    2561 scope.go:117] "RemoveContainer" containerID="7a9868560b37fdf568881ffe12f402c8da610ed45a96697ed4ca26313305a5c0"
Aug 05 18:17:27.074518 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:17:27.074494    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a9868560b37fdf568881ffe12f402c8da610ed45a96697ed4ca26313305a5c0\": container with ID starting with 7a9868560b37fdf568881ffe12f402c8da610ed45a96697ed4ca26313305a5c0 not found: ID does not exist" containerID="7a9868560b37fdf568881ffe12f402c8da610ed45a96697ed4ca26313305a5c0"
Aug 05 18:17:27.074633 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.074531    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a9868560b37fdf568881ffe12f402c8da610ed45a96697ed4ca26313305a5c0"} err="failed to get container status \"7a9868560b37fdf568881ffe12f402c8da610ed45a96697ed4ca26313305a5c0\": rpc error: code = NotFound desc = could not find container \"7a9868560b37fdf568881ffe12f402c8da610ed45a96697ed4ca26313305a5c0\": container with ID starting with 7a9868560b37fdf568881ffe12f402c8da610ed45a96697ed4ca26313305a5c0 not found: ID does not exist"
Aug 05 18:17:27.074633 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.074552    2561 scope.go:117] "RemoveContainer" containerID="42ff87c3a640b1fa475c6bb736f9da92e86fad2d5581e0b3594a4f2b19b8a77f"
Aug 05 18:17:27.074802 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:17:27.074784    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42ff87c3a640b1fa475c6bb736f9da92e86fad2d5581e0b3594a4f2b19b8a77f\": container with ID starting with 42ff87c3a640b1fa475c6bb736f9da92e86fad2d5581e0b3594a4f2b19b8a77f not found: ID does not exist" containerID="42ff87c3a640b1fa475c6bb736f9da92e86fad2d5581e0b3594a4f2b19b8a77f"
Aug 05 18:17:27.074841 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.074808    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42ff87c3a640b1fa475c6bb736f9da92e86fad2d5581e0b3594a4f2b19b8a77f"} err="failed to get container status \"42ff87c3a640b1fa475c6bb736f9da92e86fad2d5581e0b3594a4f2b19b8a77f\": rpc error: code = NotFound desc = could not find container \"42ff87c3a640b1fa475c6bb736f9da92e86fad2d5581e0b3594a4f2b19b8a77f\": container with ID starting with 42ff87c3a640b1fa475c6bb736f9da92e86fad2d5581e0b3594a4f2b19b8a77f not found: ID does not exist"
Aug 05 18:17:27.074841 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.074823    2561 scope.go:117] "RemoveContainer" containerID="4932f548081561953fe86f25ca89772bb54a5c9c3633d6ba47d4e55037cc0520"
Aug 05 18:17:27.075097 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:17:27.075081    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4932f548081561953fe86f25ca89772bb54a5c9c3633d6ba47d4e55037cc0520\": container with ID starting with 4932f548081561953fe86f25ca89772bb54a5c9c3633d6ba47d4e55037cc0520 not found: ID does not exist" containerID="4932f548081561953fe86f25ca89772bb54a5c9c3633d6ba47d4e55037cc0520"
Aug 05 18:17:27.075155 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.075101    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4932f548081561953fe86f25ca89772bb54a5c9c3633d6ba47d4e55037cc0520"} err="failed to get container status \"4932f548081561953fe86f25ca89772bb54a5c9c3633d6ba47d4e55037cc0520\": rpc error: code = NotFound desc = could not find container \"4932f548081561953fe86f25ca89772bb54a5c9c3633d6ba47d4e55037cc0520\": container with ID starting with 4932f548081561953fe86f25ca89772bb54a5c9c3633d6ba47d4e55037cc0520 not found: ID does not exist"
Aug 05 18:17:27.075155 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.075116    2561 scope.go:117] "RemoveContainer" containerID="7a9868560b37fdf568881ffe12f402c8da610ed45a96697ed4ca26313305a5c0"
Aug 05 18:17:27.075336 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.075321    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a9868560b37fdf568881ffe12f402c8da610ed45a96697ed4ca26313305a5c0"} err="failed to get container status \"7a9868560b37fdf568881ffe12f402c8da610ed45a96697ed4ca26313305a5c0\": rpc error: code = NotFound desc = could not find container \"7a9868560b37fdf568881ffe12f402c8da610ed45a96697ed4ca26313305a5c0\": container with ID starting with 7a9868560b37fdf568881ffe12f402c8da610ed45a96697ed4ca26313305a5c0 not found: ID does not exist"
Aug 05 18:17:27.075379 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.075337    2561 scope.go:117] "RemoveContainer" containerID="42ff87c3a640b1fa475c6bb736f9da92e86fad2d5581e0b3594a4f2b19b8a77f"
Aug 05 18:17:27.075566 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.075549    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42ff87c3a640b1fa475c6bb736f9da92e86fad2d5581e0b3594a4f2b19b8a77f"} err="failed to get container status \"42ff87c3a640b1fa475c6bb736f9da92e86fad2d5581e0b3594a4f2b19b8a77f\": rpc error: code = NotFound desc = could not find container \"42ff87c3a640b1fa475c6bb736f9da92e86fad2d5581e0b3594a4f2b19b8a77f\": container with ID starting with 42ff87c3a640b1fa475c6bb736f9da92e86fad2d5581e0b3594a4f2b19b8a77f not found: ID does not exist"
Aug 05 18:17:27.075612 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.075568    2561 scope.go:117] "RemoveContainer" containerID="4932f548081561953fe86f25ca89772bb54a5c9c3633d6ba47d4e55037cc0520"
Aug 05 18:17:27.075798 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.075779    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4932f548081561953fe86f25ca89772bb54a5c9c3633d6ba47d4e55037cc0520"} err="failed to get container status \"4932f548081561953fe86f25ca89772bb54a5c9c3633d6ba47d4e55037cc0520\": rpc error: code = NotFound desc = could not find container \"4932f548081561953fe86f25ca89772bb54a5c9c3633d6ba47d4e55037cc0520\": container with ID starting with 4932f548081561953fe86f25ca89772bb54a5c9c3633d6ba47d4e55037cc0520 not found: ID does not exist"
Aug 05 18:17:27.091215 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.091185    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c99d3ec-3f22-4230-976f-e1507d64a94b-kserve-provision-location\") pod \"2c99d3ec-3f22-4230-976f-e1507d64a94b\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") "
Aug 05 18:17:27.091544 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.091227    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c99d3ec-3f22-4230-976f-e1507d64a94b-home\") pod \"2c99d3ec-3f22-4230-976f-e1507d64a94b\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") "
Aug 05 18:17:27.091544 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.091275    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6nhj\" (UniqueName: \"kubernetes.io/projected/2c99d3ec-3f22-4230-976f-e1507d64a94b-kube-api-access-c6nhj\") pod \"2c99d3ec-3f22-4230-976f-e1507d64a94b\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") "
Aug 05 18:17:27.091544 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.091303    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c99d3ec-3f22-4230-976f-e1507d64a94b-tls-certs\") pod \"2c99d3ec-3f22-4230-976f-e1507d64a94b\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") "
Aug 05 18:17:27.091544 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.091331    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c99d3ec-3f22-4230-976f-e1507d64a94b-dshm\") pod \"2c99d3ec-3f22-4230-976f-e1507d64a94b\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") "
Aug 05 18:17:27.091544 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.091345    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c99d3ec-3f22-4230-976f-e1507d64a94b-model-cache\") pod \"2c99d3ec-3f22-4230-976f-e1507d64a94b\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") "
Aug 05 18:17:27.091544 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.091387    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c99d3ec-3f22-4230-976f-e1507d64a94b-cabundle-cert\") pod \"2c99d3ec-3f22-4230-976f-e1507d64a94b\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") "
Aug 05 18:17:27.091544 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.091412    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c99d3ec-3f22-4230-976f-e1507d64a94b-tmp-dir\") pod \"2c99d3ec-3f22-4230-976f-e1507d64a94b\" (UID: \"2c99d3ec-3f22-4230-976f-e1507d64a94b\") "
Aug 05 18:17:27.091916 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.091832    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c99d3ec-3f22-4230-976f-e1507d64a94b-model-cache" (OuterVolumeSpecName: "model-cache") pod "2c99d3ec-3f22-4230-976f-e1507d64a94b" (UID: "2c99d3ec-3f22-4230-976f-e1507d64a94b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:17:27.091916 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.091881    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c99d3ec-3f22-4230-976f-e1507d64a94b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2c99d3ec-3f22-4230-976f-e1507d64a94b" (UID: "2c99d3ec-3f22-4230-976f-e1507d64a94b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 18:17:27.092075 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.091999    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c99d3ec-3f22-4230-976f-e1507d64a94b-home" (OuterVolumeSpecName: "home") pod "2c99d3ec-3f22-4230-976f-e1507d64a94b" (UID: "2c99d3ec-3f22-4230-976f-e1507d64a94b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:17:27.093615 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.093586    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c99d3ec-3f22-4230-976f-e1507d64a94b-kube-api-access-c6nhj" (OuterVolumeSpecName: "kube-api-access-c6nhj") pod "2c99d3ec-3f22-4230-976f-e1507d64a94b" (UID: "2c99d3ec-3f22-4230-976f-e1507d64a94b"). InnerVolumeSpecName "kube-api-access-c6nhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 18:17:27.093722 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.093673    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c99d3ec-3f22-4230-976f-e1507d64a94b-dshm" (OuterVolumeSpecName: "dshm") pod "2c99d3ec-3f22-4230-976f-e1507d64a94b" (UID: "2c99d3ec-3f22-4230-976f-e1507d64a94b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:17:27.093722 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.093695    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c99d3ec-3f22-4230-976f-e1507d64a94b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c99d3ec-3f22-4230-976f-e1507d64a94b" (UID: "2c99d3ec-3f22-4230-976f-e1507d64a94b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 18:17:27.105566 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.105546    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c99d3ec-3f22-4230-976f-e1507d64a94b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2c99d3ec-3f22-4230-976f-e1507d64a94b" (UID: "2c99d3ec-3f22-4230-976f-e1507d64a94b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:17:27.115625 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.115601    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c99d3ec-3f22-4230-976f-e1507d64a94b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2c99d3ec-3f22-4230-976f-e1507d64a94b" (UID: "2c99d3ec-3f22-4230-976f-e1507d64a94b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:17:27.192392 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.192305    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6nhj\" (UniqueName: \"kubernetes.io/projected/2c99d3ec-3f22-4230-976f-e1507d64a94b-kube-api-access-c6nhj\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:17:27.192392 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.192334    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c99d3ec-3f22-4230-976f-e1507d64a94b-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:17:27.192392 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.192346    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c99d3ec-3f22-4230-976f-e1507d64a94b-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:17:27.192392 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.192357    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c99d3ec-3f22-4230-976f-e1507d64a94b-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:17:27.192392 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.192367    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c99d3ec-3f22-4230-976f-e1507d64a94b-cabundle-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:17:27.192392 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.192377    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c99d3ec-3f22-4230-976f-e1507d64a94b-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:17:27.192392 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.192390    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c99d3ec-3f22-4230-976f-e1507d64a94b-kserve-provision-location\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:17:27.192704 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.192401    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c99d3ec-3f22-4230-976f-e1507d64a94b-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:17:27.372830 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.372801    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"]
Aug 05 18:17:27.376506 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:27.376481    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-7ffb7f8f74-8gdrm"]
Aug 05 18:17:28.729893 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:28.729857    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c99d3ec-3f22-4230-976f-e1507d64a94b" path="/var/lib/kubelet/pods/2c99d3ec-3f22-4230-976f-e1507d64a94b/volumes"
Aug 05 18:17:28.834751 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:28.834714    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b" podUID="c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 05 18:17:36.086804 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:36.086766    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"]
Aug 05 18:17:36.087376 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:36.086986    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5" podUID="8d9bd4a8-4d15-442b-a299-d0530b0be122" containerName="istio-proxy" containerID="cri-o://464fb709e163b176a47eafaf1e5c5c6dddd3e67f86f69bc2000ac7a1157b29b9" gracePeriod=30
Aug 05 18:17:37.815851 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.815817    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"]
Aug 05 18:17:37.816385 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.816364    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c99d3ec-3f22-4230-976f-e1507d64a94b" containerName="llm-d-routing-sidecar"
Aug 05 18:17:37.816511 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.816389    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c99d3ec-3f22-4230-976f-e1507d64a94b" containerName="llm-d-routing-sidecar"
Aug 05 18:17:37.816511 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.816401    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c99d3ec-3f22-4230-976f-e1507d64a94b" containerName="main"
Aug 05 18:17:37.816511 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.816410    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c99d3ec-3f22-4230-976f-e1507d64a94b" containerName="main"
Aug 05 18:17:37.816511 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.816448    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c99d3ec-3f22-4230-976f-e1507d64a94b" containerName="storage-initializer"
Aug 05 18:17:37.816511 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.816458    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c99d3ec-3f22-4230-976f-e1507d64a94b" containerName="storage-initializer"
Aug 05 18:17:37.816768 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.816556    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c99d3ec-3f22-4230-976f-e1507d64a94b" containerName="main"
Aug 05 18:17:37.816768 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.816570    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c99d3ec-3f22-4230-976f-e1507d64a94b" containerName="llm-d-routing-sidecar"
Aug 05 18:17:37.821691 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.821669    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:17:37.825020 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.824439    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-xnlcr\""
Aug 05 18:17:37.825020 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.824689    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Aug 05 18:17:37.825020 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.824696    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Aug 05 18:17:37.825020 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.824828    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Aug 05 18:17:37.825020 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.824786    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Aug 05 18:17:37.825292 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.825120    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Aug 05 18:17:37.833343 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.833319    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"]
Aug 05 18:17:37.881068 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.881035    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8591796f-8aa7-4b7f-9acd-93067124c027-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:17:37.881254 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.881074    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8591796f-8aa7-4b7f-9acd-93067124c027-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:17:37.881254 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.881104    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptv78\" (UniqueName: \"kubernetes.io/projected/8591796f-8aa7-4b7f-9acd-93067124c027-kube-api-access-ptv78\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:17:37.881254 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.881164    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8591796f-8aa7-4b7f-9acd-93067124c027-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:17:37.881254 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.881200    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8591796f-8aa7-4b7f-9acd-93067124c027-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:17:37.881254 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.881218    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8591796f-8aa7-4b7f-9acd-93067124c027-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:17:37.881254 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.881236    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8591796f-8aa7-4b7f-9acd-93067124c027-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:17:37.881533 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.881359    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8591796f-8aa7-4b7f-9acd-93067124c027-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:17:37.982442 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.982409    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8591796f-8aa7-4b7f-9acd-93067124c027-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:17:37.982653 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.982458    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8591796f-8aa7-4b7f-9acd-93067124c027-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:17:37.982653 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.982477    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8591796f-8aa7-4b7f-9acd-93067124c027-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:17:37.982653 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.982500    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptv78\" (UniqueName: \"kubernetes.io/projected/8591796f-8aa7-4b7f-9acd-93067124c027-kube-api-access-ptv78\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:17:37.982653 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.982534    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8591796f-8aa7-4b7f-9acd-93067124c027-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:17:37.982653 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.982580    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8591796f-8aa7-4b7f-9acd-93067124c027-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:17:37.982653 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.982607    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8591796f-8aa7-4b7f-9acd-93067124c027-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:17:37.982653 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.982631    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8591796f-8aa7-4b7f-9acd-93067124c027-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:17:37.983059 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.982921    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8591796f-8aa7-4b7f-9acd-93067124c027-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:17:37.983059 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.982983    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8591796f-8aa7-4b7f-9acd-93067124c027-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:17:37.983059 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.983006    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8591796f-8aa7-4b7f-9acd-93067124c027-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:17:37.983181 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.983118    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8591796f-8aa7-4b7f-9acd-93067124c027-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:17:37.983379 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.983349    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8591796f-8aa7-4b7f-9acd-93067124c027-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:17:37.985061 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.985041    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8591796f-8aa7-4b7f-9acd-93067124c027-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:17:37.985310 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.985291    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8591796f-8aa7-4b7f-9acd-93067124c027-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:17:37.990041 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:37.990021    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptv78\" (UniqueName: \"kubernetes.io/projected/8591796f-8aa7-4b7f-9acd-93067124c027-kube-api-access-ptv78\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:17:38.133712 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:38.133625    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:17:38.282373 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:38.282332    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"]
Aug 05 18:17:38.288222 ip-10-0-135-151 kubenswrapper[2561]: W0805 18:17:38.288171    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8591796f_8aa7_4b7f_9acd_93067124c027.slice/crio-c33d9c011f29d75656f57b0962456d149ab879ccaaeb822e8ee44f5ca2e165a2 WatchSource:0}: Error finding container c33d9c011f29d75656f57b0962456d149ab879ccaaeb822e8ee44f5ca2e165a2: Status 404 returned error can't find the container with id c33d9c011f29d75656f57b0962456d149ab879ccaaeb822e8ee44f5ca2e165a2
Aug 05 18:17:38.834935 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:38.834849    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b" podUID="c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 05 18:17:39.116088 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:39.115990    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt" event={"ID":"8591796f-8aa7-4b7f-9acd-93067124c027","Type":"ContainerStarted","Data":"c9dd120ca09f47941bb011c6df90a0ce0fda768a4ade92d1ab4f6e020ab82e72"}
Aug 05 18:17:39.116088 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:39.116044    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt" event={"ID":"8591796f-8aa7-4b7f-9acd-93067124c027","Type":"ContainerStarted","Data":"c33d9c011f29d75656f57b0962456d149ab879ccaaeb822e8ee44f5ca2e165a2"}
Aug 05 18:17:39.258176 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:39.258124    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b" podUID="c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d" containerName="main" containerID="cri-o://c2f40fb795bc66620c06e37795a827f80f104c53af7f1b78fb06e84d02ba1160" gracePeriod=45
Aug 05 18:17:39.517762 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:39.517735    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b"
Aug 05 18:17:39.597377 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:39.597335    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xr7fn\" (UniqueName: \"kubernetes.io/projected/c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d-kube-api-access-xr7fn\") pod \"c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d\" (UID: \"c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d\") "
Aug 05 18:17:39.597570 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:39.597391    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d-tls-certs\") pod \"c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d\" (UID: \"c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d\") "
Aug 05 18:17:39.599564 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:39.599527    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d" (UID: "c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 18:17:39.599564 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:39.599557    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d-kube-api-access-xr7fn" (OuterVolumeSpecName: "kube-api-access-xr7fn") pod "c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d" (UID: "c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d"). InnerVolumeSpecName "kube-api-access-xr7fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 18:17:39.698355 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:39.698256    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr7fn\" (UniqueName: \"kubernetes.io/projected/c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d-kube-api-access-xr7fn\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:17:39.698355 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:39.698290    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:17:40.122429 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:40.122399    2561 generic.go:356] "Generic (PLEG): container finished" podID="c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d" containerID="c2f40fb795bc66620c06e37795a827f80f104c53af7f1b78fb06e84d02ba1160" exitCode=0
Aug 05 18:17:40.122850 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:40.122448    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b"
Aug 05 18:17:40.122850 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:40.122487    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b" event={"ID":"c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d","Type":"ContainerDied","Data":"c2f40fb795bc66620c06e37795a827f80f104c53af7f1b78fb06e84d02ba1160"}
Aug 05 18:17:40.122850 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:40.122544    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b" event={"ID":"c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d","Type":"ContainerDied","Data":"ea3ad8bf0d92097dfd3a43b7012124aa820a016d89712b86188c60c5381466c5"}
Aug 05 18:17:40.122850 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:40.122573    2561 scope.go:117] "RemoveContainer" containerID="c2f40fb795bc66620c06e37795a827f80f104c53af7f1b78fb06e84d02ba1160"
Aug 05 18:17:40.132274 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:40.132255    2561 scope.go:117] "RemoveContainer" containerID="c2f40fb795bc66620c06e37795a827f80f104c53af7f1b78fb06e84d02ba1160"
Aug 05 18:17:40.132548 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:17:40.132530    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2f40fb795bc66620c06e37795a827f80f104c53af7f1b78fb06e84d02ba1160\": container with ID starting with c2f40fb795bc66620c06e37795a827f80f104c53af7f1b78fb06e84d02ba1160 not found: ID does not exist" containerID="c2f40fb795bc66620c06e37795a827f80f104c53af7f1b78fb06e84d02ba1160"
Aug 05 18:17:40.132613 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:40.132558    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2f40fb795bc66620c06e37795a827f80f104c53af7f1b78fb06e84d02ba1160"} err="failed to get container status \"c2f40fb795bc66620c06e37795a827f80f104c53af7f1b78fb06e84d02ba1160\": rpc error: code = NotFound desc = could not find container \"c2f40fb795bc66620c06e37795a827f80f104c53af7f1b78fb06e84d02ba1160\": container with ID starting with c2f40fb795bc66620c06e37795a827f80f104c53af7f1b78fb06e84d02ba1160 not found: ID does not exist"
Aug 05 18:17:40.143800 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:40.143777    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b"]
Aug 05 18:17:40.149898 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:40.149865    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-w6x9b"]
Aug 05 18:17:40.730433 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:40.730396    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d" path="/var/lib/kubelet/pods/c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d/volumes"
Aug 05 18:17:41.130627 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:41.130595    2561 generic.go:356] "Generic (PLEG): container finished" podID="8591796f-8aa7-4b7f-9acd-93067124c027" containerID="c9dd120ca09f47941bb011c6df90a0ce0fda768a4ade92d1ab4f6e020ab82e72" exitCode=0
Aug 05 18:17:41.131049 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:41.130670    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt" event={"ID":"8591796f-8aa7-4b7f-9acd-93067124c027","Type":"ContainerDied","Data":"c9dd120ca09f47941bb011c6df90a0ce0fda768a4ade92d1ab4f6e020ab82e72"}
Aug 05 18:17:41.347301 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:41.347235    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:17:41.413612 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:41.413577    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9bd4a8-4d15-442b-a299-d0530b0be122-istio-podinfo\") pod \"8d9bd4a8-4d15-442b-a299-d0530b0be122\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") "
Aug 05 18:17:41.413819 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:41.413633    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9bd4a8-4d15-442b-a299-d0530b0be122-istio-data\") pod \"8d9bd4a8-4d15-442b-a299-d0530b0be122\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") "
Aug 05 18:17:41.413819 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:41.413661    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9bd4a8-4d15-442b-a299-d0530b0be122-workload-socket\") pod \"8d9bd4a8-4d15-442b-a299-d0530b0be122\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") "
Aug 05 18:17:41.413819 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:41.413682    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9bd4a8-4d15-442b-a299-d0530b0be122-istio-token\") pod \"8d9bd4a8-4d15-442b-a299-d0530b0be122\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") "
Aug 05 18:17:41.413819 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:41.413734    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6kmx\" (UniqueName: \"kubernetes.io/projected/8d9bd4a8-4d15-442b-a299-d0530b0be122-kube-api-access-t6kmx\") pod \"8d9bd4a8-4d15-442b-a299-d0530b0be122\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") "
Aug 05 18:17:41.413819 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:41.413758    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9bd4a8-4d15-442b-a299-d0530b0be122-istio-envoy\") pod \"8d9bd4a8-4d15-442b-a299-d0530b0be122\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") "
Aug 05 18:17:41.413819 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:41.413797    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9bd4a8-4d15-442b-a299-d0530b0be122-workload-certs\") pod \"8d9bd4a8-4d15-442b-a299-d0530b0be122\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") "
Aug 05 18:17:41.414202 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:41.413823    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9bd4a8-4d15-442b-a299-d0530b0be122-istiod-ca-cert\") pod \"8d9bd4a8-4d15-442b-a299-d0530b0be122\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") "
Aug 05 18:17:41.414202 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:41.413878    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9bd4a8-4d15-442b-a299-d0530b0be122-credential-socket\") pod \"8d9bd4a8-4d15-442b-a299-d0530b0be122\" (UID: \"8d9bd4a8-4d15-442b-a299-d0530b0be122\") "
Aug 05 18:17:41.414202 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:41.414025    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9bd4a8-4d15-442b-a299-d0530b0be122-istio-data" (OuterVolumeSpecName: "istio-data") pod "8d9bd4a8-4d15-442b-a299-d0530b0be122" (UID: "8d9bd4a8-4d15-442b-a299-d0530b0be122"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:17:41.414202 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:41.414195    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9bd4a8-4d15-442b-a299-d0530b0be122-istio-data\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:17:41.414419 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:41.414402    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9bd4a8-4d15-442b-a299-d0530b0be122-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8d9bd4a8-4d15-442b-a299-d0530b0be122" (UID: "8d9bd4a8-4d15-442b-a299-d0530b0be122"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:17:41.414844 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:41.414615    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9bd4a8-4d15-442b-a299-d0530b0be122-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8d9bd4a8-4d15-442b-a299-d0530b0be122" (UID: "8d9bd4a8-4d15-442b-a299-d0530b0be122"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:17:41.414844 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:41.414768    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d9bd4a8-4d15-442b-a299-d0530b0be122-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8d9bd4a8-4d15-442b-a299-d0530b0be122" (UID: "8d9bd4a8-4d15-442b-a299-d0530b0be122"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 18:17:41.414844 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:41.414822    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9bd4a8-4d15-442b-a299-d0530b0be122-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8d9bd4a8-4d15-442b-a299-d0530b0be122" (UID: "8d9bd4a8-4d15-442b-a299-d0530b0be122"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:17:41.416599 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:41.416570    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9bd4a8-4d15-442b-a299-d0530b0be122-istio-token" (OuterVolumeSpecName: "istio-token") pod "8d9bd4a8-4d15-442b-a299-d0530b0be122" (UID: "8d9bd4a8-4d15-442b-a299-d0530b0be122"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 18:17:41.416717 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:41.416677    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d9bd4a8-4d15-442b-a299-d0530b0be122-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8d9bd4a8-4d15-442b-a299-d0530b0be122" (UID: "8d9bd4a8-4d15-442b-a299-d0530b0be122"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Aug 05 18:17:41.416782 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:41.416713    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9bd4a8-4d15-442b-a299-d0530b0be122-kube-api-access-t6kmx" (OuterVolumeSpecName: "kube-api-access-t6kmx") pod "8d9bd4a8-4d15-442b-a299-d0530b0be122" (UID: "8d9bd4a8-4d15-442b-a299-d0530b0be122"). InnerVolumeSpecName "kube-api-access-t6kmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 18:17:41.417490 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:41.417462    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9bd4a8-4d15-442b-a299-d0530b0be122-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8d9bd4a8-4d15-442b-a299-d0530b0be122" (UID: "8d9bd4a8-4d15-442b-a299-d0530b0be122"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:17:41.515424 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:41.515385    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9bd4a8-4d15-442b-a299-d0530b0be122-credential-socket\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:17:41.515424 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:41.515416    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9bd4a8-4d15-442b-a299-d0530b0be122-istio-podinfo\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:17:41.515424 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:41.515427    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9bd4a8-4d15-442b-a299-d0530b0be122-workload-socket\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:17:41.515672 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:41.515438    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9bd4a8-4d15-442b-a299-d0530b0be122-istio-token\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:17:41.515672 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:41.515448    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6kmx\" (UniqueName: \"kubernetes.io/projected/8d9bd4a8-4d15-442b-a299-d0530b0be122-kube-api-access-t6kmx\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:17:41.515672 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:41.515457    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9bd4a8-4d15-442b-a299-d0530b0be122-istio-envoy\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:17:41.515672 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:41.515465    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9bd4a8-4d15-442b-a299-d0530b0be122-workload-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:17:41.515672 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:41.515473    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9bd4a8-4d15-442b-a299-d0530b0be122-istiod-ca-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:17:42.139796 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:42.139752    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt" event={"ID":"8591796f-8aa7-4b7f-9acd-93067124c027","Type":"ContainerStarted","Data":"4b8f3b87e3926fa053cd4b0c5bd15d8d0da985ef602aa3cc992424ae3a29db42"}
Aug 05 18:17:42.141403 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:42.141373    2561 generic.go:356] "Generic (PLEG): container finished" podID="8d9bd4a8-4d15-442b-a299-d0530b0be122" containerID="464fb709e163b176a47eafaf1e5c5c6dddd3e67f86f69bc2000ac7a1157b29b9" exitCode=0
Aug 05 18:17:42.141607 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:42.141411    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"
Aug 05 18:17:42.141607 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:42.141459    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5" event={"ID":"8d9bd4a8-4d15-442b-a299-d0530b0be122","Type":"ContainerDied","Data":"464fb709e163b176a47eafaf1e5c5c6dddd3e67f86f69bc2000ac7a1157b29b9"}
Aug 05 18:17:42.141607 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:42.141509    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5" event={"ID":"8d9bd4a8-4d15-442b-a299-d0530b0be122","Type":"ContainerDied","Data":"07e7acf02e57c56df5cc26b01b8b89c303e7ae4e8f95c3018fb1f07b8ba99325"}
Aug 05 18:17:42.141607 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:42.141538    2561 scope.go:117] "RemoveContainer" containerID="464fb709e163b176a47eafaf1e5c5c6dddd3e67f86f69bc2000ac7a1157b29b9"
Aug 05 18:17:42.151289 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:42.151269    2561 scope.go:117] "RemoveContainer" containerID="98395a4289af1b3f9b3182e9de622102a3df16cebeba62c1d11e8e13d2a2edae"
Aug 05 18:17:42.159387 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:42.159371    2561 scope.go:117] "RemoveContainer" containerID="464fb709e163b176a47eafaf1e5c5c6dddd3e67f86f69bc2000ac7a1157b29b9"
Aug 05 18:17:42.159653 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:17:42.159624    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"464fb709e163b176a47eafaf1e5c5c6dddd3e67f86f69bc2000ac7a1157b29b9\": container with ID starting with 464fb709e163b176a47eafaf1e5c5c6dddd3e67f86f69bc2000ac7a1157b29b9 not found: ID does not exist" containerID="464fb709e163b176a47eafaf1e5c5c6dddd3e67f86f69bc2000ac7a1157b29b9"
Aug 05 18:17:42.159756 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:42.159655    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"464fb709e163b176a47eafaf1e5c5c6dddd3e67f86f69bc2000ac7a1157b29b9"} err="failed to get container status \"464fb709e163b176a47eafaf1e5c5c6dddd3e67f86f69bc2000ac7a1157b29b9\": rpc error: code = NotFound desc = could not find container \"464fb709e163b176a47eafaf1e5c5c6dddd3e67f86f69bc2000ac7a1157b29b9\": container with ID starting with 464fb709e163b176a47eafaf1e5c5c6dddd3e67f86f69bc2000ac7a1157b29b9 not found: ID does not exist"
Aug 05 18:17:42.159756 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:42.159675    2561 scope.go:117] "RemoveContainer" containerID="98395a4289af1b3f9b3182e9de622102a3df16cebeba62c1d11e8e13d2a2edae"
Aug 05 18:17:42.159907 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:17:42.159890    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98395a4289af1b3f9b3182e9de622102a3df16cebeba62c1d11e8e13d2a2edae\": container with ID starting with 98395a4289af1b3f9b3182e9de622102a3df16cebeba62c1d11e8e13d2a2edae not found: ID does not exist" containerID="98395a4289af1b3f9b3182e9de622102a3df16cebeba62c1d11e8e13d2a2edae"
Aug 05 18:17:42.159988 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:42.159911    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98395a4289af1b3f9b3182e9de622102a3df16cebeba62c1d11e8e13d2a2edae"} err="failed to get container status \"98395a4289af1b3f9b3182e9de622102a3df16cebeba62c1d11e8e13d2a2edae\": rpc error: code = NotFound desc = could not find container \"98395a4289af1b3f9b3182e9de622102a3df16cebeba62c1d11e8e13d2a2edae\": container with ID starting with 98395a4289af1b3f9b3182e9de622102a3df16cebeba62c1d11e8e13d2a2edae not found: ID does not exist"
Aug 05 18:17:42.164359 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:42.164312    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt" podStartSLOduration=2.879802825 podStartE2EDuration="5.164298817s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.284495992s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:17:42.161947219 +0000 UTC m=+3917.906581562" watchObservedRunningTime="2026-08-05 18:17:42.164298817 +0000 UTC m=+3917.908933143"
Aug 05 18:17:42.175487 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:42.175461    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"]
Aug 05 18:17:42.180608 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:42.180582    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-6cb79f7856-xpkq5"]
Aug 05 18:17:42.731466 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:42.731419    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d9bd4a8-4d15-442b-a299-d0530b0be122" path="/var/lib/kubelet/pods/8d9bd4a8-4d15-442b-a299-d0530b0be122/volumes"
Aug 05 18:17:43.134696 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:43.134652    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:17:48.134160 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:48.134122    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:17:48.135813 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:48.135786    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt" podUID="8591796f-8aa7-4b7f-9acd-93067124c027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/health\": dial tcp 10.133.0.114:8000: connect: connection refused"
Aug 05 18:17:58.134561 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:17:58.134454    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt" podUID="8591796f-8aa7-4b7f-9acd-93067124c027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/health\": dial tcp 10.133.0.114:8000: connect: connection refused"
Aug 05 18:18:08.135100 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:18:08.135046    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt" podUID="8591796f-8aa7-4b7f-9acd-93067124c027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/health\": dial tcp 10.133.0.114:8000: connect: connection refused"
Aug 05 18:18:18.134603 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:18:18.134501    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt" podUID="8591796f-8aa7-4b7f-9acd-93067124c027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/health\": dial tcp 10.133.0.114:8000: connect: connection refused"
Aug 05 18:18:28.134065 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:18:28.134009    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt" podUID="8591796f-8aa7-4b7f-9acd-93067124c027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/health\": dial tcp 10.133.0.114:8000: connect: connection refused"
Aug 05 18:18:38.134711 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:18:38.134667    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt" podUID="8591796f-8aa7-4b7f-9acd-93067124c027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/health\": dial tcp 10.133.0.114:8000: connect: connection refused"
Aug 05 18:18:48.134742 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:18:48.134692    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt" podUID="8591796f-8aa7-4b7f-9acd-93067124c027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/health\": dial tcp 10.133.0.114:8000: connect: connection refused"
Aug 05 18:18:58.134879 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:18:58.134820    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt" podUID="8591796f-8aa7-4b7f-9acd-93067124c027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/health\": dial tcp 10.133.0.114:8000: connect: connection refused"
Aug 05 18:19:08.135109 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:08.135060    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt" podUID="8591796f-8aa7-4b7f-9acd-93067124c027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/health\": dial tcp 10.133.0.114:8000: connect: connection refused"
Aug 05 18:19:18.135003 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:18.134940    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt" podUID="8591796f-8aa7-4b7f-9acd-93067124c027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/health\": dial tcp 10.133.0.114:8000: connect: connection refused"
Aug 05 18:19:28.134723 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:28.134675    2561 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt" podUID="8591796f-8aa7-4b7f-9acd-93067124c027" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/health\": dial tcp 10.133.0.114:8000: connect: connection refused"
Aug 05 18:19:38.144602 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:38.144565    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:19:38.152382 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:38.152354    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:19:42.292574 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:42.292537    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"]
Aug 05 18:19:57.314830 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:57.314762    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt" podUID="8591796f-8aa7-4b7f-9acd-93067124c027" containerName="main" containerID="cri-o://4b8f3b87e3926fa053cd4b0c5bd15d8d0da985ef602aa3cc992424ae3a29db42" gracePeriod=45
Aug 05 18:19:58.068762 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.068736    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:19:58.120103 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.120017    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8591796f-8aa7-4b7f-9acd-93067124c027-model-cache\") pod \"8591796f-8aa7-4b7f-9acd-93067124c027\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") "
Aug 05 18:19:58.120103 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.120066    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8591796f-8aa7-4b7f-9acd-93067124c027-tls-certs\") pod \"8591796f-8aa7-4b7f-9acd-93067124c027\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") "
Aug 05 18:19:58.120103 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.120087    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8591796f-8aa7-4b7f-9acd-93067124c027-dshm\") pod \"8591796f-8aa7-4b7f-9acd-93067124c027\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") "
Aug 05 18:19:58.120398 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.120124    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8591796f-8aa7-4b7f-9acd-93067124c027-kserve-provision-location\") pod \"8591796f-8aa7-4b7f-9acd-93067124c027\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") "
Aug 05 18:19:58.120398 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.120145    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptv78\" (UniqueName: \"kubernetes.io/projected/8591796f-8aa7-4b7f-9acd-93067124c027-kube-api-access-ptv78\") pod \"8591796f-8aa7-4b7f-9acd-93067124c027\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") "
Aug 05 18:19:58.120398 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.120173    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8591796f-8aa7-4b7f-9acd-93067124c027-home\") pod \"8591796f-8aa7-4b7f-9acd-93067124c027\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") "
Aug 05 18:19:58.120398 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.120221    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8591796f-8aa7-4b7f-9acd-93067124c027-tmp-dir\") pod \"8591796f-8aa7-4b7f-9acd-93067124c027\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") "
Aug 05 18:19:58.120398 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.120249    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8591796f-8aa7-4b7f-9acd-93067124c027-model-cache" (OuterVolumeSpecName: "model-cache") pod "8591796f-8aa7-4b7f-9acd-93067124c027" (UID: "8591796f-8aa7-4b7f-9acd-93067124c027"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:19:58.120891 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.120865    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8591796f-8aa7-4b7f-9acd-93067124c027-home" (OuterVolumeSpecName: "home") pod "8591796f-8aa7-4b7f-9acd-93067124c027" (UID: "8591796f-8aa7-4b7f-9acd-93067124c027"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:19:58.120993 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.120937    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8591796f-8aa7-4b7f-9acd-93067124c027-cabundle-cert\") pod \"8591796f-8aa7-4b7f-9acd-93067124c027\" (UID: \"8591796f-8aa7-4b7f-9acd-93067124c027\") "
Aug 05 18:19:58.121355 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.121330    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8591796f-8aa7-4b7f-9acd-93067124c027-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8591796f-8aa7-4b7f-9acd-93067124c027" (UID: "8591796f-8aa7-4b7f-9acd-93067124c027"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 05 18:19:58.121477 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.121458    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8591796f-8aa7-4b7f-9acd-93067124c027-home\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:19:58.121559 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.121480    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8591796f-8aa7-4b7f-9acd-93067124c027-cabundle-cert\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:19:58.121559 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.121494    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8591796f-8aa7-4b7f-9acd-93067124c027-model-cache\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:19:58.122944 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.122920    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8591796f-8aa7-4b7f-9acd-93067124c027-dshm" (OuterVolumeSpecName: "dshm") pod "8591796f-8aa7-4b7f-9acd-93067124c027" (UID: "8591796f-8aa7-4b7f-9acd-93067124c027"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:19:58.123237 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.123215    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8591796f-8aa7-4b7f-9acd-93067124c027-kube-api-access-ptv78" (OuterVolumeSpecName: "kube-api-access-ptv78") pod "8591796f-8aa7-4b7f-9acd-93067124c027" (UID: "8591796f-8aa7-4b7f-9acd-93067124c027"). InnerVolumeSpecName "kube-api-access-ptv78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 18:19:58.123314 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.123256    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8591796f-8aa7-4b7f-9acd-93067124c027-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8591796f-8aa7-4b7f-9acd-93067124c027" (UID: "8591796f-8aa7-4b7f-9acd-93067124c027"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 05 18:19:58.135422 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.135400    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8591796f-8aa7-4b7f-9acd-93067124c027-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8591796f-8aa7-4b7f-9acd-93067124c027" (UID: "8591796f-8aa7-4b7f-9acd-93067124c027"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:19:58.146243 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.146217    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8591796f-8aa7-4b7f-9acd-93067124c027-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8591796f-8aa7-4b7f-9acd-93067124c027" (UID: "8591796f-8aa7-4b7f-9acd-93067124c027"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:19:58.222554 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.222522    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8591796f-8aa7-4b7f-9acd-93067124c027-dshm\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:19:58.222554 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.222548    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8591796f-8aa7-4b7f-9acd-93067124c027-kserve-provision-location\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:19:58.222554 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.222558    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptv78\" (UniqueName: \"kubernetes.io/projected/8591796f-8aa7-4b7f-9acd-93067124c027-kube-api-access-ptv78\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:19:58.222781 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.222567    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8591796f-8aa7-4b7f-9acd-93067124c027-tmp-dir\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:19:58.222781 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.222575    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8591796f-8aa7-4b7f-9acd-93067124c027-tls-certs\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:19:58.903108 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.903071    2561 generic.go:356] "Generic (PLEG): container finished" podID="8591796f-8aa7-4b7f-9acd-93067124c027" containerID="4b8f3b87e3926fa053cd4b0c5bd15d8d0da985ef602aa3cc992424ae3a29db42" exitCode=0
Aug 05 18:19:58.903546 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.903127    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"
Aug 05 18:19:58.903546 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.903136    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt" event={"ID":"8591796f-8aa7-4b7f-9acd-93067124c027","Type":"ContainerDied","Data":"4b8f3b87e3926fa053cd4b0c5bd15d8d0da985ef602aa3cc992424ae3a29db42"}
Aug 05 18:19:58.903546 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.903178    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt" event={"ID":"8591796f-8aa7-4b7f-9acd-93067124c027","Type":"ContainerDied","Data":"c33d9c011f29d75656f57b0962456d149ab879ccaaeb822e8ee44f5ca2e165a2"}
Aug 05 18:19:58.903546 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.903199    2561 scope.go:117] "RemoveContainer" containerID="4b8f3b87e3926fa053cd4b0c5bd15d8d0da985ef602aa3cc992424ae3a29db42"
Aug 05 18:19:58.912254 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.912233    2561 scope.go:117] "RemoveContainer" containerID="c9dd120ca09f47941bb011c6df90a0ce0fda768a4ade92d1ab4f6e020ab82e72"
Aug 05 18:19:58.928043 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.923366    2561 scope.go:117] "RemoveContainer" containerID="4b8f3b87e3926fa053cd4b0c5bd15d8d0da985ef602aa3cc992424ae3a29db42"
Aug 05 18:19:58.928043 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:19:58.924422    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b8f3b87e3926fa053cd4b0c5bd15d8d0da985ef602aa3cc992424ae3a29db42\": container with ID starting with 4b8f3b87e3926fa053cd4b0c5bd15d8d0da985ef602aa3cc992424ae3a29db42 not found: ID does not exist" containerID="4b8f3b87e3926fa053cd4b0c5bd15d8d0da985ef602aa3cc992424ae3a29db42"
Aug 05 18:19:58.928043 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.924456    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b8f3b87e3926fa053cd4b0c5bd15d8d0da985ef602aa3cc992424ae3a29db42"} err="failed to get container status \"4b8f3b87e3926fa053cd4b0c5bd15d8d0da985ef602aa3cc992424ae3a29db42\": rpc error: code = NotFound desc = could not find container \"4b8f3b87e3926fa053cd4b0c5bd15d8d0da985ef602aa3cc992424ae3a29db42\": container with ID starting with 4b8f3b87e3926fa053cd4b0c5bd15d8d0da985ef602aa3cc992424ae3a29db42 not found: ID does not exist"
Aug 05 18:19:58.928043 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.924482    2561 scope.go:117] "RemoveContainer" containerID="c9dd120ca09f47941bb011c6df90a0ce0fda768a4ade92d1ab4f6e020ab82e72"
Aug 05 18:19:58.928043 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:19:58.924748    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9dd120ca09f47941bb011c6df90a0ce0fda768a4ade92d1ab4f6e020ab82e72\": container with ID starting with c9dd120ca09f47941bb011c6df90a0ce0fda768a4ade92d1ab4f6e020ab82e72 not found: ID does not exist" containerID="c9dd120ca09f47941bb011c6df90a0ce0fda768a4ade92d1ab4f6e020ab82e72"
Aug 05 18:19:58.928043 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.924771    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9dd120ca09f47941bb011c6df90a0ce0fda768a4ade92d1ab4f6e020ab82e72"} err="failed to get container status \"c9dd120ca09f47941bb011c6df90a0ce0fda768a4ade92d1ab4f6e020ab82e72\": rpc error: code = NotFound desc = could not find container \"c9dd120ca09f47941bb011c6df90a0ce0fda768a4ade92d1ab4f6e020ab82e72\": container with ID starting with c9dd120ca09f47941bb011c6df90a0ce0fda768a4ade92d1ab4f6e020ab82e72 not found: ID does not exist"
Aug 05 18:19:58.928043 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.927853    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"]
Aug 05 18:19:58.929230 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:19:58.929209    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-846b5db989jldlt"]
Aug 05 18:20:00.730191 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:00.730156    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8591796f-8aa7-4b7f-9acd-93067124c027" path="/var/lib/kubelet/pods/8591796f-8aa7-4b7f-9acd-93067124c027/volumes"
Aug 05 18:20:05.811765 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:05.811729    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7l98z/must-gather-8jlgk"]
Aug 05 18:20:05.812237 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:05.812093    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d9bd4a8-4d15-442b-a299-d0530b0be122" containerName="istio-proxy"
Aug 05 18:20:05.812237 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:05.812108    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d9bd4a8-4d15-442b-a299-d0530b0be122" containerName="istio-proxy"
Aug 05 18:20:05.812237 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:05.812117    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d9bd4a8-4d15-442b-a299-d0530b0be122" containerName="istio-proxy"
Aug 05 18:20:05.812237 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:05.812122    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d9bd4a8-4d15-442b-a299-d0530b0be122" containerName="istio-proxy"
Aug 05 18:20:05.812237 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:05.812130    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8591796f-8aa7-4b7f-9acd-93067124c027" containerName="main"
Aug 05 18:20:05.812237 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:05.812135    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8591796f-8aa7-4b7f-9acd-93067124c027" containerName="main"
Aug 05 18:20:05.812237 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:05.812150    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d" containerName="main"
Aug 05 18:20:05.812237 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:05.812155    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d" containerName="main"
Aug 05 18:20:05.812237 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:05.812164    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8591796f-8aa7-4b7f-9acd-93067124c027" containerName="storage-initializer"
Aug 05 18:20:05.812237 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:05.812170    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8591796f-8aa7-4b7f-9acd-93067124c027" containerName="storage-initializer"
Aug 05 18:20:05.812237 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:05.812217    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8591796f-8aa7-4b7f-9acd-93067124c027" containerName="main"
Aug 05 18:20:05.812237 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:05.812224    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d9bd4a8-4d15-442b-a299-d0530b0be122" containerName="istio-proxy"
Aug 05 18:20:05.812237 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:05.812233    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8a3b5fa-1cb8-47fa-839f-81e06c9a1d2d" containerName="main"
Aug 05 18:20:05.812847 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:05.812335    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d9bd4a8-4d15-442b-a299-d0530b0be122" containerName="istio-proxy"
Aug 05 18:20:05.816805 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:05.816786    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7l98z/must-gather-8jlgk"
Aug 05 18:20:05.819360 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:05.819338    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7l98z\"/\"kube-root-ca.crt\""
Aug 05 18:20:05.819483 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:05.819340    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7l98z\"/\"openshift-service-ca.crt\""
Aug 05 18:20:05.820311 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:05.820296    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7l98z\"/\"default-dockercfg-m9nrx\""
Aug 05 18:20:05.824471 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:05.824451    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7l98z/must-gather-8jlgk"]
Aug 05 18:20:05.887058 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:05.887025    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/88bfb55b-de17-42ad-84db-235ecbb5ab10-must-gather-output\") pod \"must-gather-8jlgk\" (UID: \"88bfb55b-de17-42ad-84db-235ecbb5ab10\") " pod="openshift-must-gather-7l98z/must-gather-8jlgk"
Aug 05 18:20:05.887058 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:05.887061    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vw48\" (UniqueName: \"kubernetes.io/projected/88bfb55b-de17-42ad-84db-235ecbb5ab10-kube-api-access-2vw48\") pod \"must-gather-8jlgk\" (UID: \"88bfb55b-de17-42ad-84db-235ecbb5ab10\") " pod="openshift-must-gather-7l98z/must-gather-8jlgk"
Aug 05 18:20:05.988478 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:05.988442    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/88bfb55b-de17-42ad-84db-235ecbb5ab10-must-gather-output\") pod \"must-gather-8jlgk\" (UID: \"88bfb55b-de17-42ad-84db-235ecbb5ab10\") " pod="openshift-must-gather-7l98z/must-gather-8jlgk"
Aug 05 18:20:05.988478 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:05.988481    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vw48\" (UniqueName: \"kubernetes.io/projected/88bfb55b-de17-42ad-84db-235ecbb5ab10-kube-api-access-2vw48\") pod \"must-gather-8jlgk\" (UID: \"88bfb55b-de17-42ad-84db-235ecbb5ab10\") " pod="openshift-must-gather-7l98z/must-gather-8jlgk"
Aug 05 18:20:05.988869 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:05.988844    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/88bfb55b-de17-42ad-84db-235ecbb5ab10-must-gather-output\") pod \"must-gather-8jlgk\" (UID: \"88bfb55b-de17-42ad-84db-235ecbb5ab10\") " pod="openshift-must-gather-7l98z/must-gather-8jlgk"
Aug 05 18:20:05.996468 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:05.996443    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vw48\" (UniqueName: \"kubernetes.io/projected/88bfb55b-de17-42ad-84db-235ecbb5ab10-kube-api-access-2vw48\") pod \"must-gather-8jlgk\" (UID: \"88bfb55b-de17-42ad-84db-235ecbb5ab10\") " pod="openshift-must-gather-7l98z/must-gather-8jlgk"
Aug 05 18:20:06.136408 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:06.136321    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7l98z/must-gather-8jlgk"
Aug 05 18:20:06.262709 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:06.262679    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7l98z/must-gather-8jlgk"]
Aug 05 18:20:06.264342 ip-10-0-135-151 kubenswrapper[2561]: W0805 18:20:06.264315    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88bfb55b_de17_42ad_84db_235ecbb5ab10.slice/crio-34061e6265eca79c50c721ef6cfa5ba8a8d80117b5fb65c907183fb5fea5f86a WatchSource:0}: Error finding container 34061e6265eca79c50c721ef6cfa5ba8a8d80117b5fb65c907183fb5fea5f86a: Status 404 returned error can't find the container with id 34061e6265eca79c50c721ef6cfa5ba8a8d80117b5fb65c907183fb5fea5f86a
Aug 05 18:20:06.266104 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:06.266084    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 05 18:20:06.946480 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:06.946437    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7l98z/must-gather-8jlgk" event={"ID":"88bfb55b-de17-42ad-84db-235ecbb5ab10","Type":"ContainerStarted","Data":"34061e6265eca79c50c721ef6cfa5ba8a8d80117b5fb65c907183fb5fea5f86a"}
Aug 05 18:20:12.985802 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:12.985711    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7l98z/must-gather-8jlgk" event={"ID":"88bfb55b-de17-42ad-84db-235ecbb5ab10","Type":"ContainerStarted","Data":"6bae2a8eee4c7ffd1646df40d42bdfe412ae4c2f2feb171164197251246c5b84"}
Aug 05 18:20:12.985802 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:12.985759    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7l98z/must-gather-8jlgk" event={"ID":"88bfb55b-de17-42ad-84db-235ecbb5ab10","Type":"ContainerStarted","Data":"432f566c7d0fa7067cbdef45fcad389f8a87abc9a05ee4154f8fbfe953b3e0ed"}
Aug 05 18:20:13.001202 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:13.001149    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7l98z/must-gather-8jlgk" podStartSLOduration=1.950841086 podStartE2EDuration="8.00113587s" totalImagesPullingTime="6.050294784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:20:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:20:12.999889183 +0000 UTC m=+4068.744523502" watchObservedRunningTime="2026-08-05 18:20:13.00113587 +0000 UTC m=+4068.745770197"
Aug 05 18:20:28.204218 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:28.204175    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9_2ecc8132-d5f3-4c33-aca3-e7e0a30e7531/istio-proxy/0.log"
Aug 05 18:20:28.267448 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:28.267415    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9_2ecc8132-d5f3-4c33-aca3-e7e0a30e7531/istio-proxy/1.log"
Aug 05 18:20:29.162861 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:29.162825    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9_2ecc8132-d5f3-4c33-aca3-e7e0a30e7531/istio-proxy/0.log"
Aug 05 18:20:29.228857 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:29.228825    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9_2ecc8132-d5f3-4c33-aca3-e7e0a30e7531/istio-proxy/1.log"
Aug 05 18:20:30.349319 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:30.349289    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-nkm22_9ece522f-5b6b-47a1-b254-a92f050d50af/manager/0.log"
Aug 05 18:20:30.373080 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:30.373053    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-b7lz5_6efaf77d-b2f1-405c-93e3-27533d612e34/manager/0.log"
Aug 05 18:20:30.684938 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:30.684852    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-gb5pp_b9847013-4287-414a-85a2-48ba79c78131/manager/0.log"
Aug 05 18:20:30.694889 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:30.694854    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-tfk9h_a84e6912-5ce1-4155-a1f1-88c0b20dd5ff/limitador/0.log"
Aug 05 18:20:33.111970 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:33.111916    2561 generic.go:356] "Generic (PLEG): container finished" podID="88bfb55b-de17-42ad-84db-235ecbb5ab10" containerID="432f566c7d0fa7067cbdef45fcad389f8a87abc9a05ee4154f8fbfe953b3e0ed" exitCode=0
Aug 05 18:20:33.112402 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:33.111987    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7l98z/must-gather-8jlgk" event={"ID":"88bfb55b-de17-42ad-84db-235ecbb5ab10","Type":"ContainerDied","Data":"432f566c7d0fa7067cbdef45fcad389f8a87abc9a05ee4154f8fbfe953b3e0ed"}
Aug 05 18:20:33.112402 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:33.112320    2561 scope.go:117] "RemoveContainer" containerID="432f566c7d0fa7067cbdef45fcad389f8a87abc9a05ee4154f8fbfe953b3e0ed"
Aug 05 18:20:33.935891 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:33.935862    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-7l98z_must-gather-8jlgk_88bfb55b-de17-42ad-84db-235ecbb5ab10/gather/0.log"
Aug 05 18:20:34.505896 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:34.505863    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zgx78/must-gather-qfhs5"]
Aug 05 18:20:34.508869 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:34.508851    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zgx78/must-gather-qfhs5"
Aug 05 18:20:34.511558 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:34.511535    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zgx78\"/\"openshift-service-ca.crt\""
Aug 05 18:20:34.511666 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:34.511535    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zgx78\"/\"default-dockercfg-cb4lc\""
Aug 05 18:20:34.511666 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:34.511598    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zgx78\"/\"kube-root-ca.crt\""
Aug 05 18:20:34.518784 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:34.518762    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zgx78/must-gather-qfhs5"]
Aug 05 18:20:34.652998 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:34.652942    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/13a067ad-fd84-4c87-b05b-d0c8adfef61b-must-gather-output\") pod \"must-gather-qfhs5\" (UID: \"13a067ad-fd84-4c87-b05b-d0c8adfef61b\") " pod="openshift-must-gather-zgx78/must-gather-qfhs5"
Aug 05 18:20:34.653181 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:34.653008    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n48df\" (UniqueName: \"kubernetes.io/projected/13a067ad-fd84-4c87-b05b-d0c8adfef61b-kube-api-access-n48df\") pod \"must-gather-qfhs5\" (UID: \"13a067ad-fd84-4c87-b05b-d0c8adfef61b\") " pod="openshift-must-gather-zgx78/must-gather-qfhs5"
Aug 05 18:20:34.753464 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:34.753435    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/13a067ad-fd84-4c87-b05b-d0c8adfef61b-must-gather-output\") pod \"must-gather-qfhs5\" (UID: \"13a067ad-fd84-4c87-b05b-d0c8adfef61b\") " pod="openshift-must-gather-zgx78/must-gather-qfhs5"
Aug 05 18:20:34.753648 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:34.753476    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n48df\" (UniqueName: \"kubernetes.io/projected/13a067ad-fd84-4c87-b05b-d0c8adfef61b-kube-api-access-n48df\") pod \"must-gather-qfhs5\" (UID: \"13a067ad-fd84-4c87-b05b-d0c8adfef61b\") " pod="openshift-must-gather-zgx78/must-gather-qfhs5"
Aug 05 18:20:34.753774 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:34.753756    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/13a067ad-fd84-4c87-b05b-d0c8adfef61b-must-gather-output\") pod \"must-gather-qfhs5\" (UID: \"13a067ad-fd84-4c87-b05b-d0c8adfef61b\") " pod="openshift-must-gather-zgx78/must-gather-qfhs5"
Aug 05 18:20:34.761576 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:34.761515    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n48df\" (UniqueName: \"kubernetes.io/projected/13a067ad-fd84-4c87-b05b-d0c8adfef61b-kube-api-access-n48df\") pod \"must-gather-qfhs5\" (UID: \"13a067ad-fd84-4c87-b05b-d0c8adfef61b\") " pod="openshift-must-gather-zgx78/must-gather-qfhs5"
Aug 05 18:20:34.818736 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:34.818704    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zgx78/must-gather-qfhs5"
Aug 05 18:20:34.947644 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:34.947615    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zgx78/must-gather-qfhs5"]
Aug 05 18:20:34.949450 ip-10-0-135-151 kubenswrapper[2561]: W0805 18:20:34.949421    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13a067ad_fd84_4c87_b05b_d0c8adfef61b.slice/crio-9268bb8a64b6cc8456ea595b8817eccafe7ad975346002d154670532e22f2134 WatchSource:0}: Error finding container 9268bb8a64b6cc8456ea595b8817eccafe7ad975346002d154670532e22f2134: Status 404 returned error can't find the container with id 9268bb8a64b6cc8456ea595b8817eccafe7ad975346002d154670532e22f2134
Aug 05 18:20:35.122869 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:35.122830    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zgx78/must-gather-qfhs5" event={"ID":"13a067ad-fd84-4c87-b05b-d0c8adfef61b","Type":"ContainerStarted","Data":"9268bb8a64b6cc8456ea595b8817eccafe7ad975346002d154670532e22f2134"}
Aug 05 18:20:36.134804 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:36.134759    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zgx78/must-gather-qfhs5" event={"ID":"13a067ad-fd84-4c87-b05b-d0c8adfef61b","Type":"ContainerStarted","Data":"53cff2f9116c6609516d8acd337757841aba38b7b0218222499604e1dc417d30"}
Aug 05 18:20:36.135332 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:36.134820    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zgx78/must-gather-qfhs5" event={"ID":"13a067ad-fd84-4c87-b05b-d0c8adfef61b","Type":"ContainerStarted","Data":"389189e5fa5bee9c6903d1c438bd96a97b6fff2eebc9e105bf0289babb70e9e2"}
Aug 05 18:20:36.151480 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:36.151431    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zgx78/must-gather-qfhs5" podStartSLOduration=1.378865877 podStartE2EDuration="2.151413649s" totalImagesPullingTime="772.547772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-05 18:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:20:36.150796629 +0000 UTC m=+4091.895430955" watchObservedRunningTime="2026-08-05 18:20:36.151413649 +0000 UTC m=+4091.896047975"
Aug 05 18:20:37.308870 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:37.308835    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gfsf2_25132135-9fbe-49c4-9f6d-0e9db2cce6b8/global-pull-secret-syncer/0.log"
Aug 05 18:20:37.443074 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:37.443037    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8vtq4_8be1b0df-b531-4179-918f-1cae637ce3f7/konnectivity-agent/0.log"
Aug 05 18:20:37.489771 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:37.489743    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-151.ec2.internal_37213c9548f92e588dc6c041ff6110a8/haproxy/0.log"
Aug 05 18:20:39.371205 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:39.371150    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-7l98z/must-gather-8jlgk"]
Aug 05 18:20:39.372836 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:39.372034    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-7l98z/must-gather-8jlgk" podUID="88bfb55b-de17-42ad-84db-235ecbb5ab10" containerName="copy" containerID="cri-o://6bae2a8eee4c7ffd1646df40d42bdfe412ae4c2f2feb171164197251246c5b84" gracePeriod=2
Aug 05 18:20:39.379781 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:20:39.379045    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-8jlgk\" is forbidden: User \"system:node:ip-10-0-135-151.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-7l98z\": no relationship found between node 'ip-10-0-135-151.ec2.internal' and this object" podUID="88bfb55b-de17-42ad-84db-235ecbb5ab10" pod="openshift-must-gather-7l98z/must-gather-8jlgk"
Aug 05 18:20:39.379781 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:39.379718    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-7l98z/must-gather-8jlgk"]
Aug 05 18:20:39.761993 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:39.761119    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-7l98z_must-gather-8jlgk_88bfb55b-de17-42ad-84db-235ecbb5ab10/copy/0.log"
Aug 05 18:20:39.766120 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:39.764058    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7l98z/must-gather-8jlgk"
Aug 05 18:20:39.766566 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:20:39.766533    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-8jlgk\" is forbidden: User \"system:node:ip-10-0-135-151.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-7l98z\": no relationship found between node 'ip-10-0-135-151.ec2.internal' and this object" podUID="88bfb55b-de17-42ad-84db-235ecbb5ab10" pod="openshift-must-gather-7l98z/must-gather-8jlgk"
Aug 05 18:20:39.914894 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:39.909302    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vw48\" (UniqueName: \"kubernetes.io/projected/88bfb55b-de17-42ad-84db-235ecbb5ab10-kube-api-access-2vw48\") pod \"88bfb55b-de17-42ad-84db-235ecbb5ab10\" (UID: \"88bfb55b-de17-42ad-84db-235ecbb5ab10\") "
Aug 05 18:20:39.914894 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:39.909367    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/88bfb55b-de17-42ad-84db-235ecbb5ab10-must-gather-output\") pod \"88bfb55b-de17-42ad-84db-235ecbb5ab10\" (UID: \"88bfb55b-de17-42ad-84db-235ecbb5ab10\") "
Aug 05 18:20:39.914894 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:39.913645    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88bfb55b-de17-42ad-84db-235ecbb5ab10-kube-api-access-2vw48" (OuterVolumeSpecName: "kube-api-access-2vw48") pod "88bfb55b-de17-42ad-84db-235ecbb5ab10" (UID: "88bfb55b-de17-42ad-84db-235ecbb5ab10"). InnerVolumeSpecName "kube-api-access-2vw48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 05 18:20:39.934612 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:39.934575    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88bfb55b-de17-42ad-84db-235ecbb5ab10-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "88bfb55b-de17-42ad-84db-235ecbb5ab10" (UID: "88bfb55b-de17-42ad-84db-235ecbb5ab10"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 05 18:20:40.010540 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:40.010447    2561 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/88bfb55b-de17-42ad-84db-235ecbb5ab10-must-gather-output\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:20:40.010540 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:40.010487    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vw48\" (UniqueName: \"kubernetes.io/projected/88bfb55b-de17-42ad-84db-235ecbb5ab10-kube-api-access-2vw48\") on node \"ip-10-0-135-151.ec2.internal\" DevicePath \"\""
Aug 05 18:20:40.198994 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:40.197478    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-7l98z_must-gather-8jlgk_88bfb55b-de17-42ad-84db-235ecbb5ab10/copy/0.log"
Aug 05 18:20:40.201979 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:40.201164    2561 generic.go:356] "Generic (PLEG): container finished" podID="88bfb55b-de17-42ad-84db-235ecbb5ab10" containerID="6bae2a8eee4c7ffd1646df40d42bdfe412ae4c2f2feb171164197251246c5b84" exitCode=143
Aug 05 18:20:40.201979 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:40.201301    2561 scope.go:117] "RemoveContainer" containerID="6bae2a8eee4c7ffd1646df40d42bdfe412ae4c2f2feb171164197251246c5b84"
Aug 05 18:20:40.201979 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:40.201302    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7l98z/must-gather-8jlgk"
Aug 05 18:20:40.207218 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:20:40.207172    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-8jlgk\" is forbidden: User \"system:node:ip-10-0-135-151.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-7l98z\": no relationship found between node 'ip-10-0-135-151.ec2.internal' and this object" podUID="88bfb55b-de17-42ad-84db-235ecbb5ab10" pod="openshift-must-gather-7l98z/must-gather-8jlgk"
Aug 05 18:20:40.217569 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:40.217550    2561 scope.go:117] "RemoveContainer" containerID="432f566c7d0fa7067cbdef45fcad389f8a87abc9a05ee4154f8fbfe953b3e0ed"
Aug 05 18:20:40.231437 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:20:40.231215    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-8jlgk\" is forbidden: User \"system:node:ip-10-0-135-151.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-7l98z\": no relationship found between node 'ip-10-0-135-151.ec2.internal' and this object" podUID="88bfb55b-de17-42ad-84db-235ecbb5ab10" pod="openshift-must-gather-7l98z/must-gather-8jlgk"
Aug 05 18:20:40.252197 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:40.252151    2561 scope.go:117] "RemoveContainer" containerID="6bae2a8eee4c7ffd1646df40d42bdfe412ae4c2f2feb171164197251246c5b84"
Aug 05 18:20:40.252935 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:20:40.252896    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bae2a8eee4c7ffd1646df40d42bdfe412ae4c2f2feb171164197251246c5b84\": container with ID starting with 6bae2a8eee4c7ffd1646df40d42bdfe412ae4c2f2feb171164197251246c5b84 not found: ID does not exist" containerID="6bae2a8eee4c7ffd1646df40d42bdfe412ae4c2f2feb171164197251246c5b84"
Aug 05 18:20:40.253060 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:40.252965    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bae2a8eee4c7ffd1646df40d42bdfe412ae4c2f2feb171164197251246c5b84"} err="failed to get container status \"6bae2a8eee4c7ffd1646df40d42bdfe412ae4c2f2feb171164197251246c5b84\": rpc error: code = NotFound desc = could not find container \"6bae2a8eee4c7ffd1646df40d42bdfe412ae4c2f2feb171164197251246c5b84\": container with ID starting with 6bae2a8eee4c7ffd1646df40d42bdfe412ae4c2f2feb171164197251246c5b84 not found: ID does not exist"
Aug 05 18:20:40.253060 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:40.252991    2561 scope.go:117] "RemoveContainer" containerID="432f566c7d0fa7067cbdef45fcad389f8a87abc9a05ee4154f8fbfe953b3e0ed"
Aug 05 18:20:40.253387 ip-10-0-135-151 kubenswrapper[2561]: E0805 18:20:40.253363    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"432f566c7d0fa7067cbdef45fcad389f8a87abc9a05ee4154f8fbfe953b3e0ed\": container with ID starting with 432f566c7d0fa7067cbdef45fcad389f8a87abc9a05ee4154f8fbfe953b3e0ed not found: ID does not exist" containerID="432f566c7d0fa7067cbdef45fcad389f8a87abc9a05ee4154f8fbfe953b3e0ed"
Aug 05 18:20:40.253576 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:40.253550    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"432f566c7d0fa7067cbdef45fcad389f8a87abc9a05ee4154f8fbfe953b3e0ed"} err="failed to get container status \"432f566c7d0fa7067cbdef45fcad389f8a87abc9a05ee4154f8fbfe953b3e0ed\": rpc error: code = NotFound desc = could not find container \"432f566c7d0fa7067cbdef45fcad389f8a87abc9a05ee4154f8fbfe953b3e0ed\": container with ID starting with 432f566c7d0fa7067cbdef45fcad389f8a87abc9a05ee4154f8fbfe953b3e0ed not found: ID does not exist"
Aug 05 18:20:40.733934 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:40.733888    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88bfb55b-de17-42ad-84db-235ecbb5ab10" path="/var/lib/kubelet/pods/88bfb55b-de17-42ad-84db-235ecbb5ab10/volumes"
Aug 05 18:20:42.844633 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:42.844599    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-nkm22_9ece522f-5b6b-47a1-b254-a92f050d50af/manager/0.log"
Aug 05 18:20:42.876133 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:42.876100    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-b7lz5_6efaf77d-b2f1-405c-93e3-27533d612e34/manager/0.log"
Aug 05 18:20:43.154903 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:43.154023    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-gb5pp_b9847013-4287-414a-85a2-48ba79c78131/manager/0.log"
Aug 05 18:20:43.196457 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:43.196429    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-tfk9h_a84e6912-5ce1-4155-a1f1-88c0b20dd5ff/limitador/0.log"
Aug 05 18:20:44.530228 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:44.530177    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-776db5ccf4-n6kq7_b3428980-76ee-439e-963b-368def9cdb48/metrics-server/0.log"
Aug 05 18:20:44.638572 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:44.638542    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-922nb_2091dbfe-3a2a-44be-980e-d3655d607668/node-exporter/0.log"
Aug 05 18:20:44.670764 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:44.670734    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-922nb_2091dbfe-3a2a-44be-980e-d3655d607668/kube-rbac-proxy/0.log"
Aug 05 18:20:44.697103 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:44.697066    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-922nb_2091dbfe-3a2a-44be-980e-d3655d607668/init-textfile/0.log"
Aug 05 18:20:45.228498 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:45.228460    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-tcc2h_800108ba-5328-4c41-ab17-6efb3927a0ff/prometheus-operator/0.log"
Aug 05 18:20:45.245225 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:45.245193    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-tcc2h_800108ba-5328-4c41-ab17-6efb3927a0ff/kube-rbac-proxy/0.log"
Aug 05 18:20:45.269500 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:45.269465    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-66685ff6d4-vxqqd_7682b61a-d92f-4b1a-9b7f-e3404237c48f/prometheus-operator-admission-webhook/0.log"
Aug 05 18:20:47.388845 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:47.388806    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zgx78/perf-node-gather-daemonset-2qz4v"]
Aug 05 18:20:47.389349 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:47.389334    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88bfb55b-de17-42ad-84db-235ecbb5ab10" containerName="copy"
Aug 05 18:20:47.389421 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:47.389351    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="88bfb55b-de17-42ad-84db-235ecbb5ab10" containerName="copy"
Aug 05 18:20:47.389421 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:47.389362    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88bfb55b-de17-42ad-84db-235ecbb5ab10" containerName="gather"
Aug 05 18:20:47.389421 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:47.389373    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="88bfb55b-de17-42ad-84db-235ecbb5ab10" containerName="gather"
Aug 05 18:20:47.389572 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:47.389482    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="88bfb55b-de17-42ad-84db-235ecbb5ab10" containerName="gather"
Aug 05 18:20:47.389572 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:47.389501    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="88bfb55b-de17-42ad-84db-235ecbb5ab10" containerName="copy"
Aug 05 18:20:47.392493 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:47.392468    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zgx78/perf-node-gather-daemonset-2qz4v"
Aug 05 18:20:47.395288 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:47.395265    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7c4c1cbc-ea76-4e80-815d-e939a858e599-proc\") pod \"perf-node-gather-daemonset-2qz4v\" (UID: \"7c4c1cbc-ea76-4e80-815d-e939a858e599\") " pod="openshift-must-gather-zgx78/perf-node-gather-daemonset-2qz4v"
Aug 05 18:20:47.395410 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:47.395311    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7c4c1cbc-ea76-4e80-815d-e939a858e599-podres\") pod \"perf-node-gather-daemonset-2qz4v\" (UID: \"7c4c1cbc-ea76-4e80-815d-e939a858e599\") " pod="openshift-must-gather-zgx78/perf-node-gather-daemonset-2qz4v"
Aug 05 18:20:47.395462 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:47.395405    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c4c1cbc-ea76-4e80-815d-e939a858e599-sys\") pod \"perf-node-gather-daemonset-2qz4v\" (UID: \"7c4c1cbc-ea76-4e80-815d-e939a858e599\") " pod="openshift-must-gather-zgx78/perf-node-gather-daemonset-2qz4v"
Aug 05 18:20:47.395506 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:47.395469    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c4c1cbc-ea76-4e80-815d-e939a858e599-lib-modules\") pod \"perf-node-gather-daemonset-2qz4v\" (UID: \"7c4c1cbc-ea76-4e80-815d-e939a858e599\") " pod="openshift-must-gather-zgx78/perf-node-gather-daemonset-2qz4v"
Aug 05 18:20:47.395543 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:47.395503    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsckm\" (UniqueName: \"kubernetes.io/projected/7c4c1cbc-ea76-4e80-815d-e939a858e599-kube-api-access-qsckm\") pod \"perf-node-gather-daemonset-2qz4v\" (UID: \"7c4c1cbc-ea76-4e80-815d-e939a858e599\") " pod="openshift-must-gather-zgx78/perf-node-gather-daemonset-2qz4v"
Aug 05 18:20:47.404654 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:47.404619    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zgx78/perf-node-gather-daemonset-2qz4v"]
Aug 05 18:20:47.496680 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:47.496643    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c4c1cbc-ea76-4e80-815d-e939a858e599-lib-modules\") pod \"perf-node-gather-daemonset-2qz4v\" (UID: \"7c4c1cbc-ea76-4e80-815d-e939a858e599\") " pod="openshift-must-gather-zgx78/perf-node-gather-daemonset-2qz4v"
Aug 05 18:20:47.496864 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:47.496689    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsckm\" (UniqueName: \"kubernetes.io/projected/7c4c1cbc-ea76-4e80-815d-e939a858e599-kube-api-access-qsckm\") pod \"perf-node-gather-daemonset-2qz4v\" (UID: \"7c4c1cbc-ea76-4e80-815d-e939a858e599\") " pod="openshift-must-gather-zgx78/perf-node-gather-daemonset-2qz4v"
Aug 05 18:20:47.496864 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:47.496733    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7c4c1cbc-ea76-4e80-815d-e939a858e599-proc\") pod \"perf-node-gather-daemonset-2qz4v\" (UID: \"7c4c1cbc-ea76-4e80-815d-e939a858e599\") " pod="openshift-must-gather-zgx78/perf-node-gather-daemonset-2qz4v"
Aug 05 18:20:47.496864 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:47.496754    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7c4c1cbc-ea76-4e80-815d-e939a858e599-podres\") pod \"perf-node-gather-daemonset-2qz4v\" (UID: \"7c4c1cbc-ea76-4e80-815d-e939a858e599\") " pod="openshift-must-gather-zgx78/perf-node-gather-daemonset-2qz4v"
Aug 05 18:20:47.496864 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:47.496772    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c4c1cbc-ea76-4e80-815d-e939a858e599-sys\") pod \"perf-node-gather-daemonset-2qz4v\" (UID: \"7c4c1cbc-ea76-4e80-815d-e939a858e599\") " pod="openshift-must-gather-zgx78/perf-node-gather-daemonset-2qz4v"
Aug 05 18:20:47.496864 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:47.496840    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c4c1cbc-ea76-4e80-815d-e939a858e599-sys\") pod \"perf-node-gather-daemonset-2qz4v\" (UID: \"7c4c1cbc-ea76-4e80-815d-e939a858e599\") " pod="openshift-must-gather-zgx78/perf-node-gather-daemonset-2qz4v"
Aug 05 18:20:47.496864 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:47.496845    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7c4c1cbc-ea76-4e80-815d-e939a858e599-proc\") pod \"perf-node-gather-daemonset-2qz4v\" (UID: \"7c4c1cbc-ea76-4e80-815d-e939a858e599\") " pod="openshift-must-gather-zgx78/perf-node-gather-daemonset-2qz4v"
Aug 05 18:20:47.497220 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:47.496846    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c4c1cbc-ea76-4e80-815d-e939a858e599-lib-modules\") pod \"perf-node-gather-daemonset-2qz4v\" (UID: \"7c4c1cbc-ea76-4e80-815d-e939a858e599\") " pod="openshift-must-gather-zgx78/perf-node-gather-daemonset-2qz4v"
Aug 05 18:20:47.497220 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:47.496904    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7c4c1cbc-ea76-4e80-815d-e939a858e599-podres\") pod \"perf-node-gather-daemonset-2qz4v\" (UID: \"7c4c1cbc-ea76-4e80-815d-e939a858e599\") " pod="openshift-must-gather-zgx78/perf-node-gather-daemonset-2qz4v"
Aug 05 18:20:47.509428 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:47.509393    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsckm\" (UniqueName: \"kubernetes.io/projected/7c4c1cbc-ea76-4e80-815d-e939a858e599-kube-api-access-qsckm\") pod \"perf-node-gather-daemonset-2qz4v\" (UID: \"7c4c1cbc-ea76-4e80-815d-e939a858e599\") " pod="openshift-must-gather-zgx78/perf-node-gather-daemonset-2qz4v"
Aug 05 18:20:47.615831 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:47.615796    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-57d57488d-wcvsn_eae812f8-7c5a-4017-9dbf-f04df9cf563d/console/0.log"
Aug 05 18:20:47.706361 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:47.706101    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zgx78/perf-node-gather-daemonset-2qz4v"
Aug 05 18:20:47.865292 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:47.865262    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zgx78/perf-node-gather-daemonset-2qz4v"]
Aug 05 18:20:47.867265 ip-10-0-135-151 kubenswrapper[2561]: W0805 18:20:47.867230    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7c4c1cbc_ea76_4e80_815d_e939a858e599.slice/crio-2d0662487b742d4ba96abf4bbd9035248e79f409663db46b55234d27bd7d8a92 WatchSource:0}: Error finding container 2d0662487b742d4ba96abf4bbd9035248e79f409663db46b55234d27bd7d8a92: Status 404 returned error can't find the container with id 2d0662487b742d4ba96abf4bbd9035248e79f409663db46b55234d27bd7d8a92
Aug 05 18:20:48.270250 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:48.270148    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zgx78/perf-node-gather-daemonset-2qz4v" event={"ID":"7c4c1cbc-ea76-4e80-815d-e939a858e599","Type":"ContainerStarted","Data":"d20c650e42fe740b58eb94aa7dfa4d96cf5d73e00f34016598121ddf63aa882a"}
Aug 05 18:20:48.270250 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:48.270222    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zgx78/perf-node-gather-daemonset-2qz4v" event={"ID":"7c4c1cbc-ea76-4e80-815d-e939a858e599","Type":"ContainerStarted","Data":"2d0662487b742d4ba96abf4bbd9035248e79f409663db46b55234d27bd7d8a92"}
Aug 05 18:20:48.270474 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:48.270369    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zgx78/perf-node-gather-daemonset-2qz4v"
Aug 05 18:20:48.285778 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:48.285711    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zgx78/perf-node-gather-daemonset-2qz4v" podStartSLOduration=1.285693459 podStartE2EDuration="1.285693459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-05 18:20:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-05 18:20:48.285296777 +0000 UTC m=+4104.029931304" watchObservedRunningTime="2026-08-05 18:20:48.285693459 +0000 UTC m=+4104.030327787"
Aug 05 18:20:49.144263 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:49.144230    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4bxkx_00d8f640-e14d-4601-b45d-81ef6fbaf47d/dns/0.log"
Aug 05 18:20:49.162783 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:49.162752    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4bxkx_00d8f640-e14d-4601-b45d-81ef6fbaf47d/kube-rbac-proxy/0.log"
Aug 05 18:20:49.306897 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:49.306862    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vjnk7_47a3441a-cffb-4919-95b8-6393fd63c14b/dns-node-resolver/0.log"
Aug 05 18:20:49.821844 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:49.821809    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7c944f9855-dsl5q_6880ea65-e478-448a-a6fe-edc133ef596a/registry/0.log"
Aug 05 18:20:49.894467 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:49.894436    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tg9l4_eacfb8d3-1760-45ee-a594-7656a55c9d93/node-ca/0.log"
Aug 05 18:20:51.002189 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:51.002157    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9_2ecc8132-d5f3-4c33-aca3-e7e0a30e7531/istio-proxy/0.log"
Aug 05 18:20:51.066283 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:51.066249    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9_2ecc8132-d5f3-4c33-aca3-e7e0a30e7531/istio-proxy/1.log"
Aug 05 18:20:51.605675 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:51.605648    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pcknb_50a0dce8-3a32-4b5e-a07a-7a4492cd5166/serve-healthcheck-canary/0.log"
Aug 05 18:20:52.119541 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:52.119514    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hz4nf_67c22df9-3392-47e6-8097-30468199cade/kube-rbac-proxy/0.log"
Aug 05 18:20:52.141389 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:52.141357    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hz4nf_67c22df9-3392-47e6-8097-30468199cade/exporter/0.log"
Aug 05 18:20:52.170585 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:52.170560    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hz4nf_67c22df9-3392-47e6-8097-30468199cade/extractor/0.log"
Aug 05 18:20:54.287047 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:54.287015    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zgx78/perf-node-gather-daemonset-2qz4v"
Aug 05 18:20:56.024853 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:56.024819    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-6f49dcd9ff-cgtr7_8a3c78ca-b8ed-4973-b09e-55d7cac74626/manager/0.log"
Aug 05 18:20:56.605891 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:56.605861    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-54d86dcccd-69dsf_57894026-1268-467e-b264-8eb72ae47296/manager/0.log"
Aug 05 18:20:56.627195 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:56.627167    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-qzlbc_bf98fcfc-4562-48f9-8d84-4654b6cd399a/server/0.log"
Aug 05 18:20:57.320254 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:57.320221    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9_2ecc8132-d5f3-4c33-aca3-e7e0a30e7531/istio-proxy/0.log"
Aug 05 18:20:57.416943 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:57.416904    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9_2ecc8132-d5f3-4c33-aca3-e7e0a30e7531/istio-proxy/1.log"
Aug 05 18:20:58.384812 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:58.384766    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-8588677cfb-dl669_c26bd63d-5f44-405d-b181-d239062c25f4/manager/0.log"
Aug 05 18:20:58.424343 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:58.424314    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-hxqxk_c5717793-e01e-4e74-8f1b-d65fbc81682a/s3-init/0.log"
Aug 05 18:20:58.439083 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:58.439059    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-hxqxk_c5717793-e01e-4e74-8f1b-d65fbc81682a/download-hf-model/0.log"
Aug 05 18:20:58.470904 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:58.470873    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-qnq2d_c9534844-5bc2-4f1e-950a-a833156cd036/seaweedfs/0.log"
Aug 05 18:20:58.781068 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:58.781039    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9_2ecc8132-d5f3-4c33-aca3-e7e0a30e7531/istio-proxy/0.log"
Aug 05 18:20:58.836282 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:20:58.836240    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-tgmf9_2ecc8132-d5f3-4c33-aca3-e7e0a30e7531/istio-proxy/1.log"
Aug 05 18:21:02.880514 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:21:02.880483    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-7tdb9_83ff891c-1a98-4abd-a563-1308fd7326b8/migrator/0.log"
Aug 05 18:21:02.899150 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:21:02.899121    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-7tdb9_83ff891c-1a98-4abd-a563-1308fd7326b8/graceful-termination/0.log"
Aug 05 18:21:04.232743 ip-10-0-135-151 kubenswrapper[2561]: I0805 18:21:04.232711    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d7kqb_66f685fa-db6d-474f-b836-d0d9768e2d09/kube-multus-additional-cni-plugins/0.log"