Build a small request set from the application
A summarizer and a chat assistant produce different loads. Use representative input lengths and requested output lengths, then keep that test set fixed across providers. Remove sensitive data or create synthetic examples with the same length distribution.
Keep a short quality check alongside the performance test. A faster configuration that truncates responses or changes model precision without acceptable output quality has changed the experiment.
Measure what the user waits for
Time to first token captures how long a user sees no answer. Inter-token latency describes how smoothly the answer continues. Total request latency includes both phases. Aggregate tokens per second helps estimate capacity, but can conceal a slow experience for individual requests.
vLLM provides a serving benchmark with request-rate and dataset controls and latency metrics. Pin the installed version and save its command-line arguments with your results so another engineer can repeat the run.
| Measurement | Question it answers |
|---|---|
| First-token latency | How long until a response begins? |
| End-to-end latency | When is the complete result ready? |
| Successful output tokens | How much usable work finished? |
| Errors and timeouts | What did the throughput number leave out? |
References: vLLM serving benchmark options
Separate loading from steady operation
Start once with no warm worker and record the delay. Next, run a warm workload long enough to see queueing and memory growth. Finally, send a burst above the ordinary arrival rate and observe how quickly the queue recovers.
Use the same client location for each test. If your users are in India, a benchmark client beside a US GPU is not measuring their network path. Record the region, client location and whether requests cross a proxy.
Make the decision reproducible
Report median and tail latency together with sample count, duration and failed requests. Repeat the experiment at different times before interpreting a small difference. Keep the raw results; a chart without its configuration cannot settle a later disagreement.
Choose the configuration that meets the latency and quality requirements at an acceptable cost per successful request. If two offers are close, recovery behavior and ease of deployment can reasonably decide the purchase.
Questions readers ask
Is peak tokens per second enough to compare GPUs?
No. Prompt lengths, concurrency, precision and latency limits can change the result. Keep those conditions equal.
Should I include failed requests?
Yes. Report their count and any billed retries so the apparent throughput does not hide unusable work.
Sources and editorial notes
Primary documentation reviewed October 9, 2026. Provider descriptions reflect published documentation. Recommendations are our analysis; worked figures are illustrative unless explicitly stated otherwise. No independent provider benchmark or uptime ranking is claimed.
Read the editorial policy. NeuraMesh is a prototype, not live verified GPU inventory.