Skip to content
AI•5 min read

vLLM vs SGLang on a rented H100, the benchmark nobody publishes

Every impressive inference number comes from racks nobody rents. GB300 NVL72, Vera Rubin, AgentX. Meanwhile, the real question for anyone renting an H100 by the hour remains undocumented. vLLM or SGLang?

Both teams publish their own benchmarks and always win. The vLLM blog reported 954K tokens/s per GPU serving DeepSeek-V4.1-Flash with P/D disaggregation on a GB300 NVL72, within 6% of DeepSeek's own engine. The SGLang blog answered with its kernel series, going from 35 to 873 tokens/s in single-stream decode of the same model. Both are right in their context. Neither tells you what to install on your 8xH100 pod tonight.

This post doesn't give you my number. It gives you the method to obtain yours, the context behind the published numbers, and the decisions you can make today. - Source DeepSeek-V4.1-Flash P/D + Wide-EP on vLLM, SGLang kernel optimization

What the published numbers say

Before measuring, it's worth knowing what each team claims, with dates and context.

vLLM, October 2026. 5x agentic throughput on DeepSeek-V4.1-Flash three weeks after the model's release, and 7.8x per GPU on Rubin NVL72 versus GB200 NVL72. The most honest number for your scale is cost, 14.6x to 106x cheaper than Claude Opus 5 serving AgentX, though measured on their hardware. - Source vLLM x AgentX, Vera Rubin preview

SGLang, September and October 2026. 873 tokens/s single-stream on DeepSeek-V4.1-Flash after kernel optimization, PD disaggregation with 27% more throughput and half the p95 TTFT on long prompts, and low-latency megakernels with up to 44% faster per-user decode on GB300. - Source SGLang v0.5.22, v0.5.23

The warning nobody says out loud. To date, no independent benchmark compares the two engines on the same hardware with the same model and the same harness. Artificial Analysis measures providers, not engines on your GPU. Every other number is self-reported by the team with something to gain.

The minimum honest harness

A benchmark is worth its reproducibility. This is the setup I use, the same approach from my vLLM measurement guide, applied to comparing two engines.

Pin the versions. As of today, vLLM v0.33.0 and SGLang v0.5.23. Both moved big things in October, MRV2 as default and new kernels, so comparing old versions measures the past. - Source vLLM releases, SGLang releases

# benchmark-h100.sh — pin images, same model, same hardware
export MODEL="Qwen/Qwen3.5-397B-A17B-FP8"   # the one you actually serve
export IMG_VLLM="vllm/vllm:v0.33.0"
export IMG_SGLANG="lmsysorg/sglang:v0.5.23"
echo "run both containers with identical GPU limits"

A model you genuinely serve, with equivalent quantized checkpoints. Comparing vLLM with native FP8 against SGLang with AWQ measures quantization, not the engine.

The three metrics that matter. The same ones we already used to measure vLLM alone. TTFT at the 90th percentile, output tokens per second per user at the 50th, and aggregate throughput at target concurrency. An average without percentiles is marketing.

# vLLM now ships its own bench
vllm bench serve --model $MODEL --percentile-metrics ttft,tpot,itl \
  --num-prompts 200 --random-input-len 2048 --random-output-len 512

# SGLang equivalents with its benchmark
python -m sglang.bench_serving --backend sglang --model $MODEL \
  --num-prompts 200 --random-input 2048 --random-output 512

Watch one detail that ruins comparisons. Each engine's default benches ship different presets for concurrency, warmup and sampling. Set the parameters yourself, run the same job against both OpenAI-compatible endpoints, and discard the first warmup runs.

The workload has to resemble your traffic. If your users send 200 tokens and read 800, a 4K-input benchmark lies to you. Agentic workloads, long prompts and short outputs, invert the result compared to traditional chat. The cold-start paper and the AgentX post exist precisely because that pattern changed the game.

What to expect by hardware

With H100 and H200, FP8 weights plus FP8 KV cache is the base recipe on vLLM, and the TurboQuant study concluded 4-bit KV is situational, not a default. If you rent Blackwell, B200 or B300, NVFP4 enters the conversation with day-0 checkpoints of Mistral Large 4 and Qwen3.8. In practice, the engine that best supports your KV dtype on your specific architecture wins the round, and that changes per model. - Source TurboQuant KV study, FP8 KV cache

The other real factor is the feature ecosystem. vLLM ships /v1/decisions, token embeddings with late chunking, and a concurrent API server with --api-server-count adding 43 to 85% requests/s on short requests. SGLang answers with PD disaggregation that swaps roles live without restart, and HiCache with up to 15.5x input throughput on KV tiering. Neither wins everything, and your workload decides which feature pays the rent. - Source vLLM v0.33.0 release, SGLang v0.5.21

Signals to decide without your own benchmark

If you can't afford the GPU hours today, these are the signals you can read in advance.

Model with an official drafter or MTP. vLLM's speculative decoding ecosystem, EAGLE, DFlash, DSpark and MTP, is more integrated, and the draft-slot budget fix in v0.33 took DeepSeek-V4.1-Flash from 1373 to 3390 output tokens/s on GB300. If your model ships a drafter, start with vLLM. - Source v0.33.0 release notes

Long multi-turn with shared KV cache. SGLang leads on KV tiering and swapped prefill/decode roles live in v0.5.21. If your product is agents with kilometer-long conversations, that set weighs. - Source SGLang v0.5.21

Structured tools with odd parsers. vLLM's generic hf parser reads the checkpoint's response_template and eliminates per-model parsers. If your tool calls broke over a parser, that problem changed size in v0.32. - Source v0.32.0 release

AMD in the mix. vLLM maintains the most complete ROCm pipeline, and the Kimi K3 benchmark on MI355X matched NVIDIA's HGX B300 per GPU. If your provider has cheaper MI325X or MI355X, that math changed this quarter. - Source Kimi K3 on MI355X

Tips for when you run yours

  1. Pin the image and the dtype. Keep the container tag, model commit and exact flags in the repo. A benchmark without that is an anecdote.
  2. Discard warmup, three runs, take the median. The first batches pay compilation and autotuning. Publish the median and p90, never your best run.
  3. Same sampler on both engines. temperature, top_p and max_tokens identical. Sampling affects output length and with it throughput.
  4. Hourly GPU budget, result per dollar. The engine 15% faster that eats 40% more VRAM closes your batch-size door. Measure cost per million tokens served.
  5. Publish your harness. The ecosystem gap is reproducibility. A repo with your script is worth more than your number.

My honest forecast is that on H100 with FP8 models you'll find double-digit differences, not orders of magnitude, with a winner that depends on traffic pattern. When I run my full series on 8xH100 I'll publish it here, harness included.

Which engine are you serving on today, and what made you choose it? Tell me in the comments.


> More posts