Cold starts on serverless GPU, the problem vLLM is learning to solve
Your agent receives 3 requests per hour. With a dedicated H100 pod you pay 720 hours a month to use 3. The alternative, serverless GPU, bills by the second of real GPU time, but every cold start loads the model from scratch, and loading 30 GB of weights before the first token is exactly the kind of wait that kills a conversational product.
The vLLM cold-start problem is well known and was thinly documented. An academic paper in June analyzed it formally, and since September vLLM itself has been closing the gap with tools that change the math. - Source Breaking the Ice, vLLM cold start analysis
Where the cold start comes from
Loading vLLM adds four costs. Importing Python and its dependencies, reading weights from disk into RAM, quantizing and preparing tensors, and capturing CUDA graphs with kernel autotuning. On an H100 with a 30 GB model, that total easily crosses a minute.
The part that surprises first-time measurers is CUDA graph capture. Before the GC optimization, capture took 12 seconds and full initialization 28.9 on H200. Freezing the garbage collector during capture brought those down to 2 and 8.2 seconds respectively. - Source PR #54646
And all of this happens per container. If your serverless platform scales to zero and boots replicas on demand, every replica pays the full toll.
What vLLM ships now to attack it
Three new tools from the last quarter attack the problem from different angles.
vllm preload and the weight daemon. v0.30 introduced a persistent weight cache that keeps post-quantized tensors GPU-resident across restarts, with a health endpoint to know when it's ready. A restart stops reading 30 GB from disk while the daemon lives. - Source v0.30.0 release
vllm snapshot create/restore. v0.31 added, experimentally, full-engine snapshots via CRIU. An initialized TP1 engine restores from the snapshot without repeating load or warmup. For serverless with warm tiers it's the natural path, though it's TP1-only today and worth testing on your hardware before committing. - Source PR #51360
Pre-compiled Python bytecode. vLLM images now ship bytecode to avoid recompiling imports at boot. The measured saving is about 18 seconds per startup on H100 servers. Free, just update the image. - Source Reducing vLLM startup time
# boot with preload active
vllm serve $MODEL --load-format ipc_cache
# experimental snapshot of an initialized TP1 engine
vllm snapshot create --snapshot-path /mnt/cache/engine.snap
vllm snapshot restore --snapshot-path /mnt/cache/engine.snap
The real decision, dedicated pod vs serverless
The math I run for my RunPod pods, with November 2026 numbers.
A dedicated H100 runs around 2 to 3 dollars an hour depending on the provider, 1440 to 2160 dollars a month at 24/7. If your service holds constant traffic, dedicated always wins, no cold start and the full VRAM is yours.
Serverless bills per execution second plus a cold-boot price, and platforms like RunPod Flash promise boots without the docker layer. With sporadic traffic, say 500 requests a day of 10 seconds each, real compute is 5000 seconds a month, about 1.4 hours. Even paying 5 cold starts of 60 seconds a day, the bill lands in the tens of dollars against thousands for the dedicated pod. - Source RunPod Flash, dockerless serverless
The break-even shows up in cold starts multiplied by frequency. A model that takes 90 seconds to boot and receives a request every 5 minutes is cold 70% of the time. With vllm preload and snapshots that cold time drops, and the band where serverless pays widens. My practical rule after measuring, under one request per minute in valley hours, serverless. Between one and ten per minute sustained, dedicated pod with manual autoscaling. Above that, pure dedicated.
The hybrid pattern I use
For products with office-hours traffic, daytime peaks and quiet nights, the pattern that works best combines both.
The dedicated pod keeps the model warm through real-traffic hours and shuts down at night. A serverless tier with snapshots covers the night requests, accepting degraded latency for that marginal traffic. The router in LiteLLM decides by hour and by queue. It's more pieces than one pod, but the bill dropped 60 to 70% on the products where I tested it, and the nighttime p95 stayed under 30 seconds with snapshots.
Tips for your setup
- Measure your real cold start before deciding. A cron that boots the container and times until the first successful token. That number decides, not the paper's.
- Keep the HF cache on a persistent volume. On RunPod, a volume mounted at
~/.cache/huggingfaceavoids re-downloading 30 GB per boot. It's the difference between 90 and 200 second cold starts. - Warm up with synthetic requests. After booting, send 3 real requests from your pattern before opening traffic. Kernel autotuning on the first real batch is slow.
- Snapshot only if your engine is TP1. The experimental CRIU support covers TP1 today. With TP8, weight preload remains your best tool.
- Watch the Q4 roadmap. The official list includes FP4 KV expansion with disk offloading and extended KV connectors. The cold keeps dropping. - Source Roadmap Q4 2026
Serverless GPU for LLMs stopped being a promise and started being arithmetic. With preload and snapshots, the arithmetic increasingly favors whoever doesn't want to pay for empty hours.
Does your traffic justify a dedicated pod, or are you paying for empty hours without knowing it? Tell me in the comments.