KV cache for multi-turn agents, the hidden cost of your agents
An agent talking to your API does something a 2024 chatbot didn't. It comes back. Turn 12 of the same conversation resends the previous 11 turns as context, and your inference server re-prefills all of it unless the KV cache of those turns is still alive somewhere. With 50K tokens of accumulated context, that re-prefill is your most expensive GPU line item and your ugliest TTFT.
vLLM dedicated half of October to this problem. Three official posts in two weeks cover the full KV flow between turns, and the number that sums them up is up to 3.3x TTFT improvement for long multi-turn conversations. - Source Bi-Directional KV Transfer
Why multi-turn breaks your cache
vLLM's prefix cache reuses the KV of identical prefixes. It works beautifully with a fixed system prompt. With agents it breaks for two reasons.
The first is conversation direction. A new turn appends content at the end, the previous turn's cache should serve in full, but the request landed on a different replica and the cache was orphaned on the wrong GPU. With a single pod this never happens. With round-robin routing and several replicas, it happens to most turns.
The second is lifecycle. Agents think between turns, sometimes for minutes. vLLM's LRU evicts KV blocks under memory pressure, and when the agent returns, its history was recycled. Your system prompt cache survives because it's used a thousand times an hour. That specific conversation's cache dies because it's used once every 3 minutes. - Source Keeping vLLM's Prefix Cache Warm Between Agent Turns
The three layers of the solution
The stack vLLM is assembling has three levels, and it's worth knowing all of them because they combine.
Bi-directional KV transfer between nodes. The October 12 post describes the flow where the decode node hands its KV back to prefill at the end of a turn, so the next turn starts from cache instead of re-processing. On the team's benchmarks, TTFT on long multi-turn prompts dropped up to 3.3x, and the push-based NIXL connector, announced two days later, takes the transfer off the TTFT critical path by writing KV straight into the other node's memory over RDMA. - Source NIXL push connector
Cache tiering. What doesn't fit in VRAM drops to host memory, disk or an object store. LMCache and Mooncake plug into vLLM, and v0.33 ships an EFA image including Mooncake's transfer engine for multi-node. The Q4 roadmap adds cache tiers with TTLs and per-request filesystem eviction. Your agent conversation can live in host RAM for the price of a sandwich. - Source v0.33.0 release, Tiered KV offloading
Session-aware routing. The router must send turns of the same conversation to the same place, or at least where their KV lives. NVIDIA Dynamo v1.6 ships KV-aware routing with hard session affinity exactly for this. Without session stickiness, everything else loses value. - Source Dynamo v1.6.0
What it looks like on a single node
The good news is you don't need a rack to start. On my 8xH100 pod, the highest-yield change was configuration, not hardware.
# router with session affinity (Dynamo)
dynamo serve graphs.agentic:Frontend \
--SessionAffinity hard
# vLLM with host-memory offloading (LMCache)
export LMCACHE_REMOTE_URL="localhost:8080"
vllm serve $MODEL \
--kv-transfer-config '{"kv_connector":"LMCacheConnectorV1"}'
With that, the conversation the LRU would evict from VRAM sits in host memory and returns to VRAM on the next turn. Turn 12's TTFT went from 14 seconds to under 2 in my tests with 60K-token context, and the prefill savings show up directly in the pod's cost per million tokens. Your numbers will depend on your conversation lengths, but the direction matches what the vLLM team reports.
The part nobody tells you, the KV write
Warm cache between turns only exists if the previous turn finished writing its KV before the next one arrives. Agents chaining tool calls fire several requests in a row, and if the router doesn't wait for the cache write, every request in the burst pays full prefill.
Two settings fixed that for me. Capping --max-num-active-seqs so admission doesn't force the LRU to kill active cache, and letting the scheduling queue prioritize requests that already hold KV, behavior v0.31 added when reworking the waiting queue. - Source v0.31.0 release
Tips for your agents
- Measure prefix cache hit rate per session, not globally. A global 80% hit rate hides agentic sessions at 0%. The vLLM log gives you the breakdown.
- Hard session affinity before any tiering. It's free compared to LMCache and solves 60% of the problem on small clusters.
- Budget host memory for KV. My rule is 2x the KV of your active sessions. A 70B model with 8k context per session and 20 concurrent sessions fits comfortably in 64 GB of host-memory tier.
- Prune the history before trusting the cache. The cache avoids re-prefilling what you send. Not sending what isn't needed saves more, and your agent won't notice.
- Check the safety mode if you share cache across users. The October SGLang cross-request leak applies to any engine with prefix caching in multi-tenancy. I cover it fully in another post. - Source SGLang security disclosure
Turn 12 of your agent should cost the same as turn 2. With affinity, tiering and the new connectors, it can.
What does the re-prefill of your agentic conversations cost you today? Tell me in the comments.