KV-Repurpose: Segment-wise KV-cache reuse beats prefix caching by 20× at 30 GB
An open-source vLLM extension published on arXiv on 12 October 2026 relaxes exact-prefix KV-cache matching to per-segment reuse. Under a Zipfian RAG workload at 30 GB, it records a 63.9% token-level hit rate where prefix caching manages 3.3%.
What happened
KV-Repurpose, an open-source system built on vLLM for segment-wise KV-cache reuse in LLM serving, was published on arXiv on 12 October 2026. Under a Zipfian RAG-style workload at 30 GB cache capacity, it records a 63.9% token-level cache hit rate where standard prefix caching manages 3.3%.
Context
KV-cache reuse in LLM serving carries a hard constraint: the new prompt must be an exact extension of a previously seen prefix. In RAG pipelines and multi-turn conversations, the same retrieved document routinely appears after different system prompts, different conversation histories, or different dynamic metadata. Each positional variation triggers a full recomputation of the attention tensors for that document. vLLM already ships PagedAttention-based prefix caching, so the cache-management infrastructure exists. What is missing is the matching logic that recognises a segment outside its original positional context.
How it works
KV-Repurpose relaxes the exact-prefix requirement to segment-wise matching. An individual block of the prompt, such as a retrieved passage, can be identified and its KV tensors reused regardless of the surrounding context. The system sits on top of vLLM and adds five components to the existing PagedAttention layer: cache-aware segment scheduling, prompt construction, capacity management, workload generation, and runtime monitoring.
The evaluation uses a Zipfian-distributed RAG workload in which 20% of requests share one fixed prefix and the remaining 80% vary. At 30 GB of cache capacity, KV-Repurpose records a 63.9% token-level hit rate; prefix caching under the same workload records 3.3%. Scaling cache to 1 TB lifts prefix caching to 17.9%, but the gap persists. The higher hit rate translates to lower time-to-first-token.
Quality results are mixed. Naive segment reuse largely preserves output quality on single-document question answering and summarization. It exposes clear limitations in cross-document reasoning.
Our read
The 20× hit-rate gap will land in a slide deck. The 1 TB data point is more informative. Giving prefix caching 33× the cache headroom and still capping it at 17.9% suggests the bottleneck is structural, not capacity. You cannot solve a matching-logic problem by buying more VRAM. The economics of serving shift from "how much cache can you fit" to "how smart is your cache indexer," and those are different engineering problems.
The cross-document reasoning caveat deserves more weight than the abstract gives it. If a studio's RAG loop is single-document QA over a shot-list or a script reference, segment reuse is close to a free lunch. If the loop requires synthesising across three retrieved passages before generating, the quality degradation is not a footnote; it is the reason the pipeline exists. The brief does not specify which base model or which long-context benchmarks produced the quality results, so the claim is bounded to whatever configuration the authors tested.
This is a research and evaluation artifact, not a drop-in production replacement. The components read as a test harness with a scheduling layer, not a serving library you wire into a latency-critical path without further hardening. But the Apache-2.0 code answers one concrete question immediately: on your workload and your GPU, does segment reuse buy the TTFT reduction the paper reports?
What this changes
If the studio already runs vLLM as the inference backend for a local LLM—script generation, prompt expansion, RAG over a shot-list knowledge base—the Apache-2.0 code is immediately usable for benchmarking whether segment-wise reuse cuts TTFT on that specific workload without the cross-document quality loss the paper flags. The 30 GB data point sits in the range of a single 24–32 GB GPU, which makes it directly relevant to hardware planning.
If the pipeline is dominated by single-prompt image or video generation in ComfyUI with no LLM RAG loop, this changes nothing on Monday. The benefit is gated on having a vLLM-served LLM with repeated document content in the prompt.
License
The source code is released under the Apache-2.0 license. Commercial use is permitted; no revenue ceiling or attribution-only restriction is stated in the source.
Key takeaways
- Segment-wise KV-cache reuse outperforms exact-prefix caching by roughly 20× in token-level hit rate under a Zipfian RAG workload at 30 GB cache (63.9% vs 3.3%).
- Scaling prefix-cache capacity from 30 GB to 1 TB lifts its hit rate only to 17.9%, indicating the bottleneck is matching logic, not storage.
- Naive segment reuse preserves quality on single-document QA and summarization but degrades on cross-document reasoning.
- KV-Repurpose is a vLLM-based research and evaluation tool, not a production serving library.
- The Apache-2.0 license makes the code immediately usable for benchmarking segment reuse on a studio's own workload and hardware.
Sources
How this post was made
Drafted from clustered primary sources by the models below, then read, edited and approved by a human before it was published. The sources are listed in full at the end of the article.
- Drafted
- Independent sources
- 1
- cluster pair
- clef:27b
- cluster label
- gemma4:12b
- radar brief
- gemma4:12b
- research brief
- qwen3.8:27b
- draft article
- qwen3.8:27b
- short script
- qwen3.8:27b
- seo pack
- gemma4:12b
- Run
- editorial-20261012T135816Z