KVBoost: Chunk-Level Key-Value Cache Reuse with Deviation-Guided Recomputation for Efficient Large Language Model Inference

Computer Science > Artificial Intelligence
[Submitted on 21 May 2026]
Title:KVBoost: Chunk-Level Key-Value Cache Reuse with Deviation-Guided Recomputation for Efficient Large Language Model Inference
View PDF HTML (experimental)Abstract:Transformer-based large language models (LLMs) incur high prefill latency because key-value (KV) tensors must be recomputed for each request. Existing prefix-caching systems reduce this cost but require prompts to share a leading contiguous prefix, limiting effectiveness when shared content appears at arbitrary positions. We present KVBoost, a chunk-level KV cache reuse system for HuggingFace-compatible decoder models that enables reuse regardless of content position. KVBoost introduces a dual-hash keying scheme that separates positional identity (prefix hash) from content identity (content hash), supporting both exact and approximate cache matches. To address attention boundary errors from independently cached chunks, KVBoost employs two repair strategies: SelectiveRecompute, which re-encodes boundary regions, and CacheBlendRecompute, which identifies and recomputes high-deviation tokens after a probe pass. The system further incorporates asymmetric KV quantization (int8/int4), adaptive chunk boundary splitting, and importance-weighted eviction under a fixed memory budget. Evaluated on Qwen/Qwen2.5-3B over 1,000 bug-localization samples, KVBoost achieves a 4.49x reduction in time-to-first-token (142.4 ms vs.\ 639.1 ms) and outperforms prefix caching by 16%, with no loss in accuracy (99.2% vs.\ 99.1%). KVBoost provides a practical, memory-bounded inference acceleration layer compatible with RoPE-based models without architectural modification.
Submission history
From: Srihari Unnikrishnan [view email][v1] Thu, 21 May 2026 02:16:13 UTC (585 KB)
References & Citations
Loading...
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Connected Papers (What is Connected Papers?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
alphaXiv (What is alphaXiv?)
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Hugging Face (What is Huggingface?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

* KVBoost is a chunk-level KV cache reuse system for HuggingFace-compatible decoder models.
* It enables cache reuse irrespective of content position.
* The system uses a dual-hash keying scheme separating positional identity (prefix hash) from content identity (content hash).
* Repair strategies include SelectiveRecompute to re-encode boundary regions and CacheBlendRecompute to recompute high-deviation tokens after a probe pass.
* The system incorporates asymmetric KV quantization (int8/int4), adaptive chunk boundary splitting, and importance-weighted eviction under a fixed memory budget.
* Evaluation was conducted on Qwen/Qwen2.5-3B over 1,000 bug-localization samples.
* KVBoost achieved a 4.49x reduction in time-to-first-token (142.4 ms vs. 639.1 ms).
* It outperformed prefix caching by 16% in inference time.
* Accuracy remained high, with 99.2% accuracy versus 99.1%.

KVBoost is a chunk-level KV cache reuse system designed for HuggingFace-compatible decoder models to reduce prefill latency in large language model inference by enabling cache reuse regardless of where shared content appears. The system employs a dual-hash keying scheme to separate positional identity from content identity, allowing for both exact and approximate cache matches. To mitigate attention boundary errors arising from independently cached chunks, KVBoost implements two repair strategies: SelectiveRecompute, which re-encodes boundary regions, and CacheBlendRecompute, which recomputes high-deviation tokens after a probe pass. The framework also incorporates asymmetric KV quantization (int8/int4), adaptive chunk boundary splitting, and importance-weighted eviction within a fixed memory budget. Evaluation on Qwen/Qwen2.5-3B over 1,000 bug-localization samples demonstrated a 4.49x reduction in time-to-first-token (142.4 ms versus 639.1 ms) and a 16% performance improvement over prefix caching, with no loss in accuracy (99.2% vs. 99.1%).

The work addresses a fundamental bottleneck in LLM inference—the redundant recomputation of Key-Value (KV) tensors for every request—by introducing a cache management layer that respects content locality rather than strict prefix structure. The use of dual hashing to manage positional and content identity suggests an attempt to bridge the gap between sequential processing demands and semantic caching needs, which is a common tension in sequence-based computation. The repair mechanisms, SelectiveRecompute and CacheBlendRecompute, indicate an acknowledgment that simple caching fails at attention boundaries; this moves the problem from simple data storage to dynamic error correction within memory management. The practical integration of quantization and eviction under a memory budget grounds the theoretical efficiency gains in real-world resource constraints, showing an understanding of deployment realities. The implication is that achieving true inference acceleration requires not just storing data, but intelligently managing the integrity and relevance of that stored state across arbitrary content distributions. Future work should explore how these deviation-guided recomputations scale to much larger contexts or different attention mechanisms, testing if this chunk-level abstraction provides universal benefits beyond boundary error correction specific to the tested model family.

KVBoost: Chunk-Level Key-Value Cache Reuse with Deviation-Guided Recomputation for Efficient Large Language Model Inference

Facts Only

Executive Summary

Full Take

KVBoost: Chunk-Level Key-Value Cache Reuse with Deviation-Guided Recomputation for Efficient Large Language Model Inference

Facts Only

Executive Summary

Full Take