Computer Science > Artificial Intelligence
[Submitted on 17 Sep 2026]
Title:RBS-Attention: Radius-Bounded Sparse Prefill for Long-Context Large Language Models
View PDF HTML (experimental)Abstract:Long-context large language model inference is increasingly limited by prefill, where dense self-attention processes the entire prompt before generation begins. Sparse block selection can reduce this cost, but a block centroid may hide a highly relevant token among many irrelevant ones. We call this failure mode mean dilution and propose RBS-Attention, a training-free sparse-prefill method with two complementary selection branches. A centroid base branch captures average relevance, while a rescue branch uses the maximum key-block radius and its prompt-, layer-, and head-dependent distribution to identify blocks at risk of underestimation. Independently thresholding the two branches and combining their masks controls the contribution of rescue blocks while preserving regular block-sparse FlashAttention execution. On H100 GPUs, RBS-Attention achieves 20.65$\times$ standalone prefill-attention speedup, 11.92$\times$ vLLM prefill-attention speedup, and 5.97$\times$ end-to-end time-to-first-token speedup at 128K on Qwen3-30B-A3B-Instruct-2507-FP8. On the dense Qwen3-32B model, it obtains 88.65 overall RULER accuracy versus 89.52 for dense attention; LongBench-v2, InfiniteBench, and Video-MME provide additional quality evaluation. Supporting experiments measure actual retention, compare selectors at matched density, and characterize block-size, threshold, and memory behavior. Together, these results support radius-adaptive dual-branch selection as an effective approach to long-context prefill.
References & Citations
Loading...
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Connected Papers (What is Connected Papers?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
alphaXiv (What is alphaXiv?)
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Hugging Face (What is Huggingface?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Facts Only
* The method is named RBS-Attention: Radius-Bounded Sparse Prefill for Long-Context Large Language Models.
* Sparse block selection reduces the cost of prefill by avoiding dense self-attention over the entire prompt.
* RBS-Attention uses two complementary selection branches: a centroid base branch and a rescue branch.
* The centroid branch captures average relevance, while the rescue branch identifies blocks at risk of underestimation using maximum key-block radius distribution information.
* Masks from the two branches are independently thresholded and combined to control block contribution during execution.
* On H100 GPUs, RBS-Attention yielded a $20.65\times$ standalone prefill-attention speedup.
* It achieved an $11.92\times$ vLLM prefill-attention speedup and a $5.97\times$ end-to-end time-to-first-token speedup at 128K context length on Qwen3-30B-A3B-Instruct-2507-FP8.
* On the dense Qwen3-32B model, it achieved $88.65$ overall RULER accuracy versus $89.52$ for dense attention.
* Quality evaluation included LongBench-v2, InfiniteBench, and Video-MME.
* Experiments measured actual retention, compared selectors at matched density, and characterized block-size, threshold, and memory behavior.
Executive Summary
Full Take
The innovation in RBS-Attention lies in addressing the potential failure mode of simple sparse methods—the dilution effect where centroid selection might overlook highly relevant tokens. The dual-branch mechanism directly confronts this by combining an estimation of average relevance (centroid branch) with a targeted intervention for risk mitigation (rescue branch). This layered approach moves beyond static sparsity to introduce adaptive, context-aware gating based on geometric properties (radius) and distribution characteristics across prompt, layer, and head dimensions.
The success metrics suggest that the cost reduction translates directly into tangible performance gains, not just latency improvements, evidenced by speedups in attention mechanisms and end-to-end inference time. The fact that the method improves accuracy ($88.65$ vs $89.52$) on a dense model suggests that the adaptive masking mechanism is not merely an optimization trick but a genuine way to manage context information effectively during sparse execution. The implication is that in long-context scenarios, overlooking specific blocks leads to systemic degradation, and true efficiency requires recognizing which sparse selections are genuinely safe versus those requiring specific attention. Future work must focus on characterizing the learned distributions used by the rescue branch more deeply to further refine this adaptive risk assessment across different model architectures.
Sentinel — Human
The text exhibits the characteristic density and precision of peer-reviewed or experimental research, strongly suggesting human authorship rather than generic synthetic generation.