Bridging LLM Serving and CXL-SSDs with Chunk

Computer Science > Hardware Architecture
[Submitted on 20 Sep 2026]
Title:Bridging LLM Serving and CXL-SSDs with Chunk-Aware KV Cache Management
View PDF HTML (experimental)Abstract:NAND-backed storage offers the capacity needed to scale LLM prefix caching, but its block I/O path incurs CPU cache contention and host-DRAM staging in addition to NAND latency. Our characterization shows that these interface costs persist even with DRAM as the storage medium, motivating CXL-SSDs for byte-addressable access to NAND-backed capacity. Surprisingly, however, a stock CXL-SSD remains about 3$\times$ slower than local DRAM and no faster than an NVMe SSD, while generic prefetching provides little benefit. We present LM-CXD, a CXL-SSD specialized for LLM prefix caching. LM-CXD bridges the semantic gap between the serving engine, which knows which KV chunks will be consumed, and the device, which controls their placement and movement. It makes KV chunks device-visible I/O units, exposes NAND-to-DRAM progress to the serving engine, and uses device DRAM as a GPU-accessible buffer. LM-CXD further coordinates request scheduling with windowed prefetching and pipelines layerwise KV movement with GPU computation to hide NAND latency under limited device DRAM. Across five LLM models, LM-CXD reduces average TTFT over a stock CXL-SSD by up to 2.6$\times$ with compute asynchronous prefetching and 4.03$\times$ with layerwise prefetching, achieving TTFT within 1.5$\times$ of local DRAM on average.
Current browse context:
cs.AR
References & Citations
Loading...
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Connected Papers (What is Connected Papers?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
alphaXiv (What is alphaXiv?)
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Hugging Face (What is Huggingface?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Facts Only

* NAND-backed storage is used for scaling LLM prefix caching.
* The block I/O path incurs CPU cache contention and host-DRAM staging in addition to NAND latency.
* A stock CXL-SSD is about $3\times$ slower than local DRAM and no faster than an NVMe SSD.
* Generic prefetching provides little benefit.
* LM-CXD is a CXL-SSD specialized for LLM prefix caching.
* LM-CXD makes KV chunks device-visible I/O units.
* LM-CXD exposes NAND-to-DRAM progress to the serving engine.
* LM-CXD uses device DRAM as a GPU-accessible buffer.
* LM-CXD coordinates request scheduling with windowed prefetching and pipelines layerwise KV movement with GPU computation.
* LM-CXD reduces average TTFT over a stock CXL-SSD by up to $2.6\times$ with compute asynchronous prefetching.
* LM-CXD reduces average TTFT by $4.03\times$ with layerwise prefetching.
* Achieved TTFT is within $1.5\times$ of local DRAM on average across five LLM models.

Executive Summary

NAND-backed storage for LLM prefix caching introduces overhead from CPU cache contention and host-DRAM staging, alongside NAND latency. Despite this, a standard CXL-SSD is about three times slower than local DRAM and no faster than an NVMe SSD, and generic prefetching offers minimal benefit. LM-CXD is proposed as a CXL-SSD specialized for LLM prefix caching to bridge the gap between serving engines and devices regarding KV chunk placement. This specialization makes KV chunks device-visible I/O units and exposes NAND-to-DRAM progress. The system utilizes device DRAM as a GPU-accessible buffer, coordinates scheduling with windowed prefetching, and pipelines layerwise KV movement with GPU computation to mitigate NAND latency. Results across five LLM models show LM-CXD reduces average Time To First Token (TTFT) over a stock CXL-SSD by up to $2.6\times$ with compute asynchronous prefetching and $4.03\times$ with layerwise prefetching, achieving TTFT within $1.5\times$ of local DRAM on average.

Full Take

The work addresses a fundamental architectural bottleneck: the impedance mismatch between high-capacity, slow storage (NAND) and fast compute requirements in LLM serving. The core tension lies in moving from traditional block I/O paradigms to byte-addressable access via CXL-SSDs, which introduces interface costs that are not entirely mitigated by using DRAM as an intermediary layer. The surprise finding that a standard CXL-SSD does not offer asymptotic speedup over existing NVMe technology suggests that the bottleneck is not solely storage transfer but the host-device coherence and management overhead associated with memory hierarchy contention during I/O. LM-CXD’s success appears rooted in explicitly modeling the execution flow, transforming latent latency into manageable, prefetchable work using compute-aware scheduling (layerwise pipelining) rather than just raw bandwidth. This suggests that future LLM infrastructure design requires treating cache management not as a passive staging problem but as an active computational coordination problem where memory visibility and asynchronous GPU computation are tightly coupled to device I/O operations to hide inherent latency costs effectively. The next step is understanding if this layerwise hiding strategy scales across different workload types and storage fabrics beyond the specific context of LLM prefix caching.

Sentinel — Human

Confidence

The text exhibits the structure and density characteristic of academic or highly specialized engineering research, suggesting it originates from domain experts rather than general synthetic generation.

Signals Detected

Moderate sentence length variance; technical but flows logically.

High internal consistency; the technical claims map directly to the proposed solution (LM-CXD).

Clear structure typical of a research abstract, referencing specific performance metrics and mechanisms.

The density of technical terms (CXL-SSD, KV Cache, TTFT) suggests genuine domain expertise, although the presentation is highly polished.

Human Indicators

Use of specific, bleeding-edge technical terminology and the proposal of a novel system (LM-CXD) indicate primary source engagement with the field.