HBF for High-Throughput LLM Serving (UC Berkeley, FuriosaAI)

Large language model serving faces bottlenecks related to memory capacity and bandwidth due to the storage of model weights and KV caches, which become increasingly constrained as models grow and contexts lengthen. High-bandwidth flash (HBF) is presented as a method to expand accelerator memory capacity for LLM serving, although it introduces challenges related to access costs and limited write endurance. The research evaluates HBF across system design and scheduling choices specifically for high-throughput agentic workloads requiring repeated interactions over growing contexts.

The research centers on resolving the trade-off between performance gains and physical constraints inherent in emerging memory technologies like High Bandwidth Flash (HBF) within large-scale AI serving infrastructure. The finding that scheduling and data placement can yield substantial improvements—reducing completion time by over 36% and achieving significant energy savings—suggests that the limitation is less about raw capacity of HBF and more about inefficient utilization and management of its access characteristics. The stark contrast between the performance benefits and the write endurance concerns highlights a critical tension in hardware-software co-design: maximizing throughput while preserving physical longevity. The extension of estimated HBF write lifetime by nearly threefold through scheduling indicates that software orchestration is a powerful lever for extending hardware utility, moving the problem from purely physical limits to architectural optimization. This suggests that future advancements in LLM serving efficiency will depend less on increasing raw memory bandwidth and more on sophisticated, context-aware management layers that can dynamically account for operational demands when interacting with novel storage media.