HiCache 与 HiSparse：稀疏注意力不解决显存容量瓶颈

AI 导读

HiCache 通过将 KV cache 从 HBM 卸载到容量更高、带宽更低的下层内存，仅在计算时加载最相关的 token，以缓解显存容量瓶颈。稀疏注意力只能缓解显存带宽问题，无法解决容量瓶颈——计算时仍需保留上下文中的全部 token，最终仍要载入 HBM。HiSparse 结合缓存规则，将最相关的部分加载进 HBM，从而发挥作用。

正文 · 原文

Sparse attention fixes memory bandwidth, not memory capacity.

“The whole idea of HiCache is trying to offload KV cache from the HBM to a lower tier, higher capacity but lower bandwidth memory, and only load the most relevant set of tokens during computation.”

“Sparse attention doesn’t mitigate the memory capacity bottleneck. It’s a technique that helps a lot with memory bandwidth.”

“During the sparse attention computation, you still need to select the tokens. So you still have to keep all the tokens in context, and you cannot discard them. At some point, it still has to be loaded into HBM.”

“That’s where HiSparse becomes useful, because it can apply the typical caching rules so that you could load the most relevant parts into HBM.”