Hybrid Latent Attention 让循环 LLM 吞吐提升至 8.8 倍

AI 导读

Virginia Tech 论文提出 Hybrid Latent Attention，将旧 token 压缩为紧凑向量供 attention 直接读取，使循环 LLM 每 GPU 吞吐提升 4.0 至 8.8 倍且精度损失极小。

正文 · 原文

New Virginia Tech paper shows how cache older tokens as compact vectors that attention reads directly, and a looped LLM fits 4.0 to 8.8 times as many requests per GPU at little accuracy cost.

Looped models run each token through the same layers several times, and every pass adds to the KV cache, so fewer requests fit on a GPU.

Hybrid Latent Attention keeps the last 128 tokens exact and squeezes older ones into small vectors that every loop reads without rebuilding anything.

On Ouro models, throughput rose up to 7.4 times at 16K tokens, and accuracy stayed above 97% of the original on math, knowledge and reasoning.

Gains are largest on long contexts. You can retrofit an existing looped checkpoint by training only the new parts.

– arxiv. org/abs/2610.07940

Title: "Hybrid Latent Attention for Looped Language Models"