Sakana AI 提出 Continuous Memory Machine 双记忆机制
Sakana AI 论文提出 Continuous Memory Machine，为循环模型配备短期与长期两个记忆矩阵，由 Transformer 每步读写，解决单一隐藏向量中短期计算与长期存储争抢空间的问题。
Interesting paper from Sakana AI on memory for recurrent models.
Recurrent models are good at state tracking, but they usually keep everything in one hidden vector, so short-term computation and long-term storage compete for the same space.
The Continuous Memory Machine gives the model two memory matrices. One short-term memory tracks recent neuron activity, and the other long-term memory stores information for later steps. A Transformer reads and writes both at every step.
It builds on Sakana's Continuous Thought Machine and beats LSTM, DNC, RMC, and CTM baselines on copy, associative recall, sorting, few-shot regression, and maze solving. It also generalizes to longer inputs than earlier memory-augmented networks.
The attention maps show the model uses long-term memory for algorithmic and in-context tasks and skips it when the task does not need it.
Paper: academy.dair.ai/papers/conti…
来源：dair_ai · x.com