NYU 与 Amazon 论文提出 SGUID 筛选技能库方法

AI 导读

New NYU and Amazon paper finds that distilling a few skills that keep producing a useful training signal matches or beats distilling a skill bank up to 11× larger. Skills are short written tips, like a rule for counting cases, that a model absorbs by learning from a copy of itself that reads them. Picked by topic match, under 25% of them gave any useful signal across 3 Qwen models. SGUID keeps only skills that help early in training and still help late. With 6 such skills, 3 of 4 models matched or beat the full bank of 30 to 71 skills on math contest tests. A 2nd round with 3 new skills lifted Qwen3-8B from 64.3% to 66.3%.

正文 · AI 翻译

一篇挑选技能文件的好论文。

在蒸馏技能库之前，先记录哪些技能能提供稳定的训练信号，其余的丢弃。

纽约大学和亚马逊的一篇新论文发现，蒸馏少数几个能持续产生有用训练信号的技能，效果可以匹敌甚至超过蒸馏一个规模大至 11 倍的技能库。

技能是简短的书面提示，比如一条统计用例的规则，模型通过学习一个阅读这些提示的自身副本将其吸收。按主题匹配挑选后，在 3 个 Qwen 模型中，只有不到 25% 的技能提供了任何有用的信号。

SGUID 只保留那些在训练早期有帮助、且在训练后期仍然有帮助的技能。凭借 6 个这样的技能，4 个模型中有 3 个在数学竞赛测试上匹敌或超过了包含 30 到 71 个技能的完整技能库。第二轮加入 3 个新技能后，Qwen3-8B 的成绩从 64.3% 提升到了 66.3%。