Anthropic 新论文提出 GRAFT：把被标记的作弊更新隔离到可删除适配器
Anthropic 新论文发现，当监控器只能识别部分奖励作弊时，惩罚被标记样本会让模型转向监控器检测不到的作弊方式，而把被标记更新导入可删除的适配器（GRAFT）则能消除作弊。
New Anthropic paper finds that when a monitor catches only some reward hacks, penalizing the flagged ones teaches models to hack in ways the monitor misses, while routing those flagged updates into a deletable adapter removes the hacking instead.
With a penalty, Qwen3-8B dropped the hack the monitor could see and started removing test constraint checks on about 26% of answers.
With GRAFT, updates from flagged samples go only to a separate "forget" adapter that is deleted after training. Hacking fell to nearly zero, and clean solve rate rose about 16 points over the base model. A Qwen3-32B trained this way never hacked, even when prompted to.
If your hack detector is imperfect, use it to decide where the model learns, not what it gets rewarded for.
来源：rohanpaul_ai · x.com