Nvidia 论文：只在让测试首次通过的编辑步给基座模型打分，与后训练 SWE-bench Verified 分数相关性达 0.964
Nvidia 新论文指出，在 agent 运行的每一步都给基座模型打分，效果不如只在首次让测试通过的那次编辑上打分。研究者重放强 agent 的成功运行，每次编辑后重跑测试以定位关键步，再检验基座模型是否偏好该编辑、能否写出通过测试的编辑；10 个模型写出该编辑的概率排序与其后训练 SWE-bench Verified 分数相关性达 0.964。
New Nvidia paper shows scoring base models on every step of an agent run works worse than scoring them only on the edit that first makes the tests pass.
Post-training is expensive, so picking the right base model matters. Yet agent benchmarks score base models near zero, and simple coding tests barely predict later agent skill.
They replay a strong agent's successful run and rerun the tests after each edit to find the 1st edit that makes them pass. At this decisive step they test whether a base model favors that edit, picks it over failing edits, and can write a passing one.
How likely each base model was to write that exact edit ranked the same 10 models in line with their post-trained SWE-bench Verified scores, at 0.964.
When choosing a base model to post-train for agent work, run this check first. It needs only a benchmark's successful agent runs and its tests.
来源：rohanpaul_ai · x.com