ServiceNow 与 McGill 论文：AI 评审在长程计算机操作任务中仍漏判细微错误
ServiceNow 与 McGill 的新论文指出，用 LLM 评审 agent 运行时，长任务中的细微错误仍易被漏判，为其添加工具也未必有帮助。研究构建了 AgentHorizon，包含 1,373 项评审任务，数据来自人工屏幕录制的近似任务对并互换指令，使每次失败都对应一个略有差异请求的干净成功运行。
New ServiceNow and McGill paper finds that AI judges still miss subtle mistakes in long computer-use tasks, and giving them tools does not reliably help.
Teams use LLMs to grade agent runs for training and evaluation. A run of 100+ steps can look finished while it drew the wrong shape or posted to the wrong Slack channel.
They built AgentHorizon, 1,373 judging tasks from human screen recordings. People recorded pairs of near-identical tasks and swapped the instructions, so each failure is a clean run of a slightly different request.
The best judge, GPT-5.5 in Codex, scored 80.9% balanced accuracy on the hard split, where always saying yes gets 50%. Qwen 3.6 27B did worse with tools, scoring 77.6% reading the whole run directly but 70.4% as an agent.
Before you trust an LLM judge with agent runs, test it on near-miss failures inside the exact agent setup you will use.
来源：rohanpaul_ai · x.com