omarsar0· @omarsar0 · X·· 3 小时前AI 评分
DigitalOcean Signals 可将生产 trace 一键转为评测并回放验证
作者提出 agent 的生产 trace 是最好的评测数据，因为最关键的失败只在生产环境出现且很少进入测试集。DigitalOcean 的 Signals 会标记用户受挫或工具调用失败的会话，一键将其转为评测用例，再通过 Simulations 在 agent 的真实 harness 上回放验证修复效果，之后才发布。
Your agent traces are your best eval data.
The most important failures only show up in production, and they rarely make it into your test set.
DigitalOcean's Signals flags sessions where users got frustrated or tools failed and turns them into evals with one click. Simulations then replay them against your fix on your agent's real harness before it ships.
If you're running agents in production, you need this loop.
来源：omarsar0 · x.com