Anthropic 为 Claude 引入对话自我评估安全机制

AI 导读

Anthropic 训练 Claude 在回复后对当前对话进行初步安全评估，以标记可能被滥用的交互。该机制自 6 月启用以来发现了其他安全防护未针对检测的滥用行为，并利用 Claude 已读取对话上下文的特点提升效率。

正文 · 原文

We’ve trained Claude to flag potential misuse in conversations, a technique we call self-assessment. Right after it responds, Claude gives a preliminary safety assessment of the conversation.

Since we began using self-assessment in June, it has surfaced misuse that our other safeguards weren’t designed to catch.

Because Claude has already read the conversation, self-assessment is very efficient. And unlike classifiers, which target a discrete set of harms, it draws on Claude’s own judgment, so it gets more accurate as our models improve.