rohanpaul_ai· @rohanpaul_ai · X·· 3 小时前AI 评分
Inferact 分享 vLLM 在 Agent 生产负载的部署经验
Inferact 的 Yifan Qiao 在 DigitalOcean Open Intelligence Summit 的 Serve 分轨演讲中，讲解如何在 Agent 生产负载上运行 vLLM。
Agent traffic is a strange workload for an inference server.
Each turn resends a long, nearly identical prefix (system prompt, tool schemas, history) and asks for a short tool call back.
At DigitalOcean’s Open Intelligence Summit, Yifan Qiao of Inferact is giving a Serve track session on running vLLM on agentic production workloads.
That shape of traffic is why prefix caching and KV-cache reuse matter more for agents than raw tokens per second. A cache miss on a long prefix means paying for the full prefill again, on every turn of a long task.
来源：rohanpaul_ai · x.com