DeepSeek 开源生产推理框架 DeepInfer

DeepSeek just open-sourced DeepInfer, the inference framework behind their large-scale production services.

There is A LOT in there: kernels, massive expert parallelism, KV cache tricks…

But the scheduling is probably the part interesting that got me really interested 👀

DeepInfer's scheduler increasingly decides *where compute is actually worth spending* rather than treating every request/token the same.

For speculative decoding, a confidence head predicts which draft tokens are likely to be accepted, and the scheduler uses that + measured hardware throughput to decide how many tokens to verify for each request.

That means the duration of a batch becomes more predictable: no longer will one replica take longer than the others to finish and slow the whole group down. More regular batches -> more throughput.

They apply the same philosophy elsewhere:

- prefill chunks are dynamically sized from a compute budget + hardware cost model

- decode routing balances both request count and KV cache length

- CPU scheduling is overlapped with GPU and tightly integrated with speculative decoding

- cached prefixes are decoupled from compute placement, so routing can optimize for available compute instead of cache locality

This is exactly the kind of inference work I'd love to see become reusable beyond one model or one serving stack:

Transformers already has CB, paged KV caches and async batching; @remi_or_ and the team are currently working on branches to see which of DeepInfer's ideas we can generalize so that they can benefit many more architectures. Preliminary rsults in the image below!

Thanks DeepSeek for open-sourcing once again ❤️