DeepInfer: DeepSeek opens the inference engine that runs its services in production

Released under the Apache 2.0 license, DeepInfer initially supports DeepSeek-V4.1-Flash and runs on both Nvidia SM100 GPUs and Huawei's Ascend 950 chips.

DeepSeek has published on GitHub DeepInfer, the high-performance inference framework that serves its own services at scale. This first version, licensed under Apache 2.0, covers offline inference and currently only supports one model, DeepSeek-V4.1-Flash. It runs on Nvidia's SM100-generation GPUs and Huawei Ascend 950 accelerators.

These include the lab's in-house kernels (FlashMLA, DeepGEMM, DeepEP, etc.), expert parallelism with load balancing, accelerated decoding by draft tokens whose verification is scheduled according to a confidence level, an asynchronous scheduler, an inference mode with identical results regardless of the composition of the batch, as well as outputs structured by regex, JSON schema or grammar.

Several building blocks that are essential to a complete service are missing: the KV cache, the separation between prefill and decoding phases, and the load balancing between replicas. DeepSeek indicates that DeepInfer was first built on vLLM, with borrowings from SGLang and TensorRT-LLM.