What was actually released

DeepSeek describes DeepInfer as the inference framework powering its large-scale production services. That is the company's characterisation of the engine's role, not evidence that the public repository is identical to its deployed software. The release contains substantial inference functionality, but its own README identifies additional infrastructure required for a complete service.

The useful way to read this release is as three layers, only two of which are in the box.

The bottom layer is kernels: the low-level routines that do the arithmetic on a specific chip. DeepSeek has been publishing these for over a year (FlashMLA, DeepGEMM, DeepEP), and DeepInfer bundles them with TileKernels, DeepSelect and custom TileLang code. The middle layer is parallelism: how one model's work is split across many chips, including expert parallelism with load balancing for the mixture-of-experts layers, plus the decoding tricks (DSpark drafts up to five tokens ahead by default and checks them against confidence estimates) and the reproducibility features.

That middle layer is the part labs usually keep closed. Model weights get published; serving stacks mostly do not, because serving is where the cost advantage lives. DeepSeek has published its inference kernels and parallelism implementation under Apache 2.0, with documented support for NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. Supporting both hardware families is notable, although the README identifies some NVIDIA-only capabilities, and the sources reviewed do not establish feature or performance parity between them.

The layer that stayed home

Then there is the top layer, and the README is unusually plain about it. Under the heading "Required for a complete inference service (not included in this release)" it lists a KV cache system, prefill/decode disaggregation, and load balancing across data-parallel replicas.

These are service-level components, not cosmetic extras. A production KV-cache system manages cached attention state across requests and serving resources; its absence does not necessarily mean the engine lacks all KV caching. Prefill/decode disaggregation allows deployments to separate the prompt-processing and token-generation phases onto differently configured resources. Replica-level load balancing distributes requests across serving instances. Together, these capabilities matter for operating a reliable, efficient service at scale, beyond running offline inference on a configured cluster. Note also the distinction inside the word "balancing": the MoE expert load balancing (EPLB) that spreads experts across devices is included; distributing incoming requests across data-parallel replicas is not. They solve different problems.

Crypto Briefing, citing DeepSeek's own materials, reported production-service figures including 155 trillion tokens per day per cluster, a 96% input-cache hit rate, roughly 400 milliseconds to first token and generation speeds of about 230 tokens per second, rising to 300 for coding. Those are DeepSeek-reported numbers for its own deployment, carried through a secondary source. They have not been independently reproduced, and the report does not establish that a fresh installation of the public DeepInfer repository can achieve them. In particular, the published engine omits service infrastructure that could materially affect throughput, caching and latency. The figures are evidence of what DeepSeek claims about its service, not independent measurements of it, and the gap between the claim and a fresh download is exactly the engineering the missing list describes.

There are smaller limits worth noting before anyone plans around this. One model is supported, and support for DeepSeek-V4.1-Flash does not establish compatibility with other DeepSeek models or arbitrary Hugging Face checkpoints. Two chip families are supported, both current data-centre parts, and the two reproducibility features work only on NVIDIA. DeepSeek's published benchmark configurations used 4 SM100 GPUs for prefill and 64 for decode; those are reported configurations, not a proven minimum for every supported workflow. None of that diminishes the release; all of it defines who it is for.

What to do with it

If you run inference at scale, the practical move is to read DeepInfer as a reference implementation with reusable parts, not as a product. The kernels and the parallelism design can be studied, benchmarked and borrowed today; the repository is distributed under Apache 2.0, which generally permits commercial use and redistribution subject to its terms. The serving layer you still build yourself, or take from vLLM or SGLang, the two projects DeepInfer itself grew out of and drew on.

DeepSeek also pointed readers to deepseek-recipe, its encoding layer that converts Messages, Chat Completions and Responses API requests into DeepSeek prompts and parses model output back, which covers the request-formatting piece between an API and an engine. Between the weights (published 17–18 October), the encoding layer and now the engine, several important layers of the stack around a DeepSeek model are public. The one layer that is not is the one that turns an engine into DeepSeek's prices. Publishing the model weights, request-formatting code and a substantial inference engine gives developers more of the surrounding implementation to inspect and reuse. It is a meaningful contribution to the open inference ecosystem, even though reproducing DeepSeek's complete service remains a separate engineering task. It is worth naming precisely, so the headline and the download are not confused for each other.

Sources

  1. [1] DeepSeek, DeepInfer repository and README — description, supported model and hardware, feature list, prerequisites, benchmark configurations, the missing-components list, licence and vLLM acknowledgement; accessed 20 October 2026Read source
  2. [2] DeepSeek (@deepseek_ai), announcement post, X, 20 October 2026 — "Introducing DeepInfer, our in-house inference framework, now fully open source! Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency."Read source
  3. [3] Crypto Briefing, "DeepSeek open-sources DeepInfer, the inference engine behind its production models", 20 October 2026 — DeepSeek's production figures (155T tokens/day/cluster, 96% input cache hit rate, ~400ms TTFT, 230 tokens/second, up to 300 for coding), reported from DeepSeek's own materialsRead source
  4. [4] MadRobot, "DeepSeek open-sources DeepInfer, the engine behind its cheap AI", 20 October 2026 — its account of DeepSeek's April 2025 GitHub note declining to release the full inference engine, and the reasons givenRead source
  5. [5] DeepSeek, deepseek-recipe repository — the encoding layer converting Messages, Chat Completions and Responses API requests into DeepSeek prompts and parsing model output backRead source