DeepSeek has fully open-sourced its inference framework, which is compatible with NVIDIA and Huawei Ascend and reports 155T tokens daily single-cluster throughput and $38 per card per hour revenue.
DeepSeek has fully open-sourced its internal inference framework, DeepInfer, along with the Token codec component deepseek-recipe. Both are compatible with NVIDIA and Huawei Ascend platforms. Based on the framework, the daily throughput of a single DeepSeek V4.1 Flash cluster reaches 155 trillion Tokens, with revenue of 38 US dollars per card per hour calculated at off-peak electricity prices. Through techniques such as Prefill-Decode separation, KVCache reuse and speculative decoding, DeepSeek aims to improve inference efficiency and reduce computing power costs and energy consumption.
Following the open-sourcing of large models, the TileLang compilation tool and a series of inference operator libraries, DeepSeek has now announced the full open-sourcing of DeepInfer, and simultaneously released the Token codec component deepseek-recipe. The company aims to help developers and enterprises optimize large-scale model deployment and reduce the computing power costs and energy consumption of AI services.
DeepSeek said that, based on DeepInfer, the team has implemented a high-throughput, low-latency large-scale inference service for DeepSeek V4.1 Flash. The service throughput of a single cluster currently reaches 155 trillion Tokens per day, and revenue per card per hour is 38 US dollars calculated at off-peak electricity prices.
The inference framework is the key infrastructure connecting the capabilities of large models with end users. As call volumes from AI applications grow, the cost of inference services depends not only on the model itself but also on computing, communication, task scheduling and cache management. Improving effective throughput while meeting latency requirements has become an important step in reducing costs and improving efficiency for large-scale AI services.
**Compatible with NVIDIA and Huawei Ascend, with computing power optimized for different inference stages**
DeepInfer integrates high-performance operators and a deeply optimized inference engine. It supports distributed inference, speculative decoding, SWA Bounded Replay, structured output and other capabilities, and can run on NVIDIA and Huawei Ascend platforms. DeepSeek said the inference code for its new models and subsequent performance optimizations will continue to be open-sourced within this framework.
To meet large-scale AI call demand, DeepSeek has focused the optimization of its inference system on throughput and latency. The system separates Prefill and Decode deployment: the former processes input prompts, and the latter generates Tokens step by step. Because the computing loads of the two stages differ, separate deployment allows resources to be configured independently to optimize, respectively, the time to first Token and the interval between subsequent Tokens.
In terms of hardware organization, DeepSeek uses common 8-card machines for Prefill tasks and 64-to-128-card supernodes for Decode tasks. The system also dynamically adjusts the instance ratio between the two stages through automatic scaling, reducing resource mismatches caused by traffic fluctuations. For MoE model inference, it uses expert parallelism and dynamic load balancing to reduce load differences between computing cards.
**KVCache can be retained for up to 7 days to reduce redundant computing overhead**
Cache reuse is another key design element for lowering DeepSeek's inference costs. As long-context dialogue and AI Agent applications grow, the same prefix often appears repeatedly across different requests. Reprocessing this content every time would generate additional computing overhead.
DeepSeek V4.1 Flash divides KVCache into two parts, Block KVCache and SWA KVCache, and applies different storage strategies to each. Block KVCache is stored in the company's independent 3FS distributed file system, with related storage capacity of about 100 PiB. According to DeepSeek, KVCache retention time reaches 3 days for Agent traffic and 7 days for chatbot traffic.
When subsequent requests hit existing cache, the system reuses previous calculation results to reduce redundant Prefill computation. DeepSeek disclosed that the 3FS storage cluster reaches a peak aggregated read bandwidth of 10 TiB/s and a peak aggregated write bandwidth of about 850 GiB/s, meeting large-scale cache read and write requirements.
SWA KVCache, which is mainly reused over the short term within active sessions, is kept in machine memory without persistent storage. Online data shows the retention time for this cache is about 180 seconds, with a hit rate above 98%.
When SWA KVCache misses, DeepSeek uses the SWA Bounded Replay mechanism to recompute only the latest 128 Tokens and limits the attention window to the recomputation interval, reducing the additional computing cost caused by cache misses.
**Speculative decoding improves throughput; codec component also open-sourced**
Beyond cache reuse, DeepSeek improves inference efficiency through speculative decoding and asynchronous scheduling.
DeepSeek uses the DSpark speculative decoding framework, which dynamically determines the verification range based on the expected acceptance probability of candidate Tokens. This avoids spending verification computing power on candidates with low acceptance probability. DeepSeek disclosed that in online deployment, compared with fixed verification of 5 Tokens, variable-length verification increases throughput by 6%.
In scheduling, asynchronous coordination between CPU and GPU allows request scheduling, forward computation and sampling to overlap as much as possible, reducing device idle time. DeepInfer also supports structured output, which can constrain model generation according to rules such as JSON Schema and regular expressions to meet the output format requirements of specific applications.
DeepSeek also open-sourced the production-grade Token codec component deepseek-recipe. It supports mainstream interfaces including the Chat Completions API, Responses API and Messages API. It improves processing efficiency for long texts through mechanisms such as special Token boundary recognition, parallel encoding of ordinary text segments and streaming output parsing, ensuring that API processing logic in third-party deployments is consistent with the official service.
DeepSeek has not, however, open-sourced all components of its inference system. Because KVCache management and load balancing are deeply coupled with its internal infrastructure, the related code has not been fully released at this time.
From models and operator libraries to large-scale inference frameworks, this release further expands the coverage of DeepSeek's open-source technology stack. For developers and enterprises, it is expected to lower the technical threshold for building private and public cloud AI services. For the large model industry, improving inference throughput, reusing computed results and adapting to different hardware platforms are becoming important directions for controlling the cost of AI services.