Over the past few days, I saw a piece of news across WeChat accounts and tech media: DeepSeek open-sourced its own inference engine, DeepInfer, with headlines mainly emphasizing how fast it is. What is an inference engine? It refers to the underlying software that makes a large model run, including accepting our prompts and producing the corresponding answers. When we call DeepSeek’s online service, this software is what powers it behind the scenes.
If you just came across this news today and want to understand in a few minutes whether it has anything to do with you, my conclusion is: for most people, there is no need to use it directly. However, this reveals a more useful pattern: why engines specifically tailored for a single model often look faster than general-purpose engines like vLLM—and why the system layer that actually makes model serving work well is not in this open-source release.
If you want to download and try it yourself, the requirements to run it are clear at a glance. The code DeepSeek released on October 20 only supports Linux and only supports a single model, DeepSeek-V4.1-Flash. On the hardware side, it only supports NVIDIA’s B200 and GB200 generation of data center GPUs and Huawei’s Ascend 950 data center chips; it does not support consumer gaming GPUs or Mac computers.
Even if you have data center GPUs, you still need to distinguish between DeepSeek’s own online deployment and what was open-sourced this time. An inference engine works in two steps: the first step reads in the entire input prompt and computes the intermediate state, called prefill; the second step spits out the answer word by word, called generation. The official deployment FAQ describes how DeepSeek deploys it online, separating prefill and generation onto different GPUs, with a recommended minimum efficient unit of 64 GPUs for prefill + 64 GPUs for generation, or 128 GPUs in total. However, the code open-sourced this time only provides offline batch inference—meaning you feed a batch of questions in all at once and get results in bulk, rather than installing it and using it directly as an online service. The disaggregated online deployment part was not open-sourced. So the teams actually likely to use it directly are those who have a batch of next-generation data center GPUs on hand and happen to need to run this model offline.
The news reports all say DeepInfer is very fast, but there are two different metrics for measuring the speed of an inference engine. One is total throughput, which measures how quickly a batch of incoming requests can be finished and determines the compute bill; the other is time to first token, which measures how long you have to wait after sending a request before seeing the first word and determines how responsive it feels in practice. Whether a service feels good in daily use mainly depends on time to first token, yet these two metrics often conflict with each other.
Currently, the only same-machine third-party benchmark comes from GPUStack’s same-machine comparison. Because the open-source version of DeepInfer can only run offline batches, the testers had to wrap a community-written API server around it in order to compare online serving against the general-purpose engine vLLM. They ran the same model on the same eight-GPU machine, sending 100 requests at a concurrency of 16, each with 8,192 input tokens and 1,024 output tokens (tokens are the word and character fragments that a model uses to process text). The results on the two metrics went in exact opposite directions: DeepInfer’s total throughput was 25.2% higher, at 1,392 vs. 1,112 tokens per second; but its time to first token was roughly twice that of vLLM, at 4.28 seconds vs. 2.04 seconds. In this test, both sides turned off a commonly used speedup feature, and neither side was thoroughly tuned.
Within three to four days after DeepInfer was open-sourced, two general-purpose engines each published their own self-benchmark reports. On October 23, vLLM published a post saying that on 32 data center GPUs of the same generation, per-GPU generation throughput differed from DeepSeek’s officially published performance curve by no more than 6% (not a same-machine, same-configuration comparison). On October 24, another general-purpose engine, SGLang, published a post saying that on the same hardware as DeepInfer, running DeepInfer’s built-in benchmark yielded results roughly on par with DeepInfer, and slightly faster in some steps.
The speedup feature that both sides turned off in that GPUStack test is called speculative decoding. It works by having a small module guess a few words first, and then having the large model verify them all at once—if the guesses are right, it saves several steps. DeepInfer is just one example; in the AI community, we often see engineers write a dedicated engine for a specific model, such as NInfer for NVIDIA gaming GPUs, Strata for consumer machines, and ds4, originally aimed at Macs. People online often cite their numbers to claim they are several times faster than vLLM. If you want to deploy models locally, or if you want to make sense of this kind of news, you need to understand the three common sources of such advantages.
First is being able to fit and run on certain hardware. For example, Strata offloads infrequently used parameters into system memory, making it possible to use a 12GB gaming GPU plus a large amount of RAM to fit a model with roughly 125 billion parameters. When ds4 was released in May this year, the general-purpose engine llama.cpp did not yet support DeepSeek V4 Flash, but a little over a month later llama.cpp supported it.
Second is the associated weight package—that is, files created by converting and compressing the same model into a format dedicated to a specific engine. Using the same engine on my own GPU, simply switching the weight package (along with the cache compression format) made prefill 13% to 46% faster; simply upgrading the engine version made prefill only 1% to 5% faster. The speedup from switching weight packages is an order of magnitude larger than that from upgrading the engine version.
Third is speculative decoding. The official release of the general-purpose engine SGLang once lacked a key component, giving about 90 tokens per second for a single request on my machine; after merging the change that filled in that piece, running math and coding problems reached about 230, and that change landed in the official release a little over two weeks later. The period during which a single-model engine holds a lead is often just this kind of time lag. The test conditions for these sets of numbers on my machine were not completely identical, so they only give a rough idea.
If engine-level speedups are quickly caught up to by general-purpose frameworks, where is the hardest-to-replicate part of DeepSeek’s online service? On social media, the official claim is that it is a fully open-source production system, but if you look at DeepInfer’s documentation, it explicitly lists three components that were not open-sourced. The first is the caching system that persists intermediate results long-term and reuses them across machines. The second is disaggregated deployment that places prefill and generation on different machines; the corresponding functions in the code are empty in the base class. The third is distributing requests across multiple service replicas. The official explanation is that these are too tightly coupled with internal infrastructure and therefore have not been fully open-sourced for now.
An intuitive example is that a coding assistant sends a long conversation history to the model on every turn, and the engine caches the intermediate results from the previous turn to avoid re-reading it. Once, when I was running a long session with a coding assistant, it exceeded the context limit and was rejected; it then rewrote the middle of the history so the cache no longer matched, and that turn took 24 seconds to produce the first token. On the next turn, once the cache was rebuilt, it took less than 0.3 seconds. This is precisely the kind of problem that caching and routing systems have to handle: making requests hit the cache as much as possible, and dispatching each request to the machine that holds its cache. And that happens to be the part DeepInfer did not open-source.
In October, I wrote about why building your own multi-GPU routing is hard: the core tension is that cache reuse requires pinning a session to a specific GPU, while shortening queues requires spreading requests out, and the two conflict. From this, we can make an inference: engine-level optimizations will sooner or later be matched by general-purpose frameworks, whereas what is truly tied to a company’s own infrastructure is likely this un-open-sourced caching and routing system.
So the next time you see that a new engine is several times faster than vLLM, or that a major company has open-sourced a new inference framework, don’t rush to look at the benchmark charts—think through these three questions first, and the picture will become clear. First, what kind of fast does it mean, and who tested it under what conditions? High total throughput only means batch processing is cost-effective; it does not mean you wait less time for the first token. Second, does this advantage come from the engine itself, or from fitting in memory, weight packages, or speculative decoding? If it is speculative decoding or support for a new model, general-purpose engines will often close the gap within a few weeks to a month or two. Third, which layer did it open-source? Open-sourcing a single-machine engine is not the same as open-sourcing the entire serving stack; cross-machine caching and request routing are the most painstaking parts of self-hosting a service.
In July this year, I wrote an article on vLLM and specialized inference, which ended with a question: will these custom-built inference engines exist independently over the long term, or will they eventually be absorbed by general-purpose engines? Having watched the evolution over the past few months from community projects to DeepInfer, my answer now is: leads in speedups like speculative decoding will be absorbed by general-purpose engines, and support for new models will also catch up within a month or two, but the caching and routing layer will not come along automatically.