Looking through DeepSeek’s technical report for V4.1 Flash, one might notice an easily overlooked number: the team allocated 196B parameters to a module called Engram. Unlike the dense weights we usually discuss, these parameters do not perform matrix multiplications; they simply sit in lookup tables waiting for the model to retrieve them. Because they are not used for computation, they do not need to reside in GPU memory. Instead, they are stored in host memory and fetched on demand as the model runs. The empirical rule that larger parameter counts inevitably consume more GPU memory breaks down here.
The evaluation numbers disclosed in the report offer an intriguing clue. Based on the authors’ self-reported experimental results, the primary gains from this memory lookup module appeared in reasoning tasks, whereas the gains on knowledge tasks were comparatively modest. In the paper’s self-reported data, knowledge benchmarks saw MMLU rise by 3.0 and CMMLU by 4.0; reasoning benchmarks saw BBH increase by 5.0 and ARC-Challenge by 3.7; and needle-in-a-haystack long-context retrieval accuracy climbed from 84.2 to 97.0. All of these improvements remain self-reported by the authors and have not yet been independently replicated by third parties. Why a component specifically designed for memory retrieval would instead boost multi-step deduction and long-range attention is a puzzle far more thought-provoking than merely freeing up GPU memory.
When a language model answers a common-sense fact, it traverses the exact same path as when deducing a mathematical theorem: passing through deep networks layer by layer, executing matrix multiplications from bottom to top. Yet in terms of information structure, these two types of tasks are profoundly different. Rigorous reasoning demands a global view and multi-step deduction; the model must read through the entire context and advance layer by layer along a chain of logic. Recalling a static fact, by contrast, often depends only on a very short sequence of words immediately in front of it—given those few tokens, the answer is virtually locked in. In most cases, recall can be viewed as a local, deterministic mapping that requires no multi-step deduction.
Cramming two tasks with such fundamentally different mechanics into the same deep pathway means that every time the model encounters a static fact, it must mobilize multiple layers of attention and feed-forward networks to reconstruct it. In the Engram paper, DeepSeek views this design as an architectural flaw: while Mixture-of-Experts (MoE) models rely on conditional computation to expand parameter capacity, the model fundamentally lacks a native internal knowledge retrieval mechanism, leaving it no choice but to expend compute simulating lookups over and over again. The team’s solution is to add a conditional memory pathway to the model, serving as a second sparsity axis alongside MoE. The two axes divide labor: one conditionally activates computation on demand, while the other performs table lookups on demand.
Decoupling memory from computation does not, in an engineering sense, automatically equate to table lookups. Retrieval-Augmented Generation (RAG) moved memory outside early on by retrieving documents from knowledge bases; kNN-LM also explored a separation path, relying on vector similarity to match nearest neighbors. Neither path led toward static table lookups. Decoupling is merely the architectural goal; which specific mechanism achieves the separation depends on the intrinsic structure of what is being decoupled.
Returning to the structural characteristics of recall: it is a local, deterministic mapping. Building a dedicated pathway for it requires satisfying three stringent conditions simultaneously. First, it must directly swap the phrase at hand for the corresponding knowledge embedding without introducing deep computation of its own; otherwise, it is merely shifting matrix operations to another location. Second, it must retrieve results in a single memory lookup without going through any preceding search phase. Third, it must be capable of accommodating massive numbers of knowledge entries. A structure that satisfies all three constraints converges quite naturally: a direct key-to-value mapping, most intuitively realized as a table; a lookup key that requires no search and is constructed directly by concatenating the immediately preceding tokens; and, to handle the combinatorial explosion of tokens, hash functions that map arbitrary keys into fixed-length tables.
The bottlenecks encountered by RAG and kNN-LM validate the necessity of each trade-off above. While RAG moved storage outside, it left computation right where it started: the system retrieves documents from an external store based on similarity, but the retrieved text still has to be fed back into the context for the model to recompute layer by layer—static facts continue to consume deep compute resources while hogging precious context length. kNN-LM relies on similarity matching of neighbor vectors, where retrieval is itself a complex distance search; hardware cannot know the subsequent memory addresses to read in advance, making it impossible to schedule pipelined prefetching. Because both approaches require similarity matching upfront, neither can determine the target memory addresses solely from the raw input ahead of time, making it difficult to hide the retrieval latency behind ongoing computation.
The entire design logic clicks tightly into place: because recall is inherently a deterministic local lookup, a table lookup becomes the most energy- and compute-efficient implementation substrate; because the lookup key can be derived directly from context, concatenating the preceding consecutive tokens is the natural choice; and because the combinatorial space is excessively large, hash functions are introduced to fold the entries into the table. Converging on table lookups is the logical consequence deduced straight along the causal chain of recall as a local, deterministic mapping.
In its concrete implementation, the key used for Engram’s table lookup is drawn directly from the preceding token sequence. The network takes two to four consecutive tokens prior to the current generation position, concatenates them into a lookup key, and queries a massive vector table to retrieve the corresponding embedding vector. Since the permutations of two to four tokens are astronomically large and cannot be exhaustively expanded into a single table, the system uses multiple hash functions to map the same key to different slots in the table, forming a multi-head hashing structure. Even if one hash head suffers a collision, the other hash slots will typically land elsewhere, ensuring that valid feature vectors can still be reliably retrieved. Consequently, the entire table can scale to immense proportions while keeping the latency of an individual lookup strictly at constant time, independent of the total number of entries in the table.
Having solved the table structure, the next challenge is where to house this massive table. If it remains in GPU memory, the memory crunch is not relieved at all. The paper’s chosen path is to move these massive parameters off the accelerator and place them in host memory, which offers far more generous capacity at a much lower unit cost. This route—which at first glance seems poised to choke on bus bandwidth—is made viable by three interlocking prerequisites.
The primary prerequisite is that memory addresses are completely deterministic. Because the lookup keys are derived directly from previously generated tokens, the system can compute in advance exactly which physical addresses the subsequent network layers will read when model execution begins, entirely bypassing the overhead of runtime search. The second prerequisite is that real-world language follows an extreme heavy-tailed distribution. In natural text, a tiny minority of high-frequency token combinations accounts for the vast majority of accesses, allowing engineers to build a clean storage hierarchy: the most frequent feature vectors stay resident in GPU memory, the next tier resides in host memory, and sparse tail entries are left for slower but much larger storage media. The third prerequisite is hiding communication latency via data prefetching. While the GPU is still computing matrix multiplications in the earlier, shallow layers, the transfer channel is already moving the vectors needed by deeper layers into place. The technical report notes that V4.1 constructs this prefetch pipeline over RDMA channels, seamlessly overlapping data transfer with the forward pass computation of the first Transformer block. Among these three conditions, accurately calculating the target address ahead of time is the foundational pillar upon which this entire low-cost pipeline rests.
If it were merely about communication optimization, Engram might be seen as nothing more than a clever memory-saving technique. The core insight the paper really wants to convey is that model capacity itself can follow fundamentally different cost pathways. Mixture-of-Experts (MoE) scales capacity via conditional computation: each generated token must go through dynamic routing to select and activate a subset of expert networks. Engram, on the other hand, scales capacity via conditional memory: by mounting an enormous lookup table, the model accumulates a much richer reserve of static knowledge without adding a single floating-point matrix operation. Computation and memory are thus cleanly separated into two distinct ledgers, giving rise to a new engineering question: how should a finite sparse capacity be partitioned between dynamic deduction and static table lookups?
The paper names this trade-off “sparse allocation” and maps out a clear U-shaped curve through experiments. Betting all capacity on compute experts is suboptimal, but swinging entirely toward static memory lookups leaves the model ill-equipped for complex contexts; the global optimum lands right in the sweet spot between the two. The research team first scanned different ratios at 5.7B and 9.9B parameter scales, finding that allocating roughly 20% to 25% of sparse capacity to Engram yielded the best performance. Applying this rule to a 27B model resulted in superior overall performance compared to a pure MoE baseline with identical parameter count and FLOPs. To be sure, these metrics remain self-reported experimental data from the authors, and independent third-party replications have yet to appear. The central value of this exploration extends beyond point engineering optimizations: it pushes the long-standing conversation about “how many parameters a model needs” one step further—onto which hardware media should model capacity be anchored, and in what proportions should it be partitioned between computation and memory?
Intuitively, one would expect that attaching or embedding a memory lookup unit would primarily serve to help the model memorize common-sense facts, with gains concentrated on knowledge QA benchmarks. Yet the actual evaluation trends disclosed in the paper reveal a striking contrast: in the authors’ self-reported benchmarks, the biggest improvements occurred in multi-step reasoning—BBH jumped by 5.0 and ARC-Challenge by 3.7, with concurrent gains in code and math—while knowledge-oriented MMLU and CMMLU rose by 3.0 and 4.0, respectively. An even more dramatic shift appeared in needle-in-a-haystack tests, where long-context retrieval accuracy soared from 84.2 to 97.0. It should be noted that these substantial leaps in reasoning and retrieval remain self-reported data from the paper, with no independent third-party replications seen to date.
Behind this seemingly anomalous trend lies a clear mechanistic chain. In conventional autoregressive architectures, the shallow layers near the input must expend significant parameter capacity and attention budget repeatedly piecing together and reconstructing local static patterns. Once the table lookup module takes over these local mappings, the earliest layers are freed from repetitive, mechanical heavy-lifting, allowing the model’s precious computational depth to concentrate on far more challenging and complex reasoning tasks. At the same time, shifting local token-order dependencies to the lookup module relieves self-attention of the burden of modeling nearby patterns, freeing up more of its budget to capture global, long-range dependencies—thereby markedly improving long-context retrieval accuracy.
This is also the most valuable engineering takeaway of the entire work. Moving static memory out of the dense compute path delivers value far beyond saving precious GPU memory: it fundamentally alters how the remaining compute resources are utilized. Laying down a low-cost, deterministic lookup channel yields, in return, a highly efficient compute pathway dedicated to complex logical deduction.
Viewed in the broader context of computing history, this idea is hardly new. Ever since the von Neumann machine, compute and storage have resided in separate locations, with data continually shuttled between them; throughput has long been choked along this channel, giving rise to what became known as the von Neumann bottleneck. Engram represents another response to this age-old problem: because shuttling an entire table of static knowledge in and out of the compute unit is prohibitively expensive, it is kept in inexpensive host memory, fetching only the small number of vectors needed at the moment by address and using prefetching to hide transfer latency. What it separates are two inherently different types of capacity within the model: one for computing, and one for looking up.
From this perspective, this exploration closely resembles the battle-tested storage tiering mechanisms of computer architecture. Virtual memory in operating systems operates on the exact same logic: massive datasets reside on inexpensive, slower storage media, while only the frequently accessed active working set is paged into high-speed physical memory, with the system relying on the principle of locality for caching hot data and prefetching. Engram’s deterministic addressing similarly makes it well-suited for cache tiering, allowing the runtime to schedule data movement in advance; what the V4.1 report explicitly details is host memory paired with background RDMA prefetching. This design rigorously introduces industry-standard storage tiering principles into deep learning architectures.
Over the past several years, Product Key Memory (PKM), kNN-LM, and Retrieval-Augmented Generation (RAG) have all attempted to decouple memory from the backbone network, but most were bogged down by excessive retrieval latency, the overhead of training separate retrieval models, or simply stuffing retrieved text back into the context window. That this path is now viable again relies on the concurrent maturation of three engineering conditions. First, server host memory offers physical capacity vastly exceeding GPU memory, and deterministic addressing combined with a prefetch pipeline drives the overhead of reading vectors from host memory to negligible levels. Second, the retrieved embedding vectors can propagate gradients end-to-end via a gating network. Third, the architecture establishes a fine-grained allocation mechanism between compute capacity and memory capacity. Without any one of these three, this pathway would struggle to materialize.
This also explains why related explorations are not confined to a single team. Qwen3.8 Flash-Next introduced a 51B-parameter N-gram embedding layer into its architecture; its design paper explicitly notes that the approach is based on the Engram architecture and adopts its gated fusion method, with the lookup table likewise located in host memory. When two representative engineering teams converge in the same direction around the same time, it typically signals that offloading static memory has entered the mainstream consciousness.
For teams tasked with actual system deployment, the baseline dimensions for performance evaluation shift accordingly. While engineers have historically been accustomed to watching GPU memory high-water marks and peak tensor compute FLOPS, going forward they must also rigorously account for available host memory capacity, bus throughput bandwidth, and whether production workloads truly exhibit heavy-tailed access patterns. The “negligible overhead” cited in the paper’s self-reported results hinges on deterministic addressing, heavy-tailed access patterns, and the complete overlapping of prefetching with computation—ideal conditions that dynamic real-world production environments do not always sustain. A measured, cautious stance should also be maintained regarding empirical claims: the comparative performance at 27B parameters is built on author-selected baselines, and the 75/25 sparse allocation ratio likewise stems from a single paper’s self-reports; whether these hold universally across model scales and task domains remains to be seen. Engram outlines a clear, compelling roadmap, but establishing it as a standard building block for next-generation foundation models will require thorough independent replication and validation across the industry.
This article examines memory decoupling along the dimension of model capacity: how static knowledge can be systematically and safely moved out of the GPU core. For system design on the serving side regarding aggressive KV cache compression, see the first article in this series, DeepSeek V4.1 Flash: With Compute Optimization Hitting a Ceiling, the Long-Context War Moves to Memory.