During LLM inference, the context window serves as the model’s working memory. How much content the model can observe in a single inference pass—and with what clarity—directly dictates what tasks it can accomplish. Whether tracing call chains across thousands of lines of code, maintaining state in multi-step autonomous agents, or cross-referencing hundreds of pages of financial documents, all reasoning capabilities are built upon context.
However, the moment one attempts to scale up the context length, engineering implementations collide head-on with two primary bottlenecks: GPU memory running out and compute becoming intractable. The chief obstacle among them is the KV cache left in GPU memory—the model’s working scratchpad. During inference, every token in the context requires the model to maintain a key-value state in real time for attention computation. The footprint of this scratchpad depends on the model’s number of layers, KV heads, head dimension, and storage precision.
Taking a 70B-scale model (such as Llama-3-70B) as an example: processing an ultra-long context of 1 million tokens (1M) in FP16 consumes approximately 327 GB of VRAM just to store this working scratchpad. This already far exceeds the physical capacity of any single high-end GPU. Furthermore, this 327 GB accounts only for the scratchpad itself, without factoring in the model’s own weight parameters or the quadratic surge in attention compute as sequence length grows. To breach this physical wall of memory and compute, the engineering community has explored numerous avenues. Examining the three mainstream approaches proven effective today reveals that they operate at three distinct layers of the model’s technology stack:
These three solutions originated from different research teams and handle different input modalities. Yet underlying their engineering design is the very same ancient intuition.
This intuition has long existed in computer science and cognitive science. In 3D graphics rendering, engineers established mipmapping and Level of Detail (LOD) mechanisms long ago. Objects close to the camera are rendered fully with tens of thousands of polygons and high-definition textures, whereas mountains and trees far on the horizon automatically switch to simplified, coarse meshes and blurred textures. Allocating the same compute to distant background pixels as to foreground objects is a waste of rendering resources. In human vision, the fovea centralis provides high-resolution, high-color-acuity vision only for a narrow central region of about 2 degrees; peripheral vision is blurry and coarse-grained, serving primarily to detect motion and rough outlines.
Human memory follows the exact same pattern. Specific word choices and tone particles from a conversation minutes ago can be recalled vividly; yet for a meeting five years ago, the brain typically retains only a handful of key takeaways and a structural framework, with lossy memory compression having long filtered out the details.
These mechanisms all point to a single prior: with finite compute resources and storage budgets, applying uniform precision across all distances is the most wasteful approach. The most rational resource allocation strategy is to reserve maximum precision for nearby context and critical focal points, while applying lossy compression to distant or background information. The three approaches that have broken ground in long-context engineering today are precisely the independent manifestations of this prior across three distinct layers of the model tech stack.
Transformers have no inherent awareness of word order; each token must be explicitly assigned a position index. Rotary Position Embedding (RoPE, introduced by Su et al. in their 2021 paper arXiv:2104.09864) rotates vectors in space by specific angles based on their position, allowing the model to gauge token relationships through relative distances. Different dimensions are assigned different rotational speeds: fast-rotating dimensions distinguish adjacent tokens and capture phrasing and syntactic nuances; slow-rotating dimensions track macro-level positions across paragraphs. This division of labor between fast and slow dimensions is determined in a geometric progression before training as a hardcoded rule. Because the model’s weights are trained end-to-end alongside these fixed rotations, the optimization process naturally learns to read fast dimensions for local details and slow dimensions for long-range positioning.
RoPE features three key properties: relative position awareness, vector norm preservation under rotation, and no need for lookup tables. It has become the industry standard for mainstream LLMs (widely adopted by LLaMA, Mistral, Qwen, Gemma, and DeepSeek, with citation counts reaching 6,184 on Semantic Scholar). However, RoPE’s rotational speeds are calibrated within a fixed training sequence length. If a text exceeding the training length is fed directly into the model, the out-of-distribution rotational angles distort the entire coordinate ruler, causing model outputs to degrade rapidly.
Initial attempts at extension centered on Position Interpolation. This is akin to uniformly stretching the scale of the entire ruler: to expand the context by 4x, the rotational speeds across all dimensions are uniformly slowed down by a factor of 4.
The issue with this uniform global stretching is that it ignores the division of labor across dimensions. While slow-rotating dimensions are indeed brought back into readable ranges, the fast-rotating dimensions responsible for fine-grained local syntax are stretched as well. Consequently, the subtle sense of distance between nearby tokens is lost, leading to a noticeable degradation in the model’s fundamental language capabilities on standard short texts.
In 2023, researchers from Nous Research, EleutherAI, and the University of Geneva proposed YaRN (arXiv:2309.00071). Targeting the degradation of local resolution caused by Position Interpolation, its core mechanism, dubbed NTK-by-parts, applies frequency-band splitting to treat different segments of the ruler differently:
Additionally, coordinate stretching causes attention distributions to disperse, making it difficult for the model to focus. YaRN introduces attention temperature scaling to retighten the focus of attention using a temperature multiplier.
Experiments demonstrated that YaRN allowed Llama 2 7B to successfully extend its context to 128K using only ~384 A100 GPU hours across 400+200 fine-tuning steps; it achieved a 99.4% passkey retrieval accuracy at 128K length with less than 1% degradation on short-context benchmarks. Compared to pure Position Interpolation, YaRN saved approximately 10x in training tokens.
This frequency-band scaling strategy has become the standard
technique for coordinate-layer extension. The llama3 RoPE
implementation in the Llama 3.1 128K configuration is fundamentally
frequency-threshold scaling under the hood; Qwen2/2.5 as well as the
open-source model gpt-oss-20b (which explicitly specifies
rope_type "yarn" with a scale factor of 32) adopt the same
line of thinking.
What YaRN accomplishes at the coordinate layer is straightforward: leave fine-grained nearby ticks untouched, and compress coarse-grained distant ticks. This is precisely the realization of the “high fidelity nearby, lossy at a distance” intuition at the coordinate layer.
While the coordinate layer resolves the measurement problem of the positional ruler, it cannot overcome the aforementioned memory wall: taking a 70B model as an example, the working scratchpad for a single 1M token sequence alone demands 327 GB, which is unsustainable for hardware clusters. When processing long contexts, attention must continually read and write accumulating historical information—the KV cache: reading the past while writing the present. In standard Multi-Head Attention (MHA), each attention head maintains its own complete Key and Value representations, causing total historical cache size to scale linearly with the number of heads. Subsequent Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) have multiple heads share the same historical state, saving memory at the cost of coarser per-head representation and compromised expressiveness.
To address this dilemma, DeepSeek introduced Multi-Head Latent Attention (MLA) in DeepSeek-V2 (arXiv:2405.04434): instead of storing a full history per head token-by-token, it compresses the state to be cached into a compact shared latent representation, which individual heads reconstruct on the fly as needed. This design directly cuts KV cache memory by 93.3% while matching or exceeding the expressiveness of standard MHA. To prevent positional information from interfering with the compressed latents, the model decouples a separate 64-dimensional channel (decoupled key) dedicated solely to carrying RoPE rotary position embeddings. (For a more accessible explanation of the MLA mechanism, see here.)
By the time of DeepSeek-V4 (arXiv:2606.19348, 2026-04), which supports a 1M-token context window, the memory wall had already been substantially lowered by MLA. However, computational complexity remained an issue: computing pairwise attention between every token and 1 million preceding tokens is still prohibitively expensive. DeepSeek-V4 proceeded to restructure the attention mechanism at the computational pathway level, adopting a hybrid attention architecture:
This mirrors human visual perception: the 128 words currently being read right in front of you are perceived with crystal clarity; the rest of the massive book in the distance is first blurred into a background context (HCA), and whenever certain paragraphs are found to be highly relevant to the query at hand, those specific sections are brought into sharp focus for detailed reading (CSA). On DeepSeek-V4-Pro with 1.6T total parameters (49B activated), this hybrid pathway architecture reduces per-token inference FLOPs at 1M context to 27% of V3.2, and cuts KV cache memory to 10% of V3.2.
The trade-offs at the information pathway layer are a direct realization of the article’s core intuition: within a finite physical budget, keep high fidelity nearby and allow lossiness at a distance. The nearby 128 tokens maintain a full-fidelity pathway, while distant history relies on holistic compression and selective retrieval in exchange for efficiency. (For the engineering integration of DeepSeek V4 in agentic scenarios, see this article.)
Long contexts in real-world applications often stem from hundreds or thousands of pages of PDFs, research reports, contracts, or scanned documents. Transcribing these documents verbatim into standard text tokens and feeding them to LLMs not only burns through massive token budgets, but also strips away the original document layout, nested tables, and typographic hierarchy. The model needs to comprehend the entire document, yet operates under finite token throughput budgets. Existing document parsing and multimodal input approaches face a dilemma:
In October 2025, researchers released DeepSeek-OCR (arXiv:2510.18234), proposing Contexts Optical Compression, or optical 2D compression. The intuition is akin to capturing a high-density semantic thumbnail of an entire page. The LLM directly ingests the compressed 2D visual representation, bypassing the overhead of parsing 1D text streams token by token. DeepSeek-OCR comprises two core components: 1. DeepEncoder: An image encoder that maintains a low memory footprint when processing high-resolution input images, compressing full pages or long text passages into a manageable number of high-density visual tokens at high compression ratios. 2. DeepSeek3B-MoE-A570M Decoder: Reconstructs textual content and layout structures from these compressed visual tokens.
The paper demonstrates a clear trade-off between compression ratio and reconstruction accuracy. A higher compression ratio saves more tokens, but reconstruction accuracy degrades accordingly. Specific figures include: - When the original text token count is within 10x the visual token count (compression ratio under 10x), text reconstruction accuracy reaches 97%. - When the compression ratio is pushed aggressively to 20x, reconstruction accuracy remains around 60%.
On the OmniDocBench benchmark, DeepSeek-OCR outperformed GOT-OCR2.0 (which consumes 256 tokens) using only 100 visual tokens; and with fewer than 800 visual tokens, it surpassed MinerU2.0 (which consumes over 6,000 tokens per page on average). In actual production environments, a single A100-40G GPU can process and generate over 200,000 pages of training data per day.
DeepSeek-OCR’s paper explicitly notes that the objective of this work is to explore historical long-context compression and active forgetting mechanisms in LLMs. The paper envisions keeping the current focus at maximum precision while letting historical documents reside in background memory in the form of dense visual tokens. The input representation layer is likewise an embodiment of this intuition: the current page of focus maintains highest fidelity, while historical documents persist in the background as lossily compressed visual tokens.
Examining these three methods side by side reveals a clear correspondence across different layers of the technology stack. Their differences center on their representative schemes, mechanisms of action, and respective handling of nearby versus distant contexts. The table below breaks them down layer by layer:
| Tech Layer | Representative Solution | Mechanism | Nearby / Focus Handling | Distant / Background Handling | Core Bottleneck Addressed |
|---|---|---|---|---|---|
| Coordinate Layer | YaRN | Modifies rotational frequencies of positional embeddings | High-frequency ticks unstretched to preserve fine-grained local resolution | Low-frequency ticks fully interpolated with smooth transition | Coordinate extrapolation distortion and short-text performance degradation |
| Information Pathway Layer | DeepSeek V4 | Restructures attention forward computational pathways | 128-token sliding window preserves full-fidelity computation | Long-range heavy compression (HCA) and selective retrieval (CSA) | KV cache memory wall and quadratic compute explosion |
| Input Representation Layer | DeepSeek-OCR | Modifies raw inputs before entering the model | Current focal page parsed with high fidelity | Entire pages compressed into a small number of dense visual tokens | Excessive token counts in long documents and loss of formatting/layout |
There is no direct lineage or technical transplantation among these three approaches. YaRN emerged from researchers at Nous Research, EleutherAI, and the University of Geneva investigating the mathematical properties of positional rulers; DeepSeek V4 represents an attention pathway restructuring engineered by model architecture teams pushing the limits of GPU memory; and DeepSeek-OCR is an input-level reconstruction approaching the problem through optical images and document formatting.
They were proposed by different teams at different times, confronting distinct physical bottlenecks. The reason they exhibit such striking structural isomorphism is that, at their respective technological layers, they independently converged upon the same engineering intuition: under a finite physical budget, proximity demands fidelity, while distance permits lossiness.
As engineering implementations transition from indiscriminate full-sequence computation toward tiered compression and selective retrieval, a new technical challenge emerges: positional confusion. In traditional dense computation, all tokens are arranged sequentially with clear, contiguous relationships. Once sparse selection is introduced (such as CSA selecting a handful of key tokens from the prefix) or multiple documents are concatenated together, the selected tokens lose their original contiguous physical spacing. If the model computes relative positions directly, it can easily misjudge two tokens that were originally thousands of words apart—and merely selected together—as immediately adjacent. To maintain an accurate sense of position after compression and selection, recent technical solutions have developed distinct approaches:
As seen from these three paths, positional awareness is increasingly being absorbed into the model’s internal machinery: either embedded within dedicated indexers for sparse selection, or delegated to the gating and natural decay of recurrent architectures. As sequence lengths push toward millions or tens of millions of tokens, merely stretching and tweaking external coordinate rulers is no longer sufficient to address the fundamental bottlenecks of memory exhaustion and computational explosion. Coordinate-layer modifications based on RoPE base frequency fine-tuning and YaRN are gradually retreating from the primary battleground of long-context scaling to serve as supplementary techniques.
Regardless of how engineering implementations shift from external coordinate transformations to internal computational pathways, and further to frontend input representations, the underlying technical core remains unchanged. In a physical world bounded by compute and memory budgets, high fidelity nearby and lossy compression at a distance continues to serve as the steadfast and effective intuitive foundation for building ultra-long-context intelligent systems.