A Scratchpad and a Controller: Rethinking LLM Reasoning

Before OpenAI released o1 on September 12, 2024, everyone’s habit of using ChatGPT was very simple: type a line of prompt, and the screen would immediately stream out answers word by word like a gushing fountain. But on the night o1 emerged, the interaction model underwent a clear transformation. For the first time, a prominent collapsible thinking section appeared in the chat box. The model was no longer in a hurry to answer; instead, it pondered silently for dozens of seconds, generating thousands of invisible intermediate tokens in the background. The benchmark scores released afterwards demonstrated significant improvements on math and programming competition problems. By January 2025, the open-sourcing of DeepSeek-R1 turned this long thinking process into a hot topic of debate across the entire AI community.

In our previous article, A Four-Year History of Reasoning Models: What Seemed Like a Sudden Breakthrough Was Actually a Long-Concealed Thread, we traced the four-year technical evolution from 2021 Scratchpad to o1 and DeepSeek-R1. A core conclusion we reached back then was: reasoning capacity is not a genetic mutation that suddenly erupted overnight; its real watershed in business and engineering lies in the fact that vendors for the first time restructured test-time thinking time into a compute resource that can be flexibly scheduled and independently billed. If the previous article answered how this reasoning revolution was assembled step-by-step historically, this piece serves as a complement. Rather than staying on a macro timeline, we shift our focus to the model’s internal computational mechanics: What actually happens inside an autoregressive Transformer when it drafts a scratchpad? Why does lengthening a model’s output not equate to making it smarter? What exactly does reinforcement learning rearrange deep inside the model’s weights?

Our conclusion is straightforward: a Reasoning model does not suddenly grow a new brain capable of thinking. Chain of Thought (CoT) first provides the Transformer with a reviewable append-only scratchpad, allowing it to write down intermediate states and semantically override old conclusions with new text; reasoning post-training then trains a controller that allocates scratchpads, checks, backtracks, switches paths, and stops. o1 scaled and commercialized this continuous lineage, while R1 democratized and broadly diffused it; both are important, but neither is the starting point of this intellectual history.

Mechanical Essence of Autoregressive Context: A Scratchpad and a Controller

To understand the essence of reasoning models, we must first clarify a highly confusing question: Looking at the autoregressive formula $x \xrightarrow{} s_1 \xrightarrow{} [x,s_1] \xrightarrow{} s_2$, isn’t CoT using standard autoregressive decoding? Isn’t text generation in today’s ordinary LLMs also a word-by-word streaming process? What exactly did it change? The answer is: in terms of physical decoding operators, CoT is indeed 100% standard autoregressiveness. It did not modify the basic Transformer architecture, nor did it invent any mysterious new operator. What it altered is how computation is amortized across space (network hidden layers) and time (context sequence).

As is well known, every time a large language model predicts the next Token, it must pass this new Token from the first layer to the last layer of the neural network. Practical deployments utilize KV cache to cache the key and value vectors of previous attention steps, avoiding recalculation of the entire history, yet each new Token still traverses a fixed number of layers during forward propagation. Hidden here is a frequently overlooked physical constraint: the number of layers in a model is fixed, and the computational depth available for each single next-token prediction is likewise fixed. If a problem requires 20 steps of logical transformation and we require the model to output no intermediate steps—demanding the final answer on the very first token—we are effectively forcing the neural network to compress 20 mental calculation steps into the hidden space of a single forward pass. When the derivation complexity exceeds the neural network’s fixed computational depth, the hidden space bursts, leaving the model with no choice but to blindly guess the answer.

The Chain of Thought (CoT) mechanism alters this computational topology of single-step forward passes. Its essence is to decompose the hidden-layer mental calculation—originally attempted within a single forward pass—into an iteratively recurring state loop:

$$\text{Input} \xrightarrow{} s_1 \xrightarrow{} [\text{Input}, s_1] \xrightarrow{} s_2 \xrightarrow{} [\text{Input}, s_1, s_2] \xrightarrow{} \dots \xrightarrow{} \text{Output}$$

In this loop, the Transformer’s autoregressive context window acts as a scratchpad that can be reviewed. The intermediate derivation symbol $s_t$ written by the model at step $t$ (e.g., a step-one calculation result of 42) is fed back into the model as static input at step $t+1$. This means complex computations are amortized across dozens or even hundreds of autoregressive time steps. Each generation of a new Token solidifies the intermediate variable of the previous step into the Context. During the next step of generation, the attention mechanism can directly and explicitly retrieve 42 without needing to recalculate step one within the hidden layers. CoT shifts the computational bottleneck of hidden-layer depth onto the time amortization of context sequences.

From a physical property perspective, the autoregressive Context scratchpad possesses a very special characteristic: an append-only attribute that allows writing forward but never erasing backward. Because once Transformer’s autoregressive mechanism generates a Token, it becomes solidified in the context; the model cannot pick up an eraser like a human to wipe out a mistaken line. However, the model has a very interesting way of self-correction, which can be understood as semantic overriding. When the model notices an issue in its previous derivation, it does not need to erase prior text; it only needs to append a natural reflection afterwards—for example, writing: “Wait, there’s a calculation error here. 42 multiplied by 7 should equal 294, not 284. The previous assumption doesn’t hold, so we need to restart from Equation 2…”—and subsequent Tokens can take this new judgment as a conditional context to continue deriving, allowing old errors to be semantically overridden by later text. This is not automatically guaranteed: old content remains in the Context and may continue to interfere with subsequent generation.

However, we must be clear: the scratchpad itself has no intelligence; it merely provides a space to store intermediate states. If we forcibly make the model print a string of irrelevant whitespace or dots in the Context, the model’s reasoning capabilities will not automatically improve. Ablation experiments tell us that the scratchpad works because every character written upon it carries structured, effective semantic state.

So, what is the difference between ordinary prompt-induced CoT (such as adding step-by-step thinking prompt examples in the prompt) and reasoning models like o1 / R1? Ordinary prompt CoT merely leverages pre-training habits to generate sequentially forward; it remains prone to persisting down a single faulty path, making an error at one step and committing to it all the way without knowing how to check mid-course. In contrast, the reasoning post-training conducted by o1 and DeepSeek-R1 trains a strategy hub: the Controller. The scratchpad itself cannot think; the Controller trained via post-training determines which direction the model should write on the scratchpad during autoregressive generation, when to pause and verify previous derivations, when to recognize that the current path has hit a dead end and execute a backtrack to switch paths, and at what node to confirm that the derivation is complete to issue a termination signal.

Historical Evolution: How Academia Assembled the Scratchpad and Controller Step-by-Step

Looking back at the academic milestones of the past four years, you will realize that researchers were not aimlessly trying random prompts. Instead, they were step-by-step validating, refining, and solidifying the mental model of the scratchpad and Controller described above.

First came the birth of the scratchpad form. As early as 2021, Nye et al. made a critical breakthrough in their Scratchpad research: they discovered for the first time that as long as the model explicitly printed intermediate processes—such as addition carries and variable changes—in the context before producing the final answer, its capability to execute multi-digit calculations improved dramatically. This experimentally validated for the first time the immense value of persisting hidden-layer computations as static Context symbols. Shortly after in 2022, Wei et al. proposed Chain of Thought (CoT), demonstrating further that natural language itself could serve as a general-purpose scratchpad, activating the model’s drafting capabilities even without fine-tuning, purely through few-shot prompting.

Second came early explorations of Controller strategies. Once the scratchpad existed, researchers immediately realized that relying on a single naive scratchpad draft easily led astray. Thus, researchers began exploring how to better control drafting strategies: - Wang et al. (2022) proposed Self-Consistency: since a single scratchpad might contain errors, let the model generate computations along different paths across multiple scratchpads simultaneously and take a majority vote. - Zelikman et al. (2022) proposed STaR (Self-Taught Reasoner): letting the model attempt to solve problems independently, filter valid scratchpads based on correct answers, and use these high-quality scratchpads to fine-tune itself, establishing an initial loop of Controller self-improvement. - Yao et al. (2022) proposed the ReAct framework, connecting the scratchpad to external environments so that the Controller could not only draft internally but also issue Action commands calling external tools and read the resulting Observation outputs.

Finally came building infrastructure for Controller evaluation and test-time compute allocation. From 2022 to 2024, research focus shifted toward fine-grained scoring of scratchpad quality and allocating compute budgets. Lightman et al. (2023) in PRM800K and Uesato et al. (2022) in their comparison of process versus outcome feedback demonstrated that scoring each individual step in a scratchpad (Process Reward Model, PRM) is far more precise than evaluating only final answer correctness. Meanwhile, Snell et al. (24) in Scaling LLM Test-Time Compute Optimally provided quantitative proof: dynamically allocating more thinking compute during inference can beat models with vastly larger parameter counts.

From 2024 to 2025, OpenAI o1 and DeepSeek-R1 made their debuts in succession. The real breakthrough of these two milestones lies in the fact that they no longer treated scratchpads, scoring evaluations, and compute scheduling as disjointed academic experiments. Instead, using large-scale reinforcement learning, they stitched together the technical jigsaw puzzle explored over the preceding four years. For the first time, they trained a mature Controller capable of automatically scheduling, verifying, checking, and allocating compute internally within the model, completing the critical leap from academic exploration to industrial-grade productization.

Boundary Verification: Three Sets of Ablation Experiments on Scratchpad and Controller

At this point, we have established a comprehensive mental model: LLM reasoning capabilities do not stem from a qualitative mutation of the brain, but rather rely on the collaboration between the autoregressive scratchpad (state storage) and the Controller (scheduling strategy).

However, to verify whether this mental model captures the underlying physical reality or is merely an oversimplification wrapping up buzzwords, we must establish falsifiable causal tests targeting both core components—the Controller and the scratchpad. In recent years, three extraordinary sets of ablation experiments from academia precisely address the physical boundaries of these two core components:

1. Testing the Essence of the Controller (Experiments 1 and 2)

Regarding the Controller, the most common industry misconception is that post-training (SFT + RL) appears to create a brand-new problem-solving algorithm deep within the model weights. Two key experiments independently completed by academia provide answers from the dimensions of strategy and weights respectively:

Experiment 1: s1’s Wait Token — Does Forcibly Appending Control Words Create New Capabilities or Release Existing Budgets?

In the s1 experiment by Muennighoff et al. (2025), researchers attempted to forcibly insert a Wait token at the end of the model’s output to compel the model to keep writing, raising AIME test accuracy from 50.0% to 56.7%. But the critical caveat is that this result stemmed from s1-32B, which had already undergone supervised fine-tuning on 1,000 long-CoT samples, rather than an untrained base model. The paper also observed that repeatedly suppressing the stop Token caused the model to gradually collapse into repetitive loops. This demonstrates that Wait is not a universal reflection switch applicable to any arbitrary model; it acts more like a forced release of thinking budget (Budget Forcing) on an existing Controller strategy.

Experiment 2: RLVR and pass@k Decay — Is Reinforcement Learning Inventing New Algorithms or Rearranging Probabilities?

On the weight level, Liu et al. (2025) from Sea AI in Understanding R1-Zero-Like Training and Yue et al. (2025) from Tsinghua-SJTU in Does Reinforcement Learning Really Incentivize Reasoning Capacity evaluated the success rate of models under varying sample counts (pass@k): when the sample count $k=1$, the RL-tuned model holds a massive advantage; but when the sample count extends to $k=256$, the accuracy of the un-RL’d original base model rapidly increases, matching or even surpassing the RL model. This confirms that the Controller’s core mechanism primarily lies in probability rearrangement—it reweights paths that the base model already possessed at very low probabilities, allowing the model to select them with high probability on a single attempt (pass@1). Of course, ProRL by Liu et al. (2025) in ProRL additionally indicated that prolonged RL can expand practical problem-solving boundaries under limited sampling budgets.

2. Testing the Physical Boundaries of the Scratchpad (Experiment 3)

Having clarified the probability-rearrangement nature of the Controller, let us examine the second core component: the autoregressive Context scratchpad. Is a textual scratchpad omnipotent? If autoregressive scratchpads have limitations, where is their physical ceiling?

Experiment 3: Physical Limits of Pure Text Scratchpads and PoT Tool Substitution — Can Text Scratchpads Solve Everything?

In The Illusion of Thinking, Shojaee et al. (2025) from the Apple team conducted a rigorous boundary test on the scratchpad: forcing reasoning models to solve complex Tower of Hanoi disk movements inside a pure text Context. Results revealed that when facing tasks heavily reliant on precise mechanical state tracking, pure text CoT suffers from autoregressive error accumulation as output length grows, leading to distortion of the reasoning chain.

However, in the follow-up work Thinking Isn’t an Illusion and Program of Thoughts (PoT) experiments, researchers made a change: instead of forcing the model to struggle with disk positions in plain text on the scratchpad, they had the model generate a concise piece of code and hand it over to a Python interpreter for execution. As a result, the model’s solving accuracy on the Tower of Hanoi task recovered to 100%.

This experiment precisely demonstrated the physical boundaries of the scratchpad: pure textual autoregressive drafts are inherently fragile when handling mechanical state tracking. A mature Controller must learn not only how to write on an internal scratchpad, but even more importantly, when to offload fragile textual derivations to deterministic external code executors. This also points toward the next direction for the architectural evolution of reasoning models.

Positioning and Reflection: The Real Contributions of OpenAI o1 and DeepSeek-R1

After clarifying the logical chain of scratchpads, Controllers, and ablation experiments, we can establish a more objective and rational evaluation of the two major milestones: OpenAI o1 and DeepSeek-R1.

OpenAI o1: Scaling, Commercialization, and Category Definition

The primary value of OpenAI o1 lies not in inventing novel academic theory, but in superior system engineering and product packaging: 1. System-Level Engineering Integration: In Learning to Reason With LLMs, OpenAI fused large-scale reasoning reinforcement learning, inference-time thinking compute, and frontier model capabilities, achieving state-of-the-art results on AIME and Codeforces; 2. Product Form and Billing Innovations: o1 introduced the concept of reasoning token, transforming the thinking process into a compute resource that can be scheduled by systems, perceived by users, and billed via APIs, defining an entirely new product category.

At the same time, we must clarify the boundaries: OpenAI has not disclosed o1’s Base model, training data, or search details to date, so we cannot simply equate open-source community implementations with o1’s internal practices.

DeepSeek-R1: Engineering Transparency, Open-Source Decryption, and Ecosystem Diffusion

When evaluating DeepSeek, two forms must be distinguished: - DeepSeek-R1-Zero: Running pure reinforcement learning directly on DeepSeek-V3-Base (671B MoE, 37B activated, pre-trained on 14.8T Tokens) with only format guidance and rule-based outcome rewards, without prior SFT. - Official DeepSeek-R1: Detailed in the DeepSeek-R1 technical report, uncovering a four-stage pipeline: cold-start SFT $\rightarrow$ first-stage Reasoning RL $\rightarrow$ rejection sampling generating 800K data for second-stage SFT $\rightarrow$ mixed RL.

DeepSeek-R1’s value is not reinvention of reasoning capability, because the DeepSeek-V3-Base pre-training corpus already contained massive amounts of math and code. Its true contributions lie in: 1. Transparent Training Pipeline: Demonstrating that without expensive step-by-step PRM process annotation or complex tree search, a strong Base model combined with the GRPO algorithm and outcome rewards can robustly train long-thinking capabilities; 2. Open-Source Diffusion: Completely opening the weights of R1-Zero, R1, and six distilled smaller models, allowing developers worldwide to research and deploy frontier reasoning systems.

Endgame Perspective: From Internal Scratchpad to External Workspace

Standing at the present moment, the evolution of LLM reasoning will clearly not remain confined to simply stretching internal CoT longer. Zhang et al. (2025) in Recursive Language Models (RLM) demonstrated an alternative paradigm: instead of forcibly cramming ultra-long derivations inside the internal Context, states are persisted in an external environment. The root model recursively invokes sub-models through a Python REPL interface to resolve local subproblems. On long-context tasks like OOLONG-Pairs, when the recursion depth is increased to 3, the model’s performance improves significantly over depth 1 (76.0 vs 58.0).

This signifies that the evolution of reasoning frameworks is advancing toward a more holistic system architecture: the central question is no longer whether an LLM possesses a chain of thought or how long it spends thinking, but rather:

Where does the system store reasoning state? Which fragile mechanical derivations should be handed over to deterministic executors? Has the Controller learned to perform reliable and efficient scheduling across internal scratchpads, external file systems, code interpreters, and collaborative sub-models?

LLMs have not suddenly grown a new brain. The so-called reasoning revolution is an engineering evolution that transforms autoregressive context into a scratchpad, turns neural network post-training into a Controller, and integrates external tools and state management into system scheduling. What history truly rewards has never been a dramatic buzzword, but those who systematically turn simple intuitions into reliable systems step-by-step.