Over the past few days, an article with headlines like “OpenAI’s collapse” and “a trillion dollars of compute wasted” went viral on Chinese social media. It tells a complete story: OpenAI’s 2020 scaling law paper had a bug, the whole industry sprinted down the wrong road for two years, GPT-3 was a “bloated” product of that error, and even the English language itself is inefficient. The story’s final piece of evidence: a researcher found that with the same compute, a French model reaches a certain level of grammatical ability 50 to 100 times more efficiently than an English one.
Follow the article’s own links to trace that claim, and you will find it appears in no paper, nor in the body of the blog post the article cites. It is a reader comment under that blog post, published the same day the Chinese article went out. The commenter is an independent researcher, and the claim has no peer-reviewed version. Meanwhile the only peer-reviewed study that directly compared training efficiency between English and French (PAGnol, LREC 2022) points in the opposite direction: French, with its richer morphology, is less compute-efficient than English. A same-day comment from a blog’s comment section was thus promoted into blockbuster research that “measures how much compute English wastes.” That detail alone should calibrate your trust in the article.
The historical events it references, however, are real: scaling laws have indeed been revised three times in five years. That history is far more interesting than any “scam,” because the three revisions did not change the same thing. The first fixed the measurement; the last changed the question itself. Once you see that, you can also answer a question everyone using AI cares about: will there be models with similar intelligence but far fewer parameters?
The name “scaling law” sounds like physics. The question it actually answers is much more modest: given a fixed compute budget, do you get more for your money by making the model bigger, or by feeding it more data? Training cost is roughly parameters times tokens, and the budget only covers one combination, so there is a trade-off to make.
In 2020, Kaplan et al. at OpenAI fitted a curve from hundreds of small-scale experiments and concluded that models come first: as compute grows, most of it should go into parameters, with data growing only slowly. That curve directly shaped GPT-3: 175 billion parameters trained on just 300 billion tokens, fewer than 2 tokens per parameter. For the next two years, every flagship model followed this recipe and piled on parameters.
In 2022, DeepMind’s Chinchilla paper refitted the curve with larger-scale experiments and arrived at nearly the opposite ratio: parameters and data should scale together, at roughly 20 tokens per parameter. They trained the 70-billion-parameter Chinchilla accordingly, and with the same compute it beat their own 280-billion-parameter Gopher across the board. A quarter of the size, four times the data, better results. By this standard, the GPT-3 generation was indeed undertrained.
Two top labs, one question, answers ten times apart. What happened in between? This is where the “bug” narrative comes from.
In 2024, two independent teams ran factor-by-factor replications, and the answer is now fairly clear. Porian et al. identified three main factors: Kaplan’s compute accounting omitted the output layer, small models were given disproportionately long learning-rate warmup, and optimizer hyperparameters were not tuned per scale. Pearce and Song added another: Kaplan counted non-embedding parameters only, and embeddings are a large fraction of small models. Correct these items, and Kaplan’s own experimental framework reproduces Chinchilla’s conclusion almost exactly. In other words, any team running that study under 2020’s conditions would have obtained the same biased curve. As for the most widely circulated explanation, that the cosine learning-rate schedule was the culprit, it happens to be the one hypothesis the experiments rejected: Porian et al. found that even a plain constant learning rate reproduces the Chinchilla law.
More interestingly, the correctors got corrected too. In 2024, Besiroglu et al. re-examined Chinchilla’s data and found a numerical problem in one of its three fitting methods; the Chinchilla authors publicly acknowledged the mistake. After the fix, the new result actually aligned with the other two methods, making the 20:1 conclusion more solid than before.
There is no villain in this chain of corrections. Diogo Almeida, who brought the word “bug” into public view, worked on LLM optimization at OpenAI at the time. In his blog post he admits he missed the problem too, and what he asks for is a corrective note on the original paper. Search these replication papers and you will find no “mislead” or “fraud”; the words in their titles are “reconciling” and “resolving.” Kaplan 2020 is still called seminal work in all of them.
The story should have ended there: the correct ratio was found, everyone trains at 20:1. What happened after 2023 was the opposite. The entire industry moved away from Chinchilla, and kept moving further. Meta fed its 8-billion-parameter Llama 3 15 trillion tokens, nearly 1900 per parameter, more than ninety times the Chinchilla ratio.
The reason is not in the math. The question changed. Chinchilla optimizes the compute of a single training run; but a trained model goes into service, answering hundreds of millions of requests a day, and inference costs accumulate far beyond that one training bill. When Sardana et al. added inference cost into the equation, the optimum shifted wholesale: small models are cheap to serve, so paying extra training compute for a smaller model pays off once request volume is large enough. Kaplan, Chinchilla, Llama: none of the three recipes is right or wrong. They solve three different problems. The first mismeasured the problem’s conditions, the second fixed the measurement, and the third replaced the problem itself.
Once you see Act Three, the opening question answers itself: models with similar intelligence and far fewer parameters are not something to wait for. This has been happening for three years, and someone has quantified the pace. The Densing Law, published in Nature Machine Intelligence by a Tsinghua team, surveyed recent open-source models and found that the parameter count needed for a given capability halves roughly every 3.5 months. An 8-billion-parameter open model today roughly matches a flagship of tens of billions of parameters from two years ago. Three forces drive this: Llama-style overtraining far beyond the ratio, steadily improving data quality, and distilling large models into small ones.
Miniaturization does have a floor. The Physics of LLMs series measured that a language model stores roughly 2 bits of knowledge per parameter. Reasoning style can be compressed through distillation, but knowledge capacity is hard-bound to parameter count; a model that is too small simply cannot remember that many facts. So the more likely picture is a division of labor: everyday tasks go to ever-smaller models, frontier capability stays with the largest ones, and the frontier itself keeps moving.
Looking back over these five years, the most misleading thing about scaling laws is the name. They are not derived from first principles; they are curves fitted to experiments, and the exponents move whenever the experimental conditions move. An ICLR 2025 systematic survey of 51 scaling-law papers concluded that these fits are extremely sensitive to methodological choices, and that most papers underreport the details needed to reproduce them. Only by mistaking such empirical curves for laws of physics would anyone read every coefficient revision as a “collapse.”
The same history can be told two ways. “The scientific community spent four years calibrating an empirical curve layer by layer” gets no retweets. “OpenAI scammed the whole world in three steps” gets a hundred thousand views. The viral article also cites Lilian Weng’s late-June review, without mentioning its title: Scaling Laws, Carefully. Carefully. That word is more accurate than any rebuttal.