Let a large language model “pick a random number between 1 and 10,” ask it 20 times in a row — what do you think would happen?
Many technical readers’ first reaction is: since the model uses probabilistic sampling when generating text, even with a very low sampling temperature, 20 responses should scatter across several different numbers, just like rolling a die 20 times in a row.
Take a moment to choose your predicted result in your mind, then look at what we actually measured.
In day-to-day API usage, developers have long grown accustomed to treating large models as some kind of probabilistic system. When an endpoint occasionally throws an error or generation quality degrades, people can also guess that some brief failure has occurred in the infrastructure. The problem is, apart from waiting for the official status page to update or discussing “is the model dumber today” in the community, developers lack a low-cost, long-running observational tool to confirm whether the underlying inference stack of the API has actually changed.
If there were an observation method that only consumes very few tokens each day yet can sense the state of an API endpoint like a thermometer, what would it look like? The answer may just be hidden in those seemingly extremely boring random number questions.
On the first day of testing, we ran this experiment against the
gpt-5.6-sol model’s API. The results were remarkably
decisive: when asked to pick a random number between 1 and 10 for 20
consecutive times, the answers were all uniformly 7.
To confirm whether this was merely an incidental phenomenon from one specific prompt, we expanded this test into a “100 small questions per day” routine, consisting of 5 groups of different types of minimal prompts, each repeated 20 times:
7.Q.47, 4 were 57, and 1 was
73.7 and 1 was 42.These 100 small questions demonstrate the concrete shape of low-entropy short responses. They reveal that when facing seemingly free choices, the model actually carries extremely strong intrinsic preferences, rather than spreading out evenly.
There is also a nuance here: prompts like picking a random number from 1 to 100 (yielding three results — 47, 57, 73) and coin flips (yielding both heads and tails) show that the exact same prompt produced different text across multiple calls. This phenomenon weakens the hypothesis that “the gateway is simply doing static Response Cache,” though it does not yet fully ascertain exactly how the caching mechanism within the service stack works.
This concentrated preference distribution is not a quirk of a single model or a chance anomaly.
The 2026 paper One Token Is Enough tested 165 large language models. The results show that on extremely short responses, the median entropy of the response distribution across different models is only about 1.0 bit. In other words, when facing free choices, models are innately wired to concentrate most probability mass on very few options.
When we collect the response frequencies of these 100 questions every day, how do we compare distribution changes between two consecutive days or two time periods? The paper uses Jensen-Shannon divergence (JSD) as the metric. It is essentially a number that measures how dissimilar two probability distribution tables are: a value close to 0 means the distribution barely changed; a larger value means the response preferences have shifted.
The paper further validated the sensitivity of this distribution to changes: on a test set containing 40 question groups, using response distribution alone to distinguish different models achieved an Equal Error Rate (EER) as low as 7.3%; even when compressed to 8 question groups, the error rate was only around 10.6%. However, it should be made clear that the paper studied using distribution fingerprints to identify different models, while we are concerned with a different direction: on the same API endpoint, using this sensitive distribution change to detect behavioral drift of the inference stack.
Why repeat asking these 100 boring little questions every day?
Looking back at past technical incidents, for example, Anthropic once confirmed that infrastructure issues led to an increased error rate for Claude models, and many developers intuitively felt at the time that “output quality had degraded.” But between users and providers there lacked a shared, low-cost, and quantitative set of observational data. After a service adjustment took place, everyone could only guess by feel, and no one could pinpoint exactly which day the change started.
The service stack of large model APIs encompasses many components, from bottom-level sampling parameters and routing rules to upper-level safety filtering or inference protocol adjustments — any minor change has the potential to alter the model’s output distribution on low-entropy prompts.
Asking 100 small questions every day is like hanging a lightweight thermometer next to the API endpoint. While a change in the preference of a single small question cannot directly pinpoint what specific failure occurred at the bottom layer (it signals change but does not attribute), this low-cost time series can first capture unexpected drifts in the inference stack, alerting developers when to initiate deeper diagnostics.
To turn the distribution properties revealed in the paper into a persistent long-term observational practice, we established the Codex Behavior Today project and launched a public monitoring dashboard at Dashboard.
The project’s operation maintains a clear separation of privacy and data: - The local sampler automatically initiates 100 minimal prompts against the target endpoint each day and keeps data containing detailed latency and raw responses in a local SQLite database, requiring no credential sharing or public disclosure of private conversations. - Each run only submits aggregated response counts, latency summaries, and distribution difference metrics to Git, which are then automatically rendered into a public dashboard via GitHub Pages.
Whether replicating the dashboard or extending observations, the
entire workflow is relatively lightweight: developers simply clone the
repo, use uv venv .venv to create a virtual environment and
install dependencies, configure the local auth CLI, and they can run the
sampler and site build. Even without API credentials, one can directly
use the open-sourced daily aggregate data in the repository to
regenerate the site and verify the tests. I also welcome community PRs
that extend this observation mechanism to more API endpoints.
Finally, we need to calmly and objectively look at the boundaries of this thermometer.
A change in the distribution of low-entropy short responses only indicates that the service stack behavior of that API endpoint has shifted compared to the historical baseline. It does not equate to “the provider secretly swapped the model weights,” nor does it mean the model’s capability has degraded.
The 100 samples from the first day (2026-07-22) recorded our first initial observation point. Over the next 7 to 14 days, the ongoing public sampling experiment will gradually establish a stable historical baseline. Only after this period of observation can we verify whether this set of low-cost boring questions can truly become a keen and reliable canary in the coal mine for the large model API domain.