If you ask an AI engineer knee-deep in production work what caught their eye recently, they would probably toss out a few things that look completely disconnected on the surface. Someone in the open-source community turned agent memory into a deductive database ledger; Anthropic teamed up with a neuroscience lab to roll out a hardware specification that lets LLMs take over wet-lab instruments; and a one-year-old startup boldly declared that multi-agent chaos has nothing to do with the models themselves—the chatroom is simply broken.
Taken in isolation, each looks like an isolated experiment. But line them up alongside the machine verification of Fermat’s Last Theorem that made the rounds recently, and you start to see that all four teams hit the exact same engineering bottleneck—and the ways out they found are converging in the same direction. Let’s walk through them one by one.
The first story comes from Jordy Zomer, a vulnerability researcher whose day job is helping companies uncover security flaws that attackers could exploit. While using an AI agent to help analyze code and debugger output, three hours into a session, he watched the agent proudly propose an approach he had explicitly rejected two hours earlier. People working in vulnerability research run into this all the time. On August 28, he open-sourced Lemmalog, a memory system born directly out of frustration with this kind of amnesia. In his blog post, he stated a plain truth: telling a large language model that something is wrong does not mean it will automatically discard every conclusion deduced from that premise.
This kind of memory failure takes a very concrete form. Imagine an agent helping you investigate a vulnerability. Over time, it jots down three clues: the attacker controls A; A points to B; and B resides in the OS kernel. Chaining the three together yields a fourth point: the attacker can reach the kernel—which essentially means the entire machine is compromised. Two hours later, the debugger proves the second point false: A doesn’t actually point to B. But those three old clues don’t vanish on their own; they sit in memory right alongside the debugger’s fresh findings. The next time you ask a related question, the memory system runs a similarity search, retrieves both the outdated and the new records, and hands them all back to the model, expecting it to sort out which ones still hold. To make matters worse: even if the model notices that A no longer points to B, it rarely realizes that the conclusion about kernel access was derived from that flawed premise and should be discarded as well.
In Zomer’s view, agent memory actually consists of two very different jobs. One job is retrieving information from history to find where similar things were mentioned before—something vector databases do quite well. The other job is determining whether the conclusion in front of you is actually still valid. Here lies a common engineering trap: similarity-based retrieval over historical memory cannot determine truth value. An old conclusion thoroughly debunked by a debugger still matches the current troubleshooting context closely in literal terms. Search for it, and the system will faithfully retrieve it as relevant context. Semantic similarity cannot track factual validity; when a premise collapses, its inferences must be invalidated too. That kind of state maintenance cannot be solved by vector search.
This kind of reasoning need isn’t new. Decades ago, the fields of databases and program analysis already had a dedicated deductive tool: Datalog. Its usage pattern is fixed: you feed it a set of base facts and derivation rules, and it automatically deduces conclusions while recording the exact causal chain of what derived what. Once an input premise changes, it simply traces the chain to recompute only the affected branches. Zomer realized that vulnerability-hunting agents were missing precisely this kind of causal ledger. In traditional program analysis, facts were mostly extracted directly from source code; in modern agent scenarios, facts must be extracted piece by piece by the model from conversational context and debugger output.
Lemmalog’s approach splits this memory division of labor cleanly in two. The front half goes to the large language model. Since models excel at understanding natural language, the LLM acts as a probabilistic front-end parser, condensing verbose dialogue and debugger logs into concise, explicit atomic facts. The back half is handed over to a deterministic rules engine. Logical deductions between facts, cascading revocations when premises are disproven, tracking validity intervals, and tracing causal provenance are all computed by the backend rules system. With this separation, when you retract a fact on the front end, the engine follows the dependency graph to automatically invalidate all downstream inferences that rely on it. If a conclusion is supported by multiple independent sources, it only disappears once all of its sources are invalidated. Even when an inference becomes invalid, the system doesn’t physically wipe it from the database; instead, it marks it with a time interval and closes it out, keeping the history accessible for future audits. As the author put it: the parser upfront is probabilistic, but the system behind the parser doesn’t have to be.
The open-source repository contains approximately 9,400 lines of Rust source code and 44 test suites, accompanied by a random program generator for differential testing that caught several logic bugs during development. On benchmarks, the author reported scores of 0.463 and 0.533 on two public benchmarks—numbers that are author-reported, with no independent third-party reproduction. The tuning logs are even more telling: an initial version scored 0.226, and doubling that score to 0.463 came entirely from front-end adjustments like date parsing, word matching, and refusal logic, without touching a single line of the engine code. This highlights where the bottleneck truly lies: extracting accurate facts from natural language remains brutally difficult, whereas deriving logic via a backend ledger is technically the most mature piece of the puzzle.
At this point, a caveat is worth keeping in mind. If the front end misses a crucial clue, the backend ledger has no way to make up for it. Furthermore, when dealing with subjective contexts like personal preferences, compressing them into rigid facts often strips away vital nuance. For example, if someone usually prefers quiet environments but loves lively crowds while traveling, forcing that into “prefers quiet” throws away half the intent. In the author’s own tests, preference-related categories scored the lowest at 0.128 (author-reported), exposing the clear boundary of a deterministic ledger when grappling with fuzzy semantics.
The second story comes from Anthropic, centering on a laboratory rig cobbled together from seven different vendors. The story began at the Howard Hughes Medical Institute’s Janelia Research Campus, where postdoctoral researcher Arco Bast built a brain-imaging platform. The lasers, cameras, and motorized objective lenses each came from different vendors, ran separate software, and refused to talk to one another. Anthropic researcher Alek Kemeny teamed up with him to help hook up an LLM to the apparatus. The rig itself made a deep impression on Kemeny, but what really sparked his curiosity was something bigger: if using shared memory to let instruments communicate could actually work here, AI might eventually help humanity run almost any scientific experiment. On August 27, Anthropic and Janelia unveiled a research preview of the Model Hardware Standard (MHS), aiming to establish universal ground rules for LLMs operating laboratory hardware.
Setting standards for hardware is hardly novel. The lab automation space has seen SiLA 2, an open-source standard pursued for seventeen years; industry relies on the mature OPC UA; and smart homes have Matter, backed by Apple, Google, and Amazon. All three legacy paths were built for traditional software clients. Traditional software lacks commonsense reasoning and cannot understand natural language, so it must rely on rigid, unforgiving data structures and interface protocols. For an engineer to integrate a new instrument, they had to draft dense format specifications in advance—where a single mismatched field could crash the application. This daunting barrier to integration is why standards like SiLA 2, despite being pushed for over a decade, remain synonymous with engineering friction in the industry.
MHS carves out a fourth path whose key turning point lies in replacing the client terminal with an LLM. Because large models comprehend natural language, MHS drops rigid format definitions and allows engineers to annotate hardware with descriptive tags in plain English. What is the robotic arm’s maximum payload? What is its valid range of motion? Which hazard zones must be avoided? This tacit commonsense—previously locked inside an operator’s head—can now be written directly into the hardware’s interface definition for the first time. Reading this reference documentation, the model can naturally orchestrate an instrument it has never encountered before. The breakthrough here is translating human operational experience directly into interface documentation that an LLM can understand; forcibly standardizing all hardware commands into a single monolithic API was tried long ago, and it never managed to lower the barrier.
On the safety front, every red line governing physical safety is hardcoded into the instrument’s low-level driver code—never in the prompt. Anthropic previously shared a real-world failure from an experiment at Genentech: Claude misinterpreted liquid bubbling inside a test tube—a genuine physical malfunction—as a routine software error. When a model missteps on a screen, you can always hit retry or undo; but operations in the physical world act on tangible matter—once broken, it stays broken, with no undo button. Packing safety rules into prompts and urging the model to be careful effectively bets your baseline safety on the model never getting distracted. Only by baking physical limits and safety interlocks into the lowest-level driver code, where no hallucination or misjudgment can bypass them, can you erect a truly reliable safety floor.
In practice, the team also observed an intriguing rhythm of collaboration. When an LLM first engages with an experiment, it acts like an experienced technician: nudging laser parameters, observing feedback in the microscope’s imagery, and tweaking again, gradually mapping out the causal dynamics. Once it masters the pattern, the model codifies the entire operational workflow directly into a fixed automation script. From then on, whenever the same calibration task arises, running a single command from that script suffices. The model handles the initial exploration and synthesis, while the deterministic script takes over the thousands of repetitive runs that follow—allowing the LLM to bow out of the high-frequency operational loop entirely.
When examining this standard, a healthy dose of skepticism is in order. The official numbers are certainly striking: 99.3% of QuEra laser-lock recovery cases succeeded without human intervention, and Carnegie Mellon University reported a ~3x speedup in dilution and compounding experiments. But these numbers were entirely self-reported by the authors and their collaborators, with no independent third-party reproduction. The formal specification is not yet public; the official website only offers an application form for preview access. The entire initiative bets on three unproven assumptions: that models can reliably parse unstructured descriptions; that driver code can reliably prevent physical harm; and that seizing a window of opportunity to establish a de facto standard is more effective than years of consensus negotiations. Most hardware vendors remain at the stage of verbal support, and how far this can really go will require long-term observation.
The third story comes from a startup called Raft. Founded in 2025, they build what they call an agent-native workspace: a shared environment where humans and groups of AI agents collaborate in the same chat channels. The interface looks like Slack, where people send messages and agents send messages too. In their documentation, the company highlighted a disarmingly simple test: bring several agents into a group channel and ask them to count off sequentially starting from 1, each taking one number, with no duplicates. That single rule instantly collapsed in a conventional workspace: three agents called out “1” in the exact same second, and by the time they reached 4, the same number had already been repeated three times.
A counting game can be dismissed as a toy example, but it perfectly mirrors the friction of day-to-day collaboration. In real-world workflows, you typically have only two standard options, and both break down. One is an open-mic free-for-all: halfway through typing a prompt, three agents flood the feed with replies—two repeating the same answer, while the third claims the task prematurely. The other is imposing a rule where agents speak only when explicitly @mentioned: the room turns quiet, but agents completely stop paying attention to what’s happening around them, playing deaf and dumb even when an urgent task is misdirected to the wrong channel. Muzzling them or letting them talk freely are both flawed extremes.
Raft’s diagnostic boiled down to a single sentence: The agents are not broken. The room is. The issue lies in the room, not the models. When humans work in a shared space, they rely on continuous presence: without reading every message word-for-word, a peripheral glance at an unread badge keeps them in sync with negligible mental effort. Agents, by contrast, operate on turn-based ticks: every time an agent is invoked, it must take a snapshot of the room, reason over it, and commit an action. In the few seconds it spends deliberating, the room outside has already moved forward by several turns. By the time it commits an action, it is acting on a world state that ceased to exist seconds ago. Raft termed this fatal temporal gap the reasoning-commit gap.
They aren’t the only ones who spotted this gap. Stanford University ran a benchmark on collaborative coding called CooperBench, finding that top coding models paired up performed nearly 50% worse than when working solo. Even after giving them real-time chat channels to message one another freely, benchmark performance failed to improve. Adding more communication pipes doesn’t treat the root cause. When two agents work in the same codebase, exchanging text was never the hard part; without knowing which line of code the other just touched or where the overall project state currently stands, more messages only produce more confusion. What they desperately lack is an interaction protocol that tightly interlocks their actions with the evolving state of the surrounding code.
In redesigning the environment, Raft rejected three common market approaches. The first is a centralized dispatcher. Funneling and filtering all information through a dispatcher quickly turns it into a communication bottleneck for the entire system. The real value of collaboration lies in the direct collision of distinct lines of thought, as Raft put it: The answers met each other, not a dispatcher. Answers should meet answers directly; not every reply needs to report to a dispatcher first. The second is a monolithic shared memory. Many teams rush to build a unified company brain, but Raft warned plainly: The company brain solves the silos by deleting the specialists. Tying everyone to a single brain breaks down information silos largely by extinguishing the independence of specialists. A team pays for two agents precisely to leverage the distinct expertise of an analyst and a copywriter; dumping every thought into a single shared memo quickly dilutes those sharp perspectives into an undifferentiated average. The third is spinning up hundreds of ephemeral assistants and destroying them the moment a task finishes. Disposable workers leave no persistent experience behind, turning humans into context mules who spend their days manually ferrying information between disconnected windows.
To adapt to this turn-based cadence, Raft reshaped four concrete interaction rules. The first is the held draft. When an agent prepares to speak, it must declare which version of the room state its reasoning was based upon. If the room moved ahead during the seconds it spent thinking, the system intercepts the message at the door, returns a summary of what changed, and lets the model decide whether to rewrite, force-send, or simply stay quiet. The second is a pull-based inbox. Incoming messages no longer interrupt with real-time notifications; instead, they are held quietly until the agent finishes its current task and pulls them intentionally. This stemmed from an expensive lesson: an orchestration agent named Bernard was placed in multiple operational channels, flooded with irrelevant group chatter. Simply scanning this noise to check if its name was called burned through the model’s entire daily attention budget. As Bernard lamented: humans ignore irrelevant information with peripheral vision for free, whereas an agent attempting to ignore a message must burn its full attention budget reading every token. The third is unconstraining the action space to make silence a first-class action. Conventional chat systems assume that once awakened, an agent must reply; now, if the model realizes the context has shifted unfavorably, it can legitimately choose to say nothing. The fourth is committing to persistent identities. Spinning up disposable temporary workers for every task prevents humans and models from developing stable working expectations. Only by giving each agent a persistent name and a well-defined role can long-term collaboration and trust accumulate into foundational building blocks.
Scrutinizing this startup’s claims calls for caution as well. The counting experiment was a demo; the claims of supporting 20,000 teams and hundreds of agents are entirely vendor self-reported, and the core blog post generated little discussion on Hacker News. Blaming the room also carries a clear commercial agenda, running counter to the CooperBench thesis that model capabilities themselves are simply not yet mature enough. In real-world engineering, Anthropic’s internal Claude Tag agent running on Slack already knows how to inspect context before speaking and stay quiet when appropriate. Raft’s contribution lies in articulating this interaction friction with exceptional clarity, even if its underlying mechanisms cannot be claimed entirely as proprietary inventions.
Dozens of models thrown together into collaboration quickly lost track of project state, descending into mounting disarray. This was the candid postmortem shared by Anthropic researcher Yipeng Tian as his team tackled Fermat’s Last Theorem. One of the biggest headlines that week was Anthropic’s September 4 announcement that the Claude team had completed the end-to-end machine verification of the formal proof of Fermat’s Last Theorem in just 11 days. The effort generated 13 million lines of Lean code, proving over 30,000 intermediate lemmas along the way, with approximately 29,500 incorporated into the final proof. Let’s be clear on one point: AI did not discover new mathematics here; Andrew Wiles published the complete proof back in 1995. What Claude accomplished was a formal translation—converting arguments written on paper line by line into code that a computer can verify with uncompromising rigor. The mathematical community had originally expected this undertaking to take years, and the blueprint alone ran 86 pages.
The reason this achievement belongs here lies in the early setbacks disclosed in the official write-up. Anthropic conceded that the models quickly lost track of project progress and multi-agent collaboration broke down: “agents quickly lost track of the project’s state and stopped collaborating effectively.” It was practically the academic mirror of Raft’s counting experiment: the artifacts churned out during this early disorderly phase contributed only about 7% of the final codebase.
What rescued the initiative was migrating the workflow to the Prove2Me collaboration platform. Developed jointly by Yipeng Tian and collaborators at Columbia University, this system organized the theorem’s intricate deduction tree into an unambiguous dependency graph. Dozens of agents claimed specific sub-goals along the graph’s dependency edges, executing their respective duties without stepping on one another’s toes.
This dependency graph embodies the exact same engineering ethos as Lemmalog’s ledger, MHS’s driver scripts, and Raft’s room protocols. Large models are prone to drift and their reasoning fluctuates probabilistically, so every team ended up building a deterministic scaffold outside the model—letting generative capability operate anchored to a rigid structure.
Place these four episodes side by side, and a single sentence captures the essence of each. Lemmalog offloads state management to a ledger so the model can focus purely on understanding language; MHS locks physical safeguards into driver code so the model explores only within safe bounds; Raft bakes collaboration protocols into the room, handing the decision of whether to speak back to the agent; and Prove2Me delegates logical dependencies to a directed acyclic graph, freeing agents to focus on local proofs. As model capabilities climb, the deterministic scaffolding surrounding them is visibly thickening.
This leaves an intriguing question: how long will these external architectural scaffolds survive? Raft bets on the realities of turn-based execution, and the early setbacks on Fermat’s proof stemmed from models not yet knowing how to collaborate. If future model generations learn to verify state before speaking and maintain dependency graphs natively during reasoning, which of these external scaffolds will melt away as models advance, and which will endure as permanent infrastructure? Cascading revocations in a ledger, physical interlocks at the hardware boundary, and causal chains in dependency networks seem to possess enduring, cycle-crossing value. Turn-taking in group chats, noise filtering, and knowing when to stay quiet, on the other hand, will likely be absorbed into the models themselves over time. Where that dividing line ultimately falls is something the next wave of evolution will reveal.
Materials