Consider an engineering scenario (an illustrative example). You task a software agent with optimizing the throughput of an online inference service. It tweaks a few configuration parameters and launches a stress test, only to see throughput drop by 20%. Confronted with the worse numbers, it modifies another batch of functions and reruns the test, but after several iterations, it still cannot figure out where to look. The issue is not that the agent lacks execution capability; rather, the test environment returns only a coarse, aggregate end-to-end score after the test completes, leaving it with no way to deduce what actually broke in the code.
When troubleshooting system performance bottlenecks in day-to-day engineering, engineers rarely just stare at total latency on a benchmark dashboard. When throughput drops, they typically pull up timelines of compute and communication first to see whether the two overlap; if they suspect an error in computation logic, they compare calculated values against a reference implementation item by item to quantify the discrepancy; if they suspect a hot loop is dragging down the whole system, they isolate it and run a microbenchmark to measure local latency. If these diagnostic tools are not packaged for automated systems, handing over the entire repository to an agent will leave it unable to troubleshoot when scores degrade.
In Z.ai’s post-mortem published on September 17, 2026, the GLM-5.3-powered Infra Agent encountered precisely this situation. The task at the time was to deploy the GLM-5.3-Flash large model into production on a domestic chip cluster. The vendor reported that the cluster comprised over 100,000 domestic accelerators, handling all production inference workloads for the model. Engineers set performance targets and system boundaries, the Infra Agent proposed hypotheses and modified operator implementations and communication code, and the test environment supplied layered feedback, iterating until passing acceptance. The vendor stated that the system took less than two weeks from its first successful run to production readiness, reaching approximately 3x end-to-end throughput compared to the initial baseline.
What kept this engineering loop moving forward was the feedback mechanism built by engineers. After modifying code, if the test environment returns only a degraded aggregate performance score each time, the agent cannot tell whether the issue lies in compute kernels, communication libraries, or concurrency scheduling. As Z.ai wrote in its post-mortem: “End-to-end metrics can tell an agent that results got worse, but they cannot explain why.” End-to-end metrics only indicate that outcomes degraded; they cannot explain the underlying mechanisms. Understanding where an agent stalls in engineering tasks and how it breaks through is often far more valuable than simply looking at final benchmark scores. Behind this lies a simple principle: measure before optimizing. Inferences based on measurement data are always more reliable than guesswork; this holds true for humans, and holds even more true for AI, because agents depend even more heavily on the information fed to them by their environment.
When troubleshooting system faults, engineers typically follow a clear tiered mental model: to verify calculation correctness, first compare against a reference implementation; to locate latency bottlenecks, first pull up timelines to see at which handoff point compute and communication stall; to evaluate whether hardware resources are saturated, first run microbenchmarks. Organizing these troubleshooting methods—originally invoked through human experience—into a toolchain that gives automated systems multi-dimensional, immediate feedback after code changes is what the vendor terms dense feedback. The engineering methodology built around this core is the feedback engineering referenced in this article’s title. If an agent lacks such diagnostic tools when a stress test misses its target, its field of view remains trapped between static code and an isolated final score.
The core division of labor in this craft lies in separating acceptance from diagnosis. End-to-end scores answer whether the system improved; diagnostic feedback answers which layer the fault is stuck at and where the hypothesis failed. An agent armed only with scores ends up paralyzed—knowing it failed but unable to find out why; a system with only local diagnostics and no global acceptance easily gets lost in localized tweaks that drift from the objective. Only when both mechanisms work together can an agent establish causal chains across complex low-level stacks.
The dense feedback system emphasized by the vendor has three hard requirements. First, feedback must indicate the approximate location of the problem. Rather than abstractly reporting a drop in accuracy, the system should directly output numerical discrepancies before and after changes for specific inputs, guiding the agent to narrow its investigation to concrete code paths. Second, feedback must be low-cost and fast. Questions that a few-second operator test can answer should not require multi-hour cluster stress tests; short-cycle verification helps the agent rapidly eliminate false hypotheses. Third, feedback must support objective verification. Whether a code change is effective must be determined by comparisons with reference implementations, controlled tests, and quantitative metrics, rather than relying on the agent’s subjective assertions. As the vendor noted in its post-mortem: “correlations between observations alone cannot establish a root cause.” Runtime signals offer only clues; causal confirmation still requires controlled comparative experiments.
Under this system, human engineers focused their efforts on three critical areas: defining optimization objectives and system boundaries, building feedback environments accessible to the agent, and conducting human reviews on key changes involving numerical compute semantics, asynchronous concurrency, and production risks. The vendor summarized this division in the post-mortem: “Engineers defined objectives and system boundaries. The agent handled analysis, hypotheses, and code changes. The experimental environment provided layered, timely, and verifiable feedback.” (Engineers set goals and system boundaries; the agent handled analysis, hypotheses, and code changes; the experimental environment provided layered, timely, and verifiable feedback.) With this division of labor, responsibilities on both sides become clear: engineers govern direction and safety baselines, while the automated system runs autonomously within the engineered verification environment.
The hardest bugs in inference systems rarely live in a single line of code; instead, they emerge from the interaction of multiple component layers—each layer appears normal in isolation, but errors surface when they are stacked together. During the development of the GLM-5.3 infrastructure, three such cases arose, serving as ideal illustrations of how feedback engineering is applied in troubleshooting. All three followed the same path: set objectives, observe phenomena, formulate hypotheses, fix, and verify.
The first case involved the correctness of computational results. In long-context inference, the system objective was to ensure that outputs from sliced computations in Context Parallelism matched baseline numerical values without slicing. In comparative testing, the agent observed an anomaly: output error in the sliced parallel path exceeded allowable tolerances and amplified steadily as context length grew. According to the vendor’s account, it hypothesized that the error stemmed from precision loss during cross-slice state transfer and reduction, where the underlying matrix multiplications defaulted to a low-precision mode.
Tracing down this hypothesis, that was indeed where the problem lay. During cross-slice state reduction in the KDA operator, two consecutive multiplications defaulted to TF32 execution mode. The fix explicitly specified both operations as input_precision=“tf32x3”, using three combined TF32 hardware operations to approximate higher-precision floating-point accumulation and reduce accumulated error. Local regression tests subsequently confirmed that output error dropped back within acceptable tolerance. The fix was subsequently submitted to the open-source project Flash Linear Attention and merged; what was merged is a precision path disabled by default, requiring explicit activation. This fix ensures numerical correctness rather than execution speed, and cannot be credited toward end-to-end performance gains.
The second case involved troubleshooting a cross-layer concurrency bottleneck. In long-context and multi-turn conversational inference, attention states generated by preceding computations must be transferred across network channels directly to subsequent compute nodes, saving them the overhead of recomputation. This mechanism for transferring key-value caches across nodes is commonly known as KV Transfer. The engineering objective was to overlap data transfer concurrently with prefill-stage compute, keeping the performance gap after introducing KV Transfer within 5%. Benchmarks showed that the performance gap exceeded 20% (vendor-reported), lagging substantially behind the target.
Looking solely at throughput numbers, one might easily guess that the issue was network bandwidth saturation or an inefficient transfer engine. According to the vendor, the agent pulled up execution timelines and spotted an anomaly: in the benchmark scenario, Python-side transfer tasks and low-level communication scheduling never overlapped on the timeline. This indicated that the bottleneck likely lay not in the network itself, but in higher-level tasks failing to dispatch in time. Tracing down the call chain, the agent pinned the issue to the language boundary between Python and C++.
The roadblock was runtime lock contention. The Python runtime relies on the Global Interpreter Lock (GIL) to manage multi-threaded memory safety. If a worker thread enters a low-level C++ communication extension without releasing this lock, other Python threads are forced to queue up and stall while waiting for it. In the version of DeepEP used, the C++ functions for intra-node dispatch (such as intranode_dispatch and intranode_combine) did not release this lock, whereas the inter-node dispatch function had already released it long ago, with source code comments explicitly stating that failure to release would block other threads performing KV Transfer. The fix was to explicitly release the lock within the C++ scope of intra-node communication as well, allowing transfer threads to dispatch tasks concurrently.
Retesting the timeline after releasing the lock, transfer tasks overlapped smoothly with compute, narrowing the performance gap to under 1% (vendor-reported). This concurrency conflict had already been documented in the open-source community. DeepEP PR #142, merged on May 8, 2025, had resolved an interpreter lock issue when used alongside Mooncake; PR #555, targeting intra-node calls, was also submitted on January 4, 2026, and remains open. In our assessment, the agent’s contribution lay in localizing, porting, and validating the fix on a specific cluster, rather than discovering the defect for the first time. The vendor’s post-mortem also conceded: “correlations between observations alone cannot establish a root cause”; timeline tracing ruled out false hypotheses, while final causal confirmation depended on end-to-end validation following controlled fixes.
The third case centered on single-operator performance tuning and system-level trade-offs. The principle of measuring before optimizing is paramount here: a faster operator in a microbenchmark does not guarantee faster end-to-end performance; over-allocating hardware resources to a single operator can crowd out communication bandwidth and slow down the global pipeline. Every modification must therefore be confirmed in end-to-end stress tests. After introducing a replay mechanism (ReplaySSM) into the KDA decode operator, memory footprint dropped, but compute latency increased noticeably, with profiling revealing compute-bound operations as the primary bottleneck.
According to the vendor, the agent first rewrote the internal division logic based on this observation, reducing execution time by 9.6% (vendor-reported). It then discovered that tiling along the feature dimension caused floating-point normalization and gating computations to execute four times redundantly. It adjusted the tiling strategy, merging slices into a single thread block, keeping intermediate results in registers, and replacing redundant computations with warp-level reductions. By trading off a slight degree of parallelism to eliminate redundant computation, the operator achieved a 1.71x local speedup (vendor-reported). The modified operator was then returned to the end-to-end stress-testing environment to verify gains; the vendor stated that these learnings were subsequently consolidated into an optimization skeleton library for future reuse.
Using agents to optimize systems and decomposing verification methods already has precedents in industry. On September 11, 2026, Elastic wrote when introducing its automated optimization harness, atune: Benchmarks provide the verdict and profilers provide the gradient, noting that gradient information supplied by profilers is far more valuable than verdicts delivered by final benchmarks. Elastic constructed a verification ladder: sub-second probes quickly reject broken code, local tests filter environmental noise, two-way comparisons handle rigorous acceptance, and multi-hour full workloads serve as the final defense line—with costs escalating at each tier. In one experiment, an agent’s instruction consolidation caused a 26% benchmark regression; the harness used vectorization decompiler warnings to explain the cause to the agent in approximately 10 seconds.
Public records show that this paradigm of human-agent collaborative infrastructure optimization has already been unfolding across various teams. When OpenAI and Broadcom jointly announced the Jalapeño dedicated inference chip on June 24, 2026, they stated: “The same models served to users are helping improve the infrastructure used to run future models.” Models not only serve end users, but also assist in improving the underlying infrastructure that powers future models. With model participation, the chip completed its entire lifecycle from architectural design to tape-out in nine months. Earlier, DeepMind’s AlphaEvolve discovered a superior scheduling policy for Google’s Borg clusters, running in production for over a year and reclaiming an average of 0.7% of Google’s global compute; it also sped up a core Gemini kernel by 23%, reducing training time by 1% (this figure comes from official blog disclosures and cannot be directly compared under identical conditions to the GLM system’s throughput numbers).
Given these industry precedents, why does feedback engineering remain rare? Because prior discussions mostly asked how rigid the signal was, rarely asking what the signal actually communicated. In prior analyses covering the five-tier gradient of verification signals, bottlenecks in testing ground and exam supply, and the deconstruction of AlphaEvolve’s architecture, the core focus was on the certainty of verification signals: formal verification yields mathematical proof, while test suites yield pass rates. Yet no matter how high the certainty, it only indicates whether code is correct, not where the issue lies. A formal verifier throws only a rejection, and a full stress test produces only a degraded throughput number—neither indicates which layer to investigate next. Inference infrastructure spans hardware compute, cross-language glue layers, communication topologies, and concurrency scheduling—a classic zone of strong verification but zero diagnosis. The incremental value of feedback engineering lies in stringing together scattered profiling tools, execution traces, and correctness checks, enabling the agent to eliminate false hypotheses step by step and know what to investigate next at every turn.
Ultimately, this craft highlights the essential role of engineers in the age of automation. As writing and fine-tuning code increasingly shift to automated tools, the upper bound on engineering velocity is determined by who can build observation environments for agents that are fast, cost-effective, and capable of pinpointing root causes when things fail. Mastering this feedback design capability is what translates the execution power of automated systems into real system performance gains.
When approaching real-world engineering projects, teams do not need to wait for more capable future models; they can evaluate their existing verification environments today using three diagnostic questions. First question: What can each of our current verification tools answer, and what can they not answer? Taking the GLM system’s practice as a reference: end-to-end stress tests answer whether overall system performance meets specifications; execution timelines answer which concurrency handoff points consume time; and microbenchmarks answer under which data layouts operators run more efficiently. If any of these three tiers is missing, the agent falls into an inefficient cycle of blindly thrashing in place.
Second question: What is the cost and turnaround time of each feedback mechanism? Hypotheses that can be validated by a few-second operator test should not trigger a multi-hour full cluster load test every time. Designing a low-cost, agile feedback ladder allows the agent to test more hypotheses per unit of time. Third question: Which critical changes must human engineers personally guard? Changes involving numerical semantic conversions, cross-language concurrency safety, and production deployment stability must have human review gates, rather than allowing agents to merge directly into the main branch without review.
Yet when absorbing these engineering practices, one should maintain clear-eyed skepticism toward technical reports. First, all reported cases come from a single vendor’s technical post-mortem, carrying a tendency to selectively showcase successful examples. Second, the vendor did not disclose rigorous ablation experiments that removed diagnostic feedback and retained only final scores; the marginal contribution of diagnostic feedback to speedups cannot be statistically quantified in isolation. Third, the vendor-reported end-to-end throughput reaching approximately 3x of the initial baseline was the compound result of multiple combined techniques—weight quantization, mixed-precision caching, replay mechanisms, and pipeline slicing—and cannot simply be attributed to the agent’s standalone capability. Fourth, the concurrency blocking caused by the interpreter lock was a known engineering flaw in the open-source community; the agent executed concrete localization, porting, and verification, which should not be exaggerated as an original algorithmic discovery. If rigorous future benchmarks under identical compute budgets demonstrate that a score-only control group can achieve comparable speedups, this article’s assessment of the necessity of diagnostic feedback will need to be revised downward accordingly.
Distinguishing engineering mechanisms from promotional framing is itself a skill worth honing. When evaluating frontier engineering practices, one must strictly differentiate between two classes of evidence. One consists of concrete engineering mechanisms that can be verified line by line along open-source codebases and pull requests, such as the precision selection path in PR #1180 and internal locking logic within communication libraries. The other consists of composite metrics self-reported by vendors, such as reaching approximately 3x baseline end-to-end throughput or achieving production readiness within two weeks. By evaluating verifiable mechanisms separately from vendor-reported numbers, teams can extract genuine technical insights without being misled by marketing claims.
The GLM infrastructure post-mortem does not depict a sci-fi narrative of autonomous AI self-evolution. The vendor explicitly conceded in the article that “we have not yet reached recursive self-improvement,” while emphasizing that defining objectives, setting boundaries, and mitigating risks “remain human responsibilities” as a firm bottom line. The genuine practical value of this report lies in demonstrating how humans maintain control and steering over systems through feedback design as automated tools penetrate deeper into low-level engineering. As summarized in the text: “The model optimizes the system; the system runs the model.”
How much engineering labor software agents can automate depends on how clear an observational window humans can provide them. If an agent is still blindly thrashing in place after modifying code, don’t rush to swap in a larger model—first examine its verification environment.