In the past, testing AI code generation typically meant giving it a function and running a few unit tests. Now, agents have begun writing entire backends from empty directories. The question has changed with it: how do we know the whole thing actually works?
Take a push notification service. Alice creates an application — Bob shouldn’t be able to see it. After Alice deletes the application, its messages should disappear too. When the system rejects a bad request, the database must not silently accumulate dirty data. Each rule on its own is not complicated. But with the same set of endpoints, changing the user or the operation order changes what the correct answer should be. The first version of the tests can hardly cover every scenario.
BackendForge aims to solve exactly this problem. The researchers first wrote 7,250 tests, then had an agent specifically search for what those tests missed, ultimately adding 640 more. The test count grew by only +8.8%, yet the same batch of GPT-5.5 code dropped from passing 31/56 tasks to 16/56.
The code did not change — what changed was the question bank used for scoring. This is the insight BackendForge brings: when AI begins writing entire applications, benchmarks can’t simply scale up the problem size. The method of setting questions must also evolve from a one-shot written exam into an interview that keeps asking follow-ups.
HumanEval tests a single function. The model reads a requirement, writes a few dozen lines of code, and then runs unit tests.
SWE-bench places the model inside real code repositories. It must read the issue, locate the relevant files, and submit a fix patch. FEA-Bench tests at an even larger scale: the agent doesn’t just fix bugs — it must also implement a new feature spanning multiple files.
Now, the evaluation target has become the entire application. The model must handle databases, APIs, authentication, and authorization on its own, ultimately delivering a system that can start up and accept requests. BackendForge belongs to this generation of benchmarks.
Each time the task expands a level, testing gains another layer of difficulty. A function can typically be judged correct or not with a single call. An entire service requires a sequence of actions to be completed before you can tell what went wrong.
There are already quite a few benchmarks for complete applications. BaxBench has models generate backends while checking for both functionality and security issues. AppForge evaluates Android apps by loading the finished product into an emulator and actually tapping through the interface. Vibe Code Bench evaluates web applications, then has a browser agent verify them step by step according to user flows.
These benchmarks already test complete products. The extra step BackendForge takes is making the question bank continue searching for gaps before release.
It includes 56 backend tasks, encompassing 2,345 API operations in total. After models generate code, the researchers package the services into Docker and test them with real HTTP requests. The entire evaluation sent 24,798 requests in total.
These requests are chained together. The first request creates data, the second switches to another user to read it, the third deletes it, and finally the database is checked for what remains. Rules like the Alice-and-Bob scenario earlier can only be tested this way.
The original 7,250 tests already checked for these cross-request rules. This line of thinking is not a new invention — Microsoft’s RESTler has long been automatically trying different API call sequences. BackendForge’s change is having the tests proactively search for their own blind spots during benchmark construction.
BackendForge first prepares a reference service. You can think of it as the answer key. Then, three agents circle around it, repeatedly setting questions, reviewing, and correcting.
The first agent is responsible for creating questions, referred to in the paper as the Test Agent. It designs new tests based on the requirements document and the OpenAPI spec. If the reference service passes, the question found nothing new and is discarded. If the reference service fails, the question may have caught a rule the first version of the test bank missed.
The second agent is responsible for checking whether the new question has a basis in the requirements, referred to in the paper as the Review Agent. It rejected 113 untenable candidates in total. Finally, the Code Agent fixes the reference service and reruns all old and new tests. Only after everything passes does the new question enter the official question bank.
The paper calls this cycle co-evolution — tests and the reference service improving together. The 640 newly added tests all originated from specific problems:
You can think of previous evaluations as written exams. Questions for each task are written in advance; after the code is submitted, it is scored against a fixed answer key. BackendForge adds an interviewer during the question-setting stage: it first looks at where this task’s reference service went wrong, then follows that error with further questioning.
Co-evolution increased tests from 7,250 to 7,890. The 640 new tests account for only +8.8%. Though few in number, each one survived a process of first discovering a real failure, then being verified against requirements, and finally passing regression testing.
BackendForge’s scoring is strict: every test within a task must pass. Even a single failure means the entire task is considered failed.
There are 56 tasks in total. When scored with the original question bank, services generated by GPT-5.5 passed all tests for 31 tasks. With the hardened question bank, only 16 remained. In other words, only 51.6% of the originally passing tasks still passed.
The drop for Claude Opus 4.7 was even larger: originally passing 33 tasks, only 10 remained after the hardened question bank.
The models did not regenerate code, nor did they suddenly become weaker. The same code met a test suite better at finding problems, and the results changed. Some issues the first version of the question bank missed were enough to flip an entire task from pass to fail.
The paper states the authors plan to release the complete evaluation materials, but as of this article’s verification, these resources are not yet publicly available. The outside world cannot yet examine the 640 new tests item by item, so the magnitude of the score change observed here should not be directly extrapolated to all whole-application evaluations.
BackendForge customizes its follow-up questioning per repo or task. Wherever Gotify’s reference service makes mistakes, the Test Agent follows up along Gotify’s issues; for a different task, it continues questioning from the problems that service exposes. As a result, the questions ultimately added to the bank can differ from task to task. It does not swap questions on the fly for each evaluated model.
But once the question bank is finalized, the interview is over. The final tests for a given task are frozen. Whether GPT-5.5, Claude Opus 4.7, or any other model tackles the task, they all face the same set of questions. This way, follow-up questioning can target each project’s specific issues without swapping questions on the fly for any particular candidate.
So BackendForge still retains the written exam. The only difference is: before the test paper is finalized, a round of interview-style follow-up is conducted based on each task’s actual responses. After finalization, all candidates take the same exam paper and have their scores compared.
Going forward, when evaluating entire applications, this round of follow-up questioning could become a required step before release. First, let the agent search for gaps against the reference service, then have humans check whether the new questions have a basis in the requirements. After verification, freeze a unified version. The public question bank handles long-term comparison, while periodically updated hidden questions continue searching for new vulnerabilities.
From a single function to a complete application, the tasks AI receives have kept growing. BackendForge has taught evaluation to follow up as well. Standard exam papers are still necessary — they just ought to go through an interview first, before being finalized.