If you only look at SWE-bench or various public leaderboards, many engineering teams fall into an illusion when selecting AI Coding Agents: if a model achieves an 80%+ success rate on Python benchmarks, it should naturally fix bugs and submit PRs automatically when brought into company projects. Yet in real-world production codebases, Agents often either fail to interpret error traces or blindly trial-and-error through agonizingly slow build logs and stringent type constraints.
This disconnect highlights a fundamental question that spans the entire evaluation and procurement process: How should we actually evaluate AI Coding Agents to accurately predict their real-world engineering ROI in production codebases?
The primary reason generic leaderboards frequently fail to reflect enterprise reality is that they mistake problem-solving heuristics tailored for Python as universal capabilities applicable across all programming languages.
Take SWE-bench as an example—its baseline is almost entirely built upon Python open-source projects. When OpenAI audited SWE-bench Verified, they noted that over 59.4% of high-failure samples suffered from flawed test cases or problem statements, alongside significant pre-training data contamination. Yet deeper than dataset flaws lies a common intuitive assumption: because Python code is simple and Python accounts for a massive proportion of LLM pre-training corpora, an Agent performing well on Python will naturally perform well when migrated to Kotlin or Java. However, this confuses the fundamental physical differences between pre-training data density and language type system constraints.
Python’s advantage lies in its exceptionally high pre-training data density—generating Python code is almost a knee-jerk reflex for LLMs. But within an Agent’s self-directed optimization and error-correction loop, dynamic typing acts as a hidden stumbling block. Syntax errors or mismatched types in Python code trigger zero immediate feedback during editing. The Agent must execute the full test suite, and the execution flow must hit the precise code path containing the defect before a stack trace delivers an error. This runtime-dependent feedback loop is both protracted and stochastic.
Statically, strongly typed languages like Kotlin and Java are the exact opposite. Language Servers and compiler checks provide deterministic feedback on type mismatches, nullability risks, or missing arguments instantly during editing—without actually executing the code. This local, deterministic static diagnosis acts as real-time signposts along the Agent’s error-correction journey, drastically reducing its pathfinding trajectory. The real engineering friction in static languages lies in the physical overhead of end-to-end verification—every time the Agent wants to verify whether the entire workflow passes, it incurs the overhead of spinning up the Gradle Daemon, parsing KTS scripts, resolving multi-module dependency topologies, and triggering KSP code generation.
This gives rise to fundamentally distinct engineering frictions across the two ecosystems: Python relies on high-density pre-training intuition to generate code but depends on lengthy, non-deterministic dynamic execution to catch type errors; Kotlin possesses razor-sharp static diagnostics during editing, but every end-to-end verification cycle demands waiting on heavy Gradle builds.
An Agent that converges on a correct patch in Python through frequent trial-and-error will quickly be dragged down by heavy builds in a Kotlin project.
Once we understand the engineering friction inherent to compiled ecosystems, evaluating Kotlin brings into focus a variable completely masked by Python leaderboards: the outer agent harness.
On July 24, 2026, JetBrains released official data for the JetBrains
Kotlin Benchmark (source code available in the Kotlin/kotlin-swe-bench
repository). Featuring 105 real-world engineering tasks across 8
open-source projects (such as ktlint, detekt,
okhttp, and Anki-Android) containerized via
the Harbor Agent
evaluation framework, it provides the first standardized benchmark
to observe this variable.
The initial evaluation figures present a compelling comparison: powered by the exact same underlying model (Opus 4.7), Claude Code achieved an 85.71% (90/105) success rate, while JetBrains’ in-house agent Junie scored 81.9% (86/105).
With identical underlying model intelligence, swapping the outer Harness produced a nearly 4-percentage-point performance gap. Diving into how Agents handle Gradle builds on the ground reveals that this divergence stems from log “detoxification” and “tiered orchestration.”
When a Gradle build fails, the terminal often dumps hundreds of unstructured log lines filled with daemon task trees, failed task dependencies, and raw stack traces. If a Harness naively dumps this entire log into the context window, the influx of noise instantly pollutes the Agent’s context and induces hallucinations. A well-architected Harness constructs a Tiered Feedback Circuits pattern: after code edits, it first leverages the Language Server at the editor level to catch static errors, zeroing out syntax and type issues locally; only at critical milestones does it initiate a full Gradle build, precisely filtering and extracting the key lines of breaking symbol errors from the verbose build output.
This clarifies a crucial insight for agent evaluation: the minimum effective unit for evaluating Coding Agents is never just the underlying model alone, but rather Model x Agent Harness x Language Ecosystem x Real Repository. The design of the Harness in sniffing static diagnostics and sanitizing heavy build logs dictates whether the base model’s intelligence is swallowed by bloated build outputs or effectively transformed into valid code fixes.
Demystifying Harness orchestration logic answers the single biggest confusion in tool selection: Why can no public leaderboard—regardless of how authoritative it is—serve as an off-the-shelf procurement guide?
Because public benchmarks measure an Agent’s problem-solving
capability in general open-source projects (such as
okhttp), which adhere to standard, well-established public
library patterns. However, the vast majority of real-world bugs in
enterprise production environments do not boil down to generic
algorithms—they are embedded in a team’s own Implicit
Architectural Constraints.
In a specific business codebase, a code edit that appears fully compliant with Kotlin syntax and public best practices might quietly violate internal dependency injection scopes (such as custom Hilt or Koin threading constraints) or break undocumented asynchronous coroutine dispatching conventions. No matter how much open-source code an LLM saw during pre-training, it cannot magically anticipate an organization’s proprietary architectural guardrails.
To maintain absolute comparability across companies and teams, public leaderboards must strip away all enterprise-specific proprietary constraints. Yet this exact requirement guarantees that no public leaderboard can directly predict an Agent’s actual ROI in your private codebases.
Public baselines like the JetBrains Kotlin Benchmark are best positioned as Evaluation Seeds. Their true contribution is not a static leaderboard ranking, but an automated sandboxing and harness infrastructure blueprint.
What engineering teams truly need is to leverage the Harbor Task Format and verification blueprints from public seeds to transform their own historical PRs and post-mortem issues from the past six months into assertion Verifiers. This forms a tailored Micro-Eval Matrix:
From identifying ecosystem friction and understanding Harness orchestration, to leveraging Evaluation Seeds for custom Micro-Eval Matrix construction—only then does AI Coding Agent selection anchor firmly in engineering reality.