Harness EngineeringAnatomy, Architecture, and Evolution of Coding Agents -- A Source-Code Study of Eleven Systems
Across roughly four million lines of Python, TypeScript, and Rust — eleven production AI coding agents — not one of them imports a general-purpose agentic framework. No LangChain. No AutoGen. Nothing off the shelf.
Which is striking because if you asked most people what these systems are built on, the assumption would be that the frameworks exist precisely for this purpose. Someone built LangChain so you would not have to roll your own.
That's the assumption the paper by Paul Barbaste, Tristan Darrigol, Germain Vu, and Tom Wiltberger keeps pushing on. Their argument is that the interesting engineering in these systems isn't in the model; it's in what they call the harness, the runtime that couples the model to the world. And when you actually read the source code, you find that every production system built that runtime from scratch.
Walk me through what a harness actually is, because that term is doing a lot of work here.
The paper defines it cleanly: an agent equals a model plus a harness. The model is the intelligence. The harness is everything else — the loop that runs the agent, the tools it can call, how context gets managed, safety controls, orchestration across multiple agents, and surfaces for extensions and plugins. Harness engineering, which they date to February two thousand twenty-six, is the discipline of designing and evolving that runtime.
Right. My instinct is to hear that and think: okay, so it's scaffolding. Important scaffolding, but the model is still where the differentiation lives. A better model leads to a better agent.
That's exactly the assumption the study is set up to test. The method they chose is what makes this interesting — they do not run benchmarks. They read source code from eleven production systems, pinned to July two thousand twenty-six releases, audited at the file and API level.
Why does that matter? What does source code tell you that a benchmark does not?
Benchmarks tell you scores. Source code reveals how the system is built, including how the loop is structured, how tools are sandboxed, how safety is enforced, and how context gets rationed when you're running up against the model's window. Those are engineering decisions, and they are invisible in any performance number.
The corpus includes four provider-native flagships: Claude Code from Anthropic, Codex CLI from OpenAI, Gemini CLI from Google, and Mistral Vibe from Mistral. Additionally, there are seven open-source projects: OpenHands, Aider, Mini-SWE-Agent, Hermes, Pi, OpenCode, and OpenClaw. There is also a twelfth system, Omnigent from Databricks, which is analyzed separately as a meta-harness — a harness that orchestrates other harnesses. We'll come back to that.
The anatomy they build from this corpus consists of seven canonical subsystems that every harness implements in some form: the agent loop, large language model integration, tool interface, context and memory management, safety controls, multi-agent orchestration, and extension surfaces.
Seven subsystems. What does the variation actually appear like?
Start with the agent loop, because that's where the most visible divergence is. Three paradigms show up. Nine of the eleven systems use a basic iterative action and observation loop — the agent calls a tool, observes the result, decides what to do next, and repeats. Aider adds a reflection layer, re-invoking up to three times on failure. Then there's the coordinator and worker pattern, where a central agent delegates subtasks to specialized workers — Claude Code, Codex, and Gemini CLI all do this.
The coordinator-worker thing — that's where you get Claude Code's sixteen-tool whitelist for workers and the Research, Synthesis, Implementation, and Verification workflow?
Exactly. The Gemini Command Line Interface's loop has explicit stuck detection. It aborts after five identical tool calls or ten identical content chunks, with a large language model self-check kicking in after thirty turns. The session turn limit defaults to one hundred.
I want to stay on that model-centric assumption for a second. If the loop architecture is driving differentiation, you would expect performance to vary with loop type. Does it?
The paper doesn't claim that. What it does show is that loop sophistication predicts production readiness, including safety infrastructure, extensibility, and user experience — not benchmark scores. Mini-SWE-Agent implements all seven subsystems in roughly one hundred lines and reports frontier-range benchmark results. Codex CLI is six hundred twenty-one thousand lines. Gemini CLI is five hundred sixty-eight thousand.
So one hundred lines gets you competitive benchmark numbers.
Right. But Codex vendors its entire sandbox infrastructure, which is a tool called bubblewrap, directly into its source tree. Mini-SWE-Agent doesn't have that. The scale difference is almost entirely about production safety and extensibility, not raw capability.
That's the step I keep tripping on. If the benchmark numbers are similar, what exactly is the larger system providing you?
The harness is where policy lives. Here's the finding that I think reframes this: behavioral policy — the rules about what the agent will and won't do — is migrating out of system prompt prose into structured configuration files. Codex's prompts shed their no commit and anti gold plating language in favor of a feature flag called codex_git_commit. The behavior doesn't disappear. It moves into configuration.
That's a real architectural shift. A prompt can drift, while a flag is enforced.
That's the paper's claim. The safety controls in the corpus span a huge range. Hermes ships a three thousand two hundred line approval module with hard deny patterns and an optional large language model gate. Mistral Vibe has layered per tool validators and agent profile permission presets. These aren't small decisions.
Okay, now what about code retrieval? My understanding is that these systems would use vector embeddings, which is semantic search over the codebase.
None of them do. Across the entire corpus, no system uses vector-embedding retrieval-augmented generation over source code. They use ripgrep, glob patterns, tree-sitter for syntax-aware parsing, and in Hermes's case, SQLite with BM25 plus trigram indexing. Aider's RepoMap builds a PageRank style ranking over the syntax tree. It's all deterministic, lexical, or structure-aware. One system, OpenClaw, uses hybrid embeddings, but only for conversation memory, not code retrieval.
Why? The whole pitch of embeddings is semantic search at scale.
Debuggability and prompt transparency. If a deterministic tool returns a file, you know why. If an embedding retrieval returns a file, the reasoning is unclear. For a production system where you need to audit what the agent did and why, lack of clarity is costly.
That reframes the absence. It’s not that they haven’t arrived yet — it’s a deliberate architectural choice.
Now here’s the methodological piece I think is the paper’s most distinctive contribution. The original eight systems were re-pinned, fixed at an earlier snapshot, and then source-diffed across one quarter. Same systems, same methodology, three months later.
What did the diff show?
Convergence tipping into imitation. Codex adopts Claude Code's hook vocabulary verbatim and ships an importer for Claude Code's sessions and settings. OpenHands reads Claude Code's plugin format. These aren't parallel inventions; one system is copying the specific design choices of another.
Three of the observations in the April edition required substantive revision when the differential came in. The harness is changing fast enough to invalidate its own documentation in a quarter.
Which is the clearest possible evidence that the harness is a primary locus of evolution and not a stable wrapper around a model that is doing the real work.
Okay. That's the step I was missing. If the model is constant and system-level behavior shifts because the harness changed, the harness isn't just plumbing. It's where policy is created.
The extension ecosystem findings push this further. SKILL.md skills are user-visible capability bundles, essentially a directory with a metadata file declaring the name, description, triggers, and helper scripts. They are now adopted in nine of eleven systems. The Machine Control Protocol, or MCP, for tool integration, is in eight of eleven. Skills edged ahead.
Four systems now ship hosted registries and versioned catalogs for skills, including trust tiers, provenance checks, and agent-authored packages.
That's the platform turn. Harnesses are now importable software development kits and hostable runtimes. The Agent Client Protocol, or ACP, ships in six systems and acquired a third operational role: harness hosting. OpenHands can run Claude Code, Codex, or Gemini Command Line Interface as interchangeable backends. Omnigent, the Databricks meta-harness, orchestrates multiple vendor harnesses behind a single application programming interface.
That's the economic shift. If your harness becomes a platform that third-party developers build on, such as marketplaces, registries, and governance layers, then switching costs accumulate in the harness, not in the model.
The paper closes with eighteen design recommendations and a ninety-line minimum viable harness scaffold that implements ten of them directly. The practical takeaway: a simple linear loop, a middleware style policy layer, a compact toolset, and explicit context discovery. Safety as architecture from the start, not added on later.
The broader implication is what I find most striking. If the competitive frontier is in the harness, in runtime design, extension governance, and policy architecture, then the next year of progress in this field isn't won in model weights. It's won in the engineering of the runtime that wraps them.
Barbaste and colleagues are arguing this isn't a future possibility. With eight thousand or more MCP servers in the ecosystem and evidence of twenty-two to twenty-nine percent of GitHub projects already showing agent traces, the platform turn isn't coming. It already happened. This lecture was created by ennepō. Go to https://ennepo.ai to Discover, Create and Follow the latest research in your field. Read when you can. Listen when you want to.
Credits
- Source: Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents -- A Source-Code Study of Eleven SystemsPaul Barbaste, Tristan Darrigol, Germain Vu, Tom WiltbergerarXiv (Cornell University), 2026DOI: 10.48550/arxiv.2609.00006CC-BY 4.0
AI-generated lecture by ennepō.ai, adapted from the source above — restructured, summarized and explained. Not endorsed by the authors.
This lecture © ennepō.ai. See our Terms of Service.
Related lectures
- Artificial intelligence in communication impacts language and social relationships
- The sociolinguistic foundations of language modeling
- Personality, Gender, and Age in the Language of Social Media: The Open-Vocabulary Approach
- How to factor 2048 bit RSA integers in 8 hours using 20 million noisy qubits
- A Latent Variable Model Approach to PMI-based Word Embeddings
- Mixtral of Experts