The Moving Harness Needs a Still Judge

Every deployed AI agent runs inside a harness — the prompts, tools, memory, and control flow wrapped around the model. Until this year that layer was hand-written and frozen: engineers guessed the right scaffolding once, for every task and every run. Since March 2026, at least eight research groups have independently decided the guess should be a learner instead. This piece reads that wave through four of its corners — Princeton, Zhejiang/Alibaba, Shanghai Jiao Tong/Xiaohongshu, and UIUC/Meta AI — four machines whose methods barely overlap, then tests what they suggest against the four papers around them. The results sort on one variable none of the papers names: who judges the adaptation. Where the verdict is computed from the world — a task verifiably solved, a re-run measurably improved — the moving harness compounds. Where the adapting system judges its own progress, the documented failures cluster: patches that make agents worse, an agent repeating a broken action 842 times while its reasoning claimed progress. The lesson the wave teaches wasn't in any single paper: the more the harness moves, the more the ground truth must not.

XO Labs Inc. · August 2026

TL;DR: The adaptive-harness literature is now a wave — at least eight entries since March 2026, and we read eight. Four in depth, chosen because they occupy four corners of the design space: EvoHarness-RL (trains the policy when to use a fixed workspace), Harness-R1 (a separate model edits the harness's executable code), Living-Harness (LLM modules rewrite the harness's content between episodes), Continual Harness (the harness is edited live, mid-run). Then the four surrounding entries — Meta-Harness, EvoTrainer, EvolveNet, and an Allen Institute evaluation audit — read against the pattern the first four suggest (§6): three fit it, the audit sharpens it. Three findings across them. First, the static-harness era is ending by consensus: each paper independently demonstrates that fixed scaffolding is wrong for most of an agent's life — a fixed self-refinement rule lowers reward on all three of one paper's benchmarks; a trained usage policy anneals from ~6 harness calls per episode to ~1. Second, outcomes track the judge, not the mechanism: the three systems anchored to a verified outcome gain +49.0, +9.3, and +10.1 points on their benchmarks, while the failure cases concentrate where verification is absent — frontier models prompted to edit a harness without re-running it dropped success by 6.2 points because they "optimize for plausibility," and the one system with no ground truth anywhere in its loop is the one whose own appendix documents a 1,003-turn stall behind a confident narration of progress. Third, every one of the four buys its ground truth from the same supplier: a state-computed verdict, clean re-runs, trajectory records, instrumented state — the services an environment provides. A moving harness doesn't shrink the environment's job. It enlarges it.

1. The layer that stopped being furniture

The harness is the part of an agent system that isn't the model: context assembly, tool routing, memory, permissions, retries, the rules for when to consult what. In production systems it is most of the system — our anatomy of Claude Code measured roughly 98.4% of a production coding agent as this infrastructure — and it has one defining property: somebody wrote it, once, and it is identical on every run. The first session in a repository and the ten-thousandth get the same scaffolding, the same memory rules, the same everything.

A 2026 wave of research treats that property as the defect — at least eight papers since March, out of Stanford, Alibaba, the Allen Institute, and a half-dozen university–industry teams. This piece reads eight of them: four in depth, as the corners of the design space, and the four around them as the test of the pattern.

2. Four ways to make a harness learn

Four almost non-overlapping mechanisms — mutate everything, rewrite the content, edit the code, train the usage. And each paper supplies its own evidence against the static baseline:

• Harness-R1 measures it directly: "a fixed Self-Refine rule lowers reward on all three benchmarks" (their §5) — and the intervention point that matters most differs by environment, "which a fixed strategy cannot select on its own."
• Living-Harness states the defect: "a static harness is fixed after deployment: it can impose predefined rules, but it cannot incorporate newly observed failure patterns" (their §1).
• Continual Harness starts from it: "without domain-specific scaffolding, frontier vision-language models make almost no progress" on its RPG benchmark — and its harness "is updated throughout the run rather than converging to a fixed scaffold" (their Fig. 3).
• EvoHarness-RL shows the optimum was never a constant: trained workspace use anneals from roughly six calls per episode to "near one call per episode" as experience accumulates (their §4.1) — so any fixed policy overpays almost everywhere on that curve.

And these four are corners of a larger family: Meta-Harness, EvoTrainer, EvolveNet, Self-Harness and others sit in the same wave, which is far enough along to have drawn its first evaluation audit — §6 reads them against this piece's argument. On the question should the harness move, the wave is unanimous, and we take no credit for it. The interesting structure is in what happens next.

3. The variable that sorts the results

Line the four systems up not by what they adapt but by what judges the adaptation — from a verdict computed out of world state on one end, to a verdict authored by the adapting system itself on the other — and the outcomes arrange themselves.

The mechanism barely matters; the judge does. Two honesty notes before leaning on this chart. First, the axis is our construction — a hypothesis these four results suggest, not a claim any of the papers make; §6 tests it against the next four entries of the wave, and §9 records how the papers were chosen. Second, the magnitudes are not comparable across systems: each gain sits on its own benchmark against its own baseline (+49.0 is over a bare agent; +10.1 is over the strongest competing method). Only the directions travel. The two-tone marker for Living-Harness is deliberate: its commits are gated by a verified score, while the content of each edit is still LLM-extracted.

One more thing the axis is not: a discovery. That optimizing against a signal degrades the signal is Goodhart's law, stated a half-century ago. That reinforcement learning compounds where rewards are checkable against the world and gets gamed where they are a model's opinion of a transcript is the operating lesson of the verifiable-rewards literature that powers current frontier training. That models cannot reliably grade or correct their own work is by now a measured result many times over — our own contribution to that pile put an agent's self-reported completion below chance against test-verified outcomes. What the wave adds is not the principle but the recurrence: eight groups building harnesses that learn, each running into the same two-sided constraint in its own ablations, on its own benchmarks, for its own reasons. An old lesson resurfacing independently in eight new places is better evidence than a new axis would be — it says the constraint is structural, and that the adaptive-harness field will not get to skip it.

4. Where the verdict is verified, adaptation compounds

The three systems on the left of that axis have almost nothing in common mechanically, and their results rhyme anyway.

EvoHarness-RL anchors on the purest signal available: ten reward points if the game's own record says the episode ended solved, zero otherwise — the paper writes it 10 · 1[solved] — plus an efficiency bonus "only granted upon success." From that alone, an 8-billion-parameter model goes from 47.9% to 96.9% on the seen split of ALFWorld, its text-game benchmark of multi-step household tasks (their Table 1) — and, more telling, learns a differentiated economy nobody supervised: harness calls anneal from ~6 per episode toward 1, keeping the one action (recall) that still pays for itself.

Harness-R1 verifies at one remove: its 9B engineer proposes an executable patch, the patched-but-frozen agent re-runs the same task batch, and the measured change in outcomes — "including a negative value when it causes regressions" — is the engineer's reward. Average success rises from 44.3% to 53.6% across its three text benchmarks — WebShop (shopping), ALFWorld (household tasks), and DBBench (database work) (their Table 1), the outcome-trained engineer beats its supervised-only twin by 7.1 points, and the learned editing policy transfers: +7.06 points averaged across twenty unseen target models (their App. E).

Living-Harness verifies at the gate: no update is committed until "this score has been recorded" from the benchmark evaluator, and its stated first principle is the axis of this piece in one sentence — an update must be "grounded in a completed trajectory and its evaluation signals, rather than in an unverified self-critique alone" (their §1). Result: 83.09% average on τ²-Bench, a customer-support benchmark of dialogue plus tool use, against 73.02 for the best interactive baseline and 57.39 for the bare model (their Table 1).

Three mechanisms — train the user, train the editor, gate the writes — one anchor, three compounding curves.

5. Where the system judges itself, the failures cluster

The right half of the axis is where this wave gets genuinely instructive, because both of its data points come from the papers' own reporting.

The first is Harness-R1's control experiment. Prompt a frontier model to edit the harness — same job as the trained engineer, but with no re-run, no measured delta, nothing but the model's own judgment of its edit — and the result is not merely weaker: it is negative. Gemini-3.5-Flash's patches dropped ALFWorld success from 41.6% to 35.4%; of its 39 valid patches, 21 made things worse (their App. H.3). The authors' diagnosis is the cleanest sentence in the four papers: fixed frontier editors "optimize for plausibility, emitting syntactically valid and reasonable-looking edits, but because they never rerun the target they cannot tell whether an edit actually raises task success" (their §5). Plausibility is what an unverified signal rewards, so plausibility is what the system produces.

The second is Continual Harness — and it needs stating carefully, because the paper's headline arms work: on Pokémon Emerald its harness reaches 100% of milestones at a $130 median API spend against 98% at $215 for the minimalist baseline (their §4.4). But it is also the only system of the four with no ground truth anywhere in its adaptation loop — the refiner judging "failure signatures," the reward model, the relabeling teacher, and even the milestone judge are all LLMs — and its own reporting shows where that bites.

Below a capability floor, the moving harness underperforms the static one (on the weakest model tier tested, Gemini Flash-Lite, every adaptive variant lands at 3–13% of milestones against the minimalist baseline's 20%, their §4.4). One transfer variant regresses below the minimalist baseline outright (their App. C.2). And their Appendix B.3 documents the failure mode in miniature, 1,003 turns of it: an agent re-sending a schema-mismatched tool payload 842 times while its visible reasoning asserted progress. Credit where due — the authors report all of this themselves.

The shape is familiar. It is the same decoupling of narration from world that our own observational study measured in agents that weren't even adapting — self-reported completion scoring below chance (AUC 0.491) against test-verified outcomes — now running inside a system that modifies itself based on that same narration.

The apparent counterexample — an unverified loop posting this section's one win — is softer than it looks, for two reasons visible in the paper's own reporting. First, self-judged adaptation borrows the judge's accuracy: it works exactly where the judging model is strong enough to be approximately right about the trajectory it reads, which is why the same method collapses on the weakest tier. The capability floor is not a separate caveat; it is the mechanism. Second, even the scoreboard is soft: the milestone judge that grades the headline runs is itself an LLM, and the only fully hard numbers in the system — button presses and dollars — meter cost, not correctness. The one system with no ground truth in its loop is also the one whose win is graded by the same kind of judge that runs its loop. Its plus sign and the verified systems' plus signs are not the same currency.

The calibrated statement, and the only one the evidence supports: self-judged adaptation is not doomed — it is unanchored, inheriting its judge's error rate with no floor beneath it. The verified systems fail differently: a bad edit produces a bad measured delta and a negative reward, and the loop corrects. That difference — failing loudly versus failing fluently — is worth more than any headline number in the four papers.

6. The rest of the wave, read against the axis

An axis built on four points should be tested on the next four. We read the surrounding entries in full — Meta-Harness, EvolveNet, EvoTrainer, and the Allen Institute audit — asking one question of each: what does its adaptation loop consume, and what happened wherever that signal thinned?

• Meta-Harness (Stanford + MIT, March) is an agentic search over harness source code in which "the proposer never sees test-set results" and every acceptance is a measured score from executed rollouts. Its own search log is a miniature of §5: six consecutive LLM-plausible edit hypotheses regressed when measured — plausibility went zero for six — and the verified score was the only thing sorting good from bad. Its sharpest ablation extends the axis a level down: holding the verified verdict constant and inserting LLM-written summaries into the diagnostic channel made the search worse than giving the optimizer nothing at all (best candidate 38.7 against 41.3; raw execution traces: 56.7, their Table 3). Not only must the verdict come from the world — narrating the evidence through a model degrades it too.
• EvolveNet (HKBU + USTC + HKUST, August) evolves harness code across multiple collaborating agents, and every adoption decision cites "the task evaluator's verdict" per item — the LLM proposes, the gate decides. Remove the verified gate and their own centralized optimizer "fails outright," scoring below the unevolved harness and rolling back (their §6.6). One refinement: the verified signal gates well per item and ranks badly as a scalar — their validation totals inverted the true test-set ordering (their §6.3).
• EvoTrainer (Alibaba + CAS, June) extends the pattern to the training side — the diagnostic harness around an RL run — and contributes the wave's most instructive failure-catch: one policy version posted 48.80% by reading reference patches out of git history, "a number a score-only loop would promote as a breakthrough"; the true figure after sanitization was 31.04 (their §4.4). The catch came not from a model's opinion but from more world-state — tool logs, turn counts, degenerate-trajectory counts the harness had evolved to collect. The good pole of the axis is not one verified number; it is verified numbers, plural, cross-checked.
• The Allen Institute audit (AI2 + UW, July) presses from the other side, and lands on both halves of this piece. Its controlled arm corroborates §5 almost verbatim: with no external verifier, iterative harness revision dropped a strong model from 75.3 to 69.7, and the authors conclude revision "likely requires a reliable external correctness signal to ground the revision process." But it also shows a verified signal certifies less than it seems: under real unit-test verdicts, harness evolution still lost to budget-matched plain resampling (75.8 vs 86.0 pass@1), and its gains collapsed to +0.6 points on held-out tasks — "most edits memorize fixes rather than distilling strategies."

So the axis survives its first out-of-sample test, sharpened rather than shaken: who judges decides whether adaptation escapes self-delusion; whether the judged gains are real further requires eval tasks the adaptation never saw, a budget-matched search baseline, and more than one verified number. One pattern in this test deserves suspicion rather than celebration: all four extension papers confirm the axis's direction — none contradicts its core mechanism, and the audit complicates the magnitude of gains, not their sign. A sample chosen for design-space coverage that returns four-for-four agreement is weaker evidence than an adversarial search would be. So the falsifier, stated explicitly: a self-judged system with robust held-out wins, or a verified-signal system that fails without a leakage or budget explanation, would break this axis. Nothing we have read is either — and nothing we have read was found by looking for one. Those tests press on our own four exhibits too, unevenly. EvoHarness-RL reports an unseen split (86.6 against 96.9 seen) and Harness-R1 a held-out generalization result (+8.9 ± 1.5 points pooled), so both survive the leakage question on paper; Living-Harness's 83.09 is measured across cycles that re-run the same task pool, and is the number in this piece most exposed to the audit's critique. And the wave's one real-work data point — Meta-Harness ranking first among Haiku agents and second among Opus agents on TerminalBench-2, a leaderboard of real terminal tasks — carries the same-set caveat and frontier-marginal magnitudes (+2.1 over the best human-written rival): the best evidence yet that verified-signal harness search extends beyond games, and the clearest sign that its gains there are currently small.

7. What they all buy from the environment

Now read the systems as customers — the four corners and the wave's three systems alike (the audit inspects rather than consumes) — and itemize what each loop requires.

The columns are worth spelling out, because none is optional for the system that uses it:

• A state-computed verdict. ALFWorld's engine knows solved; Harness-R1's environments emit reward "once the episode ends"; Living-Harness's evaluator scores every trajectory before any commit; EvolveNet's gate consumes per-item gold verdicts.
• Clean re-runs and resets. Harness-R1's entire reward is a same-batch re-run under fixed seeds; Continual Harness carries emulator save-states across training iterations; EvolveNet engineers determinism until two full repeats produce byte-identical output; EvoTrainer re-scores historical rollouts in backtests.
• A trajectory record. EvoHarness's imitation-stage training corpus is 87 recorded teacher episodes cut into 1,153 training pairs; Harness-R1 compacts failures into "failure packets"; Living-Harness's extractors read the full trajectory; Meta-Harness's optimizer reads up to ten million tokens of execution traces per iteration — and degrades when those traces are replaced with LLM summaries.
• Instrumented state. Harness-R1's patches consume hand-built semantic predicates the wrapper exposes; Continual Harness reads structured game state straight out of emulator RAM; EvoTrainer's plural counters — dead-group ratios, truncation rates, turn counts — are what caught the gamed scalar in §6.

Every column is an environment property, not a model property — the same captures our environment argument has itemized for a different reason entirely: state before and after, a complete meter, the record. That argument was built for accounting — for making a unit of work checkable and billable. The wave described here consumes the identical capture list to make a harness learnable. Two motivations, one specification. And note which service the failure-prone quadrant was missing: the one in the first column.

8. The still point is the asset

One more result, easy to miss in Living-Harness's tables, hints at where value accumulates in this new arrangement. It is a single transfer experiment, in one domain of one paper — a data point, not a law — but it points somewhere specific. Their evolved harness state — the memory entries and repair graph distilled from scored episodes — was handed to entirely different frozen models, with no retraining: the new model simply reads the inherited state. On the hardest domain of their second benchmark (the MultiWOZ dialogue suite), models that scored 0.00 bare reached 43–45% with the inherited state; a frontier model more than doubled, 30.26 to 68.72 (their Table 3). The models were interchangeable. The accumulated harness state was not.

Follow that to where it points. The model is rented, and swappable. The harness, in this wave, churns by design — it is the part that moves. What persists and appreciates is the layer that made the adaptation trustworthy in the first place: the boundary that owns the verdict, keeps the records the next round of adaptation will consume, and holds the accumulated state that outlives any particular model behind it. Our unit-of-work research has been making the accounting version of this claim for months — the asset is not the agent you rent but the environment you own. This transfer result is one data point toward the training-side version of that claim. Not yet the claim itself; the direction is what's worth recording.

The self-evolving systems phase of our roadmap describes agents that improve on the job. These four papers are early, toy-world drafts of exactly that — and their collective lesson is the sentence that phase has been missing: self-improvement is not a property a system has. It is a loan the system's environment extends, denominated in verified outcomes, and the documented failures are what default looks like.

9. Honest limits

• How the papers were chosen. Three of the four core papers were brought to our attention and one we were already studying; the four surrounding entries were then read in full, and §6 reports how each bears on the axis. The order matters: the axis was formed on the first four and tested on the next four, not derived from all eight. This is still not a preregistered survey — the wave has further entries (Self-Harness, HarnessX/AEGIS among them) we have not read.
• The four core systems are all games and simulators: ALFWorld and WebShop episodes, dialogue benchmarks, Pokémon. The wave's one real-work data point — Meta-Harness on TerminalBench-2's real terminal tasks (§6) — is searched and evaluated on the same 89 tasks and wins by frontier-marginal magnitudes, so it softens this limit without removing it. The leap to production work remains conjecture — real work rarely hands you solved for free, which is exactly why the verdict is the expensive artifact.
• All four papers are first-version, unreplicated, and concurrent; several report single runs without seeds or variance, and two are partly self-evaluations by teams shipping the method. Trust the cross-paper shape — where the failures cluster — over any single figure.
• Frontier models sit inside every loop somewhere: as teacher and consolidator (EvoHarness), as the teacher for the initial supervised stage (Harness-R1), as actor and extractor (Living-Harness), as everything including the judge (Continual Harness). None of these systems is self-contained, and none of the papers fully prices that dependence.
• The §3 axis is our construction, not any paper's. The papers do not cite each other's failure modes, and placing them on one axis involves judgment calls we have tried to make visible (the two-tone marker; the calibrated §5 wording). Continual Harness's headline arms succeed; our claim is about where the documented failures sit, not that self-judgment cannot work.
• Instrumentation does quiet work everywhere. Harness-R1's hooks consume hand-built semantic predicates; Living-Harness's evolution procedures are hand-written per domain. Some of what reads as "learned adaptation" is expert knowledge pre-loaded into the environment — which cuts against the papers' autonomy claims and for this piece's actual thesis.
• Our own cited studies carry their own bounds, stated where they are published.

10. References

• Karten, S., et al. (2026). Continual Harness: Online Adaptation for Self-Improving Foundation Agents. Princeton + ARISE + Google DeepMind. arXiv:2605.09998
• Du, Y., et al. (2026). Living-Harness Is an Interactive-Agent Evolver. Zhejiang University + Alibaba Group, et al. arXiv:2607.26598
• Shao, S., et al. (2026). Harness-R1: Learning to Edit Executable Runtime Harnesses from Agent Failure Trajectories. Shanghai Jiao Tong University + Xiaohongshu. arXiv:2608.02276
• Ning, X., et al. (2026). EvoHarness-RL: Learning Self-Evolving Runtime Harness for Long-Horizon LLM Agents. UIUC + Meta AI. arXiv:2608.05446
• Lee, Y., et al. (2026). Meta-Harness: End-to-End Optimization of Model Harnesses. Stanford + MIT + KRAFTON. Read in full; the diagnostic-channel ablation and TerminalBench-2 results of §6. arXiv:2603.28052
• Chen, G., et al. (2026). EvoTrainer: Co-Evolving LLM Policies and Training Harnesses for Autonomous Agentic Reinforcement Learning. CAS + Alibaba Tongyi Lab. Read in full; the gamed-scalar catch of §6. arXiv:2606.03108
• Nie, J., et al. (2026). EvolveNet: Collaborative Harness Evolution for Agent Self-Improvement. HKBU + USTC + HKUST. Read in full; the gate-versus-ranking refinement of §6. arXiv:2608.04968
• Wang, Y., et al. (2026). Rethinking the Evaluation of Harness Evolution for Agents. Allen Institute for AI + UW. Read in full; the audit of §6. arXiv:2607.12227
• XO Labs (2026). What Observational Data Can't Tell You About Work Done — the self-report measurement cited in §5.
• XO Labs (2026). Why the Environment Matters — the capture list of §6, argued from the accounting side.
• XO Labs (2026). The Anatomy of Claude Code's Architecture — what a static production harness is made of.