Agent Context Research: The Evidence So Far
A guided evidence map of XO's agent-context studies, including the replication that revised the initial pilot and the questions each follow-up answered.
This page is a reading map for the research program, not a new experiment. It shows how the question changed as the evidence accumulated: from whether a richer workspace helps, to whether agents consume it, to which information is actually worth supplying.
The initial headline did not survive replication. The first pilot found a richer workspace coming out cheaper on one task. The later 84-run matrix did not reproduce that result: across two agents and two tasks, more context trended costlier. The caching observation and the measurement method remained useful, but the early claim that richer context generally lowers the token bill did not generalize.
The evidence path
The sequence below follows how the questions developed. Some later studies branch from the same result rather than forming a strict publication chronology.
| Stage | Study | Question | What changed |
|---|---|---|---|
| 1 | How the Environment Affects Agent Performance and Token Cost | Does progressively richer project context change success or token cost? | In a small pilot, every environment completed the task and richer conditions sometimes used fewer effective tokens. The full scaffold's input was heavily cached. The paper correctly labels this as a pilot. |
| 2 | The Incurious Agent | Does the cost result repeat, and what context do agents actually read? | Across 84 runs, the cheaper-context headline did not repeat. Both agents largely skipped the deepest curated material; context that went unread could not improve the outcome. This revised the pilot's main cost claim and shifted attention from authoring to consumption. |
| 3 | The Self-Sufficient Agent | If the tasks and repository get harder, does a bare agent finally need the documentation? | On this repository and task family, Codex and Claude solved all nine functional landmines without docs. They failed mainly on arbitrary project-specific formats. The useful surface for context narrowed from general capability to local convention. |
| 4a | Relevance, Not Volume | Is the amount of context the lever, or whether it contains the needed rule? | A generic contract and a same-length rule-bearing contract produced radically different conformance. On these tasks, useful context was information the agent needed and could not derive—not simply more documentation. |
| 4b | Curiosity Comparison Between Agents | Is shallow environment reading common to every agent? | Gemini read more, and read more as the workspace became richer, providing a counterexample to the earlier Codex/Claude pattern. The probe was small, and none of the three agents mined the deepest seeded memory. |
A related evaluation track
Fable 5 vs Opus 4.8: A Coding-Agent Evaluation uses the same broader discipline—pinned tasks, objective gates, captured runs, and explicit limitations—but asks a different question: how two models compare on real engineering work. Both passed the four completed easy-to-medium tasks, with an early directional efficiency and scope-discipline advantage for Fable. Six harder tasks never ran after Fable was suspended, so the study does not claim a final model ranking.
The synthesis
Across the context studies, four conditions now matter more than document count:
- Headroom: the agent must have a gap that information can close. Context cannot improve a task the bare agent already solves at the ceiling.
- Relevance: the information must supply something needed and underivable, such as a local wire-format convention.
- Consumption: the agent must actually encounter the information. A perfect memory tree that stays unopened has no treatment effect.
- Agent behavior: reading habits differ. Results from one agent cannot be assumed to describe every agent.
This is narrower than the pilot's hopeful story, but more actionable: do not ask only how much context a workspace contains. Ask what gap exists, which fact closes it, where the agent will encounter that fact, and whether the chosen agent reads that part of the environment.
Choose a reading path
- Start with the correction: read the pilot, then The Incurious Agent.
- Understand what agents already know: continue to The Self-Sufficient Agent.
- Turn the result into documentation practice: read Relevance, Not Volume.
- Compare reading behavior across agents: read Curiosity Comparison Between Agents.
- Study the evaluation method itself: read Fable 5 vs Opus 4.8.
Boundaries that still matter
The studies cover a small number of repositories, tasks, agents, and runs. Several results are directional, and the experiments deliberately use different task families to answer different questions. Treat the series as an accumulating set of bounded findings—not a universal claim that context always helps, never helps, or that one agent is categorically best.