Validation
Hypothesis-first: every empirical claim with its falsifier, bound to numbered experiments E1 through E7.
The validation program is organized hypothesis-first: each load-bearing claim is stated at full strength with its falsifier, and bound to numbered experiments, completed or scheduled.
Results below follow the two evidence modes and pre-registration discipline of methods: mock mode validates machinery, real mode measures agents, never conflated.
The hypotheses
H1: Mint integrity. Completion read from environment-captured state mints no fiction: false claims settle at rate zero, while any self-report channel settles them at approximately the false-claim rate. Falsifier: the environment arm settling any incomplete unit. Experiments: E1, E2 (mock: complete, held) · E4 (real: in progress).
H2: Ledger identification. On real agents, cost per quirq falls with unit index toward a stable floor in persistent environments and stays at cold-start in fresh contexts, holding model, prompt, and task family fixed. Falsifier: flat or rising persistent-arm trajectories, or equal decay in fresh contexts. Experiments: E3 (mock consistency: complete, held) · E4 (real: in progress).
H3: Predictive validity. Audit-corrected QER trend predicts real business outcomes (renewal, expansion, P&L attribution) better than token spend, task counts, or benchmark scores. Falsifier: QER* trend failing to outperform those baselines in instrumented deployments. Experiments: E5, E6 (scheduled).
H4: Budget governance. Under payer-set budgets, benchmark anchoring, and sampled value audits, budget inflation stays within audit tolerance under sustained optimization pressure. Falsifier: systematic budget drift in long-running ledgers despite the mitigations. Experiments: E7 (scheduled).
Completed experiments (E1 to E3, mock mode)
E1: verification source (H1). 200 identical units, an agent that falsely claims done with probability 0.05.
| Arm | False claims | Silently settled | Rate |
|---|---|---|---|
| Self-report | 7 | 7 | 3.5% |
| Environment snapshot | 7 | 0 | 0.0% |
E2: hardening (H1). 50 units, an agent that guts the test instead of fixing the code.
| Arm | Checks green | Intent actually met |
|---|---|---|
| Gameable definition of done | 100% | 0% |
| Plus one verification-surface check | 0% (all caught) | 0% |
One hash-equality check on the test file converts total silent failure into total detection.
E3: tenure (H2). Twelve similar units, persistent vs fresh environment: cost decays 85.6% to the execution floor (~800 tokens) by unit six in the persistent arm; the fresh arm stays flat at ~5,540. Mock-mode caveat in full: the scripted agent implements the cost model, so this arm demonstrates harness consistency, not agent behavior.
The roadmap (E4 to E7)
- E4 (H1, H2): real-mode replication of E1 to E3 with a production coding agent and measured tokens. The false-claim rate becomes a measurement, and the tenure curve becomes evidence. In progress.
- E5 (H3): pilot ledgers: the dashboard instrumented on real work across at least three unit types.
- E6 (H3): predictive study: QER* trend vs token spend, task counts, and benchmark scores as predictors of renewal, expansion, and P&L attribution.
- E7 (H4): longitudinal budget-drift audit under the gaming mitigations.
Results and per-run data are published here as they land.
The claims are on the record, and every one is tiered against its evidence in claims and limitations.