Everything needed to read the numbers on the overview and evidence pages: the models and their roles, the four cases, the exact scoring equation, the baseline algorithms, the seed and held-out protocol, what "validated" means, and where the raw data lives. This is a summary of a live research record; the repository is the source of truth.
Each candidate is placed in one seat of a five-agent market; the other four seats (the panel) are a frozen gemini-2.5-flash at temperature 0. A frozen gemini-2.5-flash also serves as the environment's compiler (turns accepted agreements into asset transfers) and verifier. Candidates run at temperature 0; non-Gemini models are served via OpenRouter, Gemini models via the AI-Studio native endpoint. Development leaderboard scores are pooled AER over the four cases (ratio of sums), 30 development seeds each, bootstrap 95% intervals.
| Model | Provider | Dev pooled AER | 95% CI | Held-out |
|---|---|---|---|---|
| gemini-3.5-flash | +.209 | [.176, .243] | ◇ yes | |
| kimi-k2 | Moonshot | +.156 | [.131, .183] | ◇ yes |
| deepseek-v4-flash | DeepSeek | +.116 | [.089, .144] | ◇ yes |
| greedy floor (baseline) | — | +.104 | [.092, .116] | ◇ yes |
| gemini-2.5-flash | +.083 | [.065, .101] | ◇ yes | |
| minimax-m3 | MiniMax | +.028 | [.014, .045] | ◇ yes |
| glm-4.7-flash | Z.ai | +.014 | [.005, .025] | ◇ yes |
| random (baseline) | — | +.008 | [.003, .015] | ◇ yes |
Every model and baseline now carries a private held-out score (◇). The four cross-provider models were added to the held-out set on 2026-07-22 (480 runs, ranking preserved, no overfitting signature). Exact API model versions per run are recorded in each run's run_meta.json / inference manifest.
Each case emphasizes one capability (the cases also vary in other dimensions; read them as a family, not a single-variable manipulation). All are five agents / five resources.
| Case | Emphasis | Rounds | Visibility | Settlement |
|---|---|---|---|---|
| case01 · bilateral | visible bilateral IR trade | 4 | full public | atomic commit, ≤2 rows |
| case02 · clearing | multiparty partial clearing | 4 | full public | non-atomic, ≤4 rows |
| case03 · discovery | hidden counterparties + solicitation | 6 | limited contact (sees 1 of 4) | non-atomic, ≤4 rows |
| case04 · consent | private information + sign-off | 4 | limited contact (sees 2 of 4) | non-atomic, ≤4 rows, private acceptance |
Reward is realized economic outcome (welfare) plus valid exchange, never string-matching a preferred answer. Per-case world generators are pinned by content hash; the exact distributions are in the pinned case configs in the repository.
The Aggregate Efficiency Ratio is market-level welfare (the whole five-seat market), not the tested agent's private surplus. For each case, pooled over seeds:
AER = Σs ( Wrealized − Winitial ) ⁄ Σs ( W* − Winitial )
wstar_fallback tier; W* is the god's-eye ceiling, so the score is a conservative lower bound on efficiency against any information-constrained frontier for positive gains; for value destruction it shrinks the magnitude of the negative score toward zero. The information-constrained (Bayes-achievable) tiers exact_wbayes / mc_wbayes are implemented for the bundle world and in development for these cases.Companion distribution metric (not pooled into AER). AER measures total realized welfare; a companion capture share reports how much of it the seat under test keeps: Σ agent-1 utility gain / Σ all-agent gain, from the recorded per-round utility deltas. Its reference is par: the surplus-weighted equal split among the parties to each trade, with parties determined at the transfer level (agents connected by actual transfers within a settlement; connected components separate disjoint deals compiled in one round). Reported on the evidence page for the held-out board, where full traces are retained.
Bootstrap: resampling unit is the world (seed); 2000 replicates; percentile intervals. Differences (e.g. model − baseline) are paired within-world. Confidence intervals on the pooled leaderboard are marginal; see the held-out margin-over-floor analysis on the evidence page for paired difference intervals.
The central finding, that a trivial deterministic baseline stays competitive, rests on the greedy floor, so here is exactly what it does, and exactly where its privilege lies. It may only contact the agents the case's information protocol makes visible (same contact set as the model), but within that set it reads the world's true utilities on both sides of a candidate swap; the models must infer those from dialogue. Its chosen trade is then emitted as a canonical settlement line through the same compiler/verifier/consent path the models use: every greedy episode's manifest shows compiler and verifier calls, and counterparties must consent in language (against a panel that responded too vaguely to compile, greedy's score collapsed from .150 to .000 on identical proposals; the funnel binds it).
greedy_floor(world, protocol, rounds):
gain = 0
for round in 1..rounds:
controller = rotate(controllers)
visible = agents controller may contact this round # same visibility as the LLM
swap = best mutually-IR bilateral trade among `visible` # exact, subject to the row limit
if welfare_gain(swap) > 0:
emit(swap as canonical text) # compiled + verified + consented like any model proposal
gain += welfare_gain(swap) if settled # settles only if counterparties accept
return gain # scored by the same AER as the models
So the greedy floor is not naive: each round it takes the exact welfare-maximizing one-step bilateral trade among the agents it can see. It cannot discover beyond its visible set, cannot plan across rounds, and cannot do multiparty clearing. random proposes a feasible random transfer each round; no-op proposes nothing (always scores 0, and is omitted from the boards). The greedy floor uses the same oracle-computed welfare as every model, so the comparison is on outcome, not on parsing. Its residual privileges are decision inputs (true two-sided utilities, exact feasibility) and canonical phrasing, and the phrasing half is now priced: re-rendering the same greedy decisions as a polite model-style pitch (explicit settlement rows included) collapses the floor from +.113 to +.011 on the identical 120 held-out worlds, a paired phrasing tax of +.102 [+.091, +.114], leaving the free-form floor statistically indistinguishable from the random baseline. The loss is at the consent stage (all 240 episodes compile and replay-verify; panelists simply decline verbose pitches of trades they accept when stated tersely). Every model on the board clears this interface-fair floor.
The private held-out seed set + generator salt are kept locally and never published; a leaked seed set would void the held-out claim.
Every scored run is replay-verified: re-running the recorded transcript through the scoring pipeline reproduces the numbers byte-for-byte. "Validated" on this site means replay-consistency and numerical reproducibility, not human review of every transcript, and not independent external replication. Two robustness checks are reported on the evidence page: a four-case panel swap (frozen-LLM panel → scripted-rational panel) and the private held-out re-scoring.
Known open validation items: a compiler audit (per-model false-accept / false-reject rates on a human-labeled sample), multi-provider and mixed panels, and a structured-action condition that bypasses natural-language compilation.
run_meta.json (exact per-seat model versions), the full transcript (trace.jsonl), the inference manifest, and a summary.json with per-cell AER + CI + exact n.This page and the CSV are a curated summary; the repository run records are the source of truth. Prompts, role instructions, and full per-seat inference settings are recorded per run rather than reproduced here.