AERead · Pilot V0 methods & artifacts

01 / Models & rolesThe seven-model A2A leaderboard.

Each candidate is placed in one seat of a five-agent market; the other four seats (the panel) are a frozen gemini-2.5-flash at temperature 0. A frozen gemini-2.5-flash also serves as the environment's compiler (turns accepted agreements into asset transfers) and verifier. Candidates run at temperature 0; non-Gemini models are served via OpenRouter, Gemini models via the AI-Studio native endpoint. Development leaderboard scores are pooled AER over the four cases (ratio of sums), 30 development seeds each, bootstrap 95% intervals.

Model	Provider	Dev pooled AER	95% CI	Held-out
gemini-3.5-flash	Google	+.209	[.176, .243]	◇ yes
kimi-k2	Moonshot	+.156	[.131, .183]	◇ yes
deepseek-v4-flash	DeepSeek	+.116	[.089, .144]	◇ yes
greedy floor (baseline)	—	+.104	[.092, .116]	◇ yes
gemini-2.5-flash	Google	+.083	[.065, .101]	◇ yes
minimax-m3	MiniMax	+.028	[.014, .045]	◇ yes
glm-4.7-flash	Z.ai	+.014	[.005, .025]	◇ yes
random (baseline)	—	+.008	[.003, .015]	◇ yes

Every model and baseline now carries a private held-out score (◇). The four cross-provider models were added to the held-out set on 2026-07-22 (480 runs, ranking preserved, no overfitting signature). Exact API model versions per run are recorded in each run's run_meta.json / inference manifest.

02 / CasesFour five-agent market cases (cases_v0).

Each case emphasizes one capability (the cases also vary in other dimensions — read them as a family, not a single-variable manipulation). All are five agents / five resources.

Case	Emphasis	Rounds	Visibility	Settlement
case01 · bilateral	visible bilateral IR trade	4	full public	atomic commit, ≤2 rows
case02 · clearing	multiparty partial clearing	4	full public	non-atomic, ≤4 rows
case03 · discovery	hidden counterparties + solicitation	6	limited contact — sees 1 of 4	non-atomic, ≤4 rows
case04 · consent	private information + sign-off	4	limited contact — sees 2 of 4	non-atomic, ≤4 rows, private acceptance

Reward is realized economic outcome (welfare) plus valid exchange — never string-matching a preferred answer. Per-case world generators are pinned by content hash; the exact distributions are in the pinned case configs in the repository.

03 / Scoring — the AEROne equation.

The Aggregate Efficiency Ratio is market-level welfare — the whole five-seat market — not the tested agent's private surplus. For each case, pooled over seeds:

AER = Σ_s ( W_realized − W_initial ) ⁄ Σ_s ( W* − W_initial )

W_realized — the market's final total welfare net of coordination costs, if the settlement passes the feasibility-∧-authorization gate; a failed gate contributes W_initial (that episode scores 0). Gate = the mechanism is feasible and every debited agent consented.
W* — the full-information social optimum: a social planner maximizing total welfare, computed ex-post from the realized world. Not a best-response agent, not a mechanism.
Ratio of sums (micro-pooled); an equal-weighted-across-cases macro companion is also reported so a large-opportunity case can't dominate.
Raw: value destruction stays negative; the score cannot exceed 1 (realized welfare never beats the optimum). A degenerate zero-opportunity case (denominator ≤ 0) is dropped, not scored 0.
Tier: every displayed number is the wstar_fallback tier — W* is the god's-eye ceiling, so the score is a conservative lower bound on efficiency against any information-constrained frontier for positive gains; for value destruction it shrinks the magnitude of the negative score toward zero. The information-constrained (Bayes-achievable) tiers exact_wbayes / mc_wbayes are implemented for the bundle world and in development for these cases.

Bootstrap: resampling unit is the world (seed); 2000 replicates; percentile intervals. Differences (e.g. model − baseline) are paired within-world. Confidence intervals on the pooled leaderboard are marginal — see the held-out margin-over-floor analysis on the evidence page for paired difference intervals.

04 / BaselinesThe greedy floor, defined.

The central finding — that a trivial deterministic baseline stays competitive — rests on the greedy floor, so here is exactly what it does. It sees the same visible state the model does (no god's-eye access) and never touches the language compiler; its welfare is computed directly.

greedy_floor(world, protocol, rounds):
  gain = 0
  for round in 1..rounds:
    controller = rotate(controllers)
    visible = agents controller may contact this round     # same visibility as the LLM
    swap = best mutually-IR bilateral trade among `visible` # exact, subject to the row limit
    if welfare_gain(swap) > 0:
      apply(swap); gain += welfare_gain(swap)
  return gain                                              # scored by the same AER as the models

So the greedy floor is not naive: each round it takes the exact welfare-maximizing one-step bilateral trade among the agents it can see. It cannot discover beyond its visible set, cannot plan across rounds, and cannot do multiparty clearing. random proposes a feasible random transfer each round; no-op proposes nothing (always scores 0, and is omitted from the boards). The greedy floor uses the same oracle-computed welfare as every model, so the comparison is on outcome, not on parsing.

05 / Seeds & held-outDevelopment vs private held-out.

Development: 30 pinned seeds per case (world generators pinned by content hash). Iterated against during design — the leaderboard's primary numbers.
Held-out: 30 private seeds per case, generated from a secret salt, never inspected before scoring, kept out of the public configs. They guard against us tuning the benchmark to convenient worlds, not against a model memorizing (the models are never trained).
Scope of the held-out claim: these are new seeds from the same case generators — within-generator replication, not out-of-distribution transfer. The paired held-out margin over the greedy floor is significant on bilateral, consent, and discovery (slim), but the interval spans zero on clearing (11 paired worlds). Model-vs-model ranking intervals on held-out are still to be reported.
Per-cell n: 30 development + 30 held-out worlds per (model, case), except where a provider/replay failure drops a cell (e.g. 11 paired held-out worlds on clearing). Exact per-cell counts are in each run's summary.

The private held-out seed set + generator salt are kept locally and never published; a leaked seed set would void the held-out claim.

06 / ValidationWhat "validated" means.

Every scored run is replay-verified: re-running the recorded transcript through the scoring pipeline reproduces the numbers byte-for-byte. "Validated" on this site means replay-consistency and numerical reproducibility — not human review of every transcript, and not independent external replication. Two robustness checks are reported on the evidence page: a four-case panel swap (frozen-LLM panel → scripted-rational panel) and the private held-out re-scoring.

Known open validation items: a compiler audit (per-model false-accept / false-reject rates on a human-labeled sample), multi-provider and mixed panels, and a structured-action condition that bypasses natural-language compilation.

07 / ArtifactsWhere the data lives.

Results (downloadable): results_pilot_v0.csv — per model, per case: development AER + 95% CI and the held-out point estimate.
Scoring + environment code: the AERead repository — the exchange environment, the AER scorer, the oracle carve-out, and the V1 runner, together with the four pinned case configs.
Per-run records: each run preserves run_meta.json (exact per-seat model versions), the full transcript (trace.jsonl), the inference manifest, and a summary.json with per-cell AER + CI + exact n.
Held-out generalization writeup: available in the repository (dated 2026-07-22).

This page and the CSV are a curated summary; the repository run records are the source of truth. Prompts, role instructions, and full per-seat inference settings are recorded per run rather than reproduced here.

Methods & artifacts