Methods & artifacts

01 / Models & rolesThe seven-model A2A leaderboard.

Each candidate is placed in one seat of a five-agent market; the other four seats (the panel) are a frozen gemini-2.5-flash at temperature 0. A frozen gemini-2.5-flash also serves as the environment's compiler (turns accepted agreements into asset transfers) and verifier. Candidates run at temperature 0; non-Gemini models are served via OpenRouter, Gemini models via the AI-Studio native endpoint. Development leaderboard scores are pooled AER over the four cases (ratio of sums), 30 development seeds each, bootstrap 95% intervals.

Every model and baseline now carries a private held-out score (◇). The four cross-provider models were added to the held-out set on 2026-07-22 (480 runs, ranking preserved, no overfitting signature). Exact API model versions per run are recorded in each run's run_meta.json / inference manifest.

02 / CasesFour five-agent market cases (cases_v0).

Each case emphasizes one capability (the cases also vary in other dimensions; read them as a family, not a single-variable manipulation). All are five agents / five resources.

Reward is realized economic outcome (welfare) plus valid exchange, never string-matching a preferred answer. Per-case world generators are pinned by content hash; the exact distributions are in the pinned case configs in the repository.

03 / Scoring: the AEROne equation.

The Aggregate Efficiency Ratio is market-level welfare (the whole five-seat market), not the tested agent's private surplus. For each case, pooled over seeds:

AER = Σs ( Wrealized − Winitial ) ⁄ Σs ( W* − Winitial )

• W_realized: the market's final total welfare net of coordination costs, if the settlement passes the feasibility-∧-authorization gate; a failed gate contributes W_initial (that episode scores 0). Gate = the mechanism is feasible and every debited agent consented.
• W*: the full-information social optimum: a social planner maximizing total welfare, computed ex-post from the realized world. Not a best-response agent, not a mechanism.
• Ratio of sums (micro-pooled); an equal-weighted-across-cases macro companion is also reported so a large-opportunity case can't dominate.
• Raw: value destruction stays negative; the score cannot exceed 1 (realized welfare never beats the optimum). A degenerate zero-opportunity case (denominator ≤ 0) is dropped, not scored 0.
• Tier: every displayed number is the wstar_fallback tier; W* is the god's-eye ceiling, so the score is a conservative lower bound on efficiency against any information-constrained frontier for positive gains; for value destruction it shrinks the magnitude of the negative score toward zero. The information-constrained (Bayes-achievable) tiers exact_wbayes / mc_wbayes are implemented for the bundle world and in development for these cases.

Companion distribution metric (not pooled into AER). AER measures total realized welfare; a companion capture share reports how much of it the seat under test keeps: Σ agent-1 utility gain / Σ all-agent gain, from the recorded per-round utility deltas. Its reference is par: the surplus-weighted equal split among the parties to each trade, with parties determined at the transfer level (agents connected by actual transfers within a settlement; connected components separate disjoint deals compiled in one round). Reported on the evidence page for the held-out board, where full traces are retained.

Bootstrap: resampling unit is the world (seed); 2000 replicates; percentile intervals. Differences (e.g. model − baseline) are paired within-world. Confidence intervals on the pooled leaderboard are marginal; see the held-out margin-over-floor analysis on the evidence page for paired difference intervals.

04 / BaselinesThe greedy floor, defined.

The central finding, that a trivial deterministic baseline stays competitive, rests on the greedy floor, so here is exactly what it does, and exactly where its privilege lies. It may only contact the agents the case's information protocol makes visible (same contact set as the model), but within that set it reads the world's true utilities on both sides of a candidate swap; the models must infer those from dialogue. Its chosen trade is then emitted as a canonical settlement line through the same compiler/verifier/consent path the models use: every greedy episode's manifest shows compiler and verifier calls, and counterparties must consent in language (against a panel that responded too vaguely to compile, greedy's score collapsed from .150 to .000 on identical proposals; the funnel binds it).

greedy_floor(world, protocol, rounds):
  gain = 0
  for round in 1..rounds:
    controller = rotate(controllers)
    visible = agents controller may contact this round     # same visibility as the LLM
    swap = best mutually-IR bilateral trade among `visible` # exact, subject to the row limit
    if welfare_gain(swap) > 0:
      emit(swap as canonical text)   # compiled + verified + consented like any model proposal
      gain += welfare_gain(swap) if settled                # settles only if counterparties accept
  return gain                                              # scored by the same AER as the models

So the greedy floor is not naive: each round it takes the exact welfare-maximizing one-step bilateral trade among the agents it can see. It cannot discover beyond its visible set, cannot plan across rounds, and cannot do multiparty clearing. random proposes a feasible random transfer each round; no-op proposes nothing (always scores 0, and is omitted from the boards). The greedy floor uses the same oracle-computed welfare as every model, so the comparison is on outcome, not on parsing. Its residual privileges are decision inputs (true two-sided utilities, exact feasibility) and canonical phrasing, and the phrasing half is now priced: re-rendering the same greedy decisions as a polite model-style pitch (explicit settlement rows included) collapses the floor from +.113 to +.011 on the identical 120 held-out worlds, a paired phrasing tax of +.102 [+.091, +.114], leaving the free-form floor statistically indistinguishable from the random baseline. The loss is at the consent stage (all 240 episodes compile and replay-verify; panelists simply decline verbose pitches of trades they accept when stated tersely). Every model on the board clears this interface-fair floor.

05 / Seeds & held-outDevelopment vs private held-out.

• Development: 30 pinned seeds per case (world generators pinned by content hash). Iterated against during design; the leaderboard's primary numbers.
• Held-out: 30 private seeds per case, generated from a secret salt, never inspected before scoring, kept out of the public configs. They guard against us tuning the benchmark to convenient worlds, not against a model memorizing (the models are never trained).
• Scope of the held-out claim: these are new seeds from the same case generators: within-generator replication, not out-of-distribution transfer. The paired held-out margin over the greedy floor is significant on bilateral, consent, and discovery (slim), but the interval spans zero on clearing (11 paired worlds). Model-vs-model ranking intervals on held-out are still to be reported.
• Per-cell n: 30 development + 30 held-out worlds per (model, case), except where a provider/replay failure drops a cell (e.g. 11 paired held-out worlds on clearing). Exact per-cell counts are in each run's summary.

The private held-out seed set + generator salt are kept locally and never published; a leaked seed set would void the held-out claim.

06 / ValidationWhat "validated" means.

Every scored run is replay-verified: re-running the recorded transcript through the scoring pipeline reproduces the numbers byte-for-byte. "Validated" on this site means replay-consistency and numerical reproducibility, not human review of every transcript, and not independent external replication. Two robustness checks are reported on the evidence page: a four-case panel swap (frozen-LLM panel → scripted-rational panel) and the private held-out re-scoring.

Known open validation items: a compiler audit (per-model false-accept / false-reject rates on a human-labeled sample), multi-provider and mixed panels, and a structured-action condition that bypasses natural-language compilation.

07 / ArtifactsWhere the data lives.

• Results (downloadable): results_pilot_v0.csv: per model, per case: development AER + 95% CI and the held-out point estimate.
• Scoring + environment code: the AERead repository: the exchange environment, the AER scorer, the oracle carve-out, and the V1 runner, together with the four pinned case configs.
• Per-run records: each run preserves run_meta.json (exact per-seat model versions), the full transcript (trace.jsonl), the inference manifest, and a summary.json with per-cell AER + CI + exact n.
• Held-out generalization writeup: available in the repository (dated 2026-07-22).

This page and the CSV are a curated summary; the repository run records are the source of truth. Prompts, role instructions, and full per-seat inference settings are recorded per run rather than reproduced here.

08 / ReferencesRelated work.

• Myerson & Satterthwaite (1983), Efficient mechanisms for bilateral trading: the impossibility that motivates the information-constrained denominator (in development).
• OpenAI (2025), GDPval: deployment-verdict framing for capability evaluation.
• Full bibliography in the repository.