AERead / starcraft

StarCraft: Brood War · OpenBW · (2)Challenger · 4 October 2026

Codex vs Claude, twenty games under one set of rules.

Two coding models each write a program that plays a full game of StarCraft for them, then rewrite it as the game goes on. Codex plays Terran, Claude plays Protoss. The question is not who wins; it is what each one does with resources, information and time, and every decision is on record.

Why this experiment, and what it has to do with AERead

AERead measures whether an agent can deliver value, not merely complete a task. It seats a model in a live market, lets it negotiate in ordinary language, and scores how much of the best available outcome the market actually reaches. Its pilot found the gap where it matters most: once information is hidden, every model tested collapses toward a rule-based baseline.

Two abilities decide that gap, and they are the same two that commercial agent deployments pay for and have the least evidence of: decisions made in real time while the situation changes, and plans that hold over a long horizon. A full game of StarCraft demands both from the first minute, and plays them out in a form anyone can watch and judge in minutes.

So this is the same question in a second arena, with the same evidence discipline: every order and its receipt, every program the model wrote and the reason it gave, and a native replay from each seat, open for anyone to play back. These games are prototypes, not a benchmark score. What they expose are failure classes that transfer well beyond the game:

Capability acquired, never deployed.

Codex builds tanks and parks them at the factory door while its marines die in front. Round 9 Game A, from 13:20.

A rejected action with no fallback, then a policy change.

The engine refuses Claude's attack order; its army waits. At the next rewrite Claude changes its targeting policy instead of retrying. Round 3 Game A, 7:45 and 10:18.

Right diagnosis, late action.

At 9:57 Codex's own program note names the problem: two filled bunkers, no mobile defenders, an attack approaching. Its post-game review says the same. The action came too late. Round 3 Game A.

Unspent budget and idle capacity, against throughput.

Codex banks minerals it never spends and fills bunkers it never uses, by its own reviews. Claude keeps every resource working: three buildings stream Dragoons and the next wave is walking before the first one is done. Round 3 Game A, from 10:50.

One design choice follows from the same problem. A language model is too slow to act every frame, so here it writes a program and the program acts (see Setup). A separate study asks the other half of the question, whether a small model can learn real-time, fine-grained control at all; it is a different setting, not comparable with these games, and is published in the repository.

Setup

MODELCodex or Claude PROGRAMa Python controller GAMEOpenBW, 42 ms per frame writes it, then rewrites it orders, on every observation what it can see every 3 game minutes: keep the program, or rewrite it. After each pair of games: review and plan. one loop per seat, both seats alike
SeatsCodex (Terran) vs Claude (Protoss), one seat each. An engine-side keeper handles standing production, idle workers and supply for both sides alike.
ControlA language model is too slow to act every frame, so it writes a program and the program acts. Before the game each model writes a Python controller that runs on every observation; the model may rewrite it every three game minutes.
Clock and mapOpenBW at 42 ms per frame, map (2)Challenger, 45-minute cap. Same for every game.
Between gamesEach model reviews the pair it just played, lists its own mistakes, and writes the plan it will carry into the next round.
RecordedNative replays from both seats, every order and its receipt, each program version and the reason the model gave for it.
Harness revisionsThree during the afternoon: rev 1 for rounds 1 to 4; rev 2 for rounds 5 to 7, frame pacing; rev 3 for rounds 8 to 10, expansion placement and whole-army orders. None changed the rules, the map, the clock or what a model could see.

Results

20games played
19Claude wins
0Codex wins
1timed out, no result
15/19wins inside 15:00
10:43quickest win · median 13:10
10:0020:0030:0045:00HARNESS REV 1HARNESS REV 2HARNESS REV 32026-10-04 13:19 · program mode · harness rev 1 · Round 1 Game A · Codex (Terran) vs Claude (Protoss) · Claude won at 17:181A2026-10-04 13:20 · program mode · harness rev 1 · Round 1 Game B · Codex (Terran) vs Claude (Protoss) · Claude won at 13:461B2026-10-04 13:50 · program mode · harness rev 1 · Round 2 Game A · Codex (Terran) vs Claude (Protoss) · Claude won at 10:562A2026-10-04 13:49 · program mode · harness rev 1 · Round 2 Game B · Codex (Terran) vs Claude (Protoss) · Claude won at 11:362B2026-10-04 14:11 · program mode · harness rev 1 · Round 3 Game A · Codex (Terran) vs Claude (Protoss) · Claude won at 12:173A2026-10-04 14:11 · program mode · harness rev 1 · Round 3 Game B · Codex (Terran) vs Claude (Protoss) · Claude won at 11:113B2026-10-04 15:10 · program mode · harness rev 1 · Round 4 Game A · Codex (Terran) vs Claude (Protoss) · Claude won at 11:264A2026-10-04 15:10 · program mode · harness rev 1 · Round 4 Game B · Codex (Terran) vs Claude (Protoss) · Claude won at 13:324B2026-10-04 15:39 · program mode · harness rev 2 · Round 5 Game A · Codex (Terran) vs Claude (Protoss) · Claude won at 14:355A2026-10-04 15:39 · program mode · harness rev 2 · Round 5 Game B · Codex (Terran) vs Claude (Protoss) · Claude won at 17:195B2026-10-04 16:13 · program mode · harness rev 2 · Round 6 Game A · Codex (Terran) vs Claude (Protoss) · Claude won at 11:356A2026-10-04 16:16 · program mode · harness rev 2 · Round 6 Game B · Codex (Terran) vs Claude (Protoss) · Claude won at 10:436B2026-10-04 16:33 · program mode · harness rev 2 · Round 7 Game A · Codex (Terran) vs Claude (Protoss) · Claude won at 14:257A2026-10-04 16:33 · program mode · harness rev 2 · Round 7 Game B · Codex (Terran) vs Claude (Protoss) · timed out, no result7B2026-10-04 17:34 · program mode · harness rev 3 · Round 8 Game A · Codex (Terran) vs Claude (Protoss) · Claude won at 12:258A2026-10-04 17:34 · program mode · harness rev 3 · Round 8 Game B · Codex (Terran) vs Claude (Protoss) · Claude won at 12:318B2026-10-04 17:53 · program mode · harness rev 3 · Round 9 Game A · Codex (Terran) vs Claude (Protoss) · Claude won at 20:599A2026-10-04 17:53 · program mode · harness rev 3 · Round 9 Game B · Codex (Terran) vs Claude (Protoss) · Claude won at 23:099B2026-10-04 18:28 · program mode · harness rev 3 · Round 10 Game A · Codex (Terran) vs Claude (Protoss) · Claude won at 13:1010A2026-10-04 18:28 · program mode · harness rev 3 · Round 10 Game B · Codex (Terran) vs Claude (Protoss) · Claude won at 14:0410B

Game clock at the end of each game, in play order; every solid bar is a Claude win, the hatched bar is the 45-minute timeout. Click a bar for that game.

Codex kept buying what it never used.

Its own post-game reviews record thousands of minerals banked, bunkers built but never filled, and tanks parked on the wrong side of the base. Capability acquired, not deployed.

Claude kept every resource working.

Three production buildings streamed Dragoons; the first wave of about twelve reached the Terran base before its tanks were ready, and reinforcements followed in groups.

Both rewrote themselves mid-game.

When the engine refused an order, Claude changed its targeting policy rather than retrying; Codex diagnosed its own weak defence correctly but acted on it too late. The reasoning is on each game page.

The games

Click a game for its replay, orders and the model's own reasoning. The full table with start times, harness revisions and replay files is on the games page.

All twenty games, with replays

What is not here, and why

The two program-mode games played before the add-on fix (Terran could not build Machine Shops, so Codex had no tanks), the games of 2 and 3 October in which a separate model issued the tactical orders, the scripted siege scenarios, and the Laya fine-tuning study. They are different settings and not comparable with the games above; all of them remain in the repository's evidence.