Experiments

The setup is fixed: a 20-block × 256-channel ResNet, supervised distillation from Stockfish 17 at search-depth 15, multipv=8 soft targets at temperature T = 1 pawn, 100-game evals against a calibrated Stockfish opponent at fixed UCI_Elo. The baseline scores 1,807 Elo at 800 MCTS sims/move on a 5M-position dataset.

Each experiment below changes one variable, holds everything else constant, and states the hypothesis it was meant to test, the result, and the verdict. Code links land directly on the launcher / training script for each.

The Baseline Trajectory

100-game evals vs Stockfish UCI=1350, 800 MCTS sims/move, 20-block network.

100-game evals at this anchor have a CI of roughly ±100 Elo, so the ep-10 spike (1,862) is mostly noise — not a real swing. The trend at this resolution is just “around 1,700–1,800 across epochs”; we take ep 20 = 1,807 as the canonical baseline because it’s the last training checkpoint, not because it’s deterministically best. Loss plateaus around 2.59 by epoch 10; top-1 accuracy plateaus ~0.34. This is the “1,800 ceiling” the ablations probe.

→ code: train_supervised.py · eval.py

Network Capacity — Doubling Depth Didn’t Help

Hypothesis: the ceiling is the network running out of representational capacity. Doubling the depth (40 blocks instead of 20, ~48M params instead of 24M) should buy Elo if so.

Setup: identical to the baseline except n_blocks=40. Same 5M positions, same loss, same 20 epochs. Single L40S, batch 512 (the deeper net costs ~2× per batch so we cut batch in half to keep memory sane). 13.5 min/epoch — close to the baseline because deeper-but- narrower-batch trades roughly cancel.

The UCI=1,800 anchor is where the score sits near 0.5 and the Elo CI is tightest — read that column.

Result. ~1,810 Elo, identical to the baseline. More parameters bought nothing on this data. The ceiling is not a capacity problem.

Why a deeper net didn’t help: 5M positions for a 24M-param network is already in the over-parameterized regime. Top-1 was already 0.34 with the smaller net and stays 0.34 with the bigger one. The student is hitting a teacher-signal limit, not a representation limit.

→ code: d15-40x256-eu.sh

Soft vs Hard Targets

Hypothesis: soft multipv targets (a distribution over Stockfish’s top-8 candidate moves, softmax of centipawn scores) carry more useful information than hard one-hot (“the move Stockfish actually played”). The student should learn faster from richer targets.

Setup: identical recipe in both arms — same network, same data, same epoch count. Only the policy target differs:

• Hard: π(played_move) = 1, all else 0. Standard supervised classification.
• Soft: π(move_k) = softmax(cp_k / 100). At T = 1 pawn, a 50-cp gap between two moves becomes a 62 / 38 split, not 100 / 0.

The soft version is what every other experiment on this page uses.

Result Depends on Teacher Strength

At a weak teacher, soft targets actually underperform one-hot by ~130 Elo — the student learns to hedge between moves the teacher itself wasn’t sure about, so its policy is fuzzier than necessary. At a strong teacher the soft distribution becomes sharply peaked and the hedge cost disappears; soft and hard converge.

Verdict. Soft targets help when the teacher is strong enough that its multipv ranking actually reflects move quality. Below that, the extra “information” is mostly the teacher’s own uncertainty, which hurts the student. Picking soft vs hard isn’t a free axiom — it trades off against teacher quality.

Practical consequence. Everything above d10 in this project uses soft targets. A sharper temperature (e.g. T = 0.3) is on the roadmap — it would keep some multipv ranking information but force more commitment from the student.

→ code: stockfish_data.py (the softmax(cp/100*T) construction)

Eval-Side Search — +277 Elo From Search Alone

Hypothesis: the baseline is evaluated at 800 MCTS sims/move (AlphaZero’s training-time value). At competitive play AlphaZero used many tens of thousands of sims, and Lc0 reports policy priors that transfer well to deeper inference search.

Setup: same checkpoint (d15 epoch 20), re-eval at --sims 4000. Single g6.4xlarge, 104 games, 93.6 min wallclock. No retraining, just deeper search at inference.

Result. +277 Elo from search alone, no retraining. The model sweeps 101 of 104 games at 4,000 sims; score saturation means we’re now hitting the ceiling of the opponent’s strength rather than the agent’s.

Interpretation. Distilled priors transfer well to deeper inference- time search — exactly what Lc0 demonstrated empirically. The network’s policy prior guides MCTS toward strong moves; more rollouts let MCTS refine that prior into a sharper visit distribution. The student becomes a better chess player not by learning more, but by thinking longer. Conceptually similar to the policy-improvement step in KataGo’s Playout Cap Randomization — more compute → sharper target.

Practical consequence. The “1,800 ceiling” reported from routine evals is a floor, not a ceiling. There’s ~280 Elo trivially available from deeper eval search. For honest comparison with AlphaZero, the relevant number is the search-saturated one.

A full sims sweep is in progress — same checkpoint, eval at 200 / 800 / 2,000 / 4,000 / 8,000 / 16,000 sims. The Elo-vs-log(sims) curve will reveal the saturation point. The machinery (elo_bisect.py) is in place.

→ code: eval-deep-sims.sh · eval-c-ep5-sims4000.sh

Data Scale — 6× More Positions (The Bitter Lesson?)

Hypothesis: the baseline used a 5M-position subsample of an available 30M-position dataset. Maybe the recipe is fine and we’re just data-starved.

Setup: 20×256 net (same as baseline), full ~30M positions, no MAX_POSITIONS truncation. BATCH_SIZE=2048 to keep iteration count reasonable (~14,650 batches/epoch). In-RAM via 256 GB on a g6e.8xlarge (32 vCPU + 1× L40S). 20 epochs, ~49 min/epoch, ~16 h wallclock.

Why this is 3× slower per epoch than the capacity experiment. Same single L40S in both. The wallclock difference comes from raw compute — 2× per batch (30M÷5M ≈ 1× but BATCH_SIZE=2048 vs 512 means 1.5× batches/epoch on top), so ~3× expected. Actual was ~3.6×. The extra 20% is OS page-cache cold start on epoch 1 (15 GB of memmapped files page in) plus DataLoader overhead.

Results

100-game evals vs Stockfish at UCI=1,800.

The data ablation peaks at ep 10: 2,009 Elo at the tight UCI=1,800 anchor.

Same Teacher, More Data

Same teacher, 6× more data → +335 Elo at the easier anchor.

Stronger Teacher vs More Data

Verdict: the bottleneck on the original baseline was data. Six times more positions buys ~200 Elo at the calibrated anchor — even with a weaker teacher (depth-10 vs depth-15).

The bitter-lesson echo. Sutton’s Bitter Lesson claims that, across 70 years of AI, methods that leverage computation (scale and search) consistently outperform methods that build in human structure (handcrafted features, deeper / cleverer architectures). Our three ablations all point the same way at this small scale:

• Network capacity (more structure / more parameters) → no gain.
• Eval-side search (more compute) → +277 Elo.
• Data scale (more compute spent generating training signal) → +199 Elo.

The two confirmed levers are both forms of “more compute”; the rejected lever is the “smarter network” intuition. The bitter lesson, replayed at $50 budget on a single GPU.

→ code: d10-full30m.sh · stockfish_data.py

Stacking Search + Data Scale Together

The natural follow-up: apply both confirmed ablations to the same checkpoint. The d10-30M weights at sims=4,000 give us the project’s strongest measurements:

+379 Elo from the baseline. The deep-search numbers don’t quite add: search recovered +277 on the weaker d15 5M prior but only ~+105 on top of the data-scaled prior. The headline 2,189 figure is ep 15’s better learned value combined with the deep-search rollouts; the tightest CI sits at 2,110 against the stronger UCI=2,000 opponent. The curve peaks at ep 15 and dips slightly by ep 20 — the model is already past its optimum at the 20×256 / 30M scale.

→ code: eval-c-ep5-sims4000.sh · eval-c-ep5-sims2000-uci1800.sh

Stronger Teacher at Full Data Scale — d15 at 46M

The natural follow-up. d10-30M peaked at 2,189. d15 was the original plan; we settled for d10 because the d15-250K datagen run was still in flight at the time. It finished — 426K games / 45.9M positions of Stockfish d15 multipv=8 at temperature 1, the 1.5× scale-up of the d10-30M corpus that produced the prior peak.

The 2×2 — R1/R2 × v1/v2

The d15-46M training rolled out as a 2×2 ablation on the same dataset. The two axes are network capacity and LR schedule:

The two axes test two different “is this the bottleneck?” hypotheses:

• R1 vs R2 — is the d15 ceiling a capacity issue? R1 doubles depth; R2 keeps the smaller depth but adds 10× weight decay + halved LR. Both v1 runs landed in the same 2,120–2,150 sims=4,000 band, so the answer is no — at constant LR, capacity wasn’t binding.
• v1 vs v2 — was the constant-LR plateau actually a LR-schedule issue? Cosine + warmup beat constant in both arms (R1 v2 +63 over R1 v1; R2 v2 +147 over R2 v1), so the answer is yes — constant LR was oscillating instead of converging, and the schedule recovered most of the d15-over-d10 advantage we expected on day one.

Why R1 v2 and R2 v2 were killed early

Both cosine runs were killed at 2026-05-26 17:15 UTC, before their nominal end (R1 v2 had 25 of 40 epochs left; R2 v2 had 8 of 30 left). R2 v2 was on spot at $0.66/h ($5 left to finish); R1 v2 was OD g6e.8xlarge at ~$2.69/h, which would have been ~$320 over the remaining 5 days. Three reasons it made sense to stop:

1. The peak landed early. R2 v2’s 2,285 is at ep 4 of 30; R1 v2’s 2,209 is at ep 7 of 40. Subsequent epochs at sims=800 were bouncing 1,860–2,055 with no clear upward trend in either run — late-epoch cosine convergence is unlikely to add another +100 Elo.
2. The autoeval daemon is offline (operator laptop closed since 2026-05-26 ~10:50 UTC). Even if late epochs had improved, we wouldn’t measure them. Training without measurement is a waste.
3. The cheap follow-ups give better information per dollar. A sims=8,000 eval on R2 v2 ep 4 ($5, in flight as i-04f84755196ee0163) and a sims=4,000 eval on R2 v2 ep 14 — the highest sims=800 ckpt of the run that the daemon never deep-eval’d — ($5, in flight as i-0b24ea8a7fdd7d149) tighten our knowledge of the existing peak far more than ten more training epochs would.

Everything trained is still on S3 (s3://wm-chess-library-594561963943/d15-mpv8-T1-g250000-20260519T0412Z/checkpoints/); nothing was lost by killing the trainers. If a follow-up justifies more epochs, we resume from the last saved ckpt.

sims=800 trajectory — v1 (constant LR) (UCI=1,800 anchor)

100-game evals, daemon-fired per ckpt:

Per-ckpt CIs are ±70 Elo wide at the 100-game budget, so the swings between adjacent epochs are mostly noise. The trend at this resolution: both v1 runs wandered 1,850–1,950 in the sims=800 anchor with no upward drift past epoch ~7. R2 v1’s curve is statistically indistinguishable from R1 v1’s despite half the per-epoch compute.

sims=4,000 deep-read — v1 (constant LR)

A second daemon (wm-deep-eval-daemon.sh) fires sims=4,000 follow-ups on any ckpt whose sims=800 score crosses threshold (started at 1,940; dropped to 1,920 to catch R2 v1 ckpts).

The +160 to +200 Elo gap between sims=800 and sims=4,000 confirms search recovers real strength that sims=800 systematically underestimates. Surprise: R2 v1 ep 12 (highest sims=800 score of the mid-run) came in lower at sims=4,000 than ep 6/7 — the sims=800 spike was noise, not a real peak. Both v1 runs topped out in the 2,120–2,150 sims=4,000 band, at or just below the d10 30M peak (2,189).

Head-to-head — d10 ep 15 vs R1 v1 ep 7 at sims=4,000

To check whether the +43 Elo Stockfish-anchored “gap” between d10’s 2,189 and R1 v1’s 2,146 was real, we put the two networks in direct play with both using MCTS at sims=4,000. 104 games, alternating colors:

Every single game drew. The Stockfish-anchored “gap” was noise. At sims=4,000, d10’s top ckpt and R1 v1’s top ckpt are indistinguishable. The d15 teacher at constant LR didn’t give us strength beyond d10’s ceiling — but it also didn’t underperform.

→ code: h2h-d10-vs-d15.sh · h2h_mp.py

sims=800 trajectory — v2 (cosine LR) (UCI=1,800 anchor)

R1 v2 hit 2,044 at ep 9 and ep 12 — its top sims=800 marks before the daemon went offline at ep 13. R2 v2’s trajectory peaked at ep 14 (2,055), which never got a sims=4,000 follow-up because the daemon was already down by then — that ckpt now has a one-off sims=4,000 eval launched (i-0b24ea8a7fdd7d149).

sims=4,000 deep-reads — v2 (cosine LR)

The 2,300 line is broken. R2 v2 ep 14 at sims=4,000 vs UCI=1,800 scored 94/9/1 (score=0.947), and its 95% lower-CI bound (2,190) is strictly above the previous peak (d10 ep 15 = 2,189). Independent confirmation: ep 4 at the same anchor gave 2,285 [2,177, 2,554]. Both numbers come from the 20×256 cosine R2 v2 run; the 40×256 R1 v2 sibling tops out at ep 7 = 2,209.

Note ep 14’s sims=800 score (2,055) was the highest sims=800 of the whole R2 v2 run and never made it into the deep-eval daemon’s queue because the operator’s laptop went offline (the daemon polls from there). The one-off eval-r2v2-ep14-sims4000.sh launcher re-evaluated it after the trainer was already terminated — turning a “missed it” into the project’s strongest measurement.

The remaining in-flight eval (R2 v2 ep 4 at sims=8,000 vs UCI=2,000) will give the tightest single CI of any measurement to date (UCI=2,000 keeps the score nearer 0.5 than UCI=1,800).

Verdict and what’s still in flight

The cosine LR schedule is working — both R1 v2 and R2 v2 produce ckpts that meet or beat the d10-30M peak (2,189) at sims=4,000. The constant-LR plateau in R1 v1 + R2 v1 was a recipe artifact, not a teacher / data ceiling.

R2 v2 ep 14 at sims=4,000 vs UCI=1,800 = 2,301 Elo [2,190, 2,601] crosses the 2,300 line with a 95% lower bound that strictly clears the previous peak (d10 ep 15 = 2,189). ep 4 (2,285) confirms it within noise; R1 v2 ep 7 (2,209) sits below both. A 200-game head-to-head among the three top ckpts would tighten the ordering further — see h2h-d10-vs-d15.sh for the launcher template.

In flight as of 2026-05-26 17:30 UTC — two cheap follow-up evals designed to confirm whether the project has actually crossed 2,300 Elo:

R1 v2 and R2 v2 trainers were killed at 17:15 UTC — see #cosine-killed above for the rationale.

→ code: d15-full30m.sh · d15-20x256-lowlr.sh · eval-r2v2-ep4-sims8000-uci2000.sh · eval-r2v2-ep14-sims4000.sh · wm-deep-eval-daemon.sh

Peak Elo Across Every Checkpoint We’ve Ever Run

The full top-8 across the entire project, sorted by point estimate. All UCI=1,800 anchor, ~100 games each.

The top three numbers are within ~96 Elo and their CIs all overlap. At 100 games per measurement the score CI is ±10 percentage points, which converts to a wide Elo CI when the agent is far from the opponent. A direct head-to-head between the three top ckpts (rows 1, 2, 3) would put a much tighter bound on the ordering than the Stockfish-anchored numbers can.

Did the stronger teacher (d15) finally beat d10?

The earlier write-up of this section asked “why is d10 ahead of d15?” because R1 v1 (constant LR) and R2 v1 (constant LR) both topped out at or below the d10 peak. The cosine-LR re-runs flipped that ordering:

The cosine schedule unlocked the d15 teacher. Both cosine variants beat their constant-LR siblings (R1 v2 +63 over R1; R2 v2 +162 over R2) and both meet or exceed the d10 30M peak. The “d15 ≈ d10” verdict from the first round was a recipe artifact — constant LR=1e-3 was oscillating instead of converging.

The cosine numbers are still a 100-game-eval-noise away from being statistically distinguishable: R2 v2’s 2,285 has a 95% CI of ±200 Elo. A 200-game head-to-head between the three (d10 ep 15 vs R1 v2 ep 7 vs R2 v2 ep 4, MCTS sims=4,000 on all sides) is the only way to pin the real ordering. The launcher exists — see below.

What’s still uncertain

• R2 v2 ep 4 might be a noise spike. Later R2 v2 ckpts (ep 6, 9, 10) all came in lower at sims=4,000. That’s consistent either with “ep 4 was a lucky 104-game match” or with “cosine peaks early then decays” — only a re-eval of ep 4 at higher game count tells us which.
• R1 v2 hasn’t finished. Currently at ep 15 of a 40-epoch run. The d10 baseline added +105 Elo between ep 5 and ep 15; if R1 v2 does anything similar, the peak is still ahead of us.
• Soft-target dilution at d15 is still a plausible micro-loss (details) but it’s no longer the dominant story — the cosine schedule recovered most of what the constant LR was losing.

Head-to-head launcher — d10 best vs d15 best

To remove the Stockfish anchor noise from the comparison, run the two networks directly against each other using MCTS at the same sim count on both sides:

bash infra-eks/launchers/h2h-d10-vs-d15.sh

Defaults to d10 ep 15 vs d15 ep 7 at sims=4,000 (the current best of each), 104 games, alternating colors. Result lands at s3://wm-chess-library-594561963943/d15-mpv8-T1-g250000-20260519T0412Z/checkpoints/net-40x256/20260522T2229Z-full46M-40x256/h2h-<run-id>.txt. Override CKPT_B_S3=... and NB_B/NF_B env vars to re-run against d15’s actual peak when training completes.

Underlying script: experiments/selfplay/scripts/h2h_mp.py (extended with per-agent --n-blocks-a/--n-blocks-b so 20×256 and 40×256 can play each other).

Summary

The bitter-lesson levers (more compute via search + data) confirmed again. The “smarter network” lever (more parameters at constant data) still rejected. The “stronger teacher” lever turned out to be gated on LR schedule — d15 doesn’t outperform d10 with constant LR, but does with cosine. Strongest checkpoint to date: 2,301 Elo @ UCI=1,800 from R2 v2 (d15 46M, 20×256 cosine) at epoch 14 with sims=4,000 (CI [2,190, 2,601]) — and the 95% CI lower bound strictly clears the prior peak (d10 ep 15 = 2,189). Wider-CI sibling at the same anchor: 2,285 (R2 v2 ep 4, CI [2,177, 2,554]). 40×256 cosine variant best: 2,209 (R1 v2 ep 7, CI [2,115, 2,389]).

Self-play & MuZero — Negative Results at Fixed Compute

The distillation campaign above reached ~2,100–2,300. Two families of follow-ups tried to push past the distilled teacher. Both are negative results at one-GPU compute, and together they are the strongest evidence for the project thesis: distillation ≫ self-play / from-scratch at fixed compute.

Teacher baseline for the comparisons below, on the standard distill-soft eval.py harness at sims=800: ~2,101 [2,020, 2,224] (anchor SF UCI=1,800; vs SF-1350 it’s 0.99 → ~2,148 but saturated).

MuZero (learned world-model)

1. From-scratch MuZero — 64-ch latent, sims=400, 200 iters from random init on one L4 (run 20260527T2211Z-muzero-chess). Capped ~700–900 Elo vs SF UCI=1,350 (best eval 3 draws/20, mostly 0/20; plateaued). Shelved: from-random MuZero on a single GPU is compute-starved — sparse terminal rewards, no teacher.

2. Distill-init MuZero — reuse the 2,301 distilled teacher as a frozen representation h + frozen prediction f, and train only the dynamics g (latent-match + value-equivalence on teacher self-play transitions). f(h(obs)) reproduces the teacher exactly, so the root eval is the teacher by construction. Result: it collapses under MCTS. vs Stockfish at sims=800 the composed net scored 0.133 → 0.05 → 0/30 over rounds 0–2 and stayed at the noise floor through round 8, while the distillation loss fell then plateaued (pred_loss bottomed ~3.0, never lower). The learned g minimizes the supervised objective but produces latents that, unrolled multi-step in the search tree, drift off the manifold where the frozen f is meaningful — so 800 sims amplify confidently-wrong rollouts. Clean negative: latent-match + prediction- consistency on teacher self-play does not preserve search strength.

Self-play RL from the distilled teacher

3. Ungated self-play (attempt #7, OD, ran to completion). Bootstrapped from the R2v2-ep14 teacher, LR=1e-5, PCR, 56 games/iter, ~10 iters. The iter-10 checkpoint on the same harness: ~1,730 [1,656, 1,797] — a confirmed ~370-Elo regression below the teacher (non-overlapping CIs). So when the loop actually runs to completion it degrades the strong supervised init: the MCTS self-play targets aren’t better than what a ~2,100 net already plays, and with no guardrail the noisy low-LR updates walk it off the supervised optimum. (Earlier attempts #1–#6 never cleared iter 1 — first catastrophic forgetting at LR=1e-3/1e-4, then spot evictions.)

4. Gated self-play (MVP, 2026-05-29). Added AlphaGo-Zero arena gating (promote a candidate only if it beats the current champion ≥55% over 60 games), a KL-anchor to the teacher (trust region), a 50-shard replay window, and an in-loop Stockfish yardstick. Result: holds, doesn’t climb. Gating makes regression impossible by construction — the champion floors at the teacher’s ~2,101 — but candidates keep scoring below 55% vs the teacher (0.542 then 0.408 → both rejected; candidate ≈ or < the teacher), so nothing promotes. Gating converts “self-play destroys the teacher” (#3) into “self-play holds the teacher,” but yields no climb at this compute.

Verdict. At one-GPU compute, self-play RL does not push past the distilled teacher: from scratch it caps ~700–900 (MuZero), and from a strong init it regresses ungated and holds (no climb) gated. The binding constraint is the strength of the self-play signal at this data scale (~1.3k recorded samples/iter), not infrastructure or LR. Distillation reached 2,101/2,301; self-play cannot improve on it here.

R1 v2 and R2 v2 trainers were killed at 2026-05-26 17:15 UTC once the cosine peaks were in hand — late-epoch cosine convergence wasn’t worth the OD G burn rate, especially with the autoeval daemon offline. Two cheap follow-up evals (sims=8,000 on R2 v2 ep 4 and sims=4,000 on R2 v2 ep 14) are in flight to confirm whether the project has crossed 2,300 Elo.

Self-play RL on top of the strongest distilled prior has been attempted six times on spot and evicted before iter 1 finished each time — see the postmortem on the next steps page.

Cost & Process Lessons

May 2026 AWS spend was $3,143 (~80% EC2 GPU/CPU compute, $250 idle EKS control planes, $293 tax), spread across 5 regions (eu-central-1 $1,249, us-east-1 $1,116, + Tokyo/Ireland/Ohio). Most was legitimate research compute — data generation plus the runs that produced the 2,300-Elo teacher and the self-play/MuZero verdicts. But a few things were avoidable, and one bug nearly produced a wrong conclusion. What we’d do differently:

1. Test-before-spend. The MuZero MCTS had an inverted value-sign in child selection (ranked children in the child’s POV without negating to the parent’s — a negamax bug). It confounded the from-scratch run, the distill-init collapse, and four lever runs — nearly cementing a false “MuZero can’t learn this” result. A 2-minute local unit test caught it. Gate the core search/value/loss math behind a correctness test before any GPU run — not just a “does it run” smoke.
2. On-demand for long runs; spot only for short or restartable ones. Six self-play attempts were spot-evicted before iter 1 finished (~90-min iters vs sub-90-min eviction windows), each leaving only an iter-0 checkpoint = the prior with one SGD pass = nothing. The fix (OD, or a resume-from-S3 path) should have been the default after attempt #1.
3. Tear down standing infra when idle. The EKS autoeval control plane(s) ran much of the month (~$250) while evals were actually fired via a one-shot launcher anyway. Prefer one-shot launchers; delete idle clusters.
4. Tag every instance (experiment=<run-id>) — the bill had no tag attribution, so cost-per-experiment had to be inferred from instance type and region.
5. Reuse generated data. ~$940 of CPU spot was data-gen; cache shards in S3 and reuse across runs rather than regenerating per run.

Guardrails added 2026-06-02: an account-wide AWS Budget (monthly-spend-tracker-100) that emails at every $100 of actual monthly spend ($100 → $1,000), plus Cost Anomaly Detection (daily email on any ≥$50 spike), both to the project owner. Budgets are account-wide, so one covers every region/service. (Above $1,000 the per-$100 alerts stop — AWS threshold lists are finite — but anomaly detection still fires on spikes.)