Development Report · updated 21 Jul 2026 · Backgammon-NN
A development write-up on the neural backgammon engine: strengthening the value network, faster rollouts, and benchmarking against independent engines. A deeper network took it from behind wildbg to ahead of it; a long hunt for the real ceiling ruled out architecture and hand-crafted features and pinned it on training-signal quality. Then came the structural discovery: every label the project could generate was produced by the engine itself, and a network fitted to its own engine's labels cannot exceed that engine — which is why a dozen careful experiments all converged to the same place. The breakthrough was to learn from outside: 22.5 million positions labelled by world-class GNU Backgammon. The resulting net (v1.10.0) beats the best self-play champion 53.4% over 40,000 games at identical inference cost. Measured at equal search depth — after a benchmark bug that had been quietly comparing our static evaluation against a searching opponent was found and fixed — the engine is now level with GNU Backgammon itself: 49.1% over 3,000 money games, and 48.5% of 400 seven-point matches played with the doubling cube.
The goal
The engine's rollout search was already solid, but its static value net (a single 198→128→5 MLP) was the limiter. The plan: find out whether the net could be pushed past its apparent ceiling, and then measure the result honestly against an outside engine rather than only against our own hand-crafted evaluator (HCE).
Investigation
Three approaches failed in instructive ways before the fourth broke through. Every result is a head-to-head against the incumbent 128-unit net (0-ply, race-aware), so the difference is pure evaluation quality.
| Approach | What happened | Verdict |
|---|---|---|
| Rollout distribution fine-tuning | Fit improved a lot (equity-MSE 0.117→0.030) but play got worse — supervised fitting disrupts the ranking TD learned. | 49% · weaker |
| Race-focused fine-tuning (50%) | Gammon rate didn't move (57→58%) and overall strength dropped from capacity spent on races. | 37% · weaker |
| Wider single-layer net (256), 48k games | 60× the training and more width — yet dead-even. Width alone adds no representational class. | 48% · tie |
| Deeper net 256→128 + squared-ReLU, 150k games | Depth adds functions no 1-layer net can express. Trained long, it clears the incumbent decisively. | 58.8% · win |
Key insightThe width-only tie proved a single hidden layer was already at its capacity for this input+signal. Depth — not width, not fine-tuning — was the missing lever. Squared-ReLU and a long run did the rest.
The breakthrough
Trained from scratch on ~150k self-play games (TD, λ=1), it beats the old net on every axis — and, notably, is the first change to lower the gammon rate, confirming that gammon-leaking was a contact-evaluation weakness, not a race-play bug.
| Metric | Old 128-net | Deep net | Δ |
|---|---|---|---|
| Head-to-head (0-ply) | 41.2% | 58.8% | +17.6 |
| Self-play gammon+bg rate | 70.7% | 60.2% | −10.5 |
| vs HCE (win / PPG) | 81.7% · +1.36 | 85.3% · +1.36 | +3.6 |
Performance
The Rust evaluator ran one position at a time (a [1,198] matrix-vector product — the worst case for SIMD). Now it optimises the tract model for a symbolic batch axis and every rollout ply scores all legal moves in a single [N,198] pass: proper SIMD tiling, one dispatch, one allocation.
| Path | ms / move | Notes |
|---|---|---|
| Old — batch-1 (GEMV) | 4282 | one tract call per position |
| New — batched (GEMM) | 1694 | all moves per ply in one call |
At a fixed think-time that's ~2.5× more rollouts per second — a direct strength gain, and the axis the engine competes on.
External benchmark
wildbg is an independent neural backgammon engine (~5.9 GnuBG-2-ply error rate). To compare fairly, a purpose-built harness runs both nets through the identical bgcore search — so only evaluation quality differs — bridged via the shared board orientation and GnuBG Position ID.
| Test (our win %) | 128-net | Deep net | What it isolates |
|---|---|---|---|
| 1-ply eval | 39.3% | 46.7% | raw evaluation quality |
| Rollout ×100 (equal trials) | ~48% | 62.5% | eval + identical search |
| Rollout 200 ms (equal wall-clock) | — | 61.2% | fairest: same think-time |
The verdictThe deep net closed the raw-eval gap to near parity (46.7%) and, once search is involved, beats wildbg at both equal trials and equal wall-clock (~61%). Our cheaper, now-batched eval fits more informed rollouts into the same budget.
| Ours (deep) | wildbg | |
|---|---|---|
| Networks | 1 (all phases) | 2 — contact + race |
| Inputs | 198 | 202 / ~186 |
| Hidden | 256 → 128 | 300→250→200 / 16 |
| Activation | squared-ReLU | Hardsigmoid / Tanh |
| Output head | 5 nested (sigmoid) | 6 exclusive + softmax |
Findings
The story continues
The deep net closed the gap to wildbg; the next four experiments chased the rest. Two shipped, one failed instructively, and together they revealed where the ceiling actually is. Results are head-to-head vs the incumbent champion at 0-ply, winners re-checked at 1-ply.
| Approach | What happened | Verdict |
|---|---|---|
| Phase split — separate contact / race nets | The race net was a real 0-ply win (+0.25 PPG) but didn't help the rollout engine, so it ships only as an optional opponent. | partial |
| Output bucketing — one body, 8 pip-count heads (softmax-6) | Stockfish-NNUE style: the shared body sees every position, each head specialises by game stage — no data starvation, unlike separate nets. | 52.6% · win |
| Class-aware routing — race/crashed/contact × pip, 12 heads | gnubg's own position classification. But at equal training the routing was neutral; the gain came from a learning-rate-decay tail — a training-recipe win, not a routing win, and it barely survives search (52% @1-ply). | 55.8% @0-ply |
| Richer input features — +14 strategic (shots, containment…) | gnubg-style hand-crafted inputs. Fed at the input they hurt; fed NNUE-style after the accumulator they recovered to neutral — then tracked ~5 points behind the featureless net. | no gain |
Where the ceiling really isRouting was neutral and richer features were negative — twice over, changing what the net sees or how its heads are wired didn't help. That points squarely at training-signal quality: we train on noisy single-game outcomes, while gnubg trains on low-variance rollout labels. That is the lever now under construction.
Training signal
The through-line of the experiments above: architecture and features have hit a ceiling; gnubg's real edge is rollout-quality training labels, not its net. We train on single game outcomes — an extremely noisy label for a mid-game position; gnubg trains on rollouts — a low-variance estimate of the true value. So we labelled positions with the engine's own truncated rollouts and supervised-trained the net to a soft/hard blend of the rollout distribution and the real outcome.
The findingTrust the rollouts — strength rose monotonically as we weighted the low-variance label over the noisy outcome. From just 400k labelled positions the net reached parity with the 3M-game self-play champion (~375× fewer positions, minutes of training vs ~16h of self-play). Label quality genuinely substitutes for compute. But it capped at parity — because the labels used the champion as their own rollout leaf.
The real yardstick
Everything so far was measured against our own champion or wildbg. GNU Backgammon is the world-class reference. A direct 0-ply head-to-head — our engine generates the moves, gnubg picks by evaluating each resulting position, bridged via the shared Position ID and parallelised across 32 gnubg processes — gives the first absolute placement.
The verdictThe champion wins ~43% (−0.20 PPG) against gnubg at 0-ply: gnubg genuinely out-plays us — but we're competitive, not outclassed (43%, not 30%). A phase breakdown locates the gap in contact play; races and the bear-off are near-even (and gnubg's exact bear-off database owns the endgame anyway). This sets a concrete target: +7 points at 0-ply to reach parity.
Front line
The single-round labels capped at champion-level because the champion was their own rollout leaf. The escape is expert iteration — exactly how gnubg itself was trained: relabel each round with the improved net, so the leaf strengthens and the label ceiling rises every round.
| vs our champion | vs gnubg (0-ply) | |
|---|---|---|
| Champion (v1.8.0) | — | 42.7% |
| After loop round 1 (2.4M labels) | ~55% | 45.5% |
It gainsSix times the data broke past parity: round 1 beats the champion ~55% and, crucially, moves the absolute number 43% → 45.5% vs gnubg. Beating our own lineage translated into real external progress. But it proved to be a one-round gain — the loop then converged to a fixed point, which sent us hunting for the real ceiling (next).
Front line
Round 1's gain did not repeat. We chased the ceiling down every axis, and the pattern is now unambiguous: everything built on rollouts under the champion's own 0-ply play converges to champion parity — about 45.5% vs gnubg — no matter how much data or how the labels are blended.
| Label recipe | vs champion (0-ply) | verdict |
|---|---|---|
| Loop round 2 (relabel 1.54M with round-1 net) | ~48% | parity |
| Combined pool (3.9M positions) | ~48% | parity |
| α = 1.0 (drop the game-outcome anchor) | ~49% | parity |
| Untruncated (λ=1) rollouts, 1.37M | ~47% | worse |
| 1-ply search distillation | ~48% | parity |
The real ceilingRolling every playout to the game's end (λ=1) came out worse, not better — the tell that the ceiling was never the truncation, it's the champion's play. Extending a weak greedy policy fifty plies accumulates its blunders faster than a good static-eval leaf; bias–variance favours truncation. A stronger label therefore needs a stronger teacher, not a longer rollout.
Faster labeling, then a stronger teacher. A step-free rollout move generator (no per-move bookkeeping, copy-cheap 30-byte boards) lifted labeling to ~57 positions/sec — enough to afford the search route. The last search-based lever is distilling the champion's own n-ply search value: 1-ply is too mild (a TD net already approximates its own one-ply backup), but 2-ply corrects errors the static eval structurally cannot see — the very reason gnubg searches. A 2-ply distribution-returning expectiminimax was built and verified — and distilling it delivered the first search-robust gain of the project (next). (Tree search / PUCT is the wrong tool for backgammon — dice chance-nodes dilute simulations across 21 rolls per ply and the value net is too accurate for selective deep search to pay. Expectiminimax + rollouts is the paradigm, which is why no strong engine uses MCTS.)
The result — v1.9.0
Distilling the champion's own 2-ply search value into the net — via a distribution-returning expectiminimax that carries the win/gammon/backgammon split, not just the equity — produced the strongest net of the project, now shipped as v1.9.0.
| net | vs champion, 1-ply | vs gnubg, 0-ply |
|---|---|---|
| 1-ply distillation | parity | — |
| 2-ply distillation (v1.9.0) | ~52% | ~47% |
It survives searchThe edge is small (~2%) — but for the first time in this project it does not wash out under search. Earlier 0-ply gains (v1.7.0, v1.8.0) evaporated at 1-ply; this one holds, because distilling search value adds real eval quality, not 0-ply tricks. A search-robust small gain beats a big 0-ply one.
An honest correction. A fix to the gnubg benchmark — resolving crawling races by pip count rather than scoring them as non-wins — lifted every net by ~3 points and briefly looked like a parity breakthrough. On the corrected metric the champion itself was already ~46% vs gnubg (not 42.7%), and 2-ply distillation adds ~+1 on top. Always re-baseline when the measuring stick changes.
Exact bear-off, alongside. An exact one-sided bear-off database — all 54,264 home-board positions solved by dynamic programming (no rollouts) into exact win/gammon race equities, wired into the eval. It closes the endgame exactly — though a strong net already plays those bear-offs so well that it changes no 0-ply move; its payoff is at deeper search and as exact training labels.
The wall
Step back from the last several experiments and one pattern covers all of them. Self-play outcomes, truncated rollouts, untruncated rollouts, 1-ply distillation, 2-ply distillation — every training label the project ever used was generated by the engine itself. And a network fitted to labels its own engine produced cannot become better than that engine. Each experiment was asking the net to grade its own homework, so each one converged to the same place, and the differences between them were the differences between grading schemes.
The consequenceNo amount of architecture, data volume or label cleverness escapes a ceiling set by the labeller. The only way out is a teacher from outside the loop.
The pre-check
“World-class” is not the same as “stronger than us at the setting we can afford to run millions of times”. Measuring first turned out to be the whole experiment, because the answer depends on gnubg's search depth — and one of the two options would have taught us nothing at all.
| teacher candidate | our champion scores | verdict |
|---|---|---|
| gnubg 0-ply | 49.0% (z −0.28) | parity — useless |
| gnubg 2-ply | 43.9% (z −3.86) | clearly stronger |
Two gnubg behaviours worth recording. First, set evaluation chequerplay eval plies N does not affect the eval command — verified by byte-comparing output at N = 0, 2 and 3. gnubg always computes the full static/1-ply/2-ply table, so its strength in a bridge like ours is decided entirely by which row you parse — and the deep evaluation was being computed and thrown away all along, which is why reading it costs nothing. Second, positions gnubg answers from its bearoff databases print only the static row; a parser counting 2 ply: lines silently comes up short, blocks, and times out. In the head-to-head harness that quietly discarded whole games — and the survivors were the fast ones, a biased sample presented as a result.
The result — v1.10.0
Label positions with gnubg's 2-ply evaluation — the five nested probabilities, not just the equity — and train on them directly. About 2,600 positions per second across 60 gnubg processes, so 2.5M labels cost a quarter of an hour, against six hours for the 30K rollout labels this replaces, which had come back at parity.
| labels | recipe | vs champion, 0-ply | vs champion, 1-ply |
|---|---|---|---|
| 500K | warm-start, 256×128 | 51.5% | 50.9% |
| 2.5M | warm-start, 256×128 | 52.4% | 52.3% |
| 2.5M | scratch, 512×256 | 54.3% | 53.4% |
| 17.5M | scratch, 512×256 | 54.8% | 53.6% |
| 22.5M | scratch, 256×128 | 53.9% | 53.4% (z +13.70) |
Why the smaller net shipsThe 22.5M net is a 256×128 — the same architecture and the same cost per move as the net it replaces — yet it is statistically indistinguishable from the 2.7×-larger one (49.5%, z −1.50 over 20,000 games). The big net wanted 1.58× the time per evaluation, and at equal movetime in the rollout engine its advantage could not be demonstrated at all. Price capacity in the currency the application spends.
| net | vs gnubg 2-ply | points per game |
|---|---|---|
| v1.9.0 champion | 43.4% | −0.173 |
| v1.10.0 | 46.1% | −0.113 |
Reversals
Both for the same reason: for the first time the teacher outclasses the student, and several rules of thumb turn out to have been statements about that ratio rather than about training in general.
What didn't work
Recorded because each cost real time and each looks attractive from the outside.
A measurement lessonA 40,000-game gate read 54.8% at 7,000 games and finished at 53.6%. Partial head-to-heads wander by more than the effects being measured — and a comparison run through a third party (two separate small matches against a common opponent) is weaker still than playing the two nets against each other.
Shipped
dist()td.onnx / td_latest.pt--our-ply, so our side can search tooThe gnubg-distilled net (v1.10.0) is now the engine used by the GUI, console app, and rollout opponent; the v1.9.0 net ships alongside as the Neural classic opponent, an easier rung between the hand-crafted evaluator and the current net. tract parity re-verified against both.
Next
The engine is now a complete backgammon program — cube and match play landed in v1.11.0 (below). On the evaluation side, distillation has been taken as far as it goes: the teacher is capped at gnubg's 2-ply evaluation, and volume, distribution and search depth are each closed by a direct high-sample measurement. What remains has to come from somewhere other than fitting a teacher's labels.