Backgammon-NN — Development Report

58.8%

Deep net beats the old net, head-to-head

2.5×

Faster rollouts via batched inference

61.2%

vs wildbg at equal think-time

70→60%

Self-play gammon rate, dropped

The goal

Make the static evaluator genuinely stronger

The engine's rollout search was already solid, but its static value net (a single 198→128→5 MLP) was the limiter. The plan: find out whether the net could be pushed past its apparent ceiling, and then measure the result honestly against an outside engine rather than only against our own hand-crafted evaluator (HCE).

Investigation

Four experiments to strengthen the net

Three approaches failed in instructive ways before the fourth broke through. Every result is a head-to-head against the incumbent 128-unit net (0-ply, race-aware), so the difference is pure evaluation quality.

Head-to-head vs the 128-unit net, 600 games each, seats mirrored.
Approach	What happened	Verdict
Rollout distribution fine-tuning	Fit improved a lot (equity-MSE 0.117→0.030) but play got worse — supervised fitting disrupts the ranking TD learned.	49% · weaker
Race-focused fine-tuning (50%)	Gammon rate didn't move (57→58%) and overall strength dropped from capacity spent on races.	37% · weaker
Wider single-layer net (256), 48k games	60× the training and more width — yet dead-even. Width alone adds no representational class.	48% · tie
Deeper net 256→128 + squared-ReLU, 150k games	Depth adds functions no 1-layer net can express. Trained long, it clears the incumbent decisively.	58.8% · win

Key insightThe width-only tie proved a single hidden layer was already at its capacity for this input+signal. Depth — not width, not fine-tuning — was the missing lever. Squared-ReLU and a long run did the rest.

The breakthrough

A deeper net — 198→256→128→5, squared-ReLU

Trained from scratch on ~150k self-play games (TD, λ=1), it beats the old net on every axis — and, notably, is the first change to lower the gammon rate, confirming that gammon-leaking was a contact-evaluation weakness, not a race-play bug.

Widths 256/128 are multiples of 8/16/32 — bigger *and* cleaner for SIMD than a reference net's 300/250/200.
Metric	Old 128-net	Deep net	Δ
Head-to-head (0-ply)	41.2%	58.8%	+17.6
Self-play gammon+bg rate	70.7%	60.2%	−10.5
vs HCE (win / PPG)	81.7% · +1.36	85.3% · +1.36	+3.6

Performance

Batched inference — 2.5× faster rollouts

The Rust evaluator ran one position at a time (a [1,198] matrix-vector product — the worst case for SIMD). Now it optimises the tract model for a symbolic batch axis and every rollout ply scores all legal moves in a single [N,198] pass: proper SIMD tiling, one dispatch, one allocation.

400 trials × 9-ply truncated, single-thread (isolating the inference gain). Parity & determinism unchanged.
Path	ms / move	Notes
Old — batch-1 (GEMV)	4282	one tract call per position
New — batched (GEMM)	1694	all moves per ply in one call

At a fixed think-time that's ~2.5× more rollouts per second — a direct strength gain, and the axis the engine competes on.

External benchmark

Placing the engine on an absolute scale — vs wildbg

wildbg is an independent neural backgammon engine (~5.9 GnuBG-2-ply error rate). To compare fairly, a purpose-built harness runs both nets through the identical bgcore search — so only evaluation quality differs — bridged via the shared board orientation and GnuBG Position ID.

Mirrored-dice pairs. Equal-time row: both engines measured at ~205 ms/move. Rollout legs are 80–120 games (±~9–11%).
Test (our win %)	128-net	Deep net	What it isolates
1-ply eval	39.3%	46.7%	raw evaluation quality
Rollout ×100 (equal trials)	~48%	62.5%	eval + identical search
Rollout 200 ms (equal wall-clock)	—	61.2%	fairest: same think-time

The verdictThe deep net closed the raw-eval gap to near parity (46.7%) and, once search is involved, beats wildbg at both equal trials and equal wall-clock (~61%). Our cheaper, now-batched eval fits more informed rollouts into the same budget.

Architecture, side by side

Inputs are near-identical Tesauro encodings; wildbg's remaining edge is phase-specialisation and a softmax head — the next lever.
	Ours (deep)	wildbg
Networks	1 (all phases)	2 — contact + race
Inputs	198	202 / ~186
Hidden	256 → 128	300→250→200 / 16
Activation	squared-ReLU	Hardsigmoid / Tanh
Output head	5 nested (sigmoid)	6 exclusive + softmax

Findings

What we learned

Depth beats width and fine-tuning. A wider single layer tied; a second hidden layer + long training won. Fine-tuning a converged TD net consistently hurt it.
Gammon-leaking is a contact-eval weakness. No race trick moved it; a better evaluator dropped it 10 points. The residual ~60% is structural to cubeless play — a doubling cube would end most of those games early.
Search amplifies evaluation. Near-parity eval + rollout search flips to a clear lead. Batched inference (2.5×) compounds this by buying more rollouts per second.
SIMD-friendly widths are free wins. 256/128 tile cleanly (multiples of 8/16/32); the batch dimension matters far more than layer width for utilisation.
External validation matters. Beating our own HCE saturates and stops discriminating; the wildbg harness gives an honest, anchored strength number.

Shipped

Committed & live

3b67c9eRollouts: configurable movetime & thread count

63a3c63Outcome-distribution rollouts (win/gammon/backgammon) + dist()

41bf30bRace-aware playout — competent bear-off & gammon-saving

61abe12Batched NN inference in rollouts — 2.5× faster

63dbf68Deeper nets + squared-ReLU option (198-256-128-5)

04fca7ePromote deep net to the live td.onnx / td_latest.pt

The deep net is now the engine used by the GUI, console app, and rollout opponent. tract parity re-verified against it.

Phase-split nets. Separate contact/race networks — wildbg's last edge at raw eval.
Softmax-6 + cross-entropy head. A cleaner probabilistic formulation than five independent sigmoids.
Go deeper / longer still. 256,256,128 and more games, now that depth is proven to pay.
Firm up the wildbg number. A few hundred equal-time games to tighten the ±11% on the 61%.

← Back to Backgammon-NN

Breaking the value-net ceiling — and out-playing a reference bot

Make the static evaluator genuinely stronger

Four experiments to strengthen the net

A deeper net — 198→256→128→5, squared-ReLU

Batched inference — 2.5× faster rollouts

Placing the engine on an absolute scale — vs wildbg

Architecture, side by side

What we learned

Committed & live

Remaining levers