chorcat

/

rukh-medium

Like 0

Hi Borja,

Training a 115M move-decoder from scratch, achieving 99.4% unmasked legality on raw weights, and publishing full ONNX/WASM parity benchmarks alongside honest acceptance metrics is exceptional engineering. The transparency in documenting the 1091 Elo ceiling and int8 logit drift sets a fantastic standard for open work.

Looking at your sequence modeling setup (16 layers, 768 hidden size, learned positional embeddings) and the 200-ply ceiling:

1. Reconstructing board state vs recurrent associative memory:
   In pure causal transformers, the network has no dedicated state buffer, so attention heads must repeatedly re-scan the entire 200-token move history to implicitly deduce square occupancy, pin status, and king safety before predicting step t.
   Chess is fundamentally a discrete state-transition problem where each UCI token represents a sparse delta applied to an underlying board graph.
   In an open architecture project called Maba v2 (101M reference release: https://huggingface.co/AndrewThompson1233/maba-v2-architecture), we handle sequential state updates using Decoupled Gated Delta Attention (DGDA):
   DGDA maintains an associative recurrent matrix state (S_t) that updates via closed-form delta rules. In a chess setting, this recurrent matrix naturally acts as a persistent virtual board register. Instead of re-attending to 100 past moves with quadratic compute, the recurrent layers carry forward the cumulative board geometry in constant O(1) state memory.

2. Context truncation in marathon endgames:
   Your eval notes that 4 out of 160 games hit the 200-ply ceiling and had to be adjudicated. Because the model relies on learned positional embeddings up to block=200, it cannot extrapolate into extended endgames without catastrophic degradation.
   Pairing linear recurrence with recurrent decay (NoPE) removes positional embedding cliffs entirely. A recurrent chess backbone can process 300+ ply endgames in browser WASM/WebGPU without allocating additional KV-cache memory or slowing down inference as moves accumulate.

3. INT8 parity drift in browser runtimes:
   The 3.5% argmax move divergence on model-int8.onnx often stems from softmax temperature sensitivity over long history tensors. Linear recurrent layers exhibit much more uniform eigenvalue spectra across steps, which tends to keep post-quantization argmax decisions substantially closer to FP32 baselines.

Did your attention head probes show specific heads specializing in piece tracking across the 16 layers?

Best,
Andrew