sovythos

/

Sovythos-303M-Instruct

Like 0

Hi Mahmoud,

Pretraining a 303M causal LLaMA-style transformer from scratch in pure PyTorch (without leaning on high-level training wrappers) on free-tier Google Colab, and transparently sharing an intermediate SFT checkpoint with the full validation loss curve (4.86 val loss / ~129 perplexity), is an inspiring open-source engineering effort. Documenting the reality of session disconnects and unhedged limitations sets a great benchmark for honest research.

A few technical observations on your architecture and training dynamics:

1. The token fertility tax on 32k trilingual vocabularies:
   Covering English, Modern Standard Arabic (MSA), and dialectal Egyptian Arabic within a 32,000-token BPE tokenizer creates significant tokenization pressure.
   Because Arabic words carry rich clitic and morphological prefixes/suffixes, general 32k vocabularies often suffer from high token fertility (frequently fragmenting a single Arabic word into 2 to 4 sub-tokens).
   In practice, your 1,024-token SFT window may only represent 350 to 450 actual words in Egyptian Arabic. In multi-turn chat templates with system prompts and user turns, the available conversational horizon shrinks rapidly, making it harder for the model to retain context across dialogue turns.

2. Backward pass activation memory across 24 layers on Colab:
   At 24 layers with hidden dimension 1,024 and 16 query heads, backpropagating through standard multi-head attention on a free Colab GPU (like a Tesla T4) creates heavy activation memory pressure.
   Managing batch size 8 with gradient accumulation 4 pushes VRAM close to hardware limits, which explains why extending the context beyond 1,024 during training becomes difficult on single consumer instances without running into out-of-memory errors.

3. Constant O(1) state transitions and memory reduction:
   In an open architecture project called Maba v2 (reference release: https://huggingface.co/AndrewThompson1233/maba-v2-architecture), we explore sub-quadratic architectures specifically designed for compute-constrained training:
   We route 75% of depth through Decoupled Gated Delta Attention (DGDA) paired with MABA-SA sparse attention and native NoPE:
   Because linear recurrence updates an associative state matrix rather than storing full historical attention matrices, backward activation memory is substantially lower. This allows much larger sequence lengths and higher throughput on single consumer cards.
   During generation, DGDA updates in constant O(1) time, while MABA-SA provides sharp retrieval boundaries to keep conversational responses grounded without drifting into repetitive attractor loops.

If you are planning a Sovythos v3 or looking to squeeze more context out of Colab runs, checking out the hybrid state coordination in the Maba v2 repo might offer some practical ideas for reducing memory overhead.

What is the measured token fertility (tokens per word) for your 32k tokenizer on the Egyptian Arabic OASST2 split compared to English Alpaca?

Best,
Andrew