AltSlate Labs

Efficient, honest AI — small models trained from scratch, with the full recipe published.

We build compact language models that punch above their weight, and we release everything needed to trust and reproduce them: code, weights, evaluations, and reports.

🪰 The Jugnu family

Tiny language models pretrained from scratch (jugnu — जुगनू — means "firefly": small, but it glows). All under 150M parameters, all trained on 4× Blackwell GPUs, all Apache-2.0.

JugnuLM-110M-R2+ is our flagship — the kept R2 recipe (value residuals + Muon) scaled to 25B tokens under a WSD schedule. It posts the family's best BLiMP and perplexity, beats the baseline on every metric, and ranks #1 on the Tiny-ML Leaderboard efficiency score — a hair ahead of GPT-X2-125M, at a smaller size. (It's a value-residual model, so load it with trust_remote_code=True.)

JugnuLM-110M's 81.3% BLiMP ≈ GPT-X2-125M at ~12% fewer parameters and ~9× fewer training tokens.

The ablation ladder

Starting from an honest 110M baseline (R0), we add one lever at a time and keep only what beats the previous rung on the metric that matters:

• R1 — value residuals (ResFormer): ARC-Easy +2.2 → kept.
• R2 — Muon optimizer: ARC-Easy +1.4, lower perplexity → kept.
• R3 — diverse data blend (+DCLM +FineMath): best BLiMP & perplexity, but ARC-Easy −2.5 → dropped. An honest negative — at this scale, FineWeb-Edu's educational filtering feeds ARC-Easy more than raw diversity does.
• R4 — logit distillation (from a 1.7B teacher): brackets a narrow operating point — R4a (heavy KD) lifts ARC-Easy to a family-best 57.0 but hurts perplexity (2.18); R4b (light KD) recovers perplexity but loses the ARC gain → both dropped. Offline KD across a ~15× teacher–student gap isn't a clean win at this scale.
• R2+ — the competitive run: take the strongest kept stack (R2) and scale it — 25B tokens + a WSD schedule → best-in-family BLiMP and perplexity, and #1 on the leaderboard efficiency score.

Training code & full recipe: github.com/AltSlate-Labs/jugnu

What we care about

• Efficiency per parameter — the sub-150M regime: on-device, low-latency, cheap-to-serve models.
• Reproducibility — public code, open weights, and one-command evals.
• Honest reporting — clean baselines and measured ablations over unvalidated clever tricks. We publish what didn't work (see R3 and R4), not just what did.

Links

• 💻 Code — https://github.com/AltSlate-Labs
• 🌐 Web — https://altslate.com
• ✉️ hello@altslate.com