CMA2-8M

Benchmark-selected checkpoint from a 7.85M-parameter Channel-Mixing Attention generalist language model using the unmodified native GPT-S 4,096-token tokenizer. It has no arithmetic-specific token splitting, place embeddings, role embeddings, or inference-time equation detection. It was selected at step 20,000 for an Open SLM Leaderboard-style average of 34.99%. WikiText normalized BPB is reported separately and is not used for selection.

Loading

This is a custom Transformers architecture. trust_remote_code=True is required because stock Hugging Face model classes do not implement CMA or this model's exact rotary convention.

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "User01110/CMA2-8M"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=True, dtype="auto"
)

Architecture

• Parameters: 7,849,134, with tied input/output embeddings
• Weights: safetensors only (model.safetensors); no PyTorch .bin weights
• Runtime: PyTorch 2.5+ for native SDPA grouped-query attention
• Tokenizer: AxiomicLabs/GPT-S-5M at revision df47402
• Vocabulary: 4,096 native tokens
• Parameter allocation: 1,179,648 tied embedding parameters and 6,669,486 non-embedding parameters
• Context: 1,024 tokens
• Width/layers: 288 / 9
• Token-attention heads: 6 query, 2 KV
• CMA: chunk=24, heads=3, expansion=2
• Contiguous-half RoPE without scaling
• No task-specific model features or tokenizer transformations

Training mixture

• FineWeb-Edu 100BT shuffled: 60.00% of trained tokens
• Cosmopedia v2: 15.00% of trained tokens
• FineWeb-HQ: 15.00% of trained tokens
• FineMath 4+: 10.00% of trained tokens

FineMath-4+ supplies high-quality mathematical explanations and reasoning as ordinary causal-language-model text, with no task-specific model features or tokenizer transformations. All four training sources are streamed natural-text corpora.

Zero-shot evaluation at step 20,000

The four lm-eval tasks use normalized accuracy when supplied by lm-eval 0.4.12, with float32 weights and softmax. ArithMark uses float32 weights and its official raw continuation log-likelihood-sum rule.

Benchmark	Accuracy
HellaSwag	27.32%
ARC-Easy	33.21%
ARC-Challenge	22.35%
PIQA	57.07%
ArithMark-2	27.80%
ARC mean	27.78%
Open SLM Leaderboard-style average	34.99%

The average is (HellaSwag + mean(ARC-Easy, ARC-Challenge) + PIQA + ArithMark-2) / 4, matching the Open SLM Leaderboard formula.

WikiText-103 validation at this step: loss 3.0840, perplexity 21.85, normalized BPB 1.3944 over 358,911 scored tokens and 1,145,226 normalized UTF-8 bytes.