56m/Dumb-1.2

Dumb 1.2

34,611,072 parameters SLM, so “dumb”.

---

BenchMark

Eval	Dumb 1.2	Dumb 1.2 RC5
ARC-Easy Acc	0.3645	0.3594
ARC-Easy AccNorm	0.3497	-
ARC-Challenge AccNorm	0.2278	-
Wikitext Word PPL	210.6	500.4
WikiText Byte PPL	2.71951	3.187
PIQA	0.5598	-
OpenBookQA AccNorm	0.2620	-
HellaSwag AccNorm	0.2624	-
BLiMP	0.7051	0.7040

• WikiText PPL has made a great progress.

---

Model can/can't

can

• Generate a continuation of plausible lies from a certain text
• Simple f**king coding
• 1 + 1 = 2
• Understanding the text

can't

• Accurately create complex calculations and coding
• Difficult sentence generation
• Instruct. This model is a Base model.

---

Dataset

First Pre Training:

• HuggingFaceFW/fineweb-edu
• HuggingFaceTB/cosmopedia-v2

Next Pre Training:

• HuggingFaceFW/fineweb-edu
• HuggingFaceTB/cosmopedia-v2
• AI-MO/NuminaMath-CoT
• flytech/python-codes-25k
• Magpie-Align/Magpie-Pro-300K-Filtered
• camel-ai/physics
• camel-ai/chemistry
• camel-ai/math
• nvidia/OpenMathInstruct-2
• bespokelabs/Bespoke-Stratos-17k
• Magpie-Align/Magpie-Reasoning-V1-150K

Training Token:

• First: 1B
• Next: ~100M SFT
• Sum: 1.1B

For more information about training, please visit Training.md.