Llama-3.2-1B-MathCodeInstruct-10k

A Llama-3.2-1B fine-tune on 10k examples from MathLLMs/MathCodeInstruct, trained to solve math word problems with step-by-step natural-language reasoning interleaved with executable Python.

This is one of three sibling models trained on {5k, 10k, 20k}-example subsets of the same dataset, to study how fine-tuning data volume trades off against both math performance and general capability. See the training write-up for the full comparison across all three.

Training details

Base model	unsloth/Llama-3.2-1B
Method	LoRA (r=16, α=16, dropout=0) on all attention + MLP projections, merged to full weights
Dataset	MathLLMs/MathCodeInstruct, 10k training examples
Epochs	1
Effective batch size	16 (batch 1 × grad. accum. 16)
Learning rate	2e-4, cosine schedule, warmup ratio 0.03
Hardware	1× RTX 4060 (8GB)
Framework	Unsloth + TRL SFTTrainer

Benchmark results

All benchmarks run with lm-evaluation-harness, each at its standard published shot count, compared against the un-tuned base model.

Benchmark	Llama-3.2-1B (base)	MathCodeInstruct-10k	Change
GSM8K	5.8%	8.7%	🟢 +2.9%
ARC-Challenge	36.9%	36.1%	🔴 -0.8%
HellaSwag	64.2%	63.8%	🔴 -0.4%
WinoGrande	60.8%	62.0%	🟢 +1.3%

Speed: 40.53 tokens/sec (base model: 40.59 tokens/sec)

MMLU by category

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "OliverSundaram/Llama-3.2-1B-MathCodeInstruct-10k"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto")

messages = [
    {"role": "system", "content": "Below is a math problem. Please solve it step by step."},
    {"role": "user", "content": "If a train travels 60 miles in 45 minutes, what is its speed in miles per hour?"},
]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
output = model.generate(inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Limitations

• Trained on a single epoch of a 10k-example subset — not intended to be a general-purpose assistant.
• MMLU/ARC/HellaSwag/WinoGrande scores reflect a small 1B-parameter base model and should be read relative to the base model's own scores, not against much larger models.
• No safety alignment or RLHF was applied beyond what the base Llama-3.2-1B already has.