PetitGPT

A 124.6M-parameter language model trained from scratch

Author: Yang Qi · Checkpoint: alpha075 · Release: research-v1

GitHub project · Technical report · Run guide · File checksums

This project trains a 124.6M-parameter language model from scratch on a single NVIDIA RTX 4090, processing approximately 13B model-input positions during pretraining. This repository publishes the selected PetitGPT checkpoint, its custom tokenizer, the model configuration, and a native PyTorch inference implementation. It is intended for studying efficient small-model training, post-training trade-offs, and evaluation.

Compatibility: the released CLI requires a CUDA GPU and does not implement a Transformers AutoModel loading path. GGUF, ONNX, vLLM, llama.cpp, and CPU inference are neither implemented nor validated in this release.

Quick start

Use a CUDA-enabled environment matching requirements-inference-tested.txt; the download command below assumes the Hugging Face CLI is already installed. The environment was Python 3.10.12, PyTorch 2.11.0+cu126, NumPy 2.2.6, tokenizers 0.22.2, and safetensors 0.8.0 on an RTX 4090. See the run guide for environment and input details.

hf download yqi0/petitgpt \
  --revision 7bf3df96e6880b242b2907d1e68093435feacd75 \
  --local-dir ./petitgpt-research-v1

(cd ./petitgpt-research-v1 && sha256sum -c SHA256SUMS)

python ./petitgpt-research-v1/inference.py \
  --model-directory ./petitgpt-research-v1 \
  --prompt "Say hello in one sentence." \
  --profile bf16_native \
  --max-new-tokens 32

The greeting prompt is only an example. Download the complete file set and keep the src/ directory in place—inference.py imports from it. The output is JSON containing the generated token IDs, the raw decoded text (including the terminal EOS token when one was produced), and the stop reason.

The same bundle also supports --profile fp32_math and chat-style input via --messages-json. Decoding is greedy; --max-new-tokens accepts 1–384; and the prompt plus the generation budget must fit within 2,048 tokens. Invalid input or context overflow raises an explicit error rather than being silently repaired or truncated. No default system message is inserted. The two numerical profiles are not claimed to produce identical output.

Model specification

Field	Value
Unique parameters	124,635,456
Layers / hidden width / FFN width	30 / 576 / 1,536
Attention	GQA: 9 query heads / 3 key-value heads; head dimension 64
Position encoding	RoPE, theta 10,000, full head rotation
Normalization and MLP	RMSNorm (epsilon 1e-6), SwiGLU
Vocabulary / context	32,000 / 2,048
Embeddings	Tied input and output
Dropout / stored weights	0.0 / FP32

The complete checkpoint-derived settings are in config.json. special_tokens_map.json describes the native token convention; it is not a Transformers loader contract.

Training and model lineage

Pretraining consumed approximately 13B tokens on a single RTX 4090: the exact retained packed count was 13,000,005,634 tokens, and optimizer updates traversed 12,999,720,960 model-input positions. It ran in two stages. Stage A (about 10B tokens) drew on FineWeb-Edu dedup (71.11%), DCLM-Edu (20.32%), FineWiki EN (5.08%), and Python-Edu (3.50%). Stage B (about 3B tokens) added structured tutorial content, PES2O, and StackExchange while retaining the Stage A source families. Stage budgets, source shares, pinned dataset revisions, and accounting definitions are in PRETRAIN_SOURCE_MIXTURE.csv and the technical report.

Ten recorded validation measurements, with no smoothing; lines connect observations within each stage. The Stage B detail panel uses a different vertical scale. See pretraining results for the values and measurement scope.

The released weights are a parameter interpolation between two post-training checkpoints. More precisely, in the released model's training path, P2 is supervised fine-tuning on concise instructions, and P3 is a further adaptation stage on basic instruction tasks with replay examples from earlier instruction data:

Pretrained Base → P2 instruction SFT → P3 step320
alpha075 = P2 + 0.75 × (P3_step320 − P2)

Later experiments explored DPO, response distillation, LoRA, and unified Base-SFT. Some brought local gains alongside losses in other capabilities; none was selected to replace alpha075. These experiments are not ancestors of the released weights. A separate soft-logit distillation lab used only external models. For exact hashes and branch history, consult MODEL_PROVENANCE.json and the technical report. Further detail is in the GitHub repository.

Evaluation

All comparisons below are against SmolLM-135M-Instruct and SmolLM2-135M-Instruct. The picture is mixed rather than uniformly favorable: under the project's fixed protocol, petitgpt-alpha075 leads both baselines on ARC-Easy and ARC-Challenge, trails both on PIQA and HellaSwag, and falls between SmolLM and SmolLM2 on IFEval.

All comparison results below were measured in this project on pinned model revisions; the SmolLM and SmolLM2 numbers are our own evaluations under the protocols described here, not scores copied from their official model cards.

Zero-shot likelihood benchmarks

Model	ARC-Easy acc / acc_norm	ARC-Challenge acc / acc_norm	PIQA acc / acc_norm	HellaSwag acc / acc_norm
petitgpt-alpha075	57.74% / 52.36%	28.16% / 32.68%	63.49% / 62.30%	31.28% / 35.60%
SmolLM-135M-Instruct	49.24% / 43.48%	25.43% / 27.22%	67.08% / 67.25%	34.60% / 41.96%
SmolLM2-135M-Instruct	54.00% / 48.82%	25.94% / 27.73%	66.70% / 66.76%	35.02% / 42.90%

All three models use the same task rows, their own tokenizers, FP32 scoring, and no chat template. acc_norm normalizes candidate log-likelihood by Unicode character count rather than token count and may differ from externally reported scores.

Likelihood prompts and scoring

Scoring is the zero-shot likelihood of each candidate answer given a raw completion prompt (Question: …\nAnswer: for ARC and PIQA; the pinned task's preprocessed context for HellaSwag), on the same rows for all three models, using each model's own tokenizer with no chat template, no BOS/EOS insertion, and no generated text. The forward pass runs in FP32 with autocast and TF32 disabled, MATH SDPA, and batch size 1; there is no sampling, and ties go to the first candidate.

acc ranks candidates by summed continuation log-likelihood. acc_norm divides that sum by the candidate text's Unicode character count, excluding its leading delimiter: the original answer text for ARC/PIQA and the task-preprocessed ending for HellaSwag. This uses character length rather than token count and may differ from externally reported acc_norm scores. The native evaluator follows pinned lm-evaluation-harness scoring conventions; it is not an installed-harness run. Exact values and dataset revisions are in the technical report.

The models' training data and compute budgets were not matched, and no significance test was run. ARC-Easy and PIQA were used during project development; no contamination audit was performed.

For context, SmolLM-135M was pretrained on 600B tokens and SmolLM2-135M on 2T tokens, each on 64 H100 GPUs, whereas PetitGPT was pretrained on about 13B tokens on one RTX 4090.

Instruction following: IFEval

Model	Prompt strict	Instruction strict	Prompt loose	Instruction loose	1,280-token cap hits
petitgpt-alpha075	17.19% (93/541)	28.54% (238/834)	17.74% (96/541)	29.98% (250/834)	40/541
SmolLM-135M-Instruct	10.35% (56/541)	21.82% (182/834)	12.01% (65/541)	24.10% (201/834)	290/541
SmolLM2-135M-Instruct	21.63% (117/541)	35.85% (299/834)	22.55% (122/541)	37.29% (311/834)	233/541

IFEval evaluates 541 prompts with native chat formatting and programmatic strict/loose scoring, with no LLM judge. Generation used a separate evaluation program with zero-shot greedy decoding and max_new_tokens=1280; the quick-start CLI above caps generation at 384 new tokens.

IFEval formatting, generation, and scoring

IFEval contains 541 prompts, 834 instructions, and 25 instruction types. Its evaluation prompts are in the dataset's only split, named train; this label does not mean they were used to train PetitGPT. Each model received the same official user prompt as a single user message, formatted by its own native chat formatter: petitgpt-alpha075's released formatter with no default system message, and the pinned SmolLM and SmolLM2 tokenizer chat templates, where SmolLM2's template inserts its own default system text.

No additional system prompt or few-shot messages were supplied. Responses were scored by the pinned IFEval strict/loose programmatic verifier with no LLM judge; no prompts were dropped or truncated. Responses that reached the 1,280-token cap are scored as-is, and the cap-hit counts are reported without any claim about what a larger budget would change. SmolLM2 scores highest on all four metrics, petitgpt-alpha075 is in the middle, and SmolLM lowest; all three show substantial instruction-following limitations at this scale.

See the IFEval results and protocol for full details.

Complete-answer diagnostics

Likelihood and IFEval scores do not establish reliable free-form answer correctness. The project separately reviewed generated answers to 189 prompts per model. The table below shows the assistant_owner_clarification_4_v1 version of that review.

Slice	Confirmed content passes	Unresolved	Total
Ordinary QA	6	2	41
Practical responses	10	2	38 scored turns
Old Python tasks	0	0	14
Matched natural tasks	3	0	64
Matched Python tasks	0	0	32

Joint success on the practical slice lies between 26/81 and 10/27 (32.10%–37.04%). Turns are first averaged within each of 27 dialogue groups, and the groups are then weighted equally; the range is the exact bound that results from keeping the two unresolved turns in the denominators, not a confidence interval.

In the Python diagnostics, the function interface was correct on 42 of 46 prompts, while the complete answer passed on 0 of 46. The 0/46 figure is a full-answer review result, not training-set accuracy or a uniform unit-test pass rate: finite tests, interface, explicit format, and full-answer content are scored as separate axes. Success and failure cases show Python outputs alongside summaries, rewrites, and dialogue responses, with complete stored answers and score-version notes.

Judgments were model-assisted rather than independent human gold labels. The review consisted of an initial pass with model metadata masked, a consistency pass with the mapping visible, and four targeted owner clarifications. See the versioned results and the technical report for full scope, uncertainty, and scoring differences.

Intended use and limitations

This release is intended for research and experimentation with small language models. Treat generated factual claims and code as unverified: answers may be incorrect, rewrites can omit or invent content, and code can be wrong despite a plausible function signature.

The model has not been validated for production use, high-stakes decisions, tool use, multilingual capability, long-context work, extended dialogue, or safety and refusal behavior.

Export verification

The native export preserves all 213 named state entries and 60 non-persistent rotary buffers. Eight fixed source/export fixture pairs—four per numerical profile—matched exactly within their profile, including generated tokens and stop reason. This establishes source/export parity in the measured environment; it does not establish portability or identical results on other hardware.

Licensing and data provenance

Author-controlled code, the model weights, and the tokenizer are released under Apache-2.0. Author-written documentation is licensed separately under CC BY 4.0. Third-party content retains its own terms and notices.

The source notice records the dataset and collection identities, tokenizer linkage, differences between component notices, and publisher observations that reflect only current pages. See the third-party notices, the run guide, and the commit-pinned technical report for the detailed research record.

Citation

If you use this model or its documentation, please cite the research record (CITATION.cff):

@misc{qi2026petitgpt,
  title  = {petitgpt: Building and Evaluating a 124.6M-Parameter Language Model},
  author = {Qi, Yang},
  year   = {2026},
  url    = {https://github.com/yangqi0/petitgpt}
}