TinyStories GPT-124M
A 124M parameter GPT-2 architecture model implemented from scratch (following Sebastian Raschka's Build a Large Language Model from Scratch) and pretrained on the TinyStories dataset.
Details
- Architecture: 12 layers, 12 heads, 768 embedding dim, 256 context length
- Vocab: GPT-2 BPE (tiktoken), 50257 tokens
- Training: ~2 epochs over ~360M tokens, AdamW (lr 3e-4, weight decay 0.1), bf16 mixed precision, on an NVIDIA DGX Spark
Loading
This is not using the prebuilt transformers library— it uses a custom GPTModel class,
included as CodingGPTFr.py.
```python import torch from CodingGPTFr import GPTModel
config = {"vocab_size": 50257, "context_length": 256, "emb_dim": 768, "n_heads": 12, "n_layers": 12, "drop_rate": 0.1, "qkv_bias": False} model = GPTModel(config) model.load_state_dict(torch.load("model_final.pth", map_location="cpu")) model.eval() ```
See generate.py for sampling with temperature and top-k.
Limitations
Trained only on synthetic children's stories. It writes fluent, simple narratives but has no factual knowledge, cannot do arithmetic, and is a base model (not instruction-tuned) — it completes text rather than answering questions. Logical consistency across a story degrades at this scale.