richarddzh/tiny-qwen3-30m

Tiny Qwen3 30M

Tiny Qwen3 30M is a 29.57M-parameter English causal language model trained from scratch on the TinyStories dataset. It is intended as a compact educational model for studying tokenizer training, pretraining, checkpointing, and text generation with the Hugging Face ecosystem.

This is a base language model, not an instruction-tuned or chat model.

Dataset

The model uses roneneldan/TinyStories, a synthetic English dataset of short stories generated to contain vocabulary and concepts that are typically understood by young children. Both the upstream train and validation splits are read in streaming mode.

Data preparation:

• The training stream is shuffled with seed 42 and a 10,000-example shuffle buffer.
• Empty stories are discarded.
• Stories are tokenized without automatically added special tokens, then separated with <|endoftext|>.
• Tokens from consecutive stories are concatenated and packed into fixed 512-token sequences without padding.
• Validation uses the same packing procedure and is limited to 256 packed sequences (131,072 tokens) per evaluation.

Tokenizer

A byte-level BPE tokenizer is trained from scratch on 100,000 TinyStories training examples. It uses NFC normalization, an 8,192-token vocabulary, and the special tokens <|endoftext|>, <unk>, <|im_start|>, and <|im_end|>. The tokenizer supports sequences up to 1,024 tokens, while training uses 512-token sequences.

Model architecture

Parameter	Value
Architecture	Qwen3ForCausalLM
Parameters	29,567,616
Vocabulary size	8,192
Hidden size	512
Transformer layers	9
Attention heads	8
Key/value heads	2 (GQA)
Head dimension	64
MLP intermediate size	1,408
Maximum position embeddings	1,024
RoPE theta	1,000,000
Tied input/output embeddings	Yes

Training method

The model is pretrained from random initialization with the standard next-token causal language-modeling objective. Training uses the Hugging Face Trainer, streaming packed data, mixed precision when supported, gradient accumulation, periodic validation, and checkpoint-based resume.

Training parameters

Parameter	Value
Optimizer	AdamW (adamw_torch_fused on CUDA)
Optimizer steps	20,000
Sequence length	512
Micro-batch size	8 sequences/device
Gradient accumulation	8 steps
Effective batch size	64 sequences / 32,768 tokens
Learning rate	5e-4
Scheduler	Cosine
Warmup steps	200
Weight decay	0.1
Adam betas	(0.9, 0.95)
Gradient clipping	1.0
Precision	BF16 when supported; otherwise FP16 on CUDA or FP32 on CPU
Evaluation interval	250 steps
Checkpoint interval	250 steps
Random seed	42

The recorded run completed 20,000 optimizer steps on an NVIDIA GeForce RTX 4070, with a final aggregate training loss of approximately 0.1317. This value is reported for reproducibility and should not be treated as a benchmark against other models.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "richarddzh/tiny-qwen3-30m"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id)

prompt = "Once upon a time,"
inputs = tokenizer(prompt, return_tensors="pt")
with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.8,
        top_p=0.9,
        repetition_penalty=1.1,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Reproducibility

The complete training and upload workflow is available in the GitHub notebook.

Limitations

This small experimental model is trained only on synthetic English children's stories. It has limited factual knowledge, reasoning ability, context handling, and multilingual capability. Generated text may be repetitive, inconsistent, factually incorrect, or reflect artifacts in the source dataset. Do not use it for high-stakes decisions or assume its outputs are accurate.