StoryGPT

A small GPT-style causal language model trained from scratch on a 50k excerpt of the TinyStories dataset.

Model Details

• Architecture: decoder-only transformer with causal self-attention and pre-LN blocks
• Parameters: ~29M
• Vocabulary: GPT-2 tokenizer (50,257 tokens)
• Context length: 512
• Layers: 4, Embedding dim: 256

Training

• Dataset: TinyStories, 50,000 stories (44 MB of text)
• Block size: 128 tokens
• Batch size: 16
• Optimizer: AdamW, lr = 3e-4
• Loss: Cross-entropy (causal LM)

Usage

Direct loading with transformers — no local files needed:

from transformers import GPT2Tokenizer, AutoConfig, AutoModel

tokenizer = GPT2Tokenizer.from_pretrained("coderian/StoryGPT")

model = AutoModel.from_pretrained("coderian/StoryGPT")
model.eval()

text = "One day, a little girl named Lily"
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(input_ids=inputs["input_ids"], max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Because StoryGPT is a custom model, register its config class once if autoloading fails:

from transformers import AutoConfig, AutoModel
from config_gpt import GPTConfig
from models import StoryGPT

AutoConfig.register("my_gpt", GPTConfig)
AutoModel.register(GPTConfig, StoryGPT)

model = AutoModel.from_pretrained("coderian/StoryGPT")

Limitations

Tiny model trained for only 2 epochs. Stories may be short, repetitive, or contain small errors in grammar or logic.