TinyWay-1.2.0

TinyWay-1.2.0 is a lightweight GPT-style causal language model (~110M parameters) trained from scratch on a mixed streaming corpus (web text, stories, and code). The model is designed for research, experimentation, and educational purposes, with an emphasis on transparent architecture and reproducible training.

⚡ Trained end-to-end using a custom PyTorch pipeline with mixed precision, gradient accumulation, and streaming datasets.

---

Model Overview

Property	Value
Model type	Decoder-only Transformer (GPT-style)
Parameters	~109.6M
Layers	10
Hidden size	768
Attention heads	12
Context length	256 tokens
Activation	GELU
Dropout	0.1
Precision	fp16 / bf16
Weight tying	Token embedding tied with LM head
Position encoding	Learned absolute embeddings

---

Training Details

Dataset

The model was trained using streaming data from:

• 🌍 Web text
• 📚 Stories
• 💻 Code

via the HuggingFace dataset:

shivendrra/consolidated-datasets

Streaming was used to avoid large local storage and to allow continuous sampling directly from HuggingFace.

---

Tokenization

• Tokenizer: GPT2TokenizerFast

• Vocabulary size: 50,257

• Special tokens:

  • bos_token_id = eos_token_id = pad_token_id = 50256

---

Training Configuration

Setting	Value
Sequence length	256
Effective batch size	64 sequences
Optimizer	AdamW
Learning rate	3e-4 (cosine decay + warmup)
Betas	(0.9, 0.95)
Weight decay	0.1
Gradient clipping	1.0
Mixed precision	AMP (fp16 / bf16)
Gradient accumulation	Yes
Training steps	~60k
Total tokens	~1B (approx)

Final training loss ≈ 3.0 Final perplexity ≈ ~20

---

Usage

Load with Transformers (Custom Code Required)

This repository uses a custom model definition (modeling_tinyway.py). Make sure it is available in your environment before loading.

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("NNEngine/TinyWay-1.2.0")
tokenizer = AutoTokenizer.from_pretrained("gpt2")

---

Text Generation Example

import torch

prompt = "Once upon a time"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=200,
    temperature=0.8,
    top_k=50,
    top_p=0.95,
    do_sample=True
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

Example Generations

The model demonstrates:

• ✅ Coherent sentence structure
• ✅ Narrative flow in stories
• ✅ Reasonable grammar and punctuation
• ⚠️ Occasional repetition and topic drift (expected for this scale)

This is a research-grade small LLM, not instruction-aligned by default.

---

Limitations

• ❌ Not instruction-tuned
• ❌ Limited reasoning depth compared to large LLMs
• ❌ Context length limited to 256 tokens
• ⚠️ May hallucinate or generate inconsistent facts
• ⚠️ Training data may contain noise from web sources

Use responsibly.

---

Intended Use

• Research experiments
• Educational purposes
• Model scaling studies
• Training pipeline benchmarking
• Custom fine-tuning experiments

Not recommended for production or safety-critical applications.

---

Reproducibility

The model was trained using:

• Custom PyTorch training loop
• Streaming datasets via HuggingFace
• Mixed precision training
• Gradient accumulation
• Periodic checkpointing
• Full monitoring (loss, perplexity, gradient norm, attention entropy)

If you’d like the full training code or configs, feel free to reach out.

---

License

This model follows the license of the underlying datasets and tokenizer. Please ensure compliance before commercial usage.

---

Acknowledgements

• HuggingFace 🤗
• PyTorch
• GPT-2 tokenizer
• Open research community