d3vmeh/bdh-50m-bytes

BDH 50M, bytes

A 56.8M-parameter Dragon Hatchling (BDH) language model that predicts one byte at a time. Base model, no tuning. One of a pair: the same structure with an 8,192-token BPE vocabulary is bdh-50m-bpe8k, and the two were trained on exactly the same text.

Architecture: the reference BDH from pathwaycom/bdh (arXiv:2509.26507). Six layers sharing one set of weights, four heads, residual width 384, 12,288 sparsely activated neurons per head, 512-position training block, vocabulary of 256 bytes.

Training

• Text: 3.07 GB, 83 % FineWeb-Edu and 17 % TinyStories (train split), sampled in 512-byte blocks with no state carried between blocks. 54 bytes of text per shared-layer parameter.
• 187,271 steps of 32 sequences (16,384 bytes per step), AdamW with weight decay 0.1, learning rate 1e-3 to 1e-4 on a cosine schedule with 1,500 warmup steps, dropout 0.1, bf16 autocast.
• One RTX 4090, 25.3 hours. One training seed. A second seed of the same recipe scored 1.102 combined (gap to the BPE twin 7.6 % versus 7.5 %).

Results

Bits per character on 3 MB of held-out text neither of the models saw in training (part of FineWeb-Edu shard 013 and the TinyStories validation split), scored in 512-position blocks:

text	bpc
FineWeb-Edu part	1.183
TinyStories part	0.707
combined	1.104

The BPE twin scores 1.019 combined on the same text (7.6 % lower). A matched rotary transformer with the same shared-layer parameter count, recipe, and text scores 1.157 (4.8 % higher than this model).

The model stores a fact planted earlier in its context (a planted word becomes thousands of times more likely to be predicted) but, untuned, almost never says it back: top-1 recall of the planted word is 2 of 30 at 64 characters and 1 of 30 at 128. Density of active neurons per position is 13.0 %. It is about 50 % for an untrained model of the same shape.

Use

import torch, bdh   # bdh.py from the code repository
ck = torch.load("bdh-50m-bytes.pt", map_location="cpu")
model = bdh.BDH(bdh.BDHConfig(**ck["config"]))
model.load_state_dict(ck["model"])

For generation, use chat_stream.py from the code repository with RAW=1 (plain completion). It streams each byte into the model's synaptic state once, so every position costs the same, and a 480-position sliding window keeps the model inside the context length it was trained on.

Limitations

Small model, 3.07 GB of text. It knows the shape of facts before the content of facts. It was trained on 512-byte blocks with no state carried across them, so without the sliding window its predictions degrade past 512 positions. Single training seed for this checkpoint.

Code: https://github.com/d3vmeh/small-dragon-hatchling. Write-up: in progress.