d3vmeh/bdh-100m-bytes

BDH 100M, bytes

A 100.9M-parameter Dragon Hatchling (BDH) language model that predicts one byte at a time. Base model, no tuning. One of a pair: the same structure with an 8,192-token BPE vocabulary is bdh-100m-bpe8k.

Architecture: the reference BDH from pathwaycom/bdh (arXiv:2509.26507). Six layers sharing one set of weights, four heads, residual width 512, 16,384 sparsely activated neurons per head, 512-position training block, vocabulary of 256 bytes.

Training

• Text: 5.4 GB, 83 % FineWeb-Edu and 17 % TinyStories (train split), sampled in 512-byte blocks with no state carried between blocks. 54 bytes of text per shared-layer parameter.
• 165,000 steps of 64 sequences (32,768 bytes per step), AdamW with weight decay 0.1, learning rate 1e-3 to 1e-4 on a cosine schedule with 2,000 warmup steps, dropout 0.1, bf16 autocast.
• One H100, about 27 hours. One training seed.

Results

Bits per character on 3 MB of held-out text neither of the models saw in training (part of FineWeb-Edu shard 013 and the TinyStories validation split), scored in 512-position blocks:

text	bpc
FineWeb-Edu part	1.118
TinyStories part	0.666
combined	1.042

The BPE twin scores 0.964 combined on the same text (7.5 % lower). A matched rotary transformer with the same shared-layer parameter count, recipe, and text scores 1.107 (5.8 % higher than this model).

The model stores a fact planted earlier in its context (a planted word becomes thousands of times more likely to be predicted) but, untuned, recalls it only at short distances: top-1 recall of the planted word is 57 % at 64 characters and 20 % at 128, from 30 trials each. Density of active neurons per position is 11.2 %. It is about 50 % for an untrained model of the same shape.

Use

import torch, bdh   # bdh.py from the code repository
ck = torch.load("bdh-100m-bytes.pt", map_location="cpu")
model = bdh.BDH(bdh.BDHConfig(**ck["config"]))
model.load_state_dict(ck["model"])

For generation, use chat_stream.py from the code repository with RAW=1 (plain completion). It streams each byte into the model's synaptic state once, so every position costs the same, and a 480-position sliding window keeps the model inside the context length it was trained on.

Limitations

Small model, 5.4 GB of text. It knows the shape of facts before the content of facts. It was trained on 512-byte blocks with no state carried across them, so without the sliding window its predictions degrade past 512 positions. Single training seed.

Code: https://github.com/d3vmeh/small-dragon-hatchling. Write-up: in progress.