diffu-moss-r6b

A full fine-tune of the DiT of MOSS-Image-Base that draws whole synthetic archive pages: Swedish handwritten records of the 16th to 19th centuries. It is the drawing model of diffu, which makes training pages for page-level handwriting recognition: a page layout (a PAGE/ALTO file, or the lines' boxes as JSON), the text to write in it (or the layout's own) and a sample page go in; a page image and its exact ALTO and PAGE labels come out.

The model reads a MOSS Layout prompt, one text item per line with its exact text and its box, and a sample page given as extra image tokens, whose paper, ink and look it follows. Only the DiT is in this repository. The text encoder, tokenizer, VAE and scheduler are MOSS-Image-Base's, unchanged.

This version (9 Nov 2026) is run B: trained on the whole ground truth (54 collections, 203 volumes) on the Leonardo supercomputer, 7,000 steps of 16 samples, the averaged (EMA) weights. It replaces the first version of this repository (eight collections, trained on one DGX Spark), which stays in the repository's history (revision 726b9fe).

Files

File	What
diffusion_pytorch_model.safetensors	the DiT's weights, bfloat16 (10.2 GB, 462 tensors, the same names and shapes as Base's DiT): the exponential moving average (decay 0.999) at step 7,000
config.json	diffu_moss.transformer.MossTransformer (diffu's own implementation of MOSS-Image's DiT)
export.json	takes_reference: true, sample_size: legible: the model takes a sample page, sized at the drawn page's writing scale
tensorboard/	the training curves (loss, gradient norm, learning rate, step time, memory; validation loss by prompt form every 500 steps): the Training metrics tab

Use

With diffu (see its docs/INFERENCE.md for running on a Ray or KubeRay cluster):

hf download OpenMOSS-Team/MOSS-Image-Base --revision 85975793d8b3d432f1b6e27f352abaa4c2a4c0d3 --local-dir base
hf download Gabriel/diffu-moss-r6b --local-dir dit
git clone https://github.com/Borg93/diffu2 && cd diffu2 && uv sync --all-packages
uv run diffu-moss-init --base ../base --transformer ../dit --out ../moss-diffu
uv run diffu-run --jobs run.jsonl --draw-settings draw.json --model ../moss-diffu --out out --gpus 1

diffu-moss-init reads export.json and writes the sample's size into the model directory, so the sample page is sized as in training (legible: a whole page's sample at the drawn page's pixel count).

With a sample page

The sample page is an ordinary scan (JPEG or PNG) of a page whose paper, ink and look the drawn page should have: best a page of the same volume as the layout, or any page whose look you want. It does not set the handwriting (see Limitations). Each page job in run.jsonl is one line of JSON: the layout (template, a PAGE/ALTO file, or layout, the lines' boxes in pixels), the text (left out: the layout's own), the style, the sample page under reference, and an id:

{"id": "court-0001", "template": "templates/0105.xml", "text": "Anno 1649 den 6 Aprilis Tå laga tingh höltz medh Allmogen ...", "style": {"provenance": {"title": "the court records of Jämtlands domsaga", "years": "1649"}, "look": {"greyscale": false, "paper": "cream", "ink": "black"}}, "reference": {"image": "samples/court-sample.jpg", "caption": "sample"}}

"caption": "sample" is the prompt form this model was trained on most (the sample page alone, nothing named; the style's title and look are then left out of the prompt). "sample_look" adds the look in words beside the sample. draw.json holds the draw settings, for example {"steps": 50, "device": "cuda", "reference_guidance_scale": 2}. Each page comes out as <id>.jpg with its <id>.alto.xml and <id>.page.xml. docs/INFERENCE.md section 5 describes every field, with a worked example of each combination.

One page from the command line, without a run file:

uv run diffu-generate --model ../moss-diffu --templates templates/0105.xml --text-from text.txt \
    --reference samples/court-sample.jpg --reference-caption sample --reference-guidance 2 --whole-page --out pages

Recommended drawing: about 1 MP, 50 steps, guidance 3, shift 5, block 22 skipped in the unconditional pass (MOSS-Image's own recipe), and reference guidance 2 (a third pass toward the sample: --reference-guidance 2 in diffu-generate, reference_guidance_scale: 2 in the draw settings), which makes grey microfilm samples give grey pages reliably.

On Blackwell GPUs the DiT can run in MXFP8, quantized from these weights as the model loads: give the draw settings {"precision": "mxfp8", "compile_blocks": true}. For SGLang Diffusion, diffu-moss-export-hf writes this DiT in the official MOSS-Image layout, and diffu's plugin diffu_sglang_moss adds the sample page (docs/INFERENCE.md section 13).

Training

• Data: the page pairs of diffu's whole Swedish archival handwriting ground truth at about 1 MP: 143,379 training rows (whole pages and two windows of whole lines from each page) from 54 collections and 203 volumes, each paired with a sample page from the same volume; 10 % of the volumes held out (11,920 rows). Court, chancery, church, police, mining, military and travel records, 16th to 19th century.
• Prompt forms: the sample page alone (60 %), the sample with the look in words (20 %), no sample (20 %); no prompt names an archive, years or a hand. Half of the sample-alone pairs were restyled the same way on page and sample (microfilm grey, colourised, tinted), so the look has to come from the sample.
• Recipe: MOSS-Image's own full fine-tune, from Base at revision 85975793: Muon on the joint blocks' matrices and AdamW elsewhere, learning rate 5e-5 (constant after 50 warmup steps), gradients clipped at 1, float32 weights under bfloat16 autocast, an exponential moving average of the weights at 0.999; text encoder and VAE frozen.
• Scale: 16 samples a step over 8 A100 64 GB (two nodes of Leonardo, CINECA; FSDP2), 7,000 steps (112,000 samples, about 0.8 of a pass), ~23 hours. Planned for 9,000 steps; stopped at 7,000 as the validation loss had been flat since step 6,000.

Validation loss at a fixed noise level and noise, by prompt form (lower is better):

Step	Sample alone	Sample with look words	No sample
500	0.6709	0.6708	0.6725
6,000	0.6584	0.6583	0.6605
7,000	0.6585	0.6586	0.6606

Evaluation

The swap test: held-out windows from six held-out volumes, each drawn with its own volume's sample and with three other volumes' samples, from the same noise, judged beside the real scans:

Check	Step 7,000 (raw weights, no reference guidance)	The first version of this repository
The page's own sample gives a page of the right kind (colour or grey)	6 of 6	4 of 5
A grey sample makes a colour page grey	7 of 7	3 of 5
A colour sample makes a grey page colour	3 of 3	—
The sample's handwriting is followed	no: by eye one hand a page, letter for letter, whatever the sample	1 of 12

With these EMA weights and reference guidance 2, on two of the windows with four samples each: every grey sample gives a grey page and every colour sample a colour page, the text exact. Against step 1,000 of the same run the writing is thinner, lighter and more slanted, closer to the real pages.

Limitations

• The handwriting does not follow the sample page. The model takes the paper, ink and look from the sample and writes its own plausible hand for the page. A specific scribe's hand comes from a hand adapter (a LoRA) trained on top of this model: tested for one scribe so far (Överståthållarämbetet's 1715 letter book), whose adapter carries his hand while the look still follows the sample; adapters for more scribes are being tested.
• Every line has to be checked: diffu reads each drawn line back and marks a line it cannot confirm as unclear in the labels.
• Trained on Swedish archival handwriting; printed forms and other kinds of pages are untested.

Licence

Apache-2.0, as MOSS-Image-Base.