LDLM flow-map semi-supervised checkpoints

GPT-2 latent diffusion language-model checkpoints trained on OpenWebText.

N=8 adaptive-SDE evaluation

All values use eight direct flow-map hops on the adaptive time grid with the gamma SDE rule (eta=1). Lower perplexity is better. The 50k columns show early-training behavior; the final columns show metrics at step 500k.

Links	Variant	PPL @ 50k	Entropy @ 50k	PPL @ final	Entropy @ final
link_final	Diagonal-only	14.30	1.4031	231.00	2.9367
link_final	Frozen AE, flow map initialized from diagonal-only 150k	54.33	4.1346	60.23	4.1387
link_final	Unfrozen AE, flow map initialized from diagonal-only 150k	110.33	4.2566	69.74	4.1420
link_final link_50k	Frozen AE, initialized from its previous final checkpoint	59.97	4.1163	60.78	4.1523
link_final link_50k	Unfrozen AE, initialized from its previous final checkpoint	92.58	4.2229	80.64	4.1702
link_final link_50k	Frozen AE, final initialization, diagonal-only teacher target	165.67	4.3389	361.59	4.1609
link_final link_50k	Unfrozen AE, final initialization, diagonal-only teacher target	849.46	2.6878	716.32	3.0548

The teacher-target checkpoints use the EMA model from the diagonal-only 500k run as the detached diagonal target. The target is represented as the teacher's predicted clean latent, which is the x0 parameterization used by the flow-map sampler for its instantaneous probability-flow velocity.

Initialization artifact: diagonal-only 150k snapshot.

Experiment logs

Downloads last month: -; Downloads are not tracked for this model. How to track