WorldRender — low-poly render → real video

A low-poly geometric render drives block-causal autoregressive generation of the corresponding real video. The render goes into the model's spatial-control slot, so no keypoint or skeleton extraction is involved.

Code: https://github.com/songyiren725/WorldRender

~4.9 fps on one H100. A 30 s rollout (60 blocks) is about as good as a 2 s one (4 blocks):

length	frames	blocks	jitter	brightness drift
2 s	45	4	1.78	0.8%
6 s	141	12	1.99	5.1%
10 s	237	20	1.79	4.3%
30 s	717	60	2.08	4.5%
training-time samples			1.82

Jitter is the median of per-frame mean|f[t+1]-f[t]|; drift is the relative change in mean brightness between the first and last eighth. 15x the block count costs 17% more jitter, and drift stops growing after about 6 s.

Files

stage1/step-10000.safetensors         802 tensors = 800 LoRA (r128) + pose_patch_embedding
stage2/step_2500_generator_lora.pt    800 tensors, LoRA only (the distilled student)
teacher/step-8100.safetensors         bidirectional teacher, only needed to redo stage 2

Loading — all three parts are required

base Wan2.1-I2V-14B-720P
  + stage1/step-10000.safetensors        LoRA + pose_patch_embedding
  + stage2/step_2500_generator_lora.pt   distilled LoRA, overlaid on top

The stage2 checkpoint does not contain pose_patch_embedding — PEFT saves only trainable parameters and that module is frozen in stage 2. Load stage2 alone and x = x + pose_patch_embedding(render_latents) becomes the identity: the model silently ignores its only control signal while still producing plausible video. Assert after loading:

w = dict(gen.named_parameters())["model.pose_patch_embedding.weight"]
assert float(w.abs().max()) > 0

The two checkpoints genuinely disagree on key prefix — not a typo:

stage1   saved from the generator wrapper  ->  model.blocks.0...   load into gen
stage2   saved from generator.model        ->        blocks.0...   load into gen.model

pose_patch_embedding, face_adapter, face_encoder and motion_encoder do not exist in the base model, so loading must use strict=False. This corpus has no faces and the face branch is short-circuited throughout — those three modules are randomly initialised and never trained, so any inference path that calls them unconditionally injects random residuals into 8 of the 40 layers.

Inference

python long_infer.py \
  --stage1-ckpt   stage1/step-10000.safetensors \
  --stage2-ckpt   stage2/step_2500_generator_lora.pt \
  --config        configs/render_dmd.yaml \
  --context-noise 150 \
  --kv-frames 12 --local-attn-size -1 \
  --decode-chunk 0 --seed 1234

denoising_step_list = [1000.0, 750.0, 500.0], and it must be float. sinusoidal_embedding_1d ends with return x.to(position.dtype), so an int64 timestep truncates the whole cos/sin embedding (values in [-1,1]) to zero and the output becomes pure noise.

context_noise=150 is the anti-drift lever: noise is added to a generated block before it is written back into the KV cache, which stops the model from becoming over-confident in its own output. Use 0 during training, 150 at inference.

Why step 2500 and not the final 5000

Stage 2 degrades after ~2500 steps. From its own training-time samples, averaged over all 8 ranks:

step:      0     1000   2000   2500   3000   3500   4500   5000
jitter    1.29   3.13   2.26   1.82   3.96   3.67   1.30   2.81
mean       122    119    124    130    140    225    237    157
std         58     61     61     57     55     25     11     48

Rising mean with collapsing std is contrast draining away; by 4500 the samples are nearly white. Re-measured under a corrected long rollout, step_5000 drifts 74% (the picture collapses toward a single tone) against 10.5% for step_2500. Step 2500 is closest to the healthy step-0 baseline on all three numbers.

Not overfitting: 9573 clips yield ~700k distinct windows and 5000 steps cover 5.7% of them. More likely a critic/generator imbalance in the distillation.

Training setup

base            Wan2.1-I2V-14B-720P (single dense DiT, 40 layers — not a MoE)
resolution      480x848 @ 24 fps
stage 1         causal adaptation with teacher forcing, 10000 steps, lr 5e-5 constant
                bidirectional DiT -> block-causal AR generator, 3 latent frames per block
                reference-image KV kept globally visible to every block
stage 2         distribution-matching distillation, 50 -> 3 steps [1000, 750, 500]
                two-pass block replay so a 14B student fits one 8x80GB node
                num_training_frames 12 (4 blocks), lr 1e-5 constant
LoRA            rank 128 on q,k,v,o,ffn.0,ffn.2
data            9573 training clips, 13.6 hours
hardware        8 x H100 80GB, single node, FSDP FULL_SHARD

Known limits

The KV cache holds a bounded window of 12 latent frames, so memory is flat in output length, but the attention span is what training saw and nothing anchors the rollout beyond the reference frame. Drift is therefore not zero — it settles around 4-5% rather than growing without bound.

Clips whose render has low contrast (measured std 40 against 51-58 for the rest) show visibly more jitter: the render reaches the model only through pose_patch_embedding, so low contrast means weak conditioning.

See docs/PITFALLS.md in the repo for the failure modes this code has actually produced, including three diverged training runs, inference emitting pure noise, and a stutter every two seconds from a KV cache wrap bug.