Motif-Technologies/motif-vae

MotifVAE

MotifVAE is a 3D causal video VAE with 4x temporal and 32x spatial compression and a 128-channel latent space, built as a high-compression tokenizer for latent video diffusion models. A 256x256x17-frame clip encodes to a (128, 5, 8, 8) latent, which is 1/16 the spatio-temporal tokens of an 8x-spatial VAE, so downstream diffusion-transformer training and inference cost less. The decoder is much larger than the encoder (about 4:1 in parameters), which helps it keep fine detail such as small text in documents.

It also runs as a plain image VAE at T=1. In its compression class it has the best document text fidelity, video reconstruction, and diffusability (how well a diffusion model converges on its latents) of the tokenizers we compared.

Architecture

Type	3D causal VAE, attention-free (pure convolution)
Compression	temporal 4x (T = 4n+1 grammar) / spatial 32x32
Latent	128 channels, deterministic encoder
Parameters	1.244B total: encoder 242M, decoder 1,002M (1:4.15)
Encoder	5 down stages (3x spatial-only, 2x spatio-temporal), 2 resblocks/stage, parameter-free global skip connections
Decoder	1.5x channel width, 3 resblocks/stage, sub-pixel (PixelShuffle + ICNR) spatial upsampling, trilinear temporal upsampling
Norm / act	LayerNorm / SiLU

Usage

The model code ships inside this repository (modeling_motifvae.py) and loads through the stock diffusers AutoModel — no extra package beyond torch, diffusers, einops:

import torch
from diffusers import AutoModel

vae = AutoModel.from_pretrained("Motif-Technologies/motif-vae", trust_remote_code=True)
vae = vae.to("cuda", dtype=torch.bfloat16).eval()

# video: (B, 3, T, H, W) in [-1, 1], T = 4n+1 (1, 5, 9, 13, 17, ...),
# H/W multiples of 32.  T=1 works as a plain image VAE.
x = torch.randn(1, 3, 17, 256, 256, device="cuda", dtype=torch.bfloat16)
with torch.no_grad():
    z = vae.encode(x).latent_dist.sample()   # (1, 128, 5, 8, 8)
    rec = vae.decode(z).sample               # (1, 3, 17, 256, 256)

For diffusion training, whiten the latent to roughly unit variance with the per-channel statistics shipped in the config — shift is the channel mean and scale the channel std (128 values each, measured over Kinetics-400 video (256x256x17) and ImageNet-1K images (256x256)). Apply per channel, reshaping both to (1, 128, 1, 1, 1):

z = vae.encode(x).latent_dist.sample()
shift = torch.tensor(vae.config.shift, device=z.device).view(1, -1, 1, 1, 1)
scale = torch.tensor(vae.config.scale, device=z.device).view(1, -1, 1, 1, 1)
z = (z - shift) / scale          # whiten for the diffusion model
# ... sample/denoise in this space, then invert before decoding:
z = z * scale + shift
rec = vae.decode(z).sample

Recompute these statistics on your own data if its distribution differs from natural video.

Evaluation

Baseline numbers are either the published values cited in each table or measured by us under the same protocol. MotifVAE is a 32x spatial tokenizer, so the comparison is against the other 32x tokenizers (LTX-Video, LTX-2, and the f32 image VAEs below); the lower-compression 16x Wan2.2 appears only as a reference upper bound.

Image reconstruction — ImageNet-1K val (256x256) & FFHQ (1024x1024)

All models below are 32x (f32) image VAEs. MotifVAE is measured by us with per-image PSNR / SSIM (the published-table convention); baseline rows are the published numbers from the Qwen-Image-VAE-2.0 technical report (Table 2, same ImageNet-256 / FFHQ protocol). Our own runs of the baselines reproduce those values to within 0.2 dB, so the comparison is direct.

Model	Setting	ImageNet PSNR ↑	SSIM ↑	FFHQ PSNR ↑	SSIM ↑
MotifVAE	f32c128	30.27	0.852	36.17	0.921
Qwen-Image-VAE-2.0	f32c128	29.69	0.842	35.91	0.918
LTX-Video	f32c128	29.57	0.833	35.56	0.905
HunyuanImage-2.1	f32c64	28.67	0.820	35.30	0.911
LTX-2	f32c128	26.06	0.793	33.63	0.906
DC-AE (Sana)	f32c32	24.82	0.690	31.35	0.830

At the f32c128 channel budget MotifVAE leads both benchmarks among 32x image VAEs, above Qwen-Image-VAE-2.0's f32c128 (29.69 / 35.91 dB), LTX-Video, LTX-2, and the lighter f32c64 / f32c32 models.

Qualitative comparison (real photographs)

GT vs frozen-tokenizer reconstruction on ImageNet-1K val photographs; the red box marks the zoom strip shown below each row.

These rows are high-frequency patterns that high-compression VAEs find hard: an armadillo's banded shell, a dense solar-panel grid, brickwork, and vehicle markings. LTX-Video and LTX-2 (same 32x class) lose the pattern, and the solar grid collapses for them entirely. MotifVAE keeps it and stays close to the 16x Wan2.2 reference, which also struggles on some of these.

Document text fidelity — OmniDoc-TokenBench (256x256)

f32-compression VAEs on OmniDoc-TokenBench (~3K text-rich documents). Baseline rows are the published numbers from the Qwen-Image-VAE-2.0 technical report (Table 3, same benchmark / resolution / OCR-NED protocol); sorted by NED.

Model	Setting	SSIM ↑	PSNR ↑	LPIPS ↓	FID ↓	OCR-NED ↑
MotifVAE	f32c128	0.849	22.40	0.052	3.43	0.825
Qwen-Image-VAE-2.0	f32c128	0.844	22.13	0.064	3.36	0.707
LTX-Video	f32c128	0.806	20.92	0.119	17.10	0.565
HunyuanImage-2.1	f32c64	0.781	19.85	0.096	5.19	0.490
LTX-2	f32c128	0.735	18.41	0.119	9.94	0.357
DC-AE (Sana)	f32c32	0.526	15.62	0.144	7.26	0.069

MotifVAE has the highest OCR-NED of any f32 VAE here: +0.12 over Qwen's f32c128 (same channel budget) and about 2.3x that of LTX-Video / LTX-2. It also leads every pixel metric in its class. The figure below covers English print, Korean print, and Korean handwriting; each row shows the full page and a zoom-in on the red-boxed region. The top two rows are OmniDoc-TokenBench (English); the bottom two are Korean printed text and handwriting from AI-Hub — Korea's national AI-data hub (operated by NIA, the National Information Society Agency), which publishes large-scale Korean AI training datasets (the table metrics above are OmniDoc-only):

Video reconstruction — Kinetics-400 (256x256) & OpenVid-1M (720p)

The same four frozen tokenizers on both benchmarks, under one pipeline: the three 32x spatial tokenizers (MotifVAE, LTX-Video, LTX-2) and the 16x Wan2.2 as a reference.

Kinetics-400 val — 17x256x256

Model	compression	PSNR ↑	SSIM ↑	LPIPS ↓
MotifVAE	4x32x32	35.37	0.954	0.046
LTX-Video	8x32x32	32.58	0.919	0.101
LTX-2	8x32x32	28.05	0.888	0.128
Wan2.2	4x16x16	37.41	0.966	0.034

OpenVid val-1000 — native 720p, full-frame

Model	compression	PSNR ↑	SSIM ↑	LPIPS ↓
MotifVAE	4x32x32	36.96	0.953	0.040
LTX-Video	8x32x32	34.40	0.925	0.055
LTX-2	8x32x32	32.94	0.923	0.044
Wan2.2	4x16x16	38.40	0.964	0.018

Among the 32x spatial tokenizers MotifVAE leads both tables: +2.7 dB over LTX-Video and +7.3 dB over LTX-2 on K400, and +2.6 / +4.0 dB on 720p. The 16x Wan2.2 reference is a little ahead; the gap to the other 32x VAEs is largest on high-motion clips (examples below).

Protocol. K400: 256 random val clips. OpenVid: the pinned public Dev-Jahn/OpenVid-1M-wds val split (1000 clips), each resized to height 720 keeping aspect (full-frame). Both use 17 frames/clip and report per-frame-averaged PSNR / SSIM / LPIPS under one pipeline; OpenVid LPIPS uses AlexNet. Absolute values aren't comparable across papers.

Video reconstruction examples

Frozen-tokenizer reconstruction on high-motion Kinetics-400 clips. Each clip is one strip: Input / MotifVAE / LTX-Video / LTX-2 / Wan2.2 (labelled). MotifVAE keeps the fast motion close to the 16x Wan2.2; LTX-Video and LTX-2 blur it.

Diffusability — UCF101 class-conditional generation

Latte-XL/1 (patch-1, rectified flow, 250K steps, no-CFG FVD-2048) over each frozen tokenizer, under the Latte-official stride-3 protocol. The Wan2.2 control (FVD 204) is in line with values reported for this setting in the literature, confirming the harness reproduces the expected scale.

Tokenizer	FVD @50K	FVD @100K	FVD @250K
MotifVAE (4x32x32)	636	385	276
LTX-2 (8x32x32)	952	556	450
LTX-Video (8x32x32)	1251	732	593
Wan2.2 (4x16x16)	656	304	204

Among the 32x spatial tokenizers MotifVAE leads at every milestone, finishing 39% below LTX-2 and 53% below LTX-Video. It reaches LTX-2's final 250K FVD by about 75K steps (~3x faster) and stays close to the 16x Wan2.2 reference. Reconstruction quality and diffusability are separate properties: LTX-Video reconstructs better than LTX-2 but is the worst of the three for diffusion.

License

MIT