JinhyukJang/lipforcing

Lip Forcing: Few-Step Autoregressive Diffusion for Real-time Lip Synchronization

Paper (arXiv:2606.11180) · Project page · Code (GitHub)

Lip Forcing is, to our knowledge, the first autoregressive diffusion method for video-to-video (V2V) lip synchronization. It distills a high-fidelity bidirectional 14B audio-conditioned teacher (OmniAvatar-LS, our lip-sync finetune of OmniAvatar) into causal few-step students that generate each chunk in just two denoising steps with no inference-time CFG - enabling real-time, streaming lip-sync on a reference video with sub-millisecond time-to-first-frame.

Files in this repository

File	What	For
lipforcing_14b.pth	14B student - merged, self-contained (base Wan + OmniAvatar-LS adapter + Self-Forcing student baked into one file)	inference
teacher/omniavatar_ls_14b.pt	OmniAvatar-LS 14B V2V adapter (frozen distillation teacher + student init)	training
1.3B student	coming soon (same repo)

Quickstart

Install the code from GitHub:

git clone https://github.com/cvlab-kaist/LipForcing.git && cd LipForcing
conda create -y -n lipforcing python=3.12 && conda activate lipforcing
pip install -e .

Download this checkpoint plus the small external encoders/decoders (commands for every component are in the repo README's Weights section), then:

python scripts/inference/inference_streaming.py \
  --ckpt_path weights/lipforcing_14b.pth \
  --vae_path weights/Wan2.1-T2V-14B/Wan2.1_VAE.pth \
  --wav2vec_path weights/wav2vec2-base-960h \
  --mask_path weights/mask.png \
  --taehv_ckpt weights/taew2_1.pth \
  --text_encoder_path weights/Wan2.1-T2V-14B/models_t5_umt5-xxl-enc-bf16.pth \
  --video_path ref.mp4 --audio_path speech.wav --output_path out.mp4

Streaming inference encodes, denoises, decodes, and composites each AR chunk on the fly - first frames arrive before the clip finishes, GPU memory stays constant for any clip length (~37 GB peak at 14B with precomputed text embeddings; ~50 GB with runtime T5 encoding), and face detection + 512×512 alignment + paste-back run automatically, so any talking-head video works as input.

Training (two stages: Diffusion-Forcing init, then Self-Forcing DMD distillation with the SyncNet reward) is documented in the repo's README and DATA.md.

Acknowledgements

Builds on NVIDIA FastGen, Self Forcing, Reward-Forcing, OmniAvatar, LatentSync, and Wan2.1. Third-party licenses ship in the code repository.

Citation

@article{cho2026lipforcing,
  title   = {Lip Forcing: Few-Step Autoregressive Diffusion for Real-time Lip Synchronization},
  author  = {Cho, Paul Hyunbin and Jang, Jinhyuk and Lee, SeokYoung and Lee, Joungbin and Jin, Siyoon and Shin, Heeseong and Yi, Jung and Park, Yunjin and Park, Chulmin and Kim, Seungryong},
  journal = {arXiv preprint arXiv:2606.11180},
  year    = {2026},
}