Upload Diffusers-format 0811 checkpoint

a88664b verified about 6 hours ago

4.24 kB

license: apache-2.0
language:
  - en
library_name: diffusers
pipeline_tag: video-to-video
tags:
  - diffusers
  - video
  - video-to-video
  - video-editing
  - reference-image-guided
  - autoregressive-diffusion
base_model: jdopensource/JoyAI-Video-Edit
inference: false

JoyAI-Video-Edit Diffusers

This repository contains the Diffusers-format release of the JoyAI-Video-Edit 0811 checkpoint. It provides the JoyVideoEditPipeline, transformer, causal video VAE, and scheduler in the standard Diffusers directory layout.

The MiMo-VL text/vision encoder is not duplicated in this repository. Load it separately from XiaomiMiMo/MiMo-VL-7B-RL-2508, as shown below.

Installation

The checkpoint requires a Diffusers build containing JoyVideoEditPipeline. Until the implementation is available in a released Diffusers version, install the development branch:

pip install --upgrade torch transformers accelerate safetensors imageio-ffmpeg
pip install --upgrade "git+https://github.com/feice-huang/diffusers.git@add_joyvideoedit"

Usage

import torch
from diffusers import JoyVideoEditPipeline
from diffusers.utils import export_to_video, load_video
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration

model_id = "jdopensource/JoyAI-Video-Edit-Diffusers"
mimo_id = "XiaomiMiMo/MiMo-VL-7B-RL-2508"

processor = AutoProcessor.from_pretrained(mimo_id)
text_encoder = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    mimo_id,
    torch_dtype=torch.bfloat16,
)

pipe = JoyVideoEditPipeline.from_pretrained(
    model_id,
    text_encoder=text_encoder,
    processor=processor,
    torch_dtype=torch.bfloat16,
)
pipe.enable_model_cpu_offload()

video = load_video("input.mp4")

result = pipe(
    video=video,
    prompt="Turn the scene into a watercolor painting.",
    num_inference_steps=2,
    generator=torch.Generator(device="cuda").manual_seed(0),
    output_type="pil",
)

export_to_video(result.frames[0], "output.mp4", fps=16)

Input and output notes

The pipeline performs flow-matching denoising and does not use classifier-free guidance. It does not accept negative_prompt or guidance_scale.
The source video frame count must be 8 * n + 1 after preprocessing.
Output height and width must be divisible by 24. If omitted, they default to the source video dimensions.
num_inference_steps=2 is the checkpoint's standard inference setting.
Supported output_type values are "pil", "np", "pt", and "latent".
ref_image is optional and enables reference-image-guided editing.
The repository does not include text_encoder, tokenizer, or processor; these are loaded from MiMo-VL or replaced with precomputed embeddings.

Repository structure

JoyAI-Video-Edit-Diffusers/
├── model_index.json
├── scheduler/
│   └── scheduler_config.json
├── transformer/
│   ├── config.json
│   ├── diffusion_pytorch_model.safetensors.index.json
│   └── diffusion_pytorch_model-00001-of-00007.safetensors ...
└── vae/
    ├── config.json
    └── diffusion_pytorch_model.safetensors

Citation

@article{xiao2026joyai,
  title={JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion},
  author={Xiao, Yicheng and Dai, Wenxun and Qin, Xinran and Song, Lin and Zhang, Maoquan and Xu, Hang and Chen, Yukang and Li, Yitong and Zhang, Guohui and Zhang, Yuan and Zhang, Xuying and Zhang, Tommy and Yuan, Jianlong and Li, Peihao and Lu, Shuai and Fu, Siming and Zhao, Chuyang and Han, Xin and Huang, Jie and Li, Wenbo and Ma, Guoqing and Huang, Wei and Qi, Xiaojuan and Huang, Haoyang and Duan, Nan},
  journal={arXiv preprint arXiv:2608.03974},
  year={2026}
}

License

Apache License 2.0. See the original project for details.

jdopensource
/

JoyAI-Video-Edit-Diffusers