kaiw7/Kai-AV-Edit

JAVEdit: LTX-2.3 Joint Audio-Visual Editing Pipeline

This repository is our reproducible training, inference, and evaluation workspace for joint audio-visual instruction-guided video editing with LTX-2.3. It contains the customized LTX trainer, JAVEdit-100k data preparation, rank-128 LoRA training, optional task-balanced sampling, resumable four-GPU JAVEditBench inference, and an isolated six-metric evaluation stack.

The reported checkpoint was trained for 30,000 steps with the LTX-2.3 22B dev backbone. Benchmark inference uses the same dev backbone and loads the LoRA as an adapter; it does not replace the backbone with the distilled checkpoint and does not permanently merge the LoRA during training.

Current artifacts

Large artifacts are stored in the private Hugging Face model repository kaiw7/JAVEdit. Authenticate with an account that has access before opening or downloading these links.

Artifact	Contents	Size	Hugging Face
Step-30k LoRA	Rank-128 LoRA weights used for inference	2.1 GB	lora_weights_step_30000.safetensors
Step-30k optimizer	Four FSDP optimizer shards plus distributed-checkpoint metadata	8.3 GB	optimizer_state_step_30000
Step-30k resume state	LoRA, optimizer, scheduler/global-step state, four rank-specific RNG states, and training config	10.5 GB	fullstate_0802
JAVEditBench inference	150 edited MP4 files generated with the dev backbone and step-30k LoRA	216 MB	javeditbench_step30000_dev
Evaluation results	Per-sample JSON, summary CSV, and Qwen retry outputs	< 3 MB	eval_results

Restore the uploaded artifacts into an existing clone while preserving the expected layout:

hf auth login
hf download kaiw7/JAVEdit \
  --include \
    "outputs/javedit_ltx23_lora_fullstate_0802/training_config.yaml" \
    "outputs/javedit_ltx23_lora_fullstate_0802/checkpoints/*step_30000*" \
    "outputs/javedit_ltx23_lora_fullstate_0802/checkpoints/optimizer_state_step_30000/**" \
    "outputs/javeditbench_step30000_dev/**" \
  --local-dir .

Repository layout

JAVEdit/
├── LTX-2/                         # bundled customized LTX source
├── javedit_ltx/                   # data and joint-editing helpers
├── configs/                       # standard and task-sampler training configs
├── scripts/                       # setup, preparation, training, inference, preflight
├── metrics/                       # JAVEditBench evaluation implementation
├── apps/                          # Gradio result browsers
├── benchmark_150.csv              # 150 benchmark rows
├── data/JAVEdit-100k/             # local dataset; ignored by Git
├── models/                        # local backbones and metric weights; ignored by Git
└── outputs/                       # checkpoints, videos, and results; ignored by Git

Datasets, pretrained weights, virtual environments, W&B state, checkpoints, and generated outputs are intentionally excluded from Git. Store them under the paths above or restore them from external storage.

Tested system

• Linux on NVIDIA GB200/aarch64
• Python 3.12
• CUDA 12.8-compatible driver
• Four GPUs for training, dev-backbone inference, and Qwen3-Omni evaluation
• ffmpeg and ffprobe available on PATH

The setup scripts use project-local virtual environments and do not modify system Python. The pinned CUDA wheels are important on aarch64, where default PyPI resolution may install CPU-only PyTorch.

1. Clone and create environments

git clone git@github.com:kaiw7/JAVEdit.git
cd JAVEdit

Create the preprocessing environment:

./scripts/setup_env.sh
source scripts/activate_javedit.sh

Create the training and inference environment with PyTorch 2.9.1, CUDA 12.8, and FlashAttention 4:

./scripts/setup_env_fa4.sh
source scripts/activate_javedit_fa4.sh

Create the isolated evaluation environment:

./scripts/setup_javedit_metrics.sh
source scripts/activate_javedit_metrics.sh

Each new shell must source the activation script for the pipeline it runs. The scripts prepend the matching environment's libtorch to avoid loading incompatible libraries from the host image.

2. Prepare data and model weights

Download and expand JAVEdit-100k:

source scripts/activate_javedit.sh

hf download Coraxor/JAVEdit-100k \
  --repo-type dataset \
  --local-dir data/JAVEdit-100k

python scripts/prepare_javedit_data.py \
  --dataset-root data/JAVEdit-100k \
  --extract-benchmark

The preparation script streams the multipart edited-video archive instead of creating another large joined archive. With --prompt-style both --reverse-mode all (the defaults), it produces:

• 105,938 valid instruction-conditioned triplets;
• 39,910 unique physical media files;
• data/JAVEdit-100k/ltx_training.jsonl;
• a root-level benchmark_videos symlink resolving benchmark_150.csv paths.

Download the training backbone and text encoder:

hf auth login
hf download Lightricks/LTX-2.3 \
  ltx-2.3-22b-dev.safetensors \
  --local-dir models/ltx-2.3

hf download google/gemma-3-12b-it-qat-q4_0-unquantized \
  --local-dir models/gemma-3-12b

The current training and reported benchmark inference both use:

models/ltx-2.3/ltx-2.3-22b-dev.safetensors
models/gemma-3-12b/

3. Precompute joint audio-video latents

LTX requires spatial dimensions divisible by 32. The pipeline center-crops the 1280×720 data to 1280×704 and keeps 121 frames at 25 FPS.

source scripts/activate_javedit.sh

CUDA_VISIBLE_DEVICES=0,1,2,3 accelerate launch --num_processes 4 \
  scripts/precompute_javedit.py \
  data/JAVEdit-100k/ltx_training.jsonl \
  --resolution-buckets "1280x704x121" \
  --model-path models/ltx-2.3/ltx-2.3-22b-dev.safetensors \
  --text-encoder-path models/gemma-3-12b \
  --output-dir data/JAVEdit-100k/.precomputed \
  --vae-tiling

The cache contains instruction embeddings, deduplicated video/audio latents, and explicit target/reference triplet indexes. Validate it before training:

python scripts/validate_javedit_precomputed.py \
  --config configs/javedit_ltx23_lora.yaml

4. Train on four GPUs

Standard data distribution:

export WANDB_API_KEY="<your-key>"
CUDA_VISIBLE_DEVICES=0,1,2,3 \
  ./scripts/train_javedit.sh configs/javedit_ltx23_lora.yaml

Optional task-balanced sampling:

export WANDB_API_KEY="<your-key>"
CUDA_VISIBLE_DEVICES=0,1,2,3 \
  ./scripts/train_javedit.sh configs/javedit_ltx23_lora_task_sampler.yaml

The sampler is optional and does not change the original pipeline when omitted or disabled. The checked-in task-sampler config uses:

Task	Probability
Background editing	32.72%
Speech editing	23.62%
Person attribute editing	17.17%
Subject removal	13.24%
Subject addition	13.24%

Before starting a new run, set these fields in the selected YAML:

model:
  load_checkpoint: null
wandb:
  run_id: null
output_dir: outputs/<new-run-name>

For exact resume, set model.load_checkpoint to a checkpoint directory containing matching LoRA, optimizer, training state, and rank-specific RNG files. Full-state resume restores the optimizer, scheduler, global step, dataloader position, RNG state, and saved W&B run ID. Training reconstructs the same dev backbone and attaches the LoRA before loading adapter weights; it does not permanently merge LoRA weights into the backbone.

Use a cluster scheduler or detached session such as tmux for long runs. A pod or node restart still terminates the process, so retain full checkpoints on persistent storage.

5. Run resumable JAVEditBench inference

This reproduces the reported 150-video step-30k inference using four GPUs and the training-matched dev backbone:

source scripts/activate_javedit_fa4.sh

python scripts/run_javeditbench_dev_inference.py \
  --bench-csv benchmark_150.csv \
  --source-video-dir data/JAVEdit-100k/benchmark_videos \
  --backbone-path models/ltx-2.3/ltx-2.3-22b-dev.safetensors \
  --gemma-root models/gemma-3-12b \
  --lora-path outputs/javedit_ltx23_lora_fullstate_0802/checkpoints/lora_weights_step_30000.safetensors \
  --output-dir outputs/javeditbench_step30000_dev \
  --gpus 0,1,2,3

Important settings are 1280×704, 121 frames, 25 FPS, 30 denoising steps, CFG 4.0, STG 1.0, STG block 29, and seed 42. The script fully decodes existing MP4 audio and video before skipping them, so rerunning the same command resumes missing or corrupted samples without overwriting valid outputs. Use --overwrite only when selected outputs should be regenerated.

Run an input/path check without loading the model:

python scripts/run_javeditbench_dev_inference.py --dry-run --gpus 0,1,2,3

6. Install and run evaluation

Create the isolated metrics environment and download all metric weights:

./scripts/setup_javedit_metrics.sh
source scripts/activate_javedit_metrics.sh
hf auth login
./scripts/download_javeditbench_weights.sh

Metric	Dimension	Model
VTSS	Visual quality	Koala-36M Training Suitability Assessment
UTMOSv2	Audio quality	UTMOSv2
SyncNet	A/V lip synchronization	LatentSync SyncNet + InsightFace
Instruction Compliance	Instruction following	Qwen3-Omni
Video Fidelity	Preservation/fidelity	Qwen3-Omni
A/V Quality	Holistic audio-video quality	Qwen3-Omni

The expected local paths are defined in metrics/path.yml. Verify the environment, weights, and complete generated set:

python scripts/preflight_javeditbench.py \
  --scope evaluation \
  --output-dir outputs/javeditbench_step30000_dev \
  --strict

Run all metrics on four GPUs:

VLLM_WORKER_MULTIPROC_METHOD=spawn \
python metrics/evaluate.py \
  --video_dir outputs/javeditbench_step30000_dev \
  --bench_csv benchmark_150.csv \
  --source_video_dir data/JAVEdit-100k/benchmark_videos \
  --output_path outputs/javeditbench_step30000_dev/eval_results \
  --metric syncnet vtss utmos av_quality instruction_compliance video_fidelity \
  --num_gpus 4 \
  --name step30000_dev

The evaluator writes detailed *_results.json and *_summary.csv files. Qwen metrics use structured JSON output; if a response exhausts its token budget, retry that sample and merge it by canonical __item_<row> filename rather than treating it as zero.

Step-30k results

The Qwen scores include the successful retries that complete all 150 samples. SyncNet reports the mean over the 123 samples for which the face/lip pipeline returned a valid confidence; it is not divided by 150. VTSS returned 149 valid scores.

Metric ↑	Step-30k result	Valid samples	Paper-reported reference	Difference
Visual Quality (VTSS)	0.0578	149/150	0.0596	-0.0018
Audio Quality (UTMOSv2)	2.61	150/150	2.42	+0.19
A/V Sync (SyncNet)	0.3107	123/150	0.3688	-0.0581
Instruction Compliance	4.13	150/150	4.07	+0.06
Video Fidelity	4.25	150/150	4.22	+0.03
A/V Quality	3.65	150/150	3.88	-0.23

The local results come from our fullstate_0802 step-30k LoRA and dev-backbone inference. They are not a bit-exact reproduction of the reference row: checkpoint lineage, inference implementation, valid-sample handling, and stochastic MLLM judging can change the result. The largest observed gaps are A/V Sync and A/V Quality; Audio Quality, Instruction Compliance, and Video Fidelity are above the reported reference values in this run.

Files used for the final table:

outputs/javeditbench_step30000_dev/eval_results/
├── step30000_dev_20260809_181526_results.json
├── step30000_dev_utmos_retry_20260809_182508_results.json
├── step30000_dev_qwen_retry2_20260809_183922_results.json
├── step30000_dev_instruction_retry_20260809_184830_results.json
└── step30000_dev_qwen_missing_retry_20260809_193401.json

7. Browse results in Gradio

The browser groups all 150 samples by editing task and displays the source video, instruction, edited video, and per-sample metrics.

source scripts/activate_javedit_metrics.sh
pip install -r apps/requirements-gradio.txt

python apps/javeditbench_browser.py \
  --video-dir outputs/javeditbench_step30000_dev \
  --bench-csv benchmark_150.csv \
  --host 0.0.0.0 \
  --port 7860

Open http://<server-address>:7860. In a remote pod, expose or forward port 7860 according to the platform's networking settings.

Credentials and reproducibility notes

• Never commit Hugging Face or W&B tokens. Use hf auth login and the WANDB_API_KEY environment variable.
• The private Hugging Face repository stores the reproducible step-30k artifacts; Git stores code and lightweight metadata.
• A LoRA file is sufficient for inference. Exact training resume also requires optimizer, training state, and per-rank RNG files from the same step.
• Keep the LTX-2.3 dev backbone identical between training and this inference path.