FuseAlign multilingual (noisy student, step 60,000)

Word-level forced aligner: given a recording and its transcript, it predicts each word's start and end on a 1 ms grid plus a presence probability. A reproduction of FuseAlign (Vaidya et al., 2026) by Zyphra: one multilingual noisy student, trained 60,000 steps from the multilingual model's CJK fine-tune on about 25,000 h of audio in 73 languages, mostly labelled on clean audio by a teacher per language and learnt from augmented audio.

Code, usage, long-form alignment and the C++/ggml CLI: github.com/Zyphra/fusealign-multilingual.

Files

File Size Use
fusealign-multilingual-ns4-step60000.safetensors 1.34 GB PyTorch package (torch.hub, fusealign_hub): FuseAlign fp32, Dia text encoder bf16
fusealign-multilingual-ns4-step60000-f32.gguf 1.35 GB ggml port: FuseAlign F32, Dia BF16, Silero VAD F32
fusealign-multilingual-ns4-step60000-q8_0.gguf 0.51 GB ggml port: transformer matmuls Q8_0, other Dia tensors BF16, the rest F32

Usage

import soundfile as sf
import torch

aligner = torch.hub.load("Zyphra/fusealign-multilingual", "fusealign", trust_repo=True, device="cuda")
wav, sr = sf.read("speech.wav", dtype="float32")
words = aligner.align(torch.from_numpy(wav), sr, "the transcript of the recording")
# [WordAlignment(word='the', start=..., end=..., presence=..., placed=True), ...]

Evaluation

% of word boundaries off by more than 50 ms (the paper's asymmetric error), against Qwen3-ForcedAligner-0.6B (qwen-asr 0.0.6, bf16, whole clips) on the same clips, transcripts and references (FuseAlign: the PyTorch package, CUDA, one clip per forward):

Corpus FuseAlign Qwen3-FA
TIMIT 7.92 20.17
Buckeye 7.23 24.60
AMI-silver 34.22 58.22
TIMIT-long 8.21 20.04
BZNSYP Mandarin (gold) 15.44 28.04
Seoul Corpus Korean (gold) 22.43 33.58
FLEURS Mandarin 17.97 11.06
FLEURS Japanese 13.34 8.38
FLEURS Korean 7.80 11.53
FLEURS Spanish 15.43 11.71
FLEURS Portuguese 9.20 10.24
FLEURS French 13.34 12.35
FLEURS German 13.99 13.91
FLEURS Russian 10.60 11.18

FLEURS references are Montreal Forced Aligner output, not hand labels; BZNSYP's and the Seoul Corpus's are hand-labelled (Seoul: interviewee turns of 40 speakers' spontaneous interviews, at most 30 s).

License

Apache-2.0. The checkpoint and GGUFs include the Dia text encoder weights from nari-labs/Dia-1.6B-0626 (Apache-2.0); the GGUFs include Silero VAD weights (MIT).

Citation

@article{vaidya2026fusealign,
  title   = {FuseAlign: Forced Alignment in the Wild},
  author  = {Vaidya, Mithilesh and Bailey, Stephen and Badam, Sumukh and Bendel, Matthew and He, Xingzhe},
  journal = {arXiv preprint arXiv:2609.33650},
  year    = {2026}
}
Downloads last month
149
GGUF
Model size
0.5B params
Architecture
fusealign
Hardware compatibility
Log In to add your hardware

8-bit

32-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Paper for Zyphra/fusealign

Article mentioning Zyphra/fusealign