TTS-Khmer-Checkpoint

A Khmer text-to-speech checkpoint based on Meta's Massively Multilingual Speech (MMS) TTS model for Khmer.

Base model

This repository hosts the weights of facebook/mms-tts-khm, Meta AI's VITS-based text-to-speech model for the Khmer language, released as part of the MMS project. All credit for the underlying architecture and pretrained weights goes to the original authors.

• Original model: https://huggingface.co/facebook/mms-tts-khm
• Original paper: Pratap et al., Scaling Speech Technology to 1,000+ Languages (Meta AI, 2023) — https://arxiv.org/abs/2305.13516
• Architecture: VITS (Conditional Variational Autoencoder with Adversarial Learning)

This checkpoint is used as the starting point for further Khmer TTS work by phonsobon.

Usage

Install dependencies:

pip install transformers torch scipy

Basic inference:

from transformers import VitsModel, AutoTokenizer
import torch
import scipy.io.wavfile

model = VitsModel.from_pretrained("phonsobon/TTS-Khmer-Checkpoint")
tokenizer = AutoTokenizer.from_pretrained("phonsobon/TTS-Khmer-Checkpoint")

text = "សួស្តី​ពិភពលោក"  # "Hello world" in Khmer
inputs = tokenizer(text, return_tensors="pt")

with torch.no_grad():
    output = model(**inputs).waveform

scipy.io.wavfile.write(
    "output.wav",
    rate=model.config.sampling_rate,
    data=output.float().numpy().squeeze(),
)

Full test script (test_model.py)

"""
Test script for phonsobon/TTS-Khmer-Checkpoint

Loads the model straight from the Hub and synthesizes a few Khmer
sentences to .wav files, so you can confirm the checkpoint works.
"""

from transformers import VitsModel, AutoTokenizer
import torch
import scipy.io.wavfile

REPO_ID = "phonsobon/TTS-Khmer-Checkpoint"

TEST_SENTENCES = [
    "សួស្តី​ពិភពលោក",              # "Hello world"
    "ថ្ងៃនេះអាកាសធាតុល្អណាស់",       # "The weather is very nice today"
    "អរគុណច្រើន",                   # "Thank you very much"
]


def main():
    print(f"Loading model from {REPO_ID} ...")
    model = VitsModel.from_pretrained(REPO_ID)
    tokenizer = AutoTokenizer.from_pretrained(REPO_ID)
    model.eval()

    for i, text in enumerate(TEST_SENTENCES, start=1):
        inputs = tokenizer(text, return_tensors="pt")

        with torch.no_grad():
            output = model(**inputs).waveform

        out_path = f"test_output_{i}.wav"
        scipy.io.wavfile.write(
            out_path,
            rate=model.config.sampling_rate,
            data=output.float().numpy().squeeze(),
        )
        print(f'  [{i}] "{text}" -> {out_path}')

    print("Done. Play the .wav files to check the audio quality.")


if __name__ == "__main__":
    main()

Run it with:

python test_model.py

Voice Cloning

facebook/mms-tts-khm (and this checkpoint) is a single-speaker VITS model — it always speaks in the one voice it was trained on, and does not accept a speaker embedding. To make it say text in a different person's voice, the standard approach is to run its output through a separate voice-conversion / voice-cloning model, using a short reference clip of the target voice.

The example below uses OpenVoice V2, a zero-shot voice cloning / tone-color-conversion model, as a post-processing step on top of the MMS-TTS-Khm output:

pip install torch torchaudio openvoice-cli
# Also download the OpenVoice V2 checkpoints as described in the OpenVoice repo

import torch
from transformers import VitsModel, AutoTokenizer
import scipy.io.wavfile
from openvoice import se_extractor
from openvoice.api import ToneColorConverter

REPO_ID = "phonsobon/TTS-Khmer-Checkpoint"
CONVERTER_CKPT_DIR = "checkpoints_v2/converter"  # from OpenVoice V2 release
REFERENCE_VOICE = "reference_voice.wav"           # short clip of the target voice

device = "cuda" if torch.cuda.is_available() else "cpu"

# 1. Synthesize Khmer speech with the base (single-speaker) TTS model
model = VitsModel.from_pretrained(REPO_ID).to(device)
tokenizer = AutoTokenizer.from_pretrained(REPO_ID)

text = "សួស្តី​ពិភពលោក"
inputs = tokenizer(text, return_tensors="pt").to(device)
with torch.no_grad():
    waveform = model(**inputs).waveform

base_wav_path = "base_tts_output.wav"
scipy.io.wavfile.write(
    base_wav_path,
    rate=model.config.sampling_rate,
    data=waveform.float().cpu().numpy().squeeze(),
)

# 2. Clone the reference voice's tone color onto the synthesized speech
tone_color_converter = ToneColorConverter(f"{CONVERTER_CKPT_DIR}/config.json", device=device)
tone_color_converter.load_ckpt(f"{CONVERTER_CKPT_DIR}/checkpoint.pth")

source_se, _ = se_extractor.get_se(base_wav_path, tone_color_converter, vad=True)
target_se, _ = se_extractor.get_se(REFERENCE_VOICE, tone_color_converter, vad=True)

tone_color_converter.convert(
    audio_src_path=base_wav_path,
    src_se=source_se,
    tgt_se=target_se,
    output_path="cloned_voice_output.wav",
)

cloned_voice_output.wav will contain the Khmer sentence spoken in the reference speaker's voice. Notes:

• This is a two-stage pipeline (TTS → voice conversion), not native multi-speaker synthesis — the base model itself is not being fine-tuned or changed.
• Voice cloning quality depends heavily on the reference clip (clean audio, 10+ seconds, single speaker, minimal background noise).
• OpenVoice was not trained on Khmer specifically, so results may vary; treat this as a starting point rather than production-ready cloning.

Citation

This model was developed by Vineel Pratap et al. at Meta AI. If you use it, please cite the MMS paper:

@article{pratap2023mms,
    title={Scaling Speech Technology to 1,000+ Languages},
    author={Vineel Pratap and Andros Tjandra and Bowen Shi and Paden Tomasello and Arun Babu and Sayani Kundu and Ali Elkahky and Zhaoheng Ni and Apoorv Vyas and Maryam Fazel-Zarandi and Alexei Baevski and Yossi Adi and Xiaohui Zhang and Wei-Ning Hsu and Alexis Conneau and Michael Auli},
    journal={arXiv},
    year={2023}
}

License

CC-BY-NC 4.0, inherited from the original facebook/mms-tts-khm model.