ThomasThebaud/PROPS_K16

PROPS K16

PROPS K16 is the fine-tuned 16-component-per-profile model from ProPS: Prompted Profile Synthesis for Natural Language-Conditioned Speaker Embedding Distributions. It maps an English natural-language speaker description to a Gaussian mixture distribution over 192-dimensional ECAPA-TDNN speaker embeddings.

This is a research checkpoint, not a Transformers model and not a text-to-speech system. It produces a distribution in speaker-embedding space; it does not synthesize audio.

Model details

• Architecture: composed Gaussian mixture mixture-density network (MDN)
• Text encoder: sentence-transformers/all-MiniLM-L6-v2, producing 384-dimensional inputs
• Target space: normalized 192-dimensional embeddings from speechbrain/spkrec-ecapa-voxceleb
• Requested precomputed components per training profile: 16
• Effective output components stored in this checkpoint: 15,072
• Hidden dimensions: 1024, 2048, 1024; GELU, LayerNorm, dropout 0.1
• Fine-tuned checkpoint: best saved epoch 5; stored train loss -343.3753 and development NLL -310.3394
• Training dataset: Capspeech_min100, assembled from Common Voice, GigaSpeech, MLS English, and Emilia English metadata and filtered to profiles represented by at least 100 utterances
• Source code: thomasthebaud/PROPS

The component means and diagonal standard deviations are included in the checkpoint. The original precomputed-GMM directory is therefore not needed for inference with the loader in this repository.

Usage

pip install -r requirements.txt
python inference.py "A young adult woman with a moderate speaking rate and an American accent"

This writes generated_gmm.npz with pi_logits, normalized pi, mu, and diagonal sigma arrays.

Python usage:

import torch
from sentence_transformers import SentenceTransformer
from model import ComposedGMMMDN

device = "cuda" if torch.cuda.is_available() else "cpu"
encoder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2", device=device)
model = ComposedGMMMDN.from_pretrained("ThomasThebaud/PROPS_K16", device=device)

text = ["A middle-aged speaker with a low-pitched, expressive voice"]
x = encoder.encode(text, convert_to_tensor=True).to(device)
pi, mu, sigma = model.distribution(x)
samples = model.sample(x, samples_per_prompt=10)

For reproducible use, review and pin the upstream text-encoder revision in your own environment.

Training procedure

The model was first pretrained to route profile-description embeddings toward components of real per-profile GMMs. It was then fine-tuned with per-utterance Capspeech descriptions by minimizing negative log likelihood of normalized ECAPA speaker embeddings. Model selection used development-set NLL.

The uploaded artifact is named 16_components_Capspeech_min100_p=1_finetune.pt, as supplied for this repository.

Intended uses

The model is intended for research on description-conditioned speaker representations, analysis of speaker-profile distributions, and generation of synthetic speaker embeddings for controlled experiments. Validate suitability on your own data and demographic groups before drawing conclusions.

Limitations, risks, and bias

• Outputs inherit biases, label noise, demographic imbalance, recording conditions, and coverage gaps from the source speech corpora, generated descriptions, SBERT, and ECAPA encoder.
• Profile labels such as age, gender, accent, pitch, speaking rate, and expressiveness are reductive and may be incorrect or socially sensitive. They should not be treated as ground truth about a person.
• Natural-language paraphrases, out-of-domain descriptions, non-English text, intersectional profiles, and underrepresented accents may behave unpredictably.
• Generated embeddings are not guaranteed to correspond to a natural, unique, or consenting speaker.
• Do not use this model for identity inference, surveillance, authentication, high-impact decisions, demographic profiling of individuals, impersonation, or deceptive synthetic media.
• The stored NLL values are checkpoint-selection metadata, not a broad benchmark of quality or fairness. Consult the paper and source repository for the full evaluation protocol.

Checkpoint integrity

SHA-256:

d8348add37fcef6050ab54a56fefd3b2cd95a65b40d7cacb8071d54fb62e7ecc  16_components_Capspeech_min100_p=1_finetune.pt

PyTorch checkpoints use pickle-based serialization. Load only files obtained from this trusted repository. The supplied loader requests weights_only=True on supported PyTorch versions.

Citation

@article{thebaud2026props,
  title={ProPS: Prompted Profile Synthesis for Natural Language-Conditioned Speaker Embedding Distributions},
  author={Thebaud, Thomas and Lee, Junhyeok and Moro-Velazquez, Laureano and Lopez, Jesus Villalba and Dehak, Najim},
  journal={arXiv preprint arXiv:2607.05276},
  year={2026}
}

No license file was present in the inspected source checkout. Add an explicit license before publishing if you intend to grant reuse rights, and verify that distribution complies with the licenses and terms of all source datasets and upstream models.