Download GPU_BATCHING.md from TuringLogic/little-tts: direct link, hf CLI and curl.
- Browser
- Download file 9.16 kB
-
https://huggingface.co/TuringLogic/little-tts/resolve/main/GPU_BATCHING.md
- Command line
-
hf download hf://TuringLogic/little-tts/GPU_BATCHING.md
-
curl -L -o GPU_BATCHING.md https://huggingface.co/TuringLogic/little-tts/resolve/main/GPU_BATCHING.md
FP32 GPU inference and batching
The batched ONNX path uses no PyTorch, FP16, TF32, or weight quantization. It reuses the released trained weights without retraining. The browser demo keeps using the original single-request WASM model; these additional graphs are for native CPU/CUDA deployments.
Measured results
Test hardware: NVIDIA GeForce RTX 5090, 32 GB, driver 580.178.04, Linux, Python 3.12.3, ONNX Runtime GPU 1.32.0 with CUDA 13 libraries. The environment contained no PyTorch installation. These are not RTX PRO 4000 measurements.
Batch latency includes pronunciation, CPU/GPU transfers, synthesis and audio returned to CPU memory. Models were loaded and warmed up first. Each row has 20 measured repeats, using two passages and both voices, repeated to fill the batch. Speaking speeds were calibrated to approximately five or ten seconds per clip; actual lengths are in the report.
| Audio length per request | Batch size | Median batch latency | p95 batch latency |
|---|---|---|---|
| ~5 seconds | 1 | 23.5 ms | 23.6 ms |
| ~5 seconds | 8 | 32.7 ms | 33.0 ms |
| ~5 seconds | 16 | 44.2 ms | 44.4 ms |
| ~5 seconds | 32 | 69.7 ms | 70.0 ms |
| ~10 seconds | 1 | 26.2 ms | 26.3 ms |
| ~10 seconds | 8 | 47.9 ms | 48.2 ms |
| ~10 seconds | 16 | 73.3 ms | 74.5 ms |
| ~10 seconds | 32 | 157.4 ms | 161.2 ms |
The 32-request, ten-second case was automatically split into two decoder groups by the default padded-frame memory budget. With 16 ten-second requests, the new batch path took about 73 ms, compared with about 1,269 ms to run those requests serially through the original CUDA model: approximately 17.3× faster for that workload. The original comparison used the same precision and texts.
The microbatch queue was also tested with 64 requests submitted together, a maximum collection size of 32 and a 2 ms collection window:
| Clip length | All 64 requests completed | Completed within 0.5 seconds |
|---|---|---|
| ~5 seconds | 157 ms | 64 / 64 |
| ~10 seconds | 343 ms | 64 / 64 |
These burst results are a single controlled burst per clip-length setting, not a production SLA or a maximum-capacity measurement. WAV encoding, HTTP, network transfer, cold starts and unrelated GPU workloads are excluded. Larger or more varied inputs and a longer queue change completion times. Throughput and simultaneous-request deadlines must not be treated as interchangeable.
The largest sampled per-process GPU residency after a benchmark run was 4,106 MiB, including the comparison model's session. This is not a sampled peak, a minimum VRAM requirement, or a guarantee for other batch/frame limits. Test on your own RTX PRO 4000 before promising the same latency.
Raw data: GPU benchmark, GPU parity, CPU parity.
Install and generate a batch
Use a fresh Python environment with an NVIDIA driver compatible with CUDA 13.
The GPU requirements install CUDA/cuDNN runtime libraries through pip; they do
not install a training framework. Do not install both onnxruntime and
onnxruntime-gpu in the same environment. The repository's training-package
installation is unnecessary for this standalone inference path.
git clone https://github.com/ub1979/little-tts
cd little-tts
python -m venv .venv-gpu
source .venv-gpu/bin/activate
pip install -r requirements-onnx-gpu.txt
python batched_onnx.py --device cuda --batch-size 16 \
--text "The morning sunlight streamed through the window, warming the quiet room." \
--text "The garden was quiet, and the air was filled with the scent of roses." \
--output batch-audio
To deploy only the GPU inference files from Hugging Face, rather than clone the training repository, use the Hub CLI:
pip install 'huggingface-hub>=1.16,<2'
hf download TuringLogic/little-tts --local-dir little-tts-gpu \
--include 'onnx/batched/*' 'phonemizer/*' 'batched_onnx.py' 'batch_queue.py' \
'requirements-onnx-gpu.txt' 'LICENSE*' 'NOTICE'
The original CPU/browser graph is still onnx/model.onnx. The new graphs are
onnx/batched/encoder.onnx and onnx/batched/decoder.onnx, with metadata in
onnx/batched/config.json. All learned floating-point weights and audio outputs remain FP32.
from batched_onnx import BatchedSynthesizer
synth = BatchedSynthesizer(".", device="cuda", max_batch=16,
max_padded_frames=16000)
results = synth.speak_batch(
["The garden was quiet, and the air was filled with roses.",
"The road curved gently through the hills."],
voices=["92", "6097"],
speeds=[1.0, 1.0],
seeds=[0, 1],
)
for waveform, durations in results:
print(len(waveform) / synth.sample_rate)
print(synth.last_stats)
Results are restored to input order, even when requests are length-sorted.
Voice 92 remains the recommended default. --device cuda fails if CUDA cannot
initialize; it does not silently switch the full session to CPU. Small shape
operations can still be assigned to CPU by ONNX Runtime.
Combine independent callers
speak_batch() processes a supplied list. For independent concurrent callers,
use the bounded queue to collect requests into actual shared batches:
from batch_queue import BatchQueue
with BatchQueue(synth, max_wait_ms=2, max_pending=256) as queue:
first = queue.submit("The morning was quiet, and the sky was clear.")
second = queue.submit("The garden was filled with the scent of roses.")
first_wave, _ = first.result()
second_wave, _ = second.result()
Use one queue per synthesizer. Direct concurrent calls to speak_batch() are
serialized to protect shared runtime state; the queue is what combines them.
Requests receive results at batch completion. Queue capacity is bounded, and
overload raises an error rather than accumulating unlimited work. Per-call
settings are validated; inference failures are propagated to the affected batch.
max_batch controls collection size. max_padded_frames limits
batch_size × longest_predicted_frame_count before allocating DSP tensors; an
oversized group is split automatically. This is a memory-control heuristic,
not an exact VRAM allocation limit. Each request is limited to 512 phoneme
tokens and 3,000 frames; split longer input into passages. For low latency with
ten-second clips, start with max_batch=16 and measure the actual workload.
What changed
- Real dynamic batches, token masks and frame masks replace batch-size-one assumptions. Each request keeps its own voice, speed, pitch controls, noise seed, durations and output length; padding cannot leak into real audio.
- Text encoding and waveform decoding are separate graphs. Only small duration arrays return to the host to select the frame extent. I/O binding keeps the large encoder hidden tensor on CUDA between stages.
- RoPE positions and other shape inputs come from cached host-defined ranges, avoiding duration-dependent GPU-to-host shape discovery inside a monolithic graph. Runtime graph optimization is enabled.
- The harmonic oscillator is evaluated in 16-harmonic tiles to bound temporary memory. The noise filter uses the original real-valued DFT matrices as dense matrix multiplications and four-way overlap-add, avoiding the slow DSP convolution/transposed-convolution route measured on the test GPU.
- CUDA uses the NC1D convolution layout, and TF32 is explicitly disabled. No TensorRT, PyTorch or quantization is required.
Rebuild and validate without PyTorch
The checked-in graphs are ready to use. To rebuild them, install the small additional graph-building requirement:
pip install -r requirements-onnx-build.txt
python scripts/build_batched_onnx.py
python scripts/validate_batched_onnx.py --device cuda \
--output evaluation/my_gpu_validation.json
The builder extracts unchanged learned tensors directly from onnx/model.onnx,
not from a PyTorch checkpoint. It records that source file's SHA256. The parity
checker exposes the original ONNX graph's noise input and supplies identical
noise to compare waveform samples. CPU and GPU checks cover both voices,
mixed lengths/speeds/pitch settings, input-order restoration, batch invariance
and forced memory-budget splitting. All tested token durations and output
lengths matched. Maximum GPU waveform error was approximately 0.1% of reference
peak amplitude; this numerical check is not a human naturalness rating.
To reproduce the latency and simultaneous-burst tests:
python scripts/benchmark_batched_onnx.py --device cuda \
--batch-sizes 1 2 4 8 16 32 --seconds 5 10 \
--repeats 20 --serial-repeats 3 --burst 64 \
--output evaluation/my_gpu_benchmark.json
For CPU-only validation, use requirements-onnx.txt and
requirements-onnx-build.txt in a separate environment, then pass --device cpu.
The existing browser demo does not use or download the batched graphs.
Weights, including these derived graphs, are CC BY-NC 4.0. Source code is Apache-2.0; third-party pronunciation assets retain their included notices.