|
Download GPU_BATCHING.md from TuringLogic/little-tts: direct link, hf CLI and curl.
- Browser
- Download file 9.16 kB
-
https://huggingface.co/TuringLogic/little-tts/resolve/main/GPU_BATCHING.md
- Command line
-
hf download hf://TuringLogic/little-tts/GPU_BATCHING.md
-
curl -L -o GPU_BATCHING.md https://huggingface.co/TuringLogic/little-tts/resolve/main/GPU_BATCHING.md
9.16 kB
| # FP32 GPU inference and batching | |
| The batched ONNX path uses **no PyTorch, FP16, TF32, or weight quantization**. | |
| It reuses the released trained weights without retraining. The browser demo keeps | |
| using the original single-request WASM model; these additional graphs are for | |
| native CPU/CUDA deployments. | |
| ## Measured results | |
| Test hardware: **NVIDIA GeForce RTX 5090, 32 GB**, driver 580.178.04, Linux, | |
| Python 3.12.3, ONNX Runtime GPU 1.32.0 with CUDA 13 libraries. The environment | |
| contained no PyTorch installation. **These are not RTX PRO 4000 measurements.** | |
| Batch latency includes pronunciation, CPU/GPU transfers, synthesis and audio | |
| returned to CPU memory. Models were loaded and warmed up first. Each row has | |
| 20 measured repeats, using two passages and both voices, repeated to fill the batch. Speaking speeds were calibrated | |
| to approximately five or ten seconds per clip; actual lengths are in the report. | |
| | Audio length per request | Batch size | Median batch latency | p95 batch latency | | |
| |---|---:|---:|---:| | |
| | ~5 seconds | 1 | 23.5 ms | 23.6 ms | | |
| | ~5 seconds | 8 | 32.7 ms | 33.0 ms | | |
| | ~5 seconds | 16 | 44.2 ms | 44.4 ms | | |
| | ~5 seconds | 32 | 69.7 ms | 70.0 ms | | |
| | ~10 seconds | 1 | 26.2 ms | 26.3 ms | | |
| | ~10 seconds | 8 | 47.9 ms | 48.2 ms | | |
| | ~10 seconds | 16 | 73.3 ms | 74.5 ms | | |
| | ~10 seconds | 32 | 157.4 ms | 161.2 ms | | |
| The 32-request, ten-second case was automatically split into two decoder groups | |
| by the default padded-frame memory budget. With 16 ten-second requests, the new | |
| batch path took about **73 ms**, compared with about **1,269 ms** to run those | |
| requests serially through the original CUDA model: approximately **17.3×** faster | |
| for that workload. The original comparison used the same precision and texts. | |
| The microbatch queue was also tested with **64 requests submitted together**, | |
| a maximum collection size of 32 and a 2 ms collection window: | |
| | Clip length | All 64 requests completed | Completed within 0.5 seconds | | |
| |---|---:|---:| | |
| | ~5 seconds | 157 ms | 64 / 64 | | |
| | ~10 seconds | 343 ms | 64 / 64 | | |
| These burst results are a single controlled burst per clip-length setting, | |
| not a production SLA or a maximum-capacity measurement. WAV encoding, HTTP, | |
| network transfer, cold starts and unrelated GPU workloads are excluded. Larger | |
| or more varied inputs and a longer queue change completion times. Throughput | |
| and simultaneous-request deadlines must not be treated as interchangeable. | |
| The largest sampled per-process GPU residency after a benchmark run was | |
| **4,106 MiB**, including the comparison model's session. This is not a sampled | |
| peak, a minimum VRAM requirement, or a guarantee for other batch/frame limits. | |
| Test on your own RTX PRO 4000 before promising the same latency. | |
| Raw data: [GPU benchmark](evaluation/gpu_batch_benchmark.json), | |
| [GPU parity](evaluation/batched_validation_cuda.json), | |
| [CPU parity](evaluation/batched_validation.json). | |
| ## Install and generate a batch | |
| Use a fresh Python environment with an NVIDIA driver compatible with CUDA 13. | |
| The GPU requirements install CUDA/cuDNN runtime libraries through pip; they do | |
| not install a training framework. Do not install both `onnxruntime` and | |
| `onnxruntime-gpu` in the same environment. The repository's training-package | |
| installation is unnecessary for this standalone inference path. | |
| ```bash | |
| git clone https://github.com/ub1979/little-tts | |
| cd little-tts | |
| python -m venv .venv-gpu | |
| source .venv-gpu/bin/activate | |
| pip install -r requirements-onnx-gpu.txt | |
| python batched_onnx.py --device cuda --batch-size 16 \ | |
| --text "The morning sunlight streamed through the window, warming the quiet room." \ | |
| --text "The garden was quiet, and the air was filled with the scent of roses." \ | |
| --output batch-audio | |
| ``` | |
| To deploy only the GPU inference files from Hugging Face, rather than clone | |
| the training repository, use the Hub CLI: | |
| ```bash | |
| pip install 'huggingface-hub>=1.16,<2' | |
| hf download TuringLogic/little-tts --local-dir little-tts-gpu \ | |
| --include 'onnx/batched/*' 'phonemizer/*' 'batched_onnx.py' 'batch_queue.py' \ | |
| 'requirements-onnx-gpu.txt' 'LICENSE*' 'NOTICE' | |
| ``` | |
| The original CPU/browser graph is still `onnx/model.onnx`. The new graphs are | |
| `onnx/batched/encoder.onnx` and `onnx/batched/decoder.onnx`, with metadata in | |
| `onnx/batched/config.json`. All learned floating-point weights and audio outputs remain FP32. | |
| ```python | |
| from batched_onnx import BatchedSynthesizer | |
| synth = BatchedSynthesizer(".", device="cuda", max_batch=16, | |
| max_padded_frames=16000) | |
| results = synth.speak_batch( | |
| ["The garden was quiet, and the air was filled with roses.", | |
| "The road curved gently through the hills."], | |
| voices=["92", "6097"], | |
| speeds=[1.0, 1.0], | |
| seeds=[0, 1], | |
| ) | |
| for waveform, durations in results: | |
| print(len(waveform) / synth.sample_rate) | |
| print(synth.last_stats) | |
| ``` | |
| Results are restored to input order, even when requests are length-sorted. | |
| Voice 92 remains the recommended default. `--device cuda` fails if CUDA cannot | |
| initialize; it does not silently switch the full session to CPU. Small shape | |
| operations can still be assigned to CPU by ONNX Runtime. | |
| ## Combine independent callers | |
| `speak_batch()` processes a supplied list. For independent concurrent callers, | |
| use the bounded queue to collect requests into actual shared batches: | |
| ```python | |
| from batch_queue import BatchQueue | |
| with BatchQueue(synth, max_wait_ms=2, max_pending=256) as queue: | |
| first = queue.submit("The morning was quiet, and the sky was clear.") | |
| second = queue.submit("The garden was filled with the scent of roses.") | |
| first_wave, _ = first.result() | |
| second_wave, _ = second.result() | |
| ``` | |
| Use one queue per synthesizer. Direct concurrent calls to `speak_batch()` are | |
| serialized to protect shared runtime state; the queue is what combines them. | |
| Requests receive results at batch completion. Queue capacity is bounded, and | |
| overload raises an error rather than accumulating unlimited work. Per-call | |
| settings are validated; inference failures are propagated to the affected batch. | |
| `max_batch` controls collection size. `max_padded_frames` limits | |
| `batch_size × longest_predicted_frame_count` before allocating DSP tensors; an | |
| oversized group is split automatically. This is a memory-control heuristic, | |
| not an exact VRAM allocation limit. Each request is limited to 512 phoneme | |
| tokens and 3,000 frames; split longer input into passages. For low latency with | |
| ten-second clips, start with `max_batch=16` and measure the actual workload. | |
| ## What changed | |
| - Real dynamic batches, token masks and frame masks replace batch-size-one | |
| assumptions. Each request keeps its own voice, speed, pitch controls, noise | |
| seed, durations and output length; padding cannot leak into real audio. | |
| - Text encoding and waveform decoding are separate graphs. Only small duration | |
| arrays return to the host to select the frame extent. I/O binding keeps the | |
| large encoder hidden tensor on CUDA between stages. | |
| - RoPE positions and other shape inputs come from cached host-defined ranges, | |
| avoiding duration-dependent GPU-to-host shape discovery inside a monolithic | |
| graph. Runtime graph optimization is enabled. | |
| - The harmonic oscillator is evaluated in 16-harmonic tiles to bound temporary | |
| memory. The noise filter uses the original real-valued DFT matrices as dense | |
| matrix multiplications and four-way overlap-add, avoiding the slow DSP | |
| convolution/transposed-convolution route measured on the test GPU. | |
| - CUDA uses the NC1D convolution layout, and TF32 is explicitly disabled. | |
| No TensorRT, PyTorch or quantization is required. | |
| ## Rebuild and validate without PyTorch | |
| The checked-in graphs are ready to use. To rebuild them, install the small | |
| additional graph-building requirement: | |
| ```bash | |
| pip install -r requirements-onnx-build.txt | |
| python scripts/build_batched_onnx.py | |
| python scripts/validate_batched_onnx.py --device cuda \ | |
| --output evaluation/my_gpu_validation.json | |
| ``` | |
| The builder extracts unchanged learned tensors directly from `onnx/model.onnx`, | |
| not from a PyTorch checkpoint. It records that source file's SHA256. The parity | |
| checker exposes the original ONNX graph's noise input and supplies identical | |
| noise to compare waveform samples. CPU and GPU checks cover both voices, | |
| mixed lengths/speeds/pitch settings, input-order restoration, batch invariance | |
| and forced memory-budget splitting. All tested token durations and output | |
| lengths matched. Maximum GPU waveform error was approximately **0.1% of reference | |
| peak amplitude**; this numerical check is not a human naturalness rating. | |
| To reproduce the latency and simultaneous-burst tests: | |
| ```bash | |
| python scripts/benchmark_batched_onnx.py --device cuda \ | |
| --batch-sizes 1 2 4 8 16 32 --seconds 5 10 \ | |
| --repeats 20 --serial-repeats 3 --burst 64 \ | |
| --output evaluation/my_gpu_benchmark.json | |
| ``` | |
| For CPU-only validation, use `requirements-onnx.txt` and | |
| `requirements-onnx-build.txt` in a separate environment, then pass `--device cpu`. | |
| The existing browser demo does not use or download the batched graphs. | |
| Weights, including these derived graphs, are CC BY-NC 4.0. Source code is | |
| Apache-2.0; third-party pronunciation assets retain their included notices. | |