Conformer Voice Turn-Taking & Barge-in Detector
A lightweight, end-to-end Conformer-based acoustic classification model (~2.1M parameters) engineered for low-latency conversational AI agents, real-time voice bots, and turn-taking pipelines.
Overview
Traditional Voice Activity Detection (VAD) relies heavily on arbitrary silence timeouts (500–800 ms), leading to awkward dead air or unintended user interruptions.
This model evaluates acoustic prosody, pitch contour, and speech dynamics over a 1.5-second sliding window to classify conversational intent in real time, delivering sub-15ms inference latency on commodity CPUs using ONNX Runtime.
Classes
| Class | Description | Actionable Agent Behavior |
|---|---|---|
complete |
The speaker has concluded their sentence. | Agent can claim the floor and synthesize response immediately. |
incomplete |
Mid-sentence hesitation or natural thinking pause. | Agent holds back and continues listening. |
barge_in |
Sudden user interruption while the agent is speaking. | Agent halts audio playback and yields the turn. |
Model Specifications
- Architecture: Conformer-based
EncDecClassificationModel(~2.1M parameters) - Audio Input: 16 kHz Mono PCM
- Window Size: 1.5 seconds (24,000 samples)
- Sliding Step: 200 ms (3,200 samples)
- Runtime Target: ONNX Runtime (<15 ms CPU latency)
- Artifacts:
turn_taking_final.onnx: Optimized ONNX computation graph for production inference.turn_taking_final.nemo: NVIDIA NeMo checkpoint containing network architecture and PyTorch weights.turn_taking_model.yaml: Feature extractor and model hyperparameters configuration.
Quickstart
ONNX Runtime Inference
import numpy as np
import onnxruntime as ort
# Load ONNX session
session = ort.InferenceSession("turn_taking_final.onnx")
# Dummy 1.5s audio slice (16 kHz mono)
audio_chunk = np.random.randn(1, 24000).astype(np.float32)
audio_length = np.array([24000], dtype=np.int64)
# Run inference
inputs = {
session.get_inputs()[0].name: audio_chunk,
session.get_inputs()[1].name: audio_length,
}
logits = session.run(None, inputs)[0]
# Compute softmax probabilities
probs = np.exp(logits) / np.sum(np.exp(logits), axis=-1, keepdims=True)
labels = ["complete", "incomplete", "barge_in"]
result = dict(zip(labels, probs[0]))
print(result)
- Downloads last month
- 1