PolyWhisper — Efficient Multilingual Indic ASR

PolyWhisper is an efficient multilingual Automatic Speech Recognition (ASR) system for Indian languages. It uses a frozen Whisper backbone with per-language LoRA adapters to achieve competitive accuracy at a fraction of the compute cost of full fine-tuning.

What is PolyWhisper?

PolyWhisper is a lightweight, production-ready ASR model for 5 major Indian languages: Hindi, Tamil, Telugu, Bengali, and Marathi. Unlike traditional approaches that fine-tune the entire model, PolyWhisper keeps the pretrained Whisper backbone frozen and trains small LoRA adapters (rank 16, ~14MB per language) that bolt onto the backbone at inference time.

Try the Demo

🌐 Live Demo on HuggingFace Spaces — upload audio or record from your microphone

Key Features

• 5 Indian languages in a single model: Hindi (hi), Tamil (ta), Telugu (te), Bengali (bn), Marathi (mr)
• Tiny adapter footprint: ~14MB per language vs ~1.5GB for full fine-tune
• Production-quality Hindi & Tamil: WER 37.2 (Hindi) and 60.4 (Tamil) on FLEURS benchmark
• Multiple backends: PyTorch (MPS/CUDA/CPU) + ONNX Runtime (CPU-only)
• CLI + Python API: polywhisper transcribe audio.wav --lang hi or from polywhisper import transcribe
• SRT subtitles: Export transcriptions as timestamped subtitles
• Batch processing: Transcribe entire folders of audio files
• ONNX export: Quantized INT8 models for CPU deployment

Quick Start

Install

pip install -e .

Requirements: Python 3.9+, PyTorch, Transformers, SoundFile. For CPU-only: pip install onnxruntime.

Transcribe an Audio File

# Hindi
polywhisper transcribe audio.wav --lang hi

# Tamil with JSON output
polywhisper transcribe audio.wav --lang ta --format json

# Auto-detect language
polywhisper transcribe audio.wav

Python API

from polywhisper import transcribe

# Single file
result = transcribe("audio.wav", lang="hi")
print(result.text)          # "नमस्ते, आप कैसे हैं?"
print(result.segments)      # [Segment(text=..., start_sec=0.0, end_sec=5.2)]

# Long audio (auto-chunks into 30s segments)
result = transcribe("long_meeting.mp3", lang="te", max_new_tokens=512)

Batch Transcription

polywhisper batch ./audio_folder/ --lang ta --output results.json

SRT Subtitles

polywhisper transcribe video_audio.wav --lang hi --format srt > subtitles.srt

Supported Languages

Language	Code	Script	FLEURS WER	Status
Hindi	hi	Devanagari	37.2	Production-ready
Tamil	ta	Tamil	60.4	Production-ready
Telugu	te	Telugu	105.7	Retraining scheduled
Bengali	bn	Bengali	196.3	Retraining scheduled
Marathi	mr	Devanagari	167.3	Retraining scheduled

Hindi and Tamil achieve better results than full model fine-tuning (expert baseline) using only ~14MB adapters. Telugu, Bengali, and Marathi are being retrained with domain-matched data for significant improvements.

Performance Comparison

Model	Params	Hindi WER	Tamil WER	Telugu WER
Whisper-Base (vanilla)	74M	131.3	93.2	185.6
Whisper-Small (vanilla)	244M	62.3	68.8	129.6
Whisper-Medium (vanilla)	769M	35.4	49.1	110.3
PolyWhisper Expert (Base+LoRA)	74M+3.5M	38.6	74.2	91.9
PolyWhisper Product (Small+LoRA)	244M+3.5M	37.2	60.4	105.7

Lower WER is better. Evaluated on FLEURS test set with punctuation-normalized scoring.

Model Architecture

┌─────────────────────────────────────────┐
│           Whisper Backbone (frozen)      │
│     244M params (encoder + decoder)     │
├─────────────────────────────────────────┤
│  LoRA Adapter (per-language, 14MB each) │
│  Rank 16, applied to:                   │
│  • Self-attention (Q, K, V, output)     │
│  • Cross-attention (Q, K, V, output)    │
│  • Encoder attention (HI only)          │
└─────────────────────────────────────────┘

• Backbone: OpenAI Whisper Small (244M parameters, frozen)
• Adapters: Low-Rank Adaptation (LoRA) with rank 16
• Total parameters: 244M (backbone) + 3.5M (adapter per language)
• Adapter size: ~14MB per language (vs ~1.5GB for full fine-tune)

Export to ONNX (CPU Deployment)

# Export all languages
polywhisper export --lang hi --variant prod --int8
polywhisper export --lang ta --variant prod --int8
polywhisper export --lang te --variant prod --int8
polywhisper export --lang bn --variant prod --int8
polywhisper export --lang mr --variant prod --int8

ONNX models are ~14MB per language with INT8 quantization. Use onnxruntime for CPU-only inference without PyTorch.

Training Details

• Training data: IndicVoices-ST (conversational speech)
• Evaluation: FLEURS (read speech)
• Optimizer: AdamW with cosine LR schedule (1e-4 peak)
• Training: 3 epochs per language, batch size 4, 2× NVIDIA T4 GPUs
• Techniques: Scheduled sampling, LoRA rank 16, encoder+decoder adaptation

Citation

@misc{polywhisper2026,
  title={PolyWhisper: Efficient Multilingual Indic ASR via Frozen Backbone + Per-Language LoRA},
  author={Eulogik Developer},
  year={2026},
  publisher={HuggingFace},
  url={https://huggingface.co/eulogik/polywhisper}
}

License

Apache 2.0. See LICENSE for details.

Links

• Model: huggingface.co/eulogik/polywhisper
• GitHub: github.com/eulogikdeveloper/PolyWhisper
• Training Data: ai4bharat/indicvoices-st