Post
127
A full voice agent — hearing, thinking, and speaking — now fits in ~1.2 GB on an iPhone. And on a Mac, it runs Gemma 4 at 214 tokens/sec, entirely on-device.
We just measured our on-device voice-agent pipeline end to end, and the numbers are the whole story.
It's one loop — voice detection → speech-to-text → language model → speech synthesis — and the only thing that changes per device is which model fills each slot:
📱 iPhone 16 Pro — whole loop resident in ~1.2 GB, speech-to-text at 0.04 RTF on the Neural Engine
📱 Galaxy S23 — ~1.5 GB, real-time on CPU
💻 Mac (M5 Pro) — ~3.6 GB, now with Gemma 4 E2B as the brain at 214 tok/s and Supertonic-3 speech synthesis at 0.03 RTF
The desktop result is the new part: a real conversational model that doesn't just answer — it drives the machine by voice, opening apps and acting on them, fully local. On paper the per-model peaks add up to ~5 GB, but measured co-resident it's ~3.6 GB — the on-device weights are memory-mapped, so they don't stack.
Audio and conversation state stay on the device: private by default, no server round-trip, no per-call cost. Perception, reasoning, and synthesis — all on the metal in front of you.
Full write-up, with the mobile and the new desktop benchmark tables and the architecture diagram:
https://soniqo.audio/blog/on-device-voice-agents
Linux/WIndows: https://github.com/soniqo/speech-core
Android: https://github.com/soniqo/speech-android
Mac/iOS: https://github.com/soniqo/speech-swift
#OnDeviceAI #EdgeAI #AppleSilicon #MLX #LocalLLM #VoiceAI #Gemma
We just measured our on-device voice-agent pipeline end to end, and the numbers are the whole story.
It's one loop — voice detection → speech-to-text → language model → speech synthesis — and the only thing that changes per device is which model fills each slot:
📱 iPhone 16 Pro — whole loop resident in ~1.2 GB, speech-to-text at 0.04 RTF on the Neural Engine
📱 Galaxy S23 — ~1.5 GB, real-time on CPU
💻 Mac (M5 Pro) — ~3.6 GB, now with Gemma 4 E2B as the brain at 214 tok/s and Supertonic-3 speech synthesis at 0.03 RTF
The desktop result is the new part: a real conversational model that doesn't just answer — it drives the machine by voice, opening apps and acting on them, fully local. On paper the per-model peaks add up to ~5 GB, but measured co-resident it's ~3.6 GB — the on-device weights are memory-mapped, so they don't stack.
Audio and conversation state stay on the device: private by default, no server round-trip, no per-call cost. Perception, reasoning, and synthesis — all on the metal in front of you.
Full write-up, with the mobile and the new desktop benchmark tables and the architecture diagram:
https://soniqo.audio/blog/on-device-voice-agents
Linux/WIndows: https://github.com/soniqo/speech-core
Android: https://github.com/soniqo/speech-android
Mac/iOS: https://github.com/soniqo/speech-swift
#OnDeviceAI #EdgeAI #AppleSilicon #MLX #LocalLLM #VoiceAI #Gemma