Voice Agents
How a voice agent actually fits together, and why decoupling the pipeline is what lets you find the layer that's breaking.
Members: watch the recordingKey takeaways
- Decouple the voice pipeline
- Structure prompts for spoken output: no Markdown
- Transition to workflows and dialogue trees
Decouple the voice pipeline
A robust voice agent has three separate, decoupled modules: speech-to-text (e.g. Whisper), the LLM (the brain / NLP), and text-to-speech (e.g. ElevenLabs or XTTS). Decoupling lets you isolate and fix failures when things break.
Structure prompts for spoken output: no Markdown
Synthetic voices struggle with asterisks, headers and emojis. Use strict system prompt harnesses to force short, terse, punctuation-light responses.
Transition to workflows and dialogue trees
Too much context in one prompt window increases hallucinations. Break logic out of the context window. Structured dialogue trees let you reset memory and keep the LLM focused on specific conversation stages.
What members walked out with
Decoupling your voice agent's pipeline lets you isolate where the conversation breaks. If you want to stop hallucinations, limit the context and move your business logic outside of a single prompt.
Bonus power prompts
Interrupt handler (real-time conversation)
“You are a real-time conversational voice assistant. If the user interrupts you or shifts topics, immediately stop your current line of thought, acknowledge the interruption with a brief phrase like 'Got it' or 'Sure,' and address the new question directly in ten words or less.”
Back-channeling (conversational flow)
“You are a voice agent. Respond with conversational back-channels like 'mm-hmm,' 'right,' or 'makes sense' only at the start of your turn to sound natural, then deliver your main point in a single, direct sentence. Avoid using lists, bullet points, or formal introductions.”
