The goal - what I'm exploring
Echo is my deep-dive into the part of voice AI that isn't the model - the ~800 milliseconds of perceived latency and the barge-in turn-taking. The question I wanted to answer: what does it actually take to make a spoken AI conversation feel alive instead of like a 1970s answering machine? The finding that made it click is that the fix is structural - overlap the pipeline stages with a sentence chunker, model turn-taking as an explicit state machine - not "buy a faster model."
How it uses AI
Echo runs on Google Gemini. The text-and-tools model (gemini-3.1-flash-lite by default) streams tokens over Server-Sent Events and calls functions - weather, time-zone, web search - mid-conversation. Speech-to-text and text-to-speech run in the browser via the Web Speech API, so the model is the only thing that ever touches the network. A second "Live" engine swaps in Gemini's native Live API for full speech-to-speech over a WebSocket. The model is the easy part; the orchestration around it - the sentence chunker that starts talking on the first sentence, the turn-taking state machine, the barge-in handling - is where the real engineering lives.
How it works
- Closes the whole spoken loop - mic → transcribe → think → speak - and lets you cut Echo off mid-sentence. Barge-in interruption is what makes it feel like a conversation instead of dictation.
- The hard part is the ~800ms latency budget, not the model: a sentence chunker starts text-to-speech on the first complete sentence while the rest of the reply is still generating.
- Gemini answers stream token-by-token over Server-Sent Events and can call weather, time-zone, and web-search tools mid-reply.
- Turn-taking is a single explicit state machine (idle → listening → thinking → speaking → barge-in), so the many out-of-order async voice events become no-ops instead of bugs.
- Two voice engines behind one toggle: a hand-built Classic pipeline (browser STT/TTS) and Gemini’s native Live API - PCM speech-to-speech over WebSocket, connected with single-use ephemeral tokens.
- Zero-setup demo on a shared key with a usage meter, plus bring-your-own-key for unlimited use; STT/TTS run in the browser, so no audio ever leaves your machine.