Voice & realtime agents

1 · What a voice agent is essential

You've almost certainly been on the wrong end of one already. You call a bank, a robot voice reads you five options, you say "representative," and it replies "I'm sorry, I didn't understand — press 1 for billing." You try to interrupt; it keeps talking. Every pause you take, it either barges in over you or leaves three seconds of dead air. That experience — stilted, deaf to interruption, laggy — is exactly what a voice agent has to not be. Understanding why that old phone tree feels robotic is the fastest way to understand what a good voice agent is actually solving.

Here's the mental model. A text chatbot has it easy: you type, you hit Enter, it thinks, it prints. Turn-taking is trivial because Enter is an unambiguous "I'm done." Strip that away — make the whole thing happen through sound, live, with no keyboard — and every one of those clean boundaries turns into a hard, real-time problem. When did the human stop talking? Can they cut you off? How fast must you answer before it feels like lag? A voice agent is a chat agent wrapped in the machinery that answers those questions continuously, dozens of times a second, while audio is flowing in both directions.

So think of it as two things bolted together. In the middle sits the same kind of reasoning agent you already know — it reads words and decides what to say. Around it sits a real-time audio harness: something that turns your voice into words for the agent, turns the agent's words back into voice for you, and — the genuinely hard part — manages the timing of the back-and-forth so it feels like talking to a person. Get the harness right and a plain LLM feels alive on the phone; get it wrong and even the smartest model feels like that bank robot. The rest of this chapter is that harness, one piece at a time. A common beginner mistake is to assume the model is the whole product; in voice, the transport and timing around the model are usually where the experience is won or lost.

A voice agent is an agent you talk to out loud and that talks back — a phone support line, a hands-free assistant, a drive-thru order taker. A realtime agent is one where the interaction is live and continuous: the user speaks, the agent responds within a fraction of a second, and either side can interrupt the other. That live, low-latency, interruptible quality is what makes it feel like a conversation instead of a walkie-talkie.

Under the hood a voice turn is a loop. Sound goes in through a microphone, becomes text or tokens, an agent decides what to say, that reply becomes sound, and it comes out a speaker. The whole loop has to close fast enough that the human doesn't feel a gap.

The dashed idea not drawn as a box: barge-in. While the speaker is playing the agent's reply, the mic is still listening. If the human starts talking, the agent must stop its own playback and listen — the loop jumps from Speaker back to Mic. Without that, talking over the agent does nothing and it feels robotic.

2 · Two architectures: pipeline vs speech-to-speech essential

Imagine two ways to translate for a friend at a foreign dinner. In the first, your friend speaks, a transcriber writes down every word, a translator reads the transcript and works out a reply, and a third person reads that reply aloud in the new language. Three specialists, each doing one clean job — and you can fire and replace any one of them without touching the others. In the second, you find one bilingual person who simply listens and speaks back directly. Faster, warmer, catches the sarcasm in your friend's voice — but if they mistranslate, you can't peek at their notes, because there aren't any. That is exactly the choice you face building a voice agent, and neither option is "the right one."

The first way is the pipeline: three separate models — speech-to-text, then the LLM, then text-to-speech — wired in series, with plain text passing between them. Its great virtue is that text is inspectable. At every hop you have a transcript you can log, run guardrails on, evaluate, and store for a compliance audit. And because the stages are independent, you can shop for the best transcriber from one vendor and the best voice from another. The cost is twofold: latency adds up hop by hop (each stage waits for the one before it), and the moment audio becomes text you throw away everything that wasn't words — the laughter, the hesitation, the rising pitch of a question.

The second way is speech-to-speech: a single model that takes audio in and emits audio out, with no text checkpoint in the middle. Because nothing has to be serialized into words and back, it's lower-latency and it keeps the prosody — the emotional texture of speech survives. The price is exactly the transparency the pipeline gave away: with no automatic transcript, logging, guardrails, and debugging all get harder, and you can't swap out just the voice or just the reasoning — it's one opaque unit. The mistake to avoid is treating this as a purely technical benchmark; the deciding factor is usually a product question — do you need an auditable record more than you need the last 200 ms and a warmer voice?

There are two ways to build the loop. The classic pipeline chains three separate models: STT (speech-to-text) transcribes audio to words, an LLM reasons over the text, and TTS (text-to-speech) speaks the answer. Each stage is swappable and debuggable, but each adds latency, and tone/emotion is lost when audio becomes plain text.

Speech-to-speech (a native realtime/voice model) takes audio in and emits audio out directly — no text bottleneck in the middle. It's lower latency and keeps prosody (laughter, hesitation, emphasis), but it's harder to inspect, log, and guardrail because you don't automatically get a transcript.

3 · Streaming transport: why not request/response intermediate

Think about how a person actually answers you. They don't listen to your entire sentence, retreat into silence to compose a complete paragraph, and then deliver it in one breath. They start talking almost as soon as they've caught your drift — "Sure, so what you'll want to do is…" — assembling the rest of the thought while their mouth is already moving. That overlap is the secret to why human conversation feels instant even though nobody is actually faster than a computer. A voice agent that answers like a normal web API — wait for the whole question, compute the whole answer, then play it — will always feel like a walkie-talkie, because it refuses to start speaking until it's completely finished thinking.

The technical reason ordinary APIs can't do this is their shape: request → wait → full response is a single round trip that only completes when the entire answer is ready. Nothing comes back early. For voice that's fatal on both ends of the loop — you can't send audio until the user stops, and you can't play audio until the reply is done — so every stage's delay stacks into one long silence. The fix is to abandon the one-shot round trip for a persistent, two-way connection that stays open for the whole call: a WebSocket or WebRTC channel. Over it, audio flows as a continuous stream of tiny chunks (frames of, say, 20 ms) in both directions at once, rather than one big upload followed by one big download.

Streaming buys you two overlaps that together hide almost all the latency. First, you send the user's speech as they talk, so transcription can begin before they've finished their sentence. Second — and this is the one people underestimate — you begin playing the reply the instant the first audio frame is ready, long before the last word has been generated. The user hears "Sure, I can help with that…" while the model is still working out the details of "that." The runnable model below makes this concrete by printing a timeline: watch how the time-to-first-audio stays tiny even as the total reply grows, because what the listener perceives as responsiveness is when the first sound arrives, not when the last one does. The classic beginner error is optimizing total response time; in streaming voice, first-audio is the number that matters.

A normal API call is request → wait → full response. That's fatal for voice: you'd record the whole sentence, upload it, wait for the whole reply, then play it — seconds of dead air. Realtime voice uses a persistent bidirectional connection (a WebSocket or WebRTC) so audio flows continuously in both directions. You send mic audio in small chunks (e.g. every 20 ms) as you capture it, and you start playing the reply as the first audio frames arrive — before the agent has finished its whole sentence. That overlap is what hides latency.

The runnable model below streams a 'reply' out in chunks and shows that the user hears the first audio long before the last chunk is produced — the core reason streaming feels instant. It is pure stdlib and prints a timeline.

streaming.pydef stream_reply(chunks, per_chunk_ms):
    """Yield (elapsed_ms, chunk) as if streaming audio out frame by frame."""
    elapsed = 0
    for chunk in chunks:
        elapsed += per_chunk_ms
        yield elapsed, chunk

reply = ["Sure,", " I can", " help", " with", " that."]
first_audio_at = None
last_audio_at = 0
for elapsed, chunk in stream_reply(reply, per_chunk_ms=120):
    if first_audio_at is None:
        first_audio_at = elapsed
    last_audio_at = elapsed

# In a NON-streaming design the user hears nothing until the whole reply is done.
print("time to FIRST audio (streaming): ", first_audio_at, "ms")
print("time to LAST audio (whole reply):", last_audio_at, "ms")
print("non-streaming would wait:        ", last_audio_at, "ms before ANY sound")

time to FIRST audio (streaming):  120 ms
time to LAST audio (whole reply): 600 ms
non-streaming would wait:         600 ms before ANY sound

4 · Turn detection: whose turn is it? intermediate

Here's a problem so basic that it's easy to miss it's a problem at all. When does the agent get to talk? In a chat window the answer is trivial — you press Enter, and Enter means "I'm done, your move." Voice hands you no Enter key. All the agent has is a stream of sound, and somewhere in that stream the human stops and expects a reply. Guess too early and you chop them off mid-sentence like an over-eager interviewer; guess too late and you leave an awkward, dead silence that makes the whole thing feel broken. Deciding whose turn it is, from sound alone, is one of the quiet make-or-break problems of voice.

The tool for this is called turn detection (or endpointing), and the workhorse underneath it is VAD — voice activity detection, which simply classifies each little slice of audio as "speech" or "silence." Once you can tell speech from silence, a first, surprisingly effective rule falls out: the turn is over when silence has lasted long enough. Not the instant the person stops making noise — because people pause constantly to breathe, to think, to say "ummm" — but once the quiet has held past some threshold. That threshold has a name worth knowing: the hangover (or silence timeout), the grace period you wait through before declaring the turn finished.

The cleanest way to think about the logic is a tiny state machine — a system that sits in one named state and hops to another when something happens. Here there are three states: LISTENING (the human is talking), MAYBE_DONE (they've gone quiet, but not for long enough to be sure), and AGENT_TURN (the silence outlasted the hangover — respond now). Any fresh burst of speech snaps you back to LISTENING and forgets the pause, which is exactly what stops a mid-sentence breath from being mistaken for the end. The lab below runs this over a fake frame stream so you can watch a short pause get correctly ignored while a long one triggers a turn. Note the trap this design avoids and re-introduces at will: a fixed silence threshold is a blunt instrument — the "right" value depends on the language, the speaker, and how noisy the line is, which is why real systems eventually replace it with a smarter, model-based endpointer.

In text chat, pressing Enter marks end-of-turn. In voice there is no Enter — the agent must infer when the human has finished speaking so it can respond, and not cut them off mid-sentence. This is turn detection (a.k.a. endpointing), usually built on VAD (voice activity detection): detect speech vs silence, and when silence lasts longer than a threshold, treat the turn as over.

Model this as a tiny state machine. States: LISTENING (user talking), MAYBE_DONE (a silence gap started), and AGENT_TURN (silence held long enough — respond). This runs offline over a fake frame stream.

turns.pySILENCE_HANGOVER_MS = 500   # how long silence must hold before the turn ends

def detect_turns(frames, frame_ms=100):
    """frames: list of 'speech'/'silence'. Emit an event when a turn ends."""
    state = "LISTENING"
    silence_ms = 0
    events = []
    for i, f in enumerate(frames):
        t = i * frame_ms
        if f == "speech":
            silence_ms = 0
            state = "LISTENING"
        else:  # silence
            silence_ms += frame_ms
            if state == "LISTENING":
                state = "MAYBE_DONE"
            if state == "MAYBE_DONE" and silence_ms >= SILENCE_HANGOVER_MS:
                state = "AGENT_TURN"
                events.append(("turn_end", t))
                state = "LISTENING"     # reset for the next turn
                silence_ms = 0
    return events

# user says something, pauses briefly (not enough), then a real long pause
stream = (["speech"] * 6 + ["silence"] * 3      # 300 ms pause: too short
          + ["speech"] * 4 + ["silence"] * 6)   # 600 ms pause: turn ends
print(detect_turns(stream))

[('turn_end', 1700)]

5 · Barge-in: letting the human interrupt advanced

Picture the moment that most annoys everyone about automated phone systems. The robot launches into a long-winded menu — "For account balances, say balance; for recent transactions, say…" — and you already know you need a human. You say "agent." Nothing. It keeps reading. You have to sit through the whole speech, blood pressure rising, before it will listen to you. Now picture a real assistant: the second you start speaking, they stop mid-word and turn their attention to you. That instinct to yield the floor the moment you're interrupted is called barge-in, and its presence or absence is the single sharpest line between a voice agent that feels human and an IVR tree that feels like a hostage situation.

Mechanically, a barge-in is a three-step scramble that has to happen almost instantly. The moment you detect the user talking over the agent, you must (1) stop TTS playback right now, (2) throw away the rest of the queued reply — all the audio the agent was about to say is now irrelevant — and (3) switch back to listening so the interruption is captured. Miss any one of these and it breaks: forget to flush the queue and the agent finishes its old sentence a beat after you've started your new one; forget to switch to listening and you drop the very words the user interrupted with.

But there's a genuinely nasty catch hiding here, and it's the reason barge-in is an "advanced" topic rather than an obvious one: echo. The microphone doesn't only hear the human — it also hears the agent's own voice coming out of the speaker. If you naively treat "any sound on the mic while speaking" as a barge-in, the agent will constantly interrupt itself, stopping dead every time it hears its own words. So a real barge-in system needs echo cancellation (AEC) to subtract the agent's known output from the mic input, so only genuine user speech triggers the interrupt. The lab below models exactly this with a session that ignores its own echo but stops for real speech. Keep in mind the deeper reason this is subtle, previewed in the callout after: the reply you're trying to kill may be buffered in the model, in the network, and in the audio device all at once — a clean interrupt has to flush every one of those, fast.

Barge-in is the ability to interrupt the agent while it is speaking — the single biggest thing that separates a natural voice agent from an IVR phone tree. When the user starts talking over the agent's playback, you must (1) immediately stop TTS playback, (2) discard the rest of the queued reply, and (3) switch to listening. The hard part is echo: the mic hears the agent's own voice from the speaker, so you need echo cancellation or you'll 'barge in' on yourself.

bargein.pyclass VoiceSession:
    def __init__(self):
        self.state = "IDLE"
        self.play_queue = []
        self.log = []

    def agent_speaks(self, chunks):
        self.state = "SPEAKING"
        self.play_queue = list(chunks)
        self.log.append(f"agent speaking, {len(self.play_queue)} chunks queued")

    def on_user_speech(self, is_agents_own_echo):
        # Echo cancellation: ignore the agent's own voice coming back in.
        if is_agents_own_echo:
            self.log.append("ignored echo of own voice")
            return
        if self.state == "SPEAKING":
            dropped = len(self.play_queue)
            self.play_queue.clear()          # stop + flush playback
            self.state = "LISTENING"
            self.log.append(f"BARGE-IN: stopped playback, dropped {dropped} chunks")

s = VoiceSession()
s.agent_speaks(["chunk1", "chunk2", "chunk3", "chunk4"])
s.on_user_speech(is_agents_own_echo=True)    # speaker bleed -> ignored
s.on_user_speech(is_agents_own_echo=False)   # real interruption -> barge-in
print("final state:", s.state)
for line in s.log:
    print("-", line)

final state: LISTENING
- agent speaking, 4 chunks queued
- ignored echo of own voice
- BARGE-IN: stopped playback, dropped 4 chunks

6 · The latency budget: why ~800 ms matters advanced

You have a lag detector built into your brain, and it is brutally sensitive. You've felt it on a bad phone call: the tiny delay where you both start talking, both stop, both apologize, both start again. Face-to-face, the gaps between conversational turns are astonishingly short — on the order of 200 milliseconds, faster than you can consciously plan a sentence. We're so tuned to that rhythm that when a reply is even a little slow, we don't think "the computer is processing" — we assume the other person didn't hear us, and we start to repeat ourselves or talk over them. That reflex is why latency isn't a nice-to-have metric for voice; it's the thing that decides whether the conversation holds together at all.

The rough rule of thumb the industry works to is a ceiling around 800 ms from the moment the user stops speaking to the moment they hear the first word back. It's not a magic constant — it varies by context and no one number is universal — but below it the exchange feels conversational, and above it the agent starts to feel sluggish and people begin stepping on it. The crucial reframing is that this 800 ms is a budget, not a single measurement: it's a fixed pot of time you have to divide across every stage in the loop — capturing and uploading the audio, finalizing the transcript, the LLM producing its first token, TTS producing its first audio, and the trip back down plus playback. Every stage spends from the same pot.

Two consequences follow, and both are easy to get wrong. First, because the stages add up, one fat stage can blow the whole budget no matter how lean the others are — so the engineering move is always to find the single biggest contributor and attack that, rather than shaving milliseconds evenly. Second, and this is the one beginners miss: the number that matters is time-to-first-token, not total generation time. Because you stream and start speaking on the first word (§3), a model that takes 350 ms to begin and then rambles for two seconds is fine for voice, while one that thinks silently for a full second before emitting anything is not — even if its total time is shorter. The calculator below sums a per-stage budget and names the worst offender; play with the numbers to feel where a real system's time actually goes.

Humans notice conversational lag. In natural conversation, gaps between turns are roughly 200 ms; once the agent's response gap climbs past about 800 ms it starts to feel sluggish and people begin to talk over it. So end-to-end latency — from the user finishing their sentence to hearing the first word back — is a hard budget you must divide across every stage: capture + network up, STT finalization, LLM time-to-first-token, TTS time-to-first-audio, network down + playback.

The tool below adds up a per-stage budget and tells you whether you fit under a target, and how much headroom is left. Change the numbers to see where the time goes.

latency.pydef latency_budget(stages, target_ms=800):
    """stages: dict of stage -> milliseconds. Report total vs target."""
    total = sum(stages.values())
    headroom = target_ms - total
    worst = max(stages, key=stages.get)
    return {
        "total_ms": total,
        "target_ms": target_ms,
        "within_budget": total <= target_ms,
        "headroom_ms": headroom,
        "biggest_stage": worst,
    }

stages = {
    "capture+upload": 60,
    "stt_finalize":   150,
    "llm_first_token": 350,
    "tts_first_audio": 120,
    "network_down+play": 80,
}
result = latency_budget(stages, target_ms=800)
for k, v in result.items():
    print(f"{k}: {v}")

total_ms: 760
target_ms: 800
within_budget: True
headroom_ms: 40
biggest_stage: llm_first_token

7 · Production concerns: VAD, echo, fallback, cost professional

Here is the story that repeats on every voice project. The demo is magical. In a quiet room, with a good headset, talking to one engineer, the agent listens, answers, gets interrupted gracefully — everyone's impressed. Then it ships to a real phone line, and it falls apart: it triggers on the hum of an air conditioner, it interrupts itself on speaker echo, it goes silent for four seconds when a backend hiccups, and the finance team asks why the bill is five figures. None of these are model problems. They're the gap between "works at my desk" and "works for a thousand strangers on bad connections," and closing that gap is what this section is about.

The failures cluster into a short, memorable checklist, and it helps to see why each one only shows up in production. VAD and endpointing that were tuned in a silent room false-trigger the moment there's background noise, so the threshold has to be re-tuned for the real noise floor and often per language. Echo cancellation is optional in a demo you drive with a headset (no speaker bleed) but mandatory the instant a real speakerphone lets the mic hear the agent — without it, barge-in (§5) fires on the agent's own voice. A fallback path never matters until the day a stage times out, and then it's the difference between a graceful "Sorry, I didn't catch that" and dead air — the single worst thing a voice agent can do, because silence reads as "the line dropped." Interruptibility has to hold everywhere, including during a slow tool call or database lookup, or barge-in quietly dies exactly when a long action makes the user most likely to want to interrupt.

The last item, cost, deserves a mental reset because voice breaks the billing intuition you built on chat. A text agent costs per token, so a terse reply is cheap. Realtime voice is billed by the minute of audio and holds an open connection for the entire call — which means a long, rambling call is expensive even if the agent barely speaks, and your capacity ceiling is set by concurrent sessions, not requests per second. Five hundred people on the line at once is five hundred live model sessions and sockets held open simultaneously; plan for peak concurrency, not average volume. The estimator below turns call-volume assumptions into a monthly bill so you can feel how directly talk-time drives the number — and why "get the caller to their answer faster" is a cost lever, not just a UX one.

A demo that works at your desk breaks on a real phone line. The production checklist:

Production voice hardening

1. VAD / endpointing tuned per language and noise floor — noisy environments false-trigger on background sound.
2. Echo cancellation (AEC) so the mic doesn't hear the speaker; without it, barge-in fires on the agent's own voice.
3. Fallback path: if speech-to-speech or a stage times out, degrade gracefully (retry, switch to the text pipeline, or 'Sorry, I didn't catch that').
4. Interruptibility everywhere: any long agent action (a tool call, a lookup) must remain interruptible, or barge-in dies mid-task.
5. Cost & concurrency: realtime audio is billed by the second and holds an open connection per call — capacity planning is about concurrent sessions, not requests.

Cost for realtime voice is dominated by connection-time, not per-request tokens. Model the monthly bill from concurrent-call assumptions:

cost.pydef voice_cost(calls_per_day, avg_call_min, cost_per_min, days=30):
    """Realtime voice is billed by audio minute (in + out)."""
    minutes = calls_per_day * avg_call_min * days
    monthly = minutes * cost_per_min
    return {
        "billed_minutes_per_month": minutes,
        "monthly_cost_usd": round(monthly, 2),
        "cost_per_call_usd": round(avg_call_min * cost_per_min, 4),
    }

print(voice_cost(calls_per_day=2000, avg_call_min=3.0, cost_per_min=0.08))

{'billed_minutes_per_month': 180000.0, 'monthly_cost_usd': 14400.0, 'cost_per_call_usd': 0.24}

8 · Tech-lead — owning a voice-agent system tech-lead

An engineer can make the demo talk; a lead has to answer for it at 2 a.m. when the phone line goes dead. The shift in altitude is the whole point of this section. So far every piece — streaming, turns, barge-in, latency, cost — has been a component you can build and test in isolation. Owning the system means turning those components into commitments you're accountable for in production, where "it usually feels fast" isn't good enough and "it works on my machine" is a liability, not a status update.

Concretely, that means four things a lead signs their name to. A latency SLO — not a vibe but a measured target, e.g. p95 first-audio under 800 ms, tracked in production so you know when you're drifting before your users tell you. An architecture decision — pipeline versus speech-to-speech (§2) — justified out loud by that SLO and by your observability needs, not by which one demoed better. A transcription path for logging and guardrails even when you chose speech-to-speech, because "we can't audit what the agent said" is an answer that ends careers in a regulated industry. And a fallback and concurrency plan so a flaky network or a traffic spike degrades into a holding phrase and a queue, not a wall of dropped calls.

Notice what's not on that list: the specific SDK. The sketch below shows real vendor wiring, and it's worth reading precisely so you can see that it contains no new ideas — stream, turn_end, audio_out, cancel_response are the streaming, turn-detection, and barge-in concepts you already modeled offline, now delivered as events over a socket. That's the lead's most useful realization: the SDK is plumbing, and plumbing gets swapped. The reasoning you can defend — why this latency budget, why this architecture, why this fallback — is the durable part, and it's what the offline labs in this chapter were quietly building all along.

A lead owns the end-to-end contract: a latency SLO (e.g. p95 first-audio < 800 ms) measured in production, an architecture decision (pipeline vs speech-to-speech) justified by that SLO and by observability needs, a transcription path for logging/guardrails even in speech-to-speech, and a fallback + concurrency plan so a bad network or a traffic spike degrades gracefully instead of dropping calls.

The real SDK usage looks like the sketch below. It is illustrative only — it needs the vendor SDK, a network connection, and credentials, so it is NOT part of the runnable labs. Every concept in it (streaming, turn events, barge-in) you already modeled offline above.

sdk_sketch.py# ILLUSTRATIVE — requires a realtime voice SDK, credentials, and a live socket.
# Do NOT expect this to run with `python file.py`; the labs above model the ideas offline.
import realtime_voice_sdk as rv   # placeholder for the actual vendor SDK

async def run_call(mic_stream, speaker):
    session = await rv.connect(model="voice-realtime", voice="warm")
    async for event in session.stream(mic_stream):     # bidirectional websocket
        if event.type == "partial_transcript":
            pass                                        # show live captions
        elif event.type == "turn_end":                 # VAD says user finished
            await session.respond()
        elif event.type == "audio_out":
            speaker.play(event.audio)                   # stream out as it arrives
        elif event.type == "user_started_speaking":
            await session.cancel_response()             # BARGE-IN: stop + flush

stages = {"asr": 150, "llm_first_token": 400, "tts_first_audio": 120}  # ms
total = sum(stages.values())
BUDGET = 800
print("total:", total, "ms")                 # 670 ms
print("within budget" if total <= BUDGET else "TOO SLOW",
      f"({BUDGET - total} ms headroom)")      # within budget (130 ms headroom)

archs = {
    "pipeline (ASR+LLM+TTS)": {
        "latency": "higher (3 hops)", "transcripts": "native (text between hops)",
        "prosody": "lost/limited", "control": "high -- swap any stage"},
    "speech-to-speech": {
        "latency": "lower (1 model)", "transcripts": "must be derived",
        "prosody": "preserved", "control": "low -- opaque"},
}
for name, a in archs.items():
    print(name); [print(f"  {k}: {v}") for k, v in a.items()]

# regulated line needs auditable transcripts + stage control:
print("\nRecommendation: PIPELINE -- transcripts are first-class and each "
      "stage is swappable/loggable, which a compliance audit requires.")

def rms(frame):
    return (sum(x * x for x in frame) / len(frame)) ** 0.5

def detect_turns(frames, threshold=0.2, hangover=3):
    speaking = False; silence = 0; turns = []; start = None
    for i, f in enumerate(frames):
        loud = rms(f) > threshold
        if loud:
            silence = 0
            if not speaking: speaking = True; start = i
        elif speaking:
            silence += 1
            if silence >= hangover:            # end turn after enough silence
                turns.append((start, i - hangover)); speaking = False
    if speaking: turns.append((start, len(frames) - 1))
    return turns

loud, quiet = [0.5, 0.5], [0.0, 0.0]
frames = [quiet, loud, loud, quiet, quiet, quiet, loud]
print(detect_turns(frames))    # [(1, 2), (6, 6)]

class VoiceAgent:
    def __init__(self): self.state = "idle"
    def on_event(self, event):
        if event == "user_speaks":
            if self.state == "speaking":
                self.state = "listening"; return "BARGE-IN: stop TTS, listen"
            self.state = "listening"; return "listening"
        if event == "user_stops":
            self.state = "thinking"; return "run LLM"
        if event == "llm_done":
            self.state = "speaking"; return "start TTS"
        if event == "tts_done":
            self.state = "idle"; return "idle"
        return self.state

a = VoiceAgent()
for e in ["user_speaks", "user_stops", "llm_done", "user_speaks", "tts_done"]:
    print(e, "->", a.on_event(e))
# the 2nd user_speaks arrives while speaking -> BARGE-IN

class VoiceService:
    def __init__(self): self.fallbacks = 0
    def respond(self, transcript, llm):
        try:
            reply = llm(transcript, timeout=1.5)   # must beat the latency budget
            if not reply:
                raise ValueError("empty reply")
            return reply
        except Exception as e:
            self.fallbacks += 1
            return "Sorry, could you repeat that?"  # holding phrase, never silence

def flaky_llm(text, timeout):
    if "trigger" in text:
        raise TimeoutError("model slow")
    return f"You said: {text}"

svc = VoiceService()
print(svc.respond("hello", flaky_llm))    # You said: hello
print(svc.respond("trigger", flaky_llm))  # holding phrase
print("fallbacks:", svc.fallbacks)        # 1

from anthropic import Anthropic

client = Anthropic()

def stream_reply(user_text, on_token):
    with client.messages.stream(
        model="claude-sonnet-4-5",
        max_tokens=200,
        messages=[{"role": "user", "content": user_text}],
    ) as stream:
        for text in stream.text_stream:      # tokens as they arrive
            on_token(text)                    # feed straight into TTS
        return stream.get_final_message()

# on_token would push each chunk to the TTS engine so audio starts
# after the FIRST sentence, not the whole reply.

Knowledge check check yourself