7 min readBrotherhood Intelligence

Inside Tanya: How We Engineered a Sub-₹2/Min Hinglish Voice Gatekeeper on Sarvam AI & Plivo

A build log from the wire up: bidirectional Plivo streams, 640-byte PCM frames, burst delivery that silently drops audio, Sarvam speech synthesis rates, turn-0 seeding, and the guardrails that keep a Hinglish voice agent cheap and honest.

Most "AI voice agent" write-ups stop at the prompt. The prompt is the easy part. What decides whether a phone call feels natural is the transport: frame sizes, pacing, sample rates, and what happens in the 300 milliseconds after the caller stops talking.

This is the build log for Tanya, our Hinglish call gatekeeper — the measured details, including the bugs that cost us days, in the hope that they save someone else the same.

The shape of the system

A call arrives on a dedicated +91 number. The carrier answers with a tiny XML document that does two things: plays a pre-cached greeting WAV, and opens a bidirectional WebSocket stream carrying raw PCM audio both ways.

<Response>
  <Play>https://…/audio/greeting/<founder>.wav?v=<hash></Play>
  <Stream bidirectional="true"
          keepCallAlive="true"
          audioTrack="inbound"
          contentType="audio/x-l16;rate=16000"
          noiseCancellation="true">wss://…/ws/plivo-stream</Stream>
</Response>

Two details in that document are non-obvious and both were learned the hard way.

<Stream> alone is not enough. An answer document containing only a stream is terminated by the carrier after about 5 seconds, with hangup_source set to the answer XML. The <Play> element is what keeps the leg open while the stream attaches. If you want the stream alone, you need to keep the call alive explicitly — and you must not "help" by adding a <Pause> element, because that is a Twilio attribute and the carrier rejects the whole document with an 8011 Invalid Answer XML.

Escape your query strings. A bare & inside the stream URL silently invalidates the XML. It must be &amp;.

Frame math: 20 ms is 640 bytes, not 1280

Audio over the stream is 16-bit signed little-endian PCM at 16 kHz. One 20-millisecond frame is therefore:

16,000 samples/sec × 0.020 sec = 320 samples
320 samples × 2 bytes = 640 bytes

Half the buffer sizes you will find in blog posts assume 16-bit stereo or 8 kHz. Ours are computed from the sample rate the carrier declares in the start event, so an 8 kHz leg produces 320-byte frames and a 16 kHz leg produces 640-byte frames without a config change.

The bug that ate every outbound frame

Here is the failure that took the longest to find, because everything looked correct.

The start event from the carrier looks roughly like this:

{
  "event": "start",
  "start": {
    "streamId": "…",
    "callId": "…",
    "mediaFormat": { "encoding": "audio/x-l16", "sampleRate": 16000 }
  }
}

Note the nesting. Our parser read event.streamId at the top level, got undefined, and then — being a well-behaved little function — dropped every outbound frame because it had nowhere to send it. The microphone path worked, transcription worked, the model produced replies, the synthesizer produced audio, and the caller heard silence. Every layer reported success.

The fix is one line — const streamId = event.start?.streamId ?? event.streamId — but the lesson is bigger: a mock that accepts both wire shapes will hide a real bug forever. Our end-to-end test now asserts the nested-only shape, which is what the carrier actually sends.

Bursts are dropped, not buffered

With the stream ID fixed, audio still did not arrive. Frames were valid, base64 was correct, the socket was open.

The cause: the media gateway expects paced delivery. Sending 200 frames at once does not queue them — it drops them. The fix is a pacer: a 20-millisecond interval that sends exactly one frame per tick, with a hard cap on catch-up after a stall so a GC pause does not turn into a burst.

sentFrames++ every tick
due = floor((now - sentStart) / 20ms) + 1 - sentFrames
send min(due, 10)   // cap catch-up at 10 frames

Ten frames is 200 milliseconds of audio — enough to recover from a hiccup, not enough to trigger the drop behaviour. When the caller interrupts, the queue is cleared and a clearAudio event is sent so the carrier flushes whatever it has buffered; without it, the AI keeps talking from the carrier's buffer even after you stopped generating.

Speech synthesis: read the WAV header, do not assume

Our speech provider's default output rate is not 16 kHz. The first version assumed it was, and the result was chipmunk audio — recognizable words at the wrong pitch.

The synthesis request now sets the sample rate explicitly per leg (16 kHz for the phone, 24 kHz for the browser playground), and the response is parsed rather than trusted: read the actual rate from the WAV header at bytes 24–27, strip the 44-byte RIFF header, and resample linearly if the provider ignored the request. Some voices hard-reject certain rates outright — 22,050 Hz came back as an error, not as audio.

Normalized samples are clamped to ±0.95 before conversion to 16-bit. At full scale, loud syllables clip and crackle on a phone speaker; the 5% headroom is inaudible except when it prevents distortion.

One more wire detail: the STT socket takes its language as an underscore query parameter (language_code) and its key in the Authorization: Bearer header. The api-key header — which appears in a lot of sample code — returns 403 on the realtime endpoint.

The conversation layer

With audio flowing, the remaining problems were conversational, not technical.

Turn 0. The greeting is played as XML, so the language model has no idea it was spoken. On the first turn it would introduce itself again: "Namaste, main Tanya hoon." The fix is to seed the history with two synthetic entries before the first real turn — a user turn describing the call start, and an assistant turn containing the exact greeting text — plus a post-processor that strips a leading greeting or self-introduction from the model's output. Both are needed: the seed fixes the common case, the stripper catches the model ignoring it.

Barge-in. On a phone line, energy-based voice detection fires on the caller's breathing, the car horn, and the echo of the AI's own voice. We gate barge-in on transcript, not energy: the AI stops only when the speech recognizer produces real words. It costs a few hundred milliseconds and eliminates nearly all false interruptions — a trade every PSTN deployment should make.

Pacing. A synthesized pause is not a pause; it is silence the caller can talk over. Sentence-level streaming keeps at most one sentence in flight, so an interruption discards less than a second of audio.

The cost model

Four line items decide whether a voice agent can be sold at ₹499 a month:

  1. Number rental — fixed, and included in the plan.
  2. Carrier minutes — inbound on an Indian number, billed per minute.
  3. Speech in and out — per recognized second, per synthesized character.
  4. LLM tokens — small, if every turn is one or two sentences.

Two decisions keep the total under ₹2 per minute of screened conversation:

  • Meter on talk seconds, bill on pulses. Credits are consumed per active second of conversation, and the dashboard reports duration in 60-second pulses — the same unit the carrier bills in. A 27-second deflection is one pulse to the founder and 27 seconds of cost to us.
  • Cache everything that repeats. The greeting and the deflection WAV are synthesized once per founder and served as static files. That removes one full synthesis round trip from the start of every call and makes the most common call path — a sales pitch, deflected in under 30 seconds — the cheapest one.

At a 75–85% gross margin, a founder paying ₹499 can burn roughly 20 minutes of screened conversation a month before the plan is upside down; the ₹899 bundle's 60 included credits cover about 90–110 short calls, which is a busy founder's month.

The guardrails are product, not policy

The last layer is the one customers never see and always feel:

  • A stack-name firewall in the system prompt, reinforced by an output filter: the agent never names its speech, model, telephony, database or hosting vendors — not even when asked directly. A caller probing the stack gets a fixed referral line and the call ends.
  • A pitch guillotine: classification runs before the model does, so a recognized sales pitch gets a scripted deflection without an LLM call at all — faster and cheaper than generating one.
  • Capture, never connect: the agent records name, number and requirement, and never promises to patch the caller through. Promise-keeping is the founder's job; the agent's job is to make the callback possible.
  • Recording off by default. Transcripts and summaries only, with the audio-recording toggle opt-in — which is also the simpler DPDP posture.

None of this is glamorous. But a voice agent is a chain of a dozen hops, and the caller experiences the weakest one. The 640-byte frame and the nested streamId do not appear in any demo — they appear in whether the phone rings, answers, and sounds like a person who works for you.

Want this on your own line? See the product this build log became · Get your own +91 number

Frequently asked questions

Why use Plivo bidirectional streams instead of Twilio for an Indian voice agent?

The important part is not the vendor, it is the shape of the transport: a WebSocket that carries raw 16-bit PCM in both directions with a keep-alive flag, so the same socket that receives caller audio can push synthesized speech back in 20-millisecond frames. Plivo's bidirectional stream exposes exactly that over Indian numbers, with a nested stream identifier that must be read from the start event.

How much does a Hinglish AI voice agent cost to run per minute?

The bill has four parts: number rental, inbound carrier minutes, speech-to-text and text-to-speech characters, and LLM tokens. With 1–2 sentence turns, low-latency models and pre-cached greetings, the blended cost per minute of screened conversation lands under ₹2 — which is what makes a ₹499/month plan with metered credits viable at a 75–85% margin.

What is the hardest bug in building a real-time voice agent?

Not the model — the audio pacing. Plivo's media gateway silently drops audio sent in bursts, so you must pace outbound frames at exactly one per 20 milliseconds and cap catch-up after stalls. Before we found that, every frame was valid and every frame was dropped.

Why cache the greeting as a WAV file?

The opening greeting is identical on every call and must play in under a second. Synthesizing it once per founder, caching it with a hash of the text and voice settings, and serving it via a plain Play element removes a full text-to-speech round trip from the start of every conversation.