Voice providers
Voice in the chat (the mic beside Send, calls, car mode) is a speech-to-speech session: the browser streams the mic to a
realtime model, the model speaks back, and everything it is asked goes to the chat’s orchestrator through one function,
ask_orchestrator. Which model, and through whom, is a voice provider. Two are built in:
VOICE_PROVIDER |
What | Credential | Wire the board speaks |
|---|---|---|---|
gemini |
The Gemini Live API directly (gemini-3.8-live). |
The organization’s own Gemini key (Settings → Connections → Advanced → Gemini key), else GEMINI_VOICE_API_KEY, else GEMINI_API_KEY. |
gemini-live: Gemini’s own BidiGenerateContent frames. |
vercel |
The Vercel AI Gateway’s realtime modality (google/gemini-3.8-live by default; any realtime model the gateway lists, LLM_VOICE_MODEL). |
The organization’s own gateway key (Settings → Connections → Advanced → AI Gateway key), else the server’s AI_GATEWAY_API_KEY (a team key from the Vercel dashboard). |
ai-sdk-realtime: the AI SDK’s normalized realtime protocol, which the gateway translates to the model’s. |
With VOICE_PROVIDER unset, voice follows the AI backend (AI backends: the org’s choice, else
LLM_PROVIDER): gateway → vercel, direct → gemini. Voice never runs on Claude or a machine, so anthropic,
machine (or no choice) takes the first provider with a key (Gemini, then the gateway); with neither key the server still runs voice on Gemini
for the organizations that bring their own Gemini key. The gateway’s voice model is the org’s (next to the AI backend
setting), else LLM_VOICE_MODEL, else AI_GATEWAY_VOICE_MODEL, else google/gemini-3.8-live. VOICE_TRANSPORT is token
(the browser connects to the provider with a one-use token the server minted) or relay (the browser connects to the
server, which holds the key and the provider’s socket; the Node server only). .env.example documents every name.
How it is put together
browser server providerVoiceControls ── voice-session start ─▶ voice-session / http-fns ─ provider.mint ─▶ a one-use token │ ◀── VoiceStart {provider, wire, url, protocols?, setup, …} ──┘ │VoiceSession ──── wire.open(setup), wire.audio(), wire.toolResponse() … ────────▶ the provider's socket (one conversation) ◀── wire.parse(frame) → ready | heard | said | audio | call | turnDone | … ──┘- Server:
apps/server/src/functions/shared/voice-providers.tsis the pluggable face. AVoiceProvidernames itswireand default model and gives:model/base/serverKeyfrom the settings,orgKey/orgHasKey(bring your own key),setup(what the browser sends first, in the wire’s shape),mint(the one call that carries the key),tokenSocketandkeySocket(where the browser, or the relay, connects, with any subprotocols), andrelayClientFrame(the relay’s view of a browser frame: the setup replaced, the rest passed or dropped).pickVoiceProvider(settings)chooses;providerKeyresolves the credential.voice-session(apps/server/src/functions/http-fns.ts) and the relay (voice-relay.ts) only use this interface.voice-core.tsis the Gemini provider’s core and what every provider shares (the prompt, the three functions, the languages, relay tickets, theVoiceStartpayload);voice-vercel.tsis the gateway’s part. - Board:
apps/board/src/lib/voice-wire.tsis the other half. AVoiceWireframes what the session wants to send (open,audio,activityStart/activityEnd,text,toolResponse) and reads the provider’s frames into the few events the session acts on.VoiceSession(voice-session.ts) is the conversation itself (turns, barge-in, the never-silent request flow, reconnects, car mode) on those events, whichever wire. The server’sVoiceStartnames the wire; a board that does not know it fails the call in words before opening a socket. - What a wire may lack. Gemini Live answers a function call several times (
willContinue) with a say-when (scheduling); the normalized protocol answers a call once. A wire says which withasyncTools; without it the session’s first response closes the call (the acknowledgement, or the answer itself when the chat was quick), the answer after an acknowledgement goes as a text turn the prompt already explains ([Andon's chat answered the earlier request] …), and progress lines are not said. - Said once. Whatever the wire, everything that makes the model speak (a result, a progress line, a late answer, a
car-mode line) waits until the model is free: not while its turn streams (until
turnDone, or it fell quiet forMODEL_QUIET_MS), not from the moment it was prompted (a result sent, a text turn, the person letting go, a response the provider started itself:response-created, hands-free) until that turn has ended (its first word can take a second or two with nothing on the wire; bounded byASKED_WAIT_MS, longer on a one-response-at-a-time wire, where a secondresponse-createwould be refused), and not while the person is talking or has just stopped in a call (its answer to them is on its way). Held things go out one per turn. Prompted mid-turn, Gemini says the thing in the turn under way and again when idle, worded differently (a fresh generation, so often with a different delivery); the prompt also tells it to say each thing once.
Adding a provider
- Same wire as one that exists (another host of Gemini Live, or another realtime model behind the gateway): add a
VoiceProvidertoVOICE_PROVIDERSinvoice-providers.tswith its settings (add the env names tovoiceSettingsFromEnvand.env.example), its mint call, socket and setup; tests beside the others inapps/server/test/functions/shared/voice-providers.test.ts(the key only ever in a header or a subprotocol, never in what the browser gets), and the relay test inapps/server/test/functions/voice-relay.test.ts. Nothing on the board changes. - A new protocol: also add a
VoiceWiretoWIRESinvoice-wire.tswith the frames in, the events out, and whether it hasasyncTools; a derived fixture underfixtures/<provider>/and tests against a fake socket inapps/board/src/lib/voice.test.ts, as the two wires have. The session should need no change: if it does, the wire is missing an event, not the session a special case. - An organization’s own key needs
integrations.providerto allow the new name (a migration, as20261026175023_gemini_byok.sqldid for Gemini and20261101210632_vercel_gateway_byok.sqlfor the gateway), a key check inintegration-secrets(apps/server/src/functions/http-fns.ts), the provider’sorgKey/orgHasKey(orgIntegrationKey/orgHasIntegrationKeywith the provider’s name), and an entry in the board’sPROVIDERS(apps/board/src/lib/integrations.ts,key: true) so Settings → Connections shows it.
Everything about the gateway was built from its published SDK and docs, not a live session: docs/internal/spikes.md (“Voice
providers”) lists what is assumed and docs/internal/manual-checks.md the check that confirms it.
