# Voice providers

Voice in the chat (the mic beside Send, calls, car mode) is a speech-to-speech session: the browser streams the mic to a
realtime model, the model speaks back, and everything it is asked goes to the chat's orchestrator through one function,
`ask_orchestrator`. Which model, and through whom, is a **voice provider**. Two are built in:

| `VOICE_PROVIDER` | What | Credential | Wire the board speaks |
|---|---|---|---|
| `gemini` | The Gemini Live API directly (`gemini-3.8-live`). | The organization's own Gemini key (Settings → Connections → Advanced → Gemini key), else `GEMINI_VOICE_API_KEY`, else `GEMINI_API_KEY`. | `gemini-live`: Gemini's own BidiGenerateContent frames. |
| `vercel` | The Vercel AI Gateway's realtime modality (`google/gemini-3.8-live` by default; any realtime model the gateway lists, `LLM_VOICE_MODEL`). | The organization's own gateway key (Settings → Connections → Advanced → AI Gateway key), else the server's `AI_GATEWAY_API_KEY` (a team key from the Vercel dashboard). | `ai-sdk-realtime`: the AI SDK's normalized realtime protocol, which the gateway translates to the model's. |

With `VOICE_PROVIDER` unset, voice follows the AI backend ([AI backends](https://andon-docs.pages.dev/machines/ai-backends.md): the org's choice, else
`LLM_PROVIDER`): `gateway` → `vercel`, `direct` → `gemini`. Voice never runs on Claude or a machine, so `anthropic`,
`machine` (or no choice) takes the first provider with a key (Gemini, then the gateway); with neither key the server still runs voice on Gemini
for the organizations that bring their own Gemini key. The gateway's voice model is the org's (next to the AI backend
setting), else `LLM_VOICE_MODEL`, else `AI_GATEWAY_VOICE_MODEL`, else `google/gemini-3.8-live`. `VOICE_TRANSPORT` is `token`
(the browser connects to the provider with a one-use token the server minted) or `relay` (the browser connects to the
server, which holds the key and the provider's socket; the Node server only). `.env.example` documents every name.

## How it is put together

```
browser                              server                                   provider
VoiceControls ── voice-session start ─▶ voice-session / http-fns ─ provider.mint ─▶ a one-use token
  │  ◀── VoiceStart {provider, wire, url, protocols?, setup, …} ──┘
  │
VoiceSession ──── wire.open(setup), wire.audio(), wire.toolResponse() … ────────▶ the provider's socket
  (one conversation)   ◀── wire.parse(frame) → ready | heard | said | audio | call | turnDone | … ──┘
```

- **Server: `apps/server/src/functions/shared/voice-providers.ts`** is the pluggable face. A `VoiceProvider` names its `wire`
  and default model and gives: `model`/`base`/`serverKey` from the settings, `orgKey`/`orgHasKey` (bring your own key),
  `setup` (what the browser sends first, in the wire's shape), `mint` (the one call that carries the key), `tokenSocket`
  and `keySocket` (where the browser, or the relay, connects, with any subprotocols), and `relayClientFrame` (the relay's
  view of a browser frame: the setup replaced, the rest passed or dropped). `pickVoiceProvider(settings)` chooses;
  `providerKey` resolves the credential. `voice-session` (`apps/server/src/functions/http-fns.ts`) and the relay
  (`voice-relay.ts`) only use this interface. `voice-core.ts` is the Gemini
  provider's core and what every provider shares (the prompt, the three functions, the languages, relay tickets, the
  `VoiceStart` payload); `voice-vercel.ts` is the gateway's part.
- **Board: `apps/board/src/lib/voice-wire.ts`** is the other half. A `VoiceWire` frames what the session wants to send
  (`open`, `audio`, `activityStart`/`activityEnd`, `text`, `toolResponse`) and reads the provider's frames into the few
  events the session acts on. `VoiceSession` (`voice-session.ts`) is the conversation itself (turns, barge-in, the
  never-silent request flow, reconnects, car mode) on those events, whichever wire. The server's `VoiceStart` names the
  wire; a board that does not know it fails the call in words before opening a socket.
- **What a wire may lack.** Gemini Live answers a function call several times (`willContinue`) with a say-when
  (`scheduling`); the normalized protocol answers a call once. A wire says which with `asyncTools`; without it the
  session's first response closes the call (the acknowledgement, or the answer itself when the chat was quick), the
  answer after an acknowledgement goes as a text turn the prompt already explains (`[Andon's chat answered the earlier
  request] …`), and progress lines are not said.
- **Said once.** Whatever the wire, everything that makes the model speak (a result, a progress line, a late answer, a
  car-mode line) waits until the model is free: not while its turn streams (until `turnDone`, or it fell quiet for
  `MODEL_QUIET_MS`), not from the moment it was prompted (a result sent, a text turn, the person letting go, a
  response the provider started itself: `response-created`, hands-free) until that turn has ended (its first word can
  take a second or two with nothing on the wire; bounded by `ASKED_WAIT_MS`, longer on a one-response-at-a-time wire,
  where a second `response-create` would be refused), and not while the person is talking or has just stopped in a
  call (its answer to them is on its way). Held things go out one per turn. Prompted mid-turn, Gemini says the thing
  in the turn under way and again when idle, worded differently (a fresh generation, so often with a different
  delivery); the prompt also tells it to say each thing once.

## Adding a provider

1. **Same wire as one that exists** (another host of Gemini Live, or another realtime model behind the gateway): add a
   `VoiceProvider` to `VOICE_PROVIDERS` in `voice-providers.ts` with its settings (add the env names to
   `voiceSettingsFromEnv` and `.env.example`), its mint call, socket and setup; tests beside the others in
   `apps/server/test/functions/shared/voice-providers.test.ts` (the key only ever in a header or a subprotocol, never
   in what the browser gets), and the relay test in `apps/server/test/functions/voice-relay.test.ts`. Nothing on the
   board changes.
2. **A new protocol:** also add a `VoiceWire` to `WIRES` in `voice-wire.ts` with the frames in, the events out, and
   whether it has `asyncTools`; a derived fixture under `fixtures/<provider>/` and tests against a fake socket in
   `apps/board/src/lib/voice.test.ts`, as the two wires have. The session should need no change: if it does, the wire
   is missing an event, not the session a special case.
3. **An organization's own key** needs `integrations.provider` to allow the new name (a migration, as
   `20261026175023_gemini_byok.sql` did for Gemini and `20261101210632_vercel_gateway_byok.sql` for the gateway), a
   key check in `integration-secrets` (`apps/server/src/functions/http-fns.ts`), the provider's `orgKey` / `orgHasKey`
   (`orgIntegrationKey` / `orgHasIntegrationKey` with the provider's name), and an entry in the board's `PROVIDERS`
   (`apps/board/src/lib/integrations.ts`, `key: true`) so Settings → Connections shows it.

Everything about the gateway was built from its published SDK and docs, not a live session: `docs/internal/spikes.md` ("Voice
providers") lists what is assumed and `docs/internal/manual-checks.md` the check that confirms it.
