# AI backends

Andon calls a model for attention triage, the assistant, recipe detection, Autopilot's placement, Start work's
guesses, the finish classifier (Auto-Done) and voice. Every one of those calls goes through one client, built on the
Vercel AI SDK, which can send it four ways. Which one is a setting, not a deploy.

## The four backends

| Backend | Where the call runs | The key it needs |
|---|---|---|
| `anthropic` | Claude, through the SDK's Anthropic provider. The default whenever there is an Anthropic key. Every feature but voice. | The org's own Anthropic key (Settings → Connections → Advanced → **Claude (Anthropic) key**), else the server's `ANTHROPIC_API_KEY`. |
| `gateway` | The Vercel AI Gateway. Models are named `creator/model`, such as `google/gemini-3.6-flash`. | The org's own AI Gateway key (Settings → Connections → Advanced), else the server's `AI_GATEWAY_API_KEY`. |
| `direct` | Gemini itself, through the SDK's Google provider. | The org's own Gemini key, else the server's `GEMINI_API_KEY` (and `GEMINI_REFERER` for a referrer-restricted key). |
| `machine` | One of the org's machines, which runs the same call with its own key. | On the machine: `andon policy llm-serve on`, and its own key (see [Serving model calls](https://andon-docs.pages.dev/machines/accounts-and-providers.md#serving-model-calls-machine-mode)). The app needs no key. |

The prompts, the answers' shape and the retries are the same on every backend. Features that ask for a JSON answer
get one in the same shape, and the assistant's tool calls work the same way.

## Which backend runs a call

The first of these that is set wins:

1. A backend the call itself asks for (a few internal calls do).
2. The org's choice: **Settings → Connections → Advanced → AI backend**. Admins change it; "Deployment default" leaves
   it to the server.
3. The server's `LLM_PROVIDER` (`anthropic`, `gateway`, `direct` or `machine`).
4. The keys the server and the org hold: `anthropic` with an Anthropic key, else `direct` with a Gemini key, else
   `gateway` with a gateway key, else `machine`.

When the chosen backend can't serve the call, the next one in line does:

- the chosen backend first;
- then the other app-side backends that hold a key, in this order: Claude, Gemini directly, the gateway;
- a machine last.

A `machine` choice tries a machine first. When no machine is online and serving, or the machine doesn't answer within
`LLM_MACHINE_TIMEOUT_MS` (45 seconds by default), the app's own keys answer instead. The server logs which backend it
picked and why, for example `llm recipe: machine wanted (org setting); no machine online → direct`.

A machine's wait comes on top of each feature's own time limit, so the app's key still has that limit to answer when
the machine doesn't. Start work's guesses while you type are the exception: they keep their 10-second limit, so a slow
machine there means no guess rather than one from the app's key.

An error from the provider itself (a quota, a refused key) is not a reason to switch: it is reported as before.

Attention triage and recipe detection start from the database, which can't read `LLM_PROVIDER`. Triage starts when
the deployment has a key (`pnpm bootstrap` turns it on), when the org holds its own Anthropic, Gemini or AI Gateway key, or when
the org chose **On a machine** and one of its machines serves model calls. A deployment with no key and only
`LLM_PROVIDER=machine` gets no triage; choose the machine backend in the org's settings instead. Recipe detection
starts with a key, the deployment's or the org's; without one a machine's own Claude picks the recipe, as before.

## Voice

Voice is a live audio session between the browser and the model, so it always needs a Gemini or AI Gateway key the app
holds. It never runs on Claude or on a machine.

- With `direct`, voice is the Gemini Live API, with `GEMINI_VOICE_MODEL` (default `gemini-3.8-live`).
- With `gateway`, voice goes through the gateway's realtime modality, with the org's voice model (next to the AI
  backend setting), else `LLM_VOICE_MODEL`, else `google/gemini-3.8-live`. The gateway names the model differently
  from Gemini, which is why it has its own setting. `AI_GATEWAY_VOICE_MODEL` is the older name and still works.
- With `anthropic` or `machine`, voice quietly uses the voice backend that holds a key: Gemini directly first, then the
  gateway. With an Anthropic key and neither of those, there is no voice.

`VOICE_PROVIDER` (`gemini` or `vercel`) still overrides all of this for voice alone.
[Voice providers](https://andon-docs.pages.dev/contributing/voice-providers.md) has the details.

## Models

Each feature's model is set in one place, with a Claude name for `anthropic`, a Gemini name for `direct` and a gateway
name for `gateway`:

| Feature | Claude (`anthropic`) | Gemini (`direct`) | Gateway |
|---|---|---|---|
| Attention triage, drive answers | `ANTHROPIC_TRIAGE_MODEL` | `GEMINI_TRIAGE_MODEL` | `AI_GATEWAY_TRIAGE_MODEL` |
| The assistant | `ANTHROPIC_ASSISTANT_MODEL` (else the triage model) | `GEMINI_ASSISTANT_MODEL` (else the triage model) | `AI_GATEWAY_ASSISTANT_MODEL` |
| Auto-Done's finish classifier | `ANTHROPIC_FINISH_MODEL` | `GEMINI_FINISH_MODEL` | `AI_GATEWAY_FINISH_MODEL` |
| Recipe detection | `ANTHROPIC_RECIPE_MODEL` | `GEMINI_RECIPE_MODEL` | `AI_GATEWAY_RECIPE_MODEL` |
| Autopilot's placement, Start work's breakdown | `ANTHROPIC_PLAN_MODEL` | `GEMINI_PLAN_MODEL` | `AI_GATEWAY_PLAN_MODEL` |
| Start work's team guess and title | `ANTHROPIC_ROUTE_MODEL` | `GEMINI_ROUTE_MODEL` | `AI_GATEWAY_ROUTE_MODEL` |

Every Claude model defaults to `ANTHROPIC_MODEL`, else `claude-opus-5-5`. Start work's guesses run while you type and
must answer within 10 seconds, so a faster model such as `claude-haiku-5-5` for `ANTHROPIC_ROUTE_MODEL` is worth a
try. Every Gemini model defaults to `gemini-3.6-flash`. Every gateway model defaults to `google/` and the feature's
Gemini model. A machine runs the model the app names, under the name for its own backend.

On Claude, settings that are Gemini's own (its thinking budget, sampling temperature) are left out: current Claude
models take none. A feature that asks for a JSON answer gets it through Claude's structured outputs.

## What travels to a machine

The app hands the call to a machine over the machine's command channel, as an `llm_generate` command: the prompt, the
model's names and the options. The machine runs it on its own key and sends back the answer. Keys never travel either
way: the app's keys stay on the server, and the machine's stay on the machine.

A machine only takes these commands from Andon itself, never from a person's command, and not when its policy requires
signed commands.

## What isn't backend-agnostic

- Voice can't run on a machine (above).
- A machine serves these calls with an Anthropic, Gemini or AI Gateway key, not with its Claude login. The older paths that ask a
  machine's own Claude (recipe detection when nobody holds a key, answers while Andon drives) are unchanged.
