Skip to content
AndonDocs
GitHub
Run machines and deploy / Deploy an instance and the desktop app

AI backends

View as MarkdownThe same text, at this page's address plus .mdOpen in ClaudeAsk questions about this pageOpen in ChatGPTAsk questions about this page

Andon calls a model for attention triage, the assistant, recipe detection, Autopilot’s placement, Start work’s guesses, the finish classifier (Auto-Done) and voice. Every one of those calls goes through one client, built on the Vercel AI SDK, which can send it four ways. Which one is a setting, not a deploy.

The four backends

Backend Where the call runs The key it needs
anthropic Claude, through the SDK’s Anthropic provider. The default whenever there is an Anthropic key. Every feature but voice. The org’s own Anthropic key (Settings → Connections → Advanced → Claude (Anthropic) key), else the server’s ANTHROPIC_API_KEY.
gateway The Vercel AI Gateway. Models are named creator/model, such as google/gemini-3.6-flash. The org’s own AI Gateway key (Settings → Connections → Advanced), else the server’s AI_GATEWAY_API_KEY.
direct Gemini itself, through the SDK’s Google provider. The org’s own Gemini key, else the server’s GEMINI_API_KEY (and GEMINI_REFERER for a referrer-restricted key).
machine One of the org’s machines, which runs the same call with its own key. On the machine: andon policy llm-serve on, and its own key (see Serving model calls). The app needs no key.

The prompts, the answers’ shape and the retries are the same on every backend. Features that ask for a JSON answer get one in the same shape, and the assistant’s tool calls work the same way.

Which backend runs a call

The first of these that is set wins:

  1. A backend the call itself asks for (a few internal calls do).
  2. The org’s choice: Settings → Connections → Advanced → AI backend. Admins change it; “Deployment default” leaves it to the server.
  3. The server’s LLM_PROVIDER (anthropic, gateway, direct or machine).
  4. The keys the server and the org hold: anthropic with an Anthropic key, else direct with a Gemini key, else gateway with a gateway key, else machine.

When the chosen backend can’t serve the call, the next one in line does:

  • the chosen backend first;
  • then the other app-side backends that hold a key, in this order: Claude, Gemini directly, the gateway;
  • a machine last.

A machine choice tries a machine first. When no machine is online and serving, or the machine doesn’t answer within LLM_MACHINE_TIMEOUT_MS (45 seconds by default), the app’s own keys answer instead. The server logs which backend it picked and why, for example llm recipe: machine wanted (org setting); no machine online → direct.

A machine’s wait comes on top of each feature’s own time limit, so the app’s key still has that limit to answer when the machine doesn’t. Start work’s guesses while you type are the exception: they keep their 10-second limit, so a slow machine there means no guess rather than one from the app’s key.

An error from the provider itself (a quota, a refused key) is not a reason to switch: it is reported as before.

Attention triage and recipe detection start from the database, which can’t read LLM_PROVIDER. Triage starts when the deployment has a key (pnpm bootstrap turns it on), when the org holds its own Anthropic, Gemini or AI Gateway key, or when the org chose On a machine and one of its machines serves model calls. A deployment with no key and only LLM_PROVIDER=machine gets no triage; choose the machine backend in the org’s settings instead. Recipe detection starts with a key, the deployment’s or the org’s; without one a machine’s own Claude picks the recipe, as before.

Voice

Voice is a live audio session between the browser and the model, so it always needs a Gemini or AI Gateway key the app holds. It never runs on Claude or on a machine.

  • With direct, voice is the Gemini Live API, with GEMINI_VOICE_MODEL (default gemini-3.8-live).
  • With gateway, voice goes through the gateway’s realtime modality, with the org’s voice model (next to the AI backend setting), else LLM_VOICE_MODEL, else google/gemini-3.8-live. The gateway names the model differently from Gemini, which is why it has its own setting. AI_GATEWAY_VOICE_MODEL is the older name and still works.
  • With anthropic or machine, voice quietly uses the voice backend that holds a key: Gemini directly first, then the gateway. With an Anthropic key and neither of those, there is no voice.

VOICE_PROVIDER (gemini or vercel) still overrides all of this for voice alone. Voice providers has the details.

Models

Each feature’s model is set in one place, with a Claude name for anthropic, a Gemini name for direct and a gateway name for gateway:

Feature Claude (anthropic) Gemini (direct) Gateway
Attention triage, drive answers ANTHROPIC_TRIAGE_MODEL GEMINI_TRIAGE_MODEL AI_GATEWAY_TRIAGE_MODEL
The assistant ANTHROPIC_ASSISTANT_MODEL (else the triage model) GEMINI_ASSISTANT_MODEL (else the triage model) AI_GATEWAY_ASSISTANT_MODEL
Auto-Done’s finish classifier ANTHROPIC_FINISH_MODEL GEMINI_FINISH_MODEL AI_GATEWAY_FINISH_MODEL
Recipe detection ANTHROPIC_RECIPE_MODEL GEMINI_RECIPE_MODEL AI_GATEWAY_RECIPE_MODEL
Autopilot’s placement, Start work’s breakdown ANTHROPIC_PLAN_MODEL GEMINI_PLAN_MODEL AI_GATEWAY_PLAN_MODEL
Start work’s team guess and title ANTHROPIC_ROUTE_MODEL GEMINI_ROUTE_MODEL AI_GATEWAY_ROUTE_MODEL

Every Claude model defaults to ANTHROPIC_MODEL, else claude-opus-5-5. Start work’s guesses run while you type and must answer within 10 seconds, so a faster model such as claude-haiku-5-5 for ANTHROPIC_ROUTE_MODEL is worth a try. Every Gemini model defaults to gemini-3.6-flash. Every gateway model defaults to google/ and the feature’s Gemini model. A machine runs the model the app names, under the name for its own backend.

On Claude, settings that are Gemini’s own (its thinking budget, sampling temperature) are left out: current Claude models take none. A feature that asks for a JSON answer gets it through Claude’s structured outputs.

What travels to a machine

The app hands the call to a machine over the machine’s command channel, as an llm_generate command: the prompt, the model’s names and the options. The machine runs it on its own key and sends back the answer. Keys never travel either way: the app’s keys stay on the server, and the machine’s stay on the machine.

A machine only takes these commands from Andon itself, never from a person’s command, and not when its policy requires signed commands.

What isn’t backend-agnostic

  • Voice can’t run on a machine (above).
  • A machine serves these calls with an Anthropic, Gemini or AI Gateway key, not with its Claude login. The older paths that ask a machine’s own Claude (recipe detection when nobody holds a key, answers while Andon drives) are unchanged.