Skip to content
AndonDocs
GitHub
Change Andon

Preview internals

View as MarkdownThe same text, at this page's address plus .mdOpen in ClaudeAsk questions about this pageOpen in ChatGPTAsk questions about this page

How a preview reaches the board, and what the review tools may do. The user’s guide is Previews and annotations. The design and its decisions are in the review-on-the-preview spec.

The token gate

A web preview goes through a token gate on the machine before the tunnel. The gate:

  • Refuses a request without the preview’s token.
  • Puts a small script first in the head of each HTML page it serves to a frame: never into the app’s files, a tab’s page, the app’s own fetches, or a native screen’s page. The script answers only the board’s origin (the agent’s boardUrl), from the board’s own frame, with the element at a point (a short CSS path, tag, role, text cut to 200 characters, and its box in CSS pixels) and the viewport. That is how a pin carries the element under it.
  • Sends the app only the cookies it set itself through the gate, matched by exact name and value. The board’s session cookie, other previews’ tokens and cookies a page sets only in script never reach it.
  • Shares a port only when every process listening on it belongs to the run, found with lsof. On machines other than macOS and Linux a port preview is refused.
  • Asks the app for framed pages unencoded (and decodes one encoded anyway). A page over 8 MB goes through unchanged.
  • Sets its cookie SameSite=Lax in a tab and SameSite=None; Secure; Partitioned in a frame, so the board can frame the preview, and a page on another site that frames it gets a cookie jar of its own and is refused.

An app whose own Content-Security-Policy forbids inline scripts, or that refuses to be framed (X-Frame-Options, frame-ancestors), gets no element on its pins.

Andon only turns off the Tailscale serve ports it added itself. Anything served with tailscale serve by hand stays.

Native screens

preview_screen serves a page that captures the screen again about twice a second while someone has it open:

  • target: ios: the booted iOS Simulator (xcrun simctl io booted screenshot; device picks one by UDID).
  • target: android: an emulator or a device over adb (adb exec-out screencap -p; device picks one by serial). adb must be on the agent’s PATH.
  • target: window, app: <name>: the largest on-screen window of that macOS app that the run owns (the process, or one it descends from, works in the run’s workspace or runs a program from it), found again when a rebuild opens a new one. An app started with open from elsewhere is refused: start its binary (App.app/Contents/MacOS/App). The agent needs Screen Recording permission; without it the page shows the window’s frame and no content.

Calling it again for the same screen keeps the one link. stop_preview(<port>) closes it, as does the session ending.

Frames

Annotate sends the run’s machine a preview_frame command at the size of the viewer’s window, taken with the same read-only capture as preview_capture. The frame is an attachment kept out of runs, with frame_of set to the link’s URL. A machine older than agent 0.3.3, or one owned by someone else, can’t take one.

The review tools

  • preview_capture(label?, port?, path?, width?, height?) takes a PNG of one of the run’s own previews and saves it on the card as an output. A web preview is loaded from its local port in a headless Chrome or Chromium found on the machine (bins.chrome in the agent config, or ANDON_CHROME, else the usual install places). Andon never installs one. The default viewport is 1280 × 800 CSS pixels at a device pixel ratio of 1.
  • preview_element(frame?, x, y) names the element at a point of a web frame, from the page loaded again at the frame’s URL, viewport and scroll.

The browser is read-only. Only GET and HEAD requests leave it; no document of another site loads (a link or a form that would go elsewhere, or post, is answered 204); dialogs are dismissed; and it uses its own profile in the run’s folder. WebSocket, WebRTC, WebTransport, shared workers and registering a service worker each throw, in the page and in its workers. The reviewer is told the same rules (PREVIEW_REVIEW_RULES).

A round reports problems on a frame as visual_findings in review_result: the frame id, a region in its pixels, a note, a severity and fixed or blocked. Each is a finding of the round and an annotation on the frame. Fix it reaches the next round as an <annotations> block. A visual finding on a frame the round didn’t capture, or outside it, fails the call before anything is recorded.

On a web frame, each visual finding also carries the element under it, like a person’s pin: before recording the round, the agent reads it at the region’s point (a rectangle’s centre) from the preview, while it is still open, as preview_element does. A finding may bring its own element instead. Either way only the known fields are kept (previewElementOf), and machine_review_result stores it in card_annotations.element (an object, its known keys, within 16 KB). A native screen, a closed preview or a failed read gets no element and fails nothing. Fix it then gives the next round the <element> block under the note.

Not built yet: the viewer’s scroll for a frame, and a round re-checking its own annotations.