# Preview internals

How a preview reaches the board, and what the review tools may do. The user's guide is
[Previews and annotations](https://andon-docs.pages.dev/guide/previews-and-annotate.md). The design and its decisions are in
[the review-on-the-preview spec](https://github.com/psci-labs/andon/blob/main/docs/internal/specs/review-on-preview.md).

## The token gate

A web preview goes through a token gate on the machine before the tunnel. The gate:

- Refuses a request without the preview's token.
- Puts a small script first in the head of each HTML page it serves to a frame: never into the app's files, a tab's
  page, the app's own fetches, or a native screen's page. The script answers only the board's origin (the agent's
  `boardUrl`), from the board's own frame, with the element at a point (a short CSS path, tag, role, text cut to 200
  characters, and its box in CSS pixels) and the viewport. That is how a pin carries the element under it.
- Sends the app only the cookies it set itself through the gate, matched by exact name and value. The board's session
  cookie, other previews' tokens and cookies a page sets only in script never reach it.
- Shares a port only when every process listening on it belongs to the run, found with `lsof`. On machines other than
  macOS and Linux a port preview is refused.
- Asks the app for framed pages unencoded (and decodes one encoded anyway). A page over 8 MB goes through unchanged.
- Sets its cookie `SameSite=Lax` in a tab and `SameSite=None; Secure; Partitioned` in a frame, so the board can frame
  the preview, and a page on another site that frames it gets a cookie jar of its own and is refused.

An app whose own `Content-Security-Policy` forbids inline scripts, or that refuses to be framed
(`X-Frame-Options`, `frame-ancestors`), gets no element on its pins.

Andon only turns off the Tailscale serve ports it added itself. Anything served with `tailscale serve` by hand stays.

## Native screens

`preview_screen` serves a page that captures the screen again about twice a second while someone has it open:

- `target: ios`: the booted iOS Simulator (`xcrun simctl io booted screenshot`; `device` picks one by UDID).
- `target: android`: an emulator or a device over adb (`adb exec-out screencap -p`; `device` picks one by serial).
  `adb` must be on the agent's PATH.
- `target: window`, `app: <name>`: the largest on-screen window of that macOS app that the run owns (the process, or
  one it descends from, works in the run's workspace or runs a program from it), found again when a rebuild opens a
  new one. An app started with `open` from elsewhere is refused: start its binary (`App.app/Contents/MacOS/App`). The agent needs **Screen Recording** permission; without it the page shows the window's frame and no
  content.

Calling it again for the same screen keeps the one link. `stop_preview(<port>)` closes it, as does the session
ending.

## Frames

**Annotate** sends the run's machine a `preview_frame` command at the size of the viewer's window, taken with the same
read-only capture as `preview_capture`. The frame is an attachment kept out of runs, with `frame_of` set to the
link's URL. A machine older than agent 0.3.3, or one owned by someone else, can't take one.

## The review tools

- `preview_capture(label?, port?, path?, width?, height?)` takes a PNG of one of the run's own previews and saves it
  on the card as an output. A web preview is loaded from its local port in a headless Chrome or Chromium found on the
  machine (`bins.chrome` in the agent config, or `ANDON_CHROME`, else the usual install places). Andon never installs
  one. The default viewport is 1280 × 800 CSS pixels at a device pixel ratio of 1.
- `preview_element(frame?, x, y)` names the element at a point of a web frame, from the page loaded again at the
  frame's URL, viewport and scroll.

The browser is read-only. Only GET and HEAD requests leave it; no document of another site loads (a link or a form
that would go elsewhere, or post, is answered 204); dialogs are dismissed; and it uses its own profile in the run's
folder. WebSocket, WebRTC, WebTransport, shared workers and registering a service worker each throw, in the page and
in its workers. The reviewer is told the same rules (`PREVIEW_REVIEW_RULES`).

A round reports problems on a frame as `visual_findings` in `review_result`: the frame id, a region in its pixels, a
note, a severity and fixed or blocked. Each is a finding of the round and an annotation on the frame. *Fix it* reaches
the next round as an `<annotations>` block. A visual finding on a frame the round didn't capture, or outside it, fails
the call before anything is recorded.

On a web frame, each visual finding also carries the element under it, like a person's pin: before recording the
round, the agent reads it at the region's point (a rectangle's centre) from the preview, while it is still open, as
`preview_element` does. A finding may bring its own `element` instead. Either way only the known fields are kept
(`previewElementOf`), and `machine_review_result` stores it in `card_annotations.element` (an object, its known keys,
within 16 KB). A native screen, a closed preview or a failed read gets no element and fails nothing. *Fix it* then
gives the next round the `<element>` block under the note.

Not built yet: the viewer's scroll for a frame, and a round re-checking its own annotations.
