Preview internals
How a preview reaches the board, and what the review tools may do. The user’s guide is Previews and annotations. The design and its decisions are in the review-on-the-preview spec.
The token gate
A web preview goes through a token gate on the machine before the tunnel. The gate:
- Refuses a request without the preview’s token.
- Puts a small script first in the head of each HTML page it serves to a frame: never into the app’s files, a tab’s
page, the app’s own fetches, or a native screen’s page. The script answers only the board’s origin (the agent’s
boardUrl), from the board’s own frame, with the element at a point (a short CSS path, tag, role, text cut to 200 characters, and its box in CSS pixels) and the viewport. That is how a pin carries the element under it. - Sends the app only the cookies it set itself through the gate, matched by exact name and value. The board’s session cookie, other previews’ tokens and cookies a page sets only in script never reach it.
- Shares a port only when every process listening on it belongs to the run, found with
lsof. On machines other than macOS and Linux a port preview is refused. - Asks the app for framed pages unencoded (and decodes one encoded anyway). A page over 8 MB goes through unchanged.
- Sets its cookie
SameSite=Laxin a tab andSameSite=None; Secure; Partitionedin a frame, so the board can frame the preview, and a page on another site that frames it gets a cookie jar of its own and is refused.
An app whose own Content-Security-Policy forbids inline scripts, or that refuses to be framed
(X-Frame-Options, frame-ancestors), gets no element on its pins.
Andon only turns off the Tailscale serve ports it added itself. Anything served with tailscale serve by hand stays.
Native screens
preview_screen serves a page that captures the screen again about twice a second while someone has it open:
target: ios: the booted iOS Simulator (xcrun simctl io booted screenshot;devicepicks one by UDID).target: android: an emulator or a device over adb (adb exec-out screencap -p;devicepicks one by serial).adbmust be on the agent’s PATH.target: window,app: <name>: the largest on-screen window of that macOS app that the run owns (the process, or one it descends from, works in the run’s workspace or runs a program from it), found again when a rebuild opens a new one. An app started withopenfrom elsewhere is refused: start its binary (App.app/Contents/MacOS/App). The agent needs Screen Recording permission; without it the page shows the window’s frame and no content.
Calling it again for the same screen keeps the one link. stop_preview(<port>) closes it, as does the session
ending.
Frames
Annotate sends the run’s machine a preview_frame command at the size of the viewer’s window, taken with the same
read-only capture as preview_capture. The frame is an attachment kept out of runs, with frame_of set to the
link’s URL. A machine older than agent 0.3.3, or one owned by someone else, can’t take one.
The review tools
preview_capture(label?, port?, path?, width?, height?)takes a PNG of one of the run’s own previews and saves it on the card as an output. A web preview is loaded from its local port in a headless Chrome or Chromium found on the machine (bins.chromein the agent config, orANDON_CHROME, else the usual install places). Andon never installs one. The default viewport is 1280 × 800 CSS pixels at a device pixel ratio of 1.preview_element(frame?, x, y)names the element at a point of a web frame, from the page loaded again at the frame’s URL, viewport and scroll.
The browser is read-only. Only GET and HEAD requests leave it; no document of another site loads (a link or a form
that would go elsewhere, or post, is answered 204); dialogs are dismissed; and it uses its own profile in the run’s
folder. WebSocket, WebRTC, WebTransport, shared workers and registering a service worker each throw, in the page and
in its workers. The reviewer is told the same rules (PREVIEW_REVIEW_RULES).
A round reports problems on a frame as visual_findings in review_result: the frame id, a region in its pixels, a
note, a severity and fixed or blocked. Each is a finding of the round and an annotation on the frame. Fix it reaches
the next round as an <annotations> block. A visual finding on a frame the round didn’t capture, or outside it, fails
the call before anything is recorded.
On a web frame, each visual finding also carries the element under it, like a person’s pin: before recording the
round, the agent reads it at the region’s point (a rectangle’s centre) from the preview, while it is still open, as
preview_element does. A finding may bring its own element instead. Either way only the known fields are kept
(previewElementOf), and machine_review_result stores it in card_annotations.element (an object, its known keys,
within 16 KB). A native screen, a closed preview or a failed read gets no element and fails nothing. Fix it then
gives the next round the <element> block under the note.
Not built yet: the viewer’s scroll for a frame, and a round re-checking its own annotations.
