Skip to content
AndonDocs
GitHub
Change Andon

Testing

View as MarkdownThe same text, at this page's address plus .mdOpen in ClaudeAsk questions about this pageOpen in ChatGPTAsk questions about this page

Andon has nine Vitest projects and a Playwright suite. Most of them need nothing running; the rest need only this worktree’s own Postgres, which they start themselves. Run the cheap ones constantly and the heavy ones on purpose.

Which suite needs which stack

Suite What it covers Command Needs
@andon/shared the pure core: verdicts, templates, schemas, the .andon/ format pnpm exec vitest run --project @andon/shared nothing
board the board’s logic: reducers, coalescing, hooks, a few components pnpm exec vitest run --project board @andon/shared and @andon/client built
agent the agent’s decisions, against a fake exec pnpm exec vitest run --project agent the same
@andon/client the supabase-js shapes and the SSE client pnpm exec vitest run --project @andon/client nothing
scripts the repository’s own scripts: the docs lint pnpm exec vitest run --project scripts nothing
docs the docs site’s plain modules: reading docs/, link rewriting, the build’s link check, code Copy text pnpm exec vitest run --project docs nothing. The site’s build (pnpm --filter @andon/docs build) also fails on a broken on-site link
desktop the desktop app’s local pages pnpm exec vitest run --project desktop nothing; cargo test in apps/desktop covers the Rust side
server the server: data API, auth, live updates, storage, functions, RLS, RPCs, triggers and the tenancy registry pnpm exec vitest run --project server this worktree’s Postgres. Its global setup starts pnpm db:local and uses an andon_test database
agent-int the agent end to end on the Andon server: a real tmux, git and ffmpeg, fake orca and claude pnpm exec vitest run --project agent-int this worktree’s Postgres (its global setup starts pnpm db:local, makes a fresh andon_agent_test database and starts the server on a free port), tmux, ffmpeg. Build the agent first (pnpm --filter @andon/agent build)
e2e (Playwright) the board in a browser at 1440, 1180 and 390 wide, and the installed PWA pnpm --filter @andon/board test:e2e nothing to start: it provisions its own database

The two packages have no Vitest config of their own, so their projects take the package name: --project shared matches nothing. On its own Vitest stops with “No projects were found”, but beside another project (--project agent --project shared) it runs the agent’s tests and quietly skips all of the shared ones.

The unit projects fail whole files with Failed to resolve entry for package "@andon/shared" in a fresh clone. That means the packages aren’t built yet, not that a test failed:

shell
pnpm --filter @andon/shared --filter @andon/client build

The e2e suite

Playwright runs against the server. scripts/e2e-db.ts makes a fresh andon_e2e database in this worktree’s Postgres, and the config starts the server, the Vite board and a production preview (for the service worker) on their own ports, then stops them. It is safe to run from any worktree. Two worktrees can run it at once with different E2E_PORT, E2E_PREVIEW_PORT and E2E_API_PORT.

The projects are desktop (1440 × 900), ipad (1180 × 820), iphone (390 × 844) and pwa. A spec joins one with a tag in its title (@ipad, @iphone, @pwa); untagged specs run on desktop. Browsers install once with pnpm --filter @andon/board exec playwright install chromium webkit.

shell
pnpm --filter @andon/board test:e2e -- --project=iphone
pnpm --filter @andon/board test:e2e -- e2e/board.spec.ts --project=desktop

The everyday loop

Run the cheapest proof first, and stop when the change is covered:

shell
pnpm exec vitest run --project board # about 30 s
pnpm exec vitest run --project agent --project @andon/shared # about 30 s
pnpm exec vitest run --project agent src/pipeline.test.ts # one file
pnpm exec vitest run --project board -t "beat the send" # tests whose name matches
pnpm exec vitest --project board # watch

Before you push, run the gate:

shell
pnpm typecheck
pnpm lint
pnpm test

pnpm test builds shared, client and agent, runs every Vitest project, then the board’s bundle budget, then Playwright. It needs nothing started first, and it takes minutes.

Also run the checks that match what you changed: pnpm check:personal always, pnpm check:docs for anything under docs/, pnpm db:check for a schema change (Schema changes) and pnpm --filter @andon/board budget for anything that adds to the board’s first load (Design system).

What kind of test for which change

Change Test Where
a pure rule: a verdict, an order, a template, a label a unit test beside the module, table-driven, with the edge cases packages/shared/src/*.test.ts
board logic: a reducer, columns, coalescing a pure unit test, no rendering apps/board/src/lib/*.test.ts
a hook that loads, patches or polls a unit test of the gate or coalescer; Playwright when people see it apps/board/src/lib/, apps/board/e2e/
a table, policy, RPC or view a server test as the real caller, a tenancy registry row, and the other-org, non-member and anon cases apps/server/test/rls/
a server endpoint, the query compiler, an auth flow a server test apps/server/test/
the agent: claiming, launching, hooks, commands a unit test for the decision, agent-int for the loop apps/agent/src/*.test.ts, apps/agent/test/*.int.test.ts
a parser for another tool’s output a unit test against a captured fixture fixtures/ and the parser’s test
a flow someone clicks through a Playwright spec with an axe check apps/board/e2e/*.spec.ts
the desktop shell cargo test and the TypeScript tests apps/desktop

Two habits that catch common gaps:

  • Test the wiring production uses. Build the object the way the app does (new Agent(config), start(), the exported factory), not a version the test assembles for itself. A test that injects the dependency production forgot passes while the feature is dead.
  • Assert as the caller the app is. A policy test run as the table owner proves nothing; run it as a member, a member of another org, a machine and anon.

A new table, view, RPC, bucket, Realtime topic or server function isn’t done until it is in the tenancy registry (apps/server/test/rls/tenancy-registry.ts). The coverage test then fails on a relation without row-level security, a policy, an org column or a registry entry, and on a function or topic the registry doesn’t know. The isolation tests seed a row of every registered table in one org, then check that nothing reaches another: no row, RPC answer, object, topic (tenancy-isolation.test.ts) or function’s answer (test/functions/tenancy-fns.test.ts).

An instance holds one org, so create_org refuses once the database has any. The suites make their many orgs with makeOrg / insertOrg (both helpers.ts) or the e2e makeOrg (backend().org), which call andon_private.insert_org as the database owner: the same rows, without the one-org check. A test of create_org’s success path needs a database of its own, as apps/server/test/api/setup.test.ts has.

Reading a result

  • The exit status is the verdict. When you pipe a run through tail or grep, keep it: cmd > log 2>&1; echo exit=$?, or set -o pipefail.
  • A run filtered by -t or a file path proves those tests, not the suite.
  • A suite you skipped because its stack wasn’t running is unverified. Say so in the pull request.
  • If a test fails before you changed anything, say it was already failing. Don’t weaken its assertion.

Known quirks

  • Many of the agent’s tests build real git repositories in a temporary folder (pipeline, andon-dir, git-workspaces, launch-files, the memory-* files). On a busy machine they hit Vitest’s 5 s timeout. Rerun the failing files alone or with --testTimeout=60000: a “Test timed out” that passes then isn’t a regression.
  • apps/agent/src/executors/codex.test.ts checks the exact environment a Codex launch gets, so ANDON_* variables in your own shell (set when you work inside an Andon run) make it fail. Unset them for the run.
  • The server and agent-int suites each use one database and run their files one at a time. A test that leaves rows behind breaks the next file, not itself: look rows up by id, never by title. CI’s unit (N/4) runs a quarter of the files, so a failure that only shows there reproduces with the same slice: pnpm exec vitest run --project '!agent-int' --shard=N/4.
  • If the server suite can’t start Postgres because a stale one holds the port, run pnpm db:local stop and try again.
  • The pwa project serves a production build, so a service worker change needs it: the dev server doesn’t register the worker.

CI

.github/workflows/ci.yml runs on every pull request into main and by hand (Actions → CI → Run workflow, or gh workflow run CI --ref <branch>). Pushes to main don’t run it. Every ready pull request gets these checks:

Check What it runs Needs
lint pnpm build, pnpm lint, check:personal, check:team-names, check:docs nothing
typecheck pnpm typecheck nothing
unit every Vitest project but agent-int (the server suite brings its own Postgres), split by --shard across four runners, unit (1/4) to unit (4/4); unit passes when all four do tmux, ffmpeg
secrets gitleaks over the whole history, with .gitleaks.toml and .gitleaksignore nothing
db-check pnpm db:check, when the pull request touches db/, scripts/db-* or apps/server/src/db/ the runner’s own Postgres

These run by hand, or on a pull request with the full-ci label:

Check What it runs Needs
andon-dir check:andon-dir, the .andon/ memory lint nothing
integration the board’s budget and Playwright e2e the server and its own Postgres
agent-linux the agent’s end-to-end suite (agent-int) Postgres, tmux
compose scripts/compose-smoke.sh Docker
linux-install, agent-macos the installer in containers; the agent on macOS Docker; a macOS runner

andon-dir, integration, agent-linux and compose still fail on main. Andon’s merge pipeline waits on every check a pull request shows, required or not, so a job that fails everywhere would send every card’s pull request to a CI fix run; each moves to every pull request once it passes on main.

A draft pull request runs nothing until it is marked ready. A change to nothing but Markdown and docs/ (outside docs/reference/, which the unit suites check against the code) runs only lint and secrets; the other jobs report “skipped”, which counts as passing. The checks still don’t cover e2e, so run pnpm typecheck, pnpm lint and pnpm test before you push.

The workflow uses no secrets and triggers on pull_request, never pull_request_target, so a pull request from a fork runs with a read-only token and none of the repository’s secrets. The desktop signing secrets live in desktop.yml’s environments, which only main and release tags may use.

A new gitleaks finding that is not a secret goes in .gitleaks.toml when a pattern describes it, or as its fingerprint in .gitleaksignore when it is a one-off. Never allowlist a real key: rotate it.

Some things need a real device, a live Orca or a hosted deployment. They are listed in Manual checks.