Testing TestLookup (contributors)

This page is for contributors who change TestLookup itself. It explains which test suites exist, how to run them, what services each one needs, and what a passing run does and does not show.

Pick the checks that cover the part of TestLookup you changed.
Read their fixtures and the services they need before you run them.
Record the commit, the command, the environment, the result and anything you left out.

A narrow suite that passes only tells you about that suite.

Test layers

Run the backend unit and regression tests separately from the integration and live tests. The default selection is -m 'not integration and not live'. Treat the frontend unit tests, type checks, the build and the Playwright browser tests as separate layers. The browser tests in CI are self-contained: they answer API calls from saved sample data, so they never touch a real backend. Run the CLI, client-library and MCP suites in their own package environments.

Next to each normal case, add cases that should fail: a wrong credential, the wrong project or a key that cannot reach it, an out-of-date version, retries and partial failures.

Before you push, run make push-check, as the contribution guide asks. It runs the CI list on your machine, including the checks that live outside the test suites, and it reports the suites listed in KNOWN_LOCAL_FAILURES without failing. The developer test guide describes it.

Service requirements

Read a suite’s fixtures before you start it. Tests differ in what they need: PostgreSQL, MongoDB, Valkey or Redis, file storage, workers, the scheduler, a browser, an AI provider or an outside integration.

Backend tests can import without a database, because pytest supplies a placeholder database address. A healthy API container does not show any of these:

that the database reached the latest migration;
that queues deliver jobs;
that the agent_children and shard queues have workers reading them;
that a report goes from upload to a finished report.

Offline checks

Label these results separately: rule-based checks (the same input always gives the same answer), mocked providers, recorded sample data, private local models, and evaluations against a live remote provider.

Checking the fields of the AI’s output, repairing broken JSON and fixed plans check the form of a result and let you trace it. They do not prove that the result is correct, or that it repeats exactly. Saved feedback is something a model can learn from, but saving it does not mean a model was trained, evaluated or promoted. A deployment check with the network blocked also needs the images, software packages and model files copied in beforehand.

Documentation checks

In the application repository, regenerate the references that are built from the code, and review the changes. Two commands check them:

python scripts/generate_reference.py --check
python scripts/check_reference_docs.py

The first confirms that the saved references can be rebuilt exactly and are tracked in Git. The second checks links, heading anchors, references to source lines, OpenAPI references, that every endpoint and data format is covered, the data dictionary’s index, and the upload examples in the API guide. Its own tests, and the checks for Mermaid diagrams, data-format documentation and the image list, are separate commands. A reference that matches the models in the code says nothing about a live database, an installation or an outside service.

What CI runs

The CI workflow runs quality checks that must not get worse, checks on the images and the offline bundle, and documentation checks. It also runs jobs for the backend, frontend, clients, MCP server, browser tests, packages, builds and deployment. Some expensive checks of queues and uploads run only when you choose the manual inputs run_large_ingestion_boundary and run_compose_queue_matrix.

The self-contained browser tests run in four parallel groups (shards). The chart tests compare screenshots with reference images made on Linux and saved in the repository, in two groups. The workflow file shows what CI is set up to run. To know what actually ran for a commit, read that commit’s CI run, its job results, and its logs or saved files.

Checks you did not run

When you report a change, list the checks you did not run, so that a reviewer knows what is still unchecked. Examples are integration, browser, model, live-provider, migration, queue, cluster, backup and restore, and hosted CI checks. Never report a check you did not run, an old result file or a workflow definition as a passing result.

The sections below come from TestLookup’s own testing guide.

Test map and scenarios

The repository combines tests of behaviour, checks on the architecture and on what the API promises, browser tests and deployment checks. A passing unit test suite cannot show that queues work in a real installation, or that a model is accurate. What the documentation checks cover is in verification.

Test map

Layer	Location and purpose	Typical command
Backend unit and regression	`backend/tests`, with mocked services and some local databases	From backend: `python -m pytest tests -m 'not integration and not live'`
Backend integration and live	`backend/tests/integration`, and tests marked live	Run only chosen tests, each with the throwaway test services set up for it
Frontend unit	Vitest tests next to the code under `frontend/src`	`npm test` from frontend
Frontend types and build	TypeScript and Vite	`npm run type-check`, `npm run build`
Browser (end to end)	`frontend/tests`, Playwright configs	`npm run test:e2e:ci`, or a live server that you set up for it
CLI	`cli/tests`	`python -m pytest cli/tests` with CLI installed
Client SDK	`client/tests` and language-specific packages	Python pytest, plus each language’s own build and test steps
MCP	`mcp/tests`	`python -m pytest mcp/tests` with MCP dependencies
Architecture checks that must not get worse	`scripts/quality_gate.py`, the files that record the current counts, and the script’s own tests	`python scripts/quality_gate.py`
Images and deployment	`scripts/release`, and checks on the Compose and Kustomize files	`python scripts/release/check_image_drift.py`
Documentation	Generated references, links, schemas and Mermaid	`python scripts/generate_reference.py --check`, `python scripts/check_reference_docs.py` (see the maintenance commands)
AI quality	Evaluation suites for agents and decisions, their manifests, and sets of answers people have confirmed	Choose the exact evaluator: offline, mocked or live. Keep a record of where each result came from

Backend pytest sets TESTING=true, turns off OpenTelemetry, and supplies a placeholder database address that leads nowhere when none is set. This lets the code import. It does not create a real database, and it does not prove that every test runs without services. Configuration and hooks set the integration and live markers. Read the fixtures before you run large suites against any real outside system. Running the SDK, MCP and CLI tests separately avoids clashes between Python import paths.

High-value regression scenarios

Tests for receiving results should cover:

the same report sent twice, or twice at the same time;
broken and empty reports;
archive limits after unpacking;
refused access across projects;
two sources that claim the same identity;
partly rejected rows;
batch sizes at the exact limits.

Tests for live sessions should cover a session that never finishes, several API processes, revoked sign-ins, updates sent in order, recovery when saving the run fails, and a sender that retries.

Agent tests should cover:

changes to a fixed plan, or to who may decide;
a lost lease;
when saved progress may be reused;
spending and time limits;
a child queue that no worker reads;
cancellation and limited retries;
the rule-based final report.

Report tests should cover rejected and replaced reviews, and the exception for pending drafts. They should also cover the exports, search, notifications and MCP server that use reports, and a review that changes before a delayed delivery.

Release and test management tests should cover:

the minimum evidence before a decision;
what each total is divided by;
the time windows for linking runs to a release;
only one current decision at a time;
out-of-date overrides;
two changes to the same test case at once;
keeping each project’s data separate.

Test the normal answers next to the failure cases: a wrong credential, a key that cannot reach the project, and an out-of-date version. Checks that only read the source text cannot replace tests of behaviour.

CI and deployment evidence

The CI workflow runs several jobs. They include quality checks that must not get worse, checks that images use exact versions and still match the image list, package tests and builds, and documentation checks. The release and deployment workflows need their own credentials, images, records of where the images came from, and environments. Some expensive checks of queues and of very large uploads run only when you choose them as manual inputs, so they do not run on every commit.

For a release, record the exact commit ID, the images and their IDs, the latest migration and the install layout you chose. Also record proof that workers read their queues, and one typical report’s full journey: received, saved, analysed, reviewed and reported. Keep this evidence with times and environment details, and say clearly whether the AI provider was mocked, offline or live. Evidence recorded for an earlier commit proves nothing about a new one.

Next, read the developer test guide for the commands and evidence that each part of TestLookup needs.