Testing and quality gates

Running the tests

Start with the smallest test that covers your change. Widen it when the change affects other parts of the system. The testing guide on GitHub gives these commands:

The Makefile also has shortcuts:

make test-backend
make test-frontend
make test-e2e
make lint
make type-check

Run each package’s tests in an environment with its real dependencies. Run the CLI, SDK, MCP and backend suites separately, so that their Python import paths don’t clash.

Before you push, the contribution policy asks you to run the full pre-push check:

make push-check         # runs what CI runs
make push-check-list    # prints the full list of checks
make install-hooks      # makes the full check run on every git push

The pre-push check runs what CI runs, including checks that live outside the test suites:

• the frontend production build, and the bundle-size and theme checks;
• the coverage ratchet, which stops test coverage from going down;
• the checks on the generated documentation;
• the project’s quality gate script, which checks architecture rules, the list of local-only paths, private markers and the list of allowed dependency licenses.

A full run takes about 30 minutes. The hook installer sets Git’s core.hooksPath to the tracked hooks folder, so the full check runs on every push.

Pushes that change only documentation skip the check, as CI does. “Only documentation” means the files in the CI workflow’s paths-ignore list:

• anything under qa/;
• CHANGELOG.md;
• Markdown files at the repository root.

A Markdown edit inside the architecture or docs folders still runs everything. To force the full check anyway, run python scripts/push_check.py --full. The quick check, make push-check-quick, is a faster but weaker first pass. Passing it doesn’t predict CI.

A few local results differ from CI on purpose:

• The generated-reference check compares only your additions against origin/main.
• For the mypy type-check ratchet, compare the files you touched, not the total.
• Suites listed in KNOWN_LOCAL_FAILURES depend on the environment. They’re reported, but they don’t fail the push.

Details: the quality gate script is scripts/quality_gate.py, and the hooks folder is .githooks/.

Run the guard checks on their own

The Makefile also runs the project-wide guard checks on their own:

In the frontend folder, these are the checks CI runs:

Documentation generation

The generated references live under docs/reference. They cover the OpenAPI document, the endpoint index, one page per API area, the schema catalogue, the data dictionary, settings, migrations and worker tasks. They’re built from the tracked code, without starting any service.

From the repository root, run the same three steps as CI’s backend job:

python scripts/generate_reference.py --check
python scripts/check_reference_docs.py
python -m pytest scripts/test_reference_docs.py -q -p no:cacheprovider

• The first step reports differences without rewriting anything. Run it without --check to regenerate the references.
• The second checks local links, heading anchors, references to source lines and OpenAPI references. It also checks which operations and schemas are covered, and the API guide’s upload examples.
• The third runs the checker’s own tests.

CI checks Mermaid diagrams in a separate job.

The reference generator and the documentation checker on GitHub define what they check. Review the generated diff. A clean generation doesn’t replace reading the text or testing the running app.

Unit tests and tests with real services

When no database address is set, backend pytest supplies a placeholder that points nowhere, so that imports succeed. That doesn’t create PostgreSQL, MongoDB, Redis, file storage, queues or outside providers. Read a test’s fixtures and markers before you assume it needs no services.

Use throwaway services for integration and live tests. When your change affects them, test these cases:

• the same upload sent twice, or two uploads at the same time;
• access refused across projects;
• retries;
• a worker losing its lease, which is its claim on a task;
• queue starvation, where tasks in one queue never get picked up;
• review decisions that are out of date;
• pass-rate denominators, which decide which tests a pass rate counts;
• wrong keys or versions.

Say clearly whether each provider was mocked, offline or live.

For a change to live uploads, a stress script runs many live SDK sessions and WebSocket watchers at once against a throwaway local copy. It refuses any host that isn’t local. It reports how long finishing a run takes, frames per second, Redis stream depth and waits for database connections.

Details: the stress script is backend/scripts/stress_live_ingest.py.

Typed AI output, JSON repair and plans that stay fixed once made all make AI results traceable and check their shape. They don’t prove that the content is correct, or that a rerun gives exactly the same result. Recording feedback gives TestLookup something to learn from. It doesn’t show that a replacement model was trained, checked or put into use.

Existing CI gates

The CI workflow on GitHub defines these jobs:

• checks that container image versions are fixed and haven’t changed unexpectedly;
• the project-wide quality gates;
• Mermaid and browser checks for diagrams;
• backend, database integration, frontend, MCP, SDK and CLI tests, and tests for each language client;
• scans of dependencies and images;
• image builds;
• a development deployment, under some conditions.

CI also runs the Playwright browser journeys in four shards, which are parts that run in parallel. Those journeys answer API calls from fixtures. The chart visual tests run against committed Linux reference images in two shards.

Some expensive jobs for queues and large uploads run only when matching paths change. You can also start them by hand with two workflow inputs:

• run_large_ingestion_boundary;
• run_compose_queue_matrix.

To see which jobs actually ran for your change, open the GitHub Actions run for your commit. Deployment and release workflows use their own credentials and environments.

Tests a fix needs

Write a test that fails without your fix before you rely on the fix. Where they apply, cover:

• success and refusal;
• out-of-date or duplicate state;
• limits;
• retries;
• sending the same request twice;
• serialization.

Architecture rules and checks on the source text help stop the code getting worse. They don’t replace tests of behavior.

For storage changes, plan the forward migration and the rollback. For queue work, test routing, which queues the workers read, retries, the dead-letter queue, cancellation and the final state. The dead-letter queue holds messages that kept failing.

For changes to the UI or a client, test the public behavior rather than the implementation details. When public behavior changes, update the generated references and the written documentation.

Evidence to attach

In your pull request, include:

• the exact commit;
• the commands you ran, and their results;
• relevant logs or reports;
• the services you used;
• the latest migration, called the migration head;
• image tags or image digests;
• timestamps;
• whether providers were mocked, offline or live.

Include the failing test that led to the change, and the final passing run.

For a release-level check, add proof that the workers read every queue. Also show one report going all the way through saving, analysis, review and the final report. Notes from checking an earlier commit describe that commit only.

The maintainers’ verification page lists what the documentation checks prove and what they don’t. For example, the checks don’t start the app, apply migrations, deliver a queued task or call a model. The page is in docs/maintainers/, next to the notes on documentation upkeep, the known limitations and a glossary.

Next: extend TestLookup, or review the contribution process.