Extending TestLookup

Routes, services and schemas

Start from the HTTP or streaming route. From there, follow:

• its request and response schemas;
• the checks on the resource, the role and the key;
• the service code and storage;
• serialization;
• the activity record;
• everything that reads the result.

Add typed contracts, which spell out what each part promises, together with the change in behavior. Add tests for refused access at the same time.

Keep a project ID in the route path, not in a query parameter. The authorization architecture test checks only routes with a project ID in the path. So a project ID passed as a query parameter escapes that check.

The codebase map on GitHub explains what each part of the repository does.

Parser and format

Treat these parts as one unit that must stay compatible: report detection, parsing, archive unpacking, validation, the normalized results, the limits and the rejected rows. A new format needs:

• reliable detection;
• tests for broken and empty input;
• tests at the size limits;
• safe archive handling, where it applies;
• a consistent source identity;
• clear client documentation.

Don’t assume TestLookup supports a format just because an SDK or upload method accepts the file. Check what the upload route accepts, and keep existing formats and error responses working. Writing results needs the QA engineer role or higher, whatever the credential.

A new live reporter should follow the shared adapter rules. Missing configuration turns the reporter off with one warning. Network errors never fail the test run.

Add an agent or tool

An agent is an automated helper that investigates or acts inside TestLookup. A new agent, tool or model capability needs all of these:

• its catalog entry;
• typed input and output contracts;
• a policy and an allow list;
• a spending limit;
• a checkpoint;
• retry and evaluation paths.

Workflow stages are listed in backend/app/services/agent_capability_registry.py. The agents.capability-has-executor quality-gate check keeps that list in step with the stage lists.

Test these cases: providers that aren’t available, refused tools, running out of time or budget, and losing a lease on a task. Also test cancellation, badly formed output and a predictable final report.

Keep drafts, findings, evidence, review decisions and outside actions separate. Optional model behavior needs a path for when the model is offline or unavailable, and users must be able to see that path.

Agents that propose changes work in three modes, which a QA lead sets for each project:

• Shadow, the default: the agent records what it would do. It proposes nothing and changes nothing.
• Suggest: the agent creates typed proposals, which wait for a person to approve or reject them. Nothing runs.
• Act: an approved proposal runs, but only for the action types turned on for that project. A person still approves every proposal.

Only three action types can run:

• quarantine.apply;
• failure.reassign;
• defect.promote.

Clients and the UI

Update everything that uses the contract:

• the Python and other language clients;
• the CLI;
• the MCP server;
• the frontend services and types, hooks and state;
• the UI pages.

Keep the credential kind, scopes, project context, streaming protocol, retry behavior and version alignment intact. When permissions are involved, add a test for a wrong scope or out-of-date state.

Discuss changes to the MCP tool list before you make them. The MCP server accepts only JWT access tokens, not API keys.

The development guide on GitHub has the route-to-UI exercise and the extension checklist.

Database changes

A storage change needs an Alembic migration, updated ORM models and serializers, a forward-compatibility review, and a rollback or recovery plan. Think about:

• backfills, indexes, defaults and uniqueness;
• keeping each project’s data separate;
• existing rows;
• whether the workers still work with the change;
• the order in which you deploy a mix of old and new versions.

Discuss schema migrations before you build them. This command generates a revision inside the backend container, for you to review and edit:

make migrate-create MSG="..."

Stage the new migration in Git before you regenerate the references. The migration list is read from git ls-files.

If the change touches MongoDB, file storage, Redis or derived indexes, document how they’re updated and recovered too. A relational migration doesn’t make an operation across several stores all-or-nothing.

Decide what project export includes

Project export includes run history as well as settings and the test catalog. So every new table, run-scoped collection or kind of stored object needs a clear export decision. Tests name anything left unclassified, and they keep the import free of side effects.

• A new table: classify it in backend/app/services/project_export_entities.py. Mark it as exported with its rules, excluded with a reason, or not tied to a project. A test walks the real model metadata and names any table that’s missing. The test is backend/tests/test_project_export_classification.py.
• A new run-scoped MongoDB collection: export it, or name it as excluded, in backend/app/services/project_export_documents.py. That file also rewrites the collection’s ID fields on import. Its test checks every member of Collections. The test also checks that the retention purge and the export agree on what a run owns. The test is backend/tests/test_project_export_documents.py.
• A new kind of stored object: follow the key rule in backend/app/services/project_export_objects.py. Only a run’s files under its validated prefix are exported, and imported keys are rebuilt under the new project. The test backend/tests/test_project_export_objects.py holds that rule in place.
• Import code: keep it free of side effects. The test backend/tests/test_project_import_side_effect_free.py walks the import code. It fails if the import can reach upload, finalization, notification, webhook, Git or Jira, defect, flaky, release-gate, agent, model or search code.

Tests, docs and compatibility

Add focused behavior tests and tests for refused access. Then run the wider package, integration, architecture and deployment checks that match your change, as Testing and quality gates describes.

When a contract changes, regenerate the references and update the client examples and user documentation. Review the generated diff rather than accepting it as it is.

python scripts/generate_reference.py

The pull request template also asks for:

• a CHANGELOG entry;
• an UPGRADING.md note for a breaking change or a manual upgrade step;
• updated documentation where behavior changed.

The documentation lives in three places:

• the in-app guide in frontend/src/content/guide/. The app serves it at /docs;
• user-guide/;
• docs/operations/.

Say how the change stays compatible, and how feature flags affect it. Keep old consumers working, or document how to move them over. Don’t rely on a source-text guard alone to prove behavior.

Use the full codebase map.