Troubleshooting and known issues

Start from the last step that you know worked, and keep its IDs. These are separate steps, and each one can fail on its own:

1. TestLookup accepts the upload.
2. It records a temporary task status.
3. It saves the run.
4. The analysis pipeline finishes.
5. A reviewer accepts the report.
6. A notification is delivered.

An accepted upload does not appear

Keep the upload task ID, and the run ID if you have one. TestLookup accepts the file with HTTP 202. That means it received the file, not that it read, saved or processed it. The task status expires from Redis after 24 hours. Sending results and live sessions needs at least the QA engineer role, so TestLookup refuses a credential with a lower role straight away.

Check these things in order:

1. You are using the same project and credential as the upload.
2. The task status.
3. Whether the file reached file storage.
4. Any error from reading the file.
5. Whether a worker reads the ingestion queues.
6. The run’s activity.
7. Whether the run exists.

A missing task status does not mean that no run was created. The JSON batch API does not use the same task status steps as file uploads.

Totals without per-test rows

Sometimes a run shows totals but no row for each test, or the Suites and Coverage pages stay empty while the Runs page fills. This usually means a worker is not reading every ingestion.shard.<i> queue. The rows for each test go to the shard queues, so a worker that reads only the plain ingestion queue causes exactly this problem.

With LIVE_INGEST_SHARD_COUNT=N, the -Q argument of each worker for the ingestion queue must list ingestion.shard.0 up to ingestion.shard.<N-1>. Release Compose and the Kubernetes base use eight shards, and source-built Compose uses none. To check, run this in a worker container:

celery -A app.worker.celery_app inspect active_queues

Once a worker reads the shard queues, a recovery task sends the affected runs through again within a couple of minutes.

If the Upload report button is missing from the Runs page, either the manual_upload flag is off or your role is below QA engineer. The flag is on for a new installation and keeps its stored value on an upgraded one. The upload API and the command-line tool never depend on that flag.

An analysis pipeline is stuck

Open the pipeline on Settings › AI › Pipeline runs to see its steps and its report. Then check:

• the next retry time;
• the leases, and the job that clears expired ones;
• the outbox and the dead-letter queue;
• whether the scheduler (beat) is running;
• whether a worker reads the pipeline’s queue, the agent_children queue and any configured shard queues;
• model timeouts;
• whether someone cancelled it.

A ready API does not prove that queued work is being delivered.

If the run’s own status looks wrong, these rules set it when the run is finished:

• A run in which no tests ran is STOPPED.
• Any failed or broken test makes it FAILED.
• Unknown outcomes, with no known failure, make it STOPPED.
• Otherwise it is PASSED.

Nothing was sent or a feature is off

If a webhook, a Jira issue, or a chat or email notification was not sent, check these in order:

1. Offline mode (AI_OFFLINE_MODE). It is on by default and overrides every integration setting, so a default installation sends nothing outside its own server.
2. The integration’s feature flag. outbound_webhooks is on for a new installation but keeps its stored value on an upgraded one. github_checks and gitlab start off.
3. Settings › Integrations › Integration health, to see whether TestLookup can reach the service and what the last error was.
4. Settings › Security & access › Audit dashboard, to see whether the event was recorded at all.

A flag that seems to do nothing usually has one of three causes. Its main switch is off (a project list alone does not turn it on), its rollout percentage is 0, or TestLookup is still using the value it read up to 30 seconds ago. The operator troubleshooting guide on GitHub lists more problems. Its note that outbound_webhooks starts off on a new installation is out of date.

AI results with less detail

Read what the run recorded:

• its mode, model size, provider and model;
• its workflow and prompt versions;
• whether evidence was available;
• the feature flags, spending limit and tool rules;
• the reasons for any skipped steps, and notes about steps that ran with less information.

When a model or document search is not available, the rules or a trained classifier can still return a simpler result. That result has the same layout but less depth. Its confidence score is not a measured probability, and it does not show that a hosted AI provider was reached.

Three causes explain most missing AI model output:

• No local model is running. On Compose, start the local AI services with make dev-llm. A site with no internet connection loads an offline model pack.
• The analysis mode is Rules or ML. Ask AI then stays hidden from the sidebar.
• A hosted provider is set up while offline mode (AI_OFFLINE_MODE) is on. Offline mode refuses any model address that is not on the same machine or on a private network.

Analysis falls back from the AI language model to the ML classifier, then to the rules, and records why. Test-case generation based on your documents does not fall back. Without usable evidence it answers HTTP 409, and when the provider produces no draft that can be reviewed, it answers 503.

A report review is refused

Check the exact report under review, its version and review state, who reviewed it, whether the review gate is on, and whether the exception for pending drafts applies. An out-of-date, rejected or replaced review does not count as accepting the current report.

By default the review gate is off (REVIEW_GATE_ENFORCED=false). TestLookup then records in the activity log that it would have refused, but it still lets the report be shared. An administrator can also set a date when the gate turns on for the whole installation, on Settings › AI › Review gate. Check both the environment variable and that page before you decide which mode applies.

Sign-in asks for another step

A sign-in can return HTTP 200 without signing you in. Instead, the response may ask you to set up two-step sign-in (MFA), enter an MFA code, or reset your password. Complete the step it asks for. If you call the API directly, read the response body. A challenge token is not an access token, so finish the step it asks for before you call other endpoints.

Sign-in fails for everyone

Two failures that affect every user have known causes:

• HTTP 503 “MFA is enabled but its stored secret cannot be read”. APP_SECRET_KEY was changed without keeping the old key in APP_SECRET_KEY_PREVIOUS. Put the previous key there and restart.
• HTTP 503 on every request after sign-in. TestLookup cannot reach the list of revoked sign-in tokens, which is kept in Valkey. When it cannot check that list, TestLookup refuses the request instead of letting it through. This is on purpose, so bring the Valkey service back. Setting AUTH_REVOCATION_FAIL_OPEN=true would let revoked tokens keep working during the outage.

Summary report numbers look inconsistent

If the summary report’s numbers do not seem to add up, check what each figure counts.

• The release filter applies to every part of the summary (the headline totals, suites, the last-run check, top failures and step figures) except the flaky count. The flaky count always covers the whole project, and the response says so on that field.
• The label that says what the pass rate is based on always reads unique_tests. It is wrong in two cases. The first is latest mode. The second is when the headline falls back to adding up the totals of runs that have no row for each test, for example when the window is filtered by suite. In those cases the label does not describe how the number was worked out.
• Flaky counts from recent history and capped totals of a selection also count different things.

Compare like with like.

API responses without a field list

In this version, 228 of the 731 API endpoints do not fully describe the data they return. That is a gap in the documentation, not a sign that those endpoints fail. The list of endpoints with undocumented responses names them. For one of them, check the generated endpoint index on GitHub and the code it links to, and look at the real response. If you change one of these endpoints, add a response model, and add a check in each program that reads it.

The sections below come from TestLookup’s own list of known limits.

More troubleshooting

Important limits

• Background work finishes a little later than the request that started it. An accepted upload (HTTP 202), its task status in Redis, the saved run, the finished pipeline, an accepted review and a delivered notification are separate steps. The upload status in Redis expires after 24 hours.

• The newer feature-flag service treats a flag it does not know as off, unless an older environment variable gives it a value. The older flag interface treats an unknown flag as on for everyone. So a feature that reads one service can behave differently from a feature that reads the other.

• The checks that limit how much traffic TestLookup accepts can let traffic through when Redis has an error. They help manage traffic, but they are not a firm limit during an outage.

• No single transaction covers PostgreSQL, MongoDB, Redis and file storage. Outboxes, catch-up jobs and markers for deleted items help recovery, but they cannot make every side effect happen exactly once.

Symptoms and first checks

Symptom	First checks
An accepted upload never appears	Check that you are looking at the right run and task IDs. Then check the file’s upload status, file storage, whether a worker reads the ingestion queues, and the run’s parsing errors and activity
The upload status returns 404	Use the same project and credential as the upload. The status expires from Redis after 24 hours, so also check whether the run itself was saved
The live view updates, but the run details lag behind	Check the jobs that save and finish live sessions, the outbox, the workers that read the ingestion queue, and that you are using the run’s permanent run ID
A pipeline stays in progress	Check that the scheduler (beat) is running and that a worker reads the pipeline’s queue. Then check the next retry time, the leases and the job that clears expired ones, the worker for agent_children, and model timeouts
The AI result has less detail than expected	Check the mode, model size and sources recorded on the run, whether evidence was available, the spending limit, the chosen workflow, and any notes about skipped steps
A feature is on, but its action is missing	Check the flag’s project and role lists and its rollout percentage, and allow for its 30-second cache. Also check whether the feature reads the older or the newer flag service, and whether it needs an AI provider
A report export is refused	Check the report under review and its review state, whether the review gate is on, whether the exception for pending drafts applies, and the user’s permissions
A release looks green or neutral when you expected a problem	Check how many runs count as evidence, any skipped, unknown or stopped runs, which runs and phase belong to the release, the saved copy of the policy that was used, and the review state
Sign-in returns HTTP 200, but you are not signed in	The response asks for an MFA code, MFA setup or a password reset instead of starting a session. Complete that step
/health/details returns 200, but something is not working	Read each service’s status in the response. Then use /health/ready, and check that workers are reading their queues
A second copy of the repository cannot start TestLookup	Container names and ports are fixed, and Docker state is shared on the machine. Stop the other copy first, or run the second one where it cannot clash with the first
/docs opens the wrong page, or Swagger shows no schema	Swagger is at /api-docs, and the in-app user guide is at /docs. Check that the ingress sends /api-docs, and the schema under it, to the backend

When you know which step fails and can make it happen again, report a bug from the community page (Report a bug). Remove passwords, keys and private data from the report first.