Day-two operations

Day-two operations are the regular checks you run once TestLookup is installed. They tell you which version is running, and they follow each job past the HTTP response into the queues and workers. They also keep enough detail for you to recover when one step fails. A healthy API process is only one of these checks.

Health checks

TestLookup has four health addresses, and each one answers a different question:

• /health/live: does the API process respond?
• /health/ready: are PostgreSQL, MongoDB and Redis ready?
• /health/version: which build is running?
• /health/details: what state is each service in? It can return HTTP 200 while one service is only partly working, so read the response body.

Before you call the system healthy, also check the workers and queues (below), and send one report all the way through to a finished report.

Metrics, monitoring and alerts

/metrics serves counters and gauges for HTTP requests and for TestLookup’s own work while METRICS_ENABLED is true (the default). Queue lengths are refreshed each time the metrics are read. Metrics about exports, imports and other data jobs include testlookup_project_data_jobs_total{kind,outcome} and testlookup_suite_parity_runs_total{status}.

Monitoring is an add-on that you turn on yourself.

On Compose, add docker-compose.monitoring.yml to the stack. It starts Prometheus, which keeps 15 days of data, and Grafana on host port 3001 with a ready-made overview dashboard. It also starts Jaeger for OpenTelemetry (OTLP) traces. Set GF_SECURITY_ADMIN_PASSWORD first, then run the command below. The Compose guide on GitHub lists the ports and logins.

docker compose -f docker-compose.yml -f docker-compose.monitoring.yml up -d

On Kubernetes, apply k8s/monitoring after the application overlay, on a cluster that runs the Prometheus Operator. It adds PodMonitors for the backend and every worker, and a PrometheusRule.

Both setups load the same 35 alert rules. They include:

• the backend or a Celery worker is down;
• a queue keeps growing;
• the agent dead-letter queue is not empty;
• the database connection pool is used up;
• Redis memory is high;
• the finalize step keeps failing;
• backups failed or are out of date;
• response-time goals for each route.

The backup rules read kube-state-metrics, and the disk and out-of-memory rules read node-exporter. Without those exporters they have no data, so check that the scrape targets are up before you trust a quiet alert list. The observability guide on GitHub covers metrics, traces, logs and decision logs. Both setups load the same alert rules file.

Queues and workers

Every queue your install mode uses needs a worker reading it: default, critical, ingestion, ai_analysis, agent_children and every configured ingestion shard queue. You also need exactly one scheduler (beat). To list the queues each worker reads, run this in a worker container:

celery -A app.worker.celery_app inspect active_queues

On source-built Compose, run it through Docker Compose:

docker compose exec worker celery -A app.worker.celery_app inspect active_queues

Source-built Compose also serves Flower, a dashboard for the workers, at http://127.0.0.1:5555. Count the API and worker processes in the PostgreSQL connection budget. Watch for queues that grow unevenly, and for workers that run out of capacity.

The ingestion queue is split into shards, and their number differs by install mode. The source-built Compose stack runs no shards (LIVE_INGEST_SHARD_COUNT=0). Release Compose and the Kubernetes base run eight, and every worker for the ingestion queue must read all of them (-Q ...,ingestion.shard.0,...,ingestion.shard.7). To change the count, change the services that send work and the workers together, following the runbook on GitHub:

1. Prepare the release.
2. Stop every service that sends work.
3. Let the database and the job queue empty.
4. Switch the count and check it, or roll back.

Scheduled jobs and time limits

In this version, the scheduler (beat) starts 58 scheduled jobs. They include:

• the retention purge and the search re-index, every hour;
• training for the ML classifier and for flaky-test confidence, every night;
• the suite-membership comparison, every day at 06:15;
• the outbox relays, and the jobs that clear expired claims (reapers);
• the check for expected runs, every 15 minutes;
• the project-data sweep, every hour at minute 35.

Each worker slot takes one task at a time, and a worker confirms a task only after it finishes. So a task stopped by a crash is delivered again. By default, a task is asked to stop after 29 minutes and is stopped by force after 31 minutes. Project history exports and imports have their own limit of three hours. The generated worker reference on GitHub lists every task with its queue and schedule.

Retention

Retention means how long TestLookup keeps old data. It is off by default, both for each project and as the default for the whole installation.

When you turn it on for a project on Settings › Project › Retention & purge:

• The purge job runs every hour, but it acts on that project only during the project’s window hour (02:00 UTC by default).
• You can pause it for 1 to 90 days.
• It always keeps a minimum: the newest 10 runs by default, the newest passing run, and each suite’s newest passing run.
• A retention hold that is in force blocks every purge and deletion it covers, until an administrator releases it and gives a reason.
• The activity log (audit) window can never be shorter than the runs window.
• The settings audit log, which holds the purge records themselves, is never purged.

Limits on AI history apply when each new version is written, even while retention is off. By default, TestLookup keeps 20 versions per run and kind, and 10 per test.

Review these together: retention, deletion jobs, how long file storage keeps objects, activity-log rules and how long Redis keeps each item. The time a run’s tests ran and the time TestLookup received them can differ. A temporary status that expired is not the same as a deletion.

Project export and import jobs

Project export and import jobs have their own time limits, and the hourly sweep-project-data-jobs job enforces them:

• An export archive is removed 72 hours after it is offered.
• An import uploaded in parts and left unfinished for 24 hours is marked failed, and its parts are removed.

While a history export builds and checks its archive, it holds two copies on the worker’s disk, so plan for about 4 GB. An import holds one copy of the assembled archive. An import sent in a single request can be up to 50 MB. Larger archives, up to 2 GB, upload in parts of at most 32 MB each, so you do not have to change any proxy limit. A project imported with run history has its retention purge paused for 90 days.

Suite-membership cutover

TestLookup is moving its record of which tests belong to which suite from older tables to the test catalog. Until the switch is complete, it writes to both. A comparison runs every day at 06:15 UTC to check that they agree, and its result shows on Settings › System › Data & storage. After seven days in a row with no differences (counted in UTC), an administrator can finish the switch. To do that, turn off the suite_membership_legacy_writes flag (on by default) on Settings › System › Feature flags.

To go back, turn the flag on again. Then run backend/scripts/reconcile_legacy_suite_memberships.py in an API container to fill in the runs received in the meantime. The script’s header shows the kubectl exec command. It only reports counts unless you add --apply.

Failed jobs and retries

Dead letters are messages that kept failing and were moved to a holding area. Before you replay anything, look at the next retry time, the history of attempts, the leases and the job that clears expired ones, the outbox and the dead-letter queue. Never start two replays of the same work at once. A policy refusal, invalid input, a rejected review, a cancellation or a used-up budget needs a fix, not another retry.

Instance administrators use these API endpoints:

• GET /api/v1/admin/maintenance/dlq reads both dead-letter stores for received results. GET /api/v1/admin/dlq does the same.
• POST /api/v1/admin/dlq/{entry_id}/replay replays one stream entry. It refuses a second replay of the same entry while one is still running.
• POST /api/v1/admin/maintenance/outbox/requeue puts failed outbox jobs back in the queue.

If a store cannot be read, the API answers 503 instead of returning an empty list. These dead letters live in Redis. The task list keeps the newest 1,000 entries and expires seven days after its last write. A restore also empties Redis, so export what you need before either happens.

Backups, restores and upgrades

Back up PostgreSQL, MongoDB and file storage together, and keep the application encryption key. Plan to rebuild Redis, the AI model files and the search index. Storage and backups has the commands. Practise the restore on a spare test installation. Check that a recent run’s report and attachments load, and that a new upload reaches a finished report. To shut down a local installation, use make stop. make clean destroys the data volumes, and it asks for CONFIRM=yes.

Every upgrade starts with a backup. Upgrade one release line at a time. In production, database updates (migrations) only move forward. On Compose, run:

make backup
make preflight TAG=vX.Y.Z
make upgrade TAG=vX.Y.Z

Afterwards, /health/ready should return ok, and /health/version should report the new commit.

An installation that moves to the bundled Valkey must let its jobs finish, export what it needs from the dead-letter queue, and start the job queue on an empty volume. The bundled versitygw file store starts empty until you copy your stored files into it. To copy them, run this inside the backend container (on Compose, put docker compose exec backend in front of it):

python /app/scripts/s3_setup.py --copy-from-endpoint <old endpoint> --copy-from-access-key <old access key> --copy-from-secret-key <old secret key>

The script can also read the old store’s details from S3_COPY_FROM_ENDPOINT, S3_COPY_FROM_ACCESS_KEY and S3_COPY_FROM_SECRET_KEY.

Record which version is running

When you write down what you saw, include:

• the commit ID of the source and the image IDs (digests);
• the /health/version response;
• the install mode and the version of your configuration;
• the latest database migration;
• the versions of the clients, models and workflows.

Keep the web app, API, MCP server, workers, database and clients on versions that work together. Report image IDs, not a tag that moves, such as latest.

What to collect when something fails

When something goes wrong, collect:

• times, with their time zone;
• project, run and task IDs, and request or correlation IDs;
• the bodies of the health responses;
• logs, with secrets removed;
• the state of the queues and workers;
• retry and dead-letter records;
• the latest database migration and the build version;
• the review or policy state;
• the last step that is known to have worked.

Remove passwords, keys and sensitive data. Write down which checks you actually ran, and which causes are still only guesses.

Runbook pointers

These operator guides on GitHub match this version of the documentation:

• the upgrade notes, for the steps each version needs;
• the shard-count change runbook linked above;
• the connection-budget guide;
• the operator troubleshooting guide;
• the administration guide, for backup, restore, retention holds and deletion.

If a user lost their authenticator app and has no recovery codes left, an operator can recover the account with the emergency script backend/scripts/mfa_breakglass.py:

• It refuses to run unless MFA_BREAKGLASS_ENABLED=true is set.
• It clears two-step sign-in (MFA) and any lockout for exactly one user. It does not change passwords, roles or sessions.
• It writes an MFA_BREAKGLASS_RESET identity event that names the operator and the reason.

Run it inside the backend container, first with --dry-run:

python /app/scripts/mfa_breakglass.py --user alice --dry-run
MFA_BREAKGLASS_ENABLED=true python /app/scripts/mfa_breakglass.py \
  --user alice@example.com --operator "on-call: sam" \
  --reason "lost phone, recovery codes exhausted, verified by video"

On Kubernetes, run the same command through kubectl:

kubectl -n testlookup exec -it deployment/testlookup-backend -- env MFA_BREAKGLASS_ENABLED=true python /app/scripts/mfa_breakglass.py --user ... --operator ... --reason ...

The sections below come from TestLookup’s own deployment guide and its troubleshooting table.

Runbooks and troubleshooting

Startup and upgrade order

1. Choose an image tag or image ID that never changes, and fix your configuration. Supply the production secrets, addresses, TLS and trusted-proxy settings. Keep the web app, API and MCP server on matching versions.
2. Set up the services that keep data, and check they have enough capacity. The PostgreSQL connection budget must count the API and worker processes, the reserved connections and the connection the migration needs. See database connection budget.
3. Run the database update job (migration) up to the target version, with the matching backend image. On Compose this is the db-migrate service, and on Kubernetes it is scripts/run-k8s-migrations.sh. A ready API does not show that the migration finished.
4. Start the API, a worker for every required queue and exactly one scheduler (beat). Check that the agent_children queue and every configured ingestion shard queue have a worker reading them.
5. Check the ingress paths, /health/version, readiness and metrics. Then send one report all the way to a finished report, and check that integrations behave as expected under your real settings.

On Compose, run make backup, then make preflight TAG=..., then make upgrade TAG=.... Together they follow this order. Read the ops scripts first. upgrade.sh pulls the new images (release stack) or rebuilds them (source stack), and runs alembic upgrade head before it replaces the application. It then restarts the services in order and runs a quick smoke check. It does not roll back on its own. The backup you took before the upgrade is the way back. After a migration that changed data, going back one database version is not a safe way to undo a release. Keep your backups and the IDs of the previous images.

Version-specific steps, such as moving an existing install from Redis to Valkey, are in the upgrade notes (UPGRADING.md). To change LIVE_INGEST_SHARD_COUNT, change the services that send work and the workers at the same time, following the ingestion shard count change runbook.

Observability and troubleshooting

Signal	What it tells you
/health/live	The API process can answer. Do not restart other services based on this alone
/health/ready	PostgreSQL, MongoDB and Valkey (or Redis) are ready. A 503 takes the pod out of traffic
/health/version	The version, commit, build and environment
/health/details	A short report on each service it depends on. It can return HTTP 200 while some of them are only partly working
/metrics	Counters and gauges for HTTP requests and TestLookup’s own work, when METRICS_ENABLED is on. Queue lengths are refreshed each time the metrics are read
Worker and beat logs, Flower	Which jobs were sent, picked up and retried. On Compose, Flower listens only on the local machine
Project activity and decision trail	What users did, and the reasons and sources behind each agent’s work
Upload, outbox, dead-letter queue and pipeline records	Which background step failed, beyond what HTTP health shows

Prometheus alert rules live in infra/monitoring/prometheus-rules/; Compose loads them through the monitoring add-on and Kubernetes through k8s/monitoring. Set alerts for pipelines that stop moving, and for leases or retries that run out. Also alert on jobs piling up in the outbox or the dead-letter queue (a holding area for messages that kept failing), uneven queues, database connections and the job queue’s memory. Installing an OpenTelemetry library does not on its own trace the Celery tasks. See the monitoring guide and the troubleshooting page.

Symptoms and first checks

Symptom	First checks
An accepted upload never appears	Check that you are looking at the right run and task IDs. Then check the file’s upload status, file storage, whether a worker reads the ingestion queues, and the run’s parsing errors and activity
The upload status returns 404	Use the same project and credential as the upload. The status expires from Redis after 24 hours, so also check whether the run itself was saved
The live view updates, but the run details lag behind	Check the jobs that save and finish live sessions, the outbox, the workers that read the ingestion queue, and that you are using the run’s permanent run ID
A pipeline stays in progress	Check that the scheduler (beat) is running and that a worker reads the pipeline’s queue. Then check the next retry time, the leases and the job that clears expired ones, the worker for agent_children, and model timeouts
The AI result has less detail than expected	Check the mode, model size and sources recorded on the run, whether evidence was available, the spending limit, the chosen workflow, and any notes about skipped steps
A feature is on, but its action is missing	Check the flag’s project and role lists and its rollout percentage, and allow for its 30-second cache. Also check whether the feature reads the older or the newer flag service, and whether it needs an AI provider
A report export is refused	Check the report under review and its review state, whether the review gate is on, whether the exception for pending drafts applies, and the user’s permissions
A release looks green or neutral when you expected a problem	Check how many runs count as evidence, any skipped, unknown or stopped runs, which runs and phase belong to the release, the saved copy of the policy that was used, and the review state
Sign-in returns HTTP 200, but you are not signed in	The response asks for an MFA code, MFA setup or a password reset instead of starting a session. Complete that step
/health/details returns 200, but something is not working	Read each service’s status in the response. Then use /health/ready, and check that workers are reading their queues
A second copy of the repository cannot start TestLookup	Container names and ports are fixed, and Docker state is shared on the machine. Stop the other copy first, or run the second one where it cannot clash with the first
/docs opens the wrong page, or Swagger shows no schema	Swagger is at /api-docs, and the in-app user guide is at /docs. Check that the ingress sends /api-docs, and the schema under it, to the backend

To trace a problem you can see back to the first step that failed, use troubleshooting.