Storage and backups

TestLookup keeps its records, and the evidence behind them, in several stores. A restore is complete only when PostgreSQL, MongoDB, file storage and the encryption key all come from the same point in time, and the search index has been rebuilt.

Where data lives

Store	What it holds
PostgreSQL	The main database: projects, users, runs and other application records, plus the records of work in progress.
MongoDB	Detailed evidence, summaries, decision documents and other documents.
File storage (S3-compatible)	Reports waiting to be processed, attachments, evidence, exports, and knowledge or training files.
ChromaDB (optional)	Search indexes built from your data and from the embedding model you chose.

The ChromaDB indexes come back in different ways after a loss:

• A full re-index rebuilds the test-result search index from PostgreSQL.
• Knowledge-source chunks come back only when you sync each source again.
• Agent-memory vectors come back only as new investigations arrive.

That is why the backup steps below treat the ChromaDB volume separately. If you lose any one of PostgreSQL, MongoDB or file storage, a run record can be left without the evidence needed to explain it.

A project export and a deletion archive are not backups. In this version, an export can include run history: up to 2,000 runs with their stored documents and attachment files, at most 2 GB in total. You can import it only as a new project. An export still leaves out raw report uploads, defects, members and instance settings, and TestLookup removes its archive from file storage after 72 hours. A deletion archive is a signed record of what was deleted, and you cannot import it again.

What Redis holds

Redis is the job queue and result store for the background workers (Celery). It also holds:

• live event streams and the copies sent to each viewer;
• session buffers and temporary upload status;
• usage limits and caches;
• the list of revoked sign-in tokens (signed-in requests get HTTP 503 while Redis cannot be reached);
• the dead-letter queue for received results, a holding area for messages that kept failing, with a seven-day expiry time;
• each user’s recent searches.

Plan recovery around how long Redis keeps each item, what it removes when memory runs low, and its append-only file saved every second (appendfsync everysec). None of these promises zero data loss. An expired upload status does not mean that the run is missing from PostgreSQL.

The bundled server is Valkey 8.1, and the REDIS_* settings keep their names. The standard backups leave Redis out on purpose. make restore empties it unless you set SKIP_REDIS_FLUSH=1, so dead-letter entries and recent searches do not survive a restore. Export any dead letters you need first.

Encryption key

Keep the application encryption key (APP_SECRET_KEY) with your protected recovery records, apart from the data archives, and note when it was last changed. A database restore without the matching key leaves stored integration, AI provider, webhook and other encrypted passwords unusable.

The same key encrypts each user’s two-step sign-in (MFA) secret. A restore under a different key makes MFA sign-in fail with HTTP 503 (“MFA is enabled but its stored secret cannot be read”). The key also signs saved copies of decision evidence and signed records (attestations).

To change the key, move the old value to APP_SECRET_KEY_PREVIOUS instead of replacing it, and record which key each backup needs. Never copy real keys into backup reports, documentation or restore examples.

Back up everything together

Back up PostgreSQL, MongoDB and file storage from the same point in time. With each backup, record the image and database versions and which key it needs. Note also that Valkey, the AI model files and the search index are rebuilt, not restored. A PostgreSQL dump on its own is not a complete backup.

On Compose, run this from the repository root:

make backup

It writes backups/testlookup-backup-<UTC time>.tar.gz. The archive holds a PostgreSQL dump, a MongoDB archive, the file-storage data, and a manifest with the version, the commit ID (git SHA) and the latest database migration (Alembic head). Add QUIESCE=1 to stop the application first, so that nothing changes while the backup runs. Add BACKUP_DIR to write somewhere else. Copy the archive to another machine.

On Kubernetes, the backup component’s scheduled job (CronJob):

• runs every day at 02:17 (17 2 * * *) and writes to a volume claim (PVC);
• keeps 14 archives by default (BACKUP_KEEP);
• exports the file storage through the S3 API, not as a disk volume;
• records a SHA-256 checksum for each file in the manifest;
• is turned on in the homelab and openshift-artifactory overlays.

The volume claim sits in the same cluster, so copy the archives off the cluster. The homelab guide shows the commands. With the monitoring add-on, the TestLookupBackupJobFailed and TestLookupBackupStale alert rules watch the backup job. They read kube-state-metrics.

Neither backup includes Valkey (or Redis), Ollama or ChromaDB. If you rely on knowledge chunks or agent-memory vectors, keep or back up ChromaDB’s volume separately.

Check a restore

Practise restores on a spare test installation that nothing else depends on.

On Compose, run:

make restore FILE=backups/testlookup-backup-<UTC time>.tar.gz

The command:

1. Checks the archive and asks you to confirm. Add CONFIRM=yes to skip the question.
2. Stops the application.
3. Restores PostgreSQL, MongoDB and file storage.
4. Empties Valkey.
5. Runs the database updates (migrations).
6. Starts TestLookup and runs a quick smoke check.

It refuses when the installed database is at a newer migration than the backup, or when the installed code does not know the backup’s migration. Add FORCE=1 only if you really mean to restore anyway.

On Kubernetes, follow the restore runbook on GitHub. The restore job needs RESTORE_CONFIRM=yes, and it checks every checksum in the manifest before it touches any data. Neither restore brings back the search index. A full re-index rebuilds it from PostgreSQL.

After a restore, check these things:

1. The database version and the build version.
2. A recent run: its report, attachments and AI analysis load. This checks the links between PostgreSQL, MongoDB and file storage.
3. Stored integration passwords and keys still work.
4. Every required worker queue has a worker reading it.
5. One new upload goes all the way to a finished report under your review settings.

On homelab local-path volumes, keep the backup off the cluster’s machines, because copies of the application do not copy node disks.

No single transaction across stores

A PostgreSQL transaction does not cover MongoDB, Redis or file storage. TestLookup uses several safeguards to lower the risk of a half-finished operation:

• outboxes, lists of follow-up jobs saved with the run so that none is lost;
• IDs that take the source into account, so a repeated delivery is recognised;
• retries, and catch-up jobs that fix anything missed;
• markers for deleted items (tombstones), so that a late job cannot bring them back;
• leases, time-limited claims that a worker holds on a job.

These safeguards still cannot make every side effect happen exactly once. When something fails, check each store and each step that sends work on, one at a time. One successful write does not prove that the whole operation finished.

The sections below come from TestLookup’s own architecture and deployment guides.

Capacity and reliability

Persistence, capacity and models

Valkey (or Redis) is more than a cache. It is the job queue for the background workers (the Celery broker), and it holds live-event streams and temporary status, so a Valkey failure affects all of them. The Compose and in-cluster setups save its data to an append-only file every second (appendfsync everysec). They use the volatile-lru policy, which removes only keys that have an expiry time (TTL) when memory runs low. Even so, some data can still be lost. If you run your own Valkey or Redis, it must save its data to disk and must never remove keys that have no expiry time. The OpenShift page lists all the requirements. Give it enough memory for your event and queue volume, and watch the levels at which TestLookup starts refusing new work. Separate child workers and the database connection budget limit how much work runs at once. If you add copies of a service without recalculating database connections, TestLookup can refuse to start or use up all the database’s connections.

AI model files are not part of the backup. They are kept outside Git, in a Compose volume or the deployment’s storage. After a restore, download them again with make dev-llm, or reload them from an offline model pack.

On K3s, follow the homelab guide to copy backups to another machine.

Reliability limits

Result screens can lag behind an accepted upload, and live totals can appear before the details of each test are saved. The data architecture and design trade-offs guides explain why.

Each extra copy of the API adds database connections, live-update readers and client connections. The number of jobs each worker runs at once must fit the database connection budget and what the AI model can handle. Separate agent_children workers stop parent pipelines from taking every worker slot while they wait for their child investigations. The queues TestLookup sends work to must match the queues the workers actually read, and these differ between install modes. See the deployment guide.

Important limits

• Background work finishes a little later than the request that started it. An accepted upload (HTTP 202), its task status in Redis, the saved run, the finished pipeline, an accepted review and a delivered notification are separate steps. The upload status in Redis expires after 24 hours, so an expired status does not mean the run is gone.

• The checks that limit how much traffic TestLookup accepts can let traffic through when Redis has an error. They help manage traffic, but they are not a firm limit during an outage.

If you run on storage that sits on the node’s own disk, adapt this plan with the homelab guide.