Application deployment guide

To install TestLookup:

1. Pick an install mode.
2. Prepare its secrets and storage.
3. Run the database updates (migrations).
4. Start the application.
5. Check it with the health checks.

If you have not chosen a mode yet, start with Pick a deployment mode further down this page. Each mode has its own guide with the step-by-step commands.

Install modes and what you need

You can install TestLookup in several ways:

• with Docker Compose on one machine;
• on Kubernetes;
• on OpenShift, Red Hat’s version of Kubernetes;
• on a single cloud VM;
• on K3s (see the homelab guide);
• as an offline install, with no internet connection.

Compose needs Docker Engine or Docker Desktop with Compose v2, and installing from a clone also needs Git, GNU Make and Bash. The system requirements page lists what you need in more detail.

Whichever mode you pick:

• Choose one exact application version, and use images whose tag never changes.
• Provide PostgreSQL, MongoDB, a Redis-compatible server, S3-compatible file storage, enough capacity, a way in for web traffic (ingress) and a trusted network.
• Keep the web app, API, MCP server and workers on the same version.

The bundled Redis-compatible server is Valkey 8.1, and the bundled S3-compatible file store is versitygw.

Each published image is tagged with the ID of the commit it was built from (:<commit-sha>), and :latest moves with every change on main. A :vX.Y.Z tag exists only after a tagged release. Use a commit ID or a release tag instead of latest.

Per-mode guides

Each install mode has its own guide on GitHub with the commands:

• Compose: the demo and development stack, the smaller dev-lite stack, release images without a clone, the setup for sites with no internet connection, and the monitoring add-on.
• Kubernetes: the base and the general-purpose overlays.
• K3s: one node or more. The homelab guide on this site walks through it.
• OpenShift: the standard setup, and a setup with no internet connection that pulls everything from one Artifactory server.
• Single VM and Cloud Run.
• Managed clouds: EKS with Terraform, GKE and AKS.

Sizing

The sizing guide on GitHub gives three starting profiles. A test result here means one result for one test.

Profile	Test results a day	Where it runs	CPU	Memory	Disk for one year
Small	Up to 50,000	One Compose host	4 vCPU	8 GB	100 GB SSD
Medium	Up to 500,000	One large host or a small Kubernetes cluster	8 vCPU	16 GB	500 GB SSD
Large	1 million or more	Kubernetes	16 or more vCPU (estimate)	32 GB or more (estimate)	1 TB or more of NVMe (estimate)

These figures exclude a local AI model. The local model add-on adds about 4 vCPU, 8 GB of memory and 10 GB of disk for the default qwen2.5:7b-instruct. make pull-llm also downloads qwen2.5:14b-instruct-q5_K_M and the embedding model, which need more disk.

The profiles are starting points from the shipped configuration, not results of load tests, and the large profile and the growth rates are estimates. After a week of real use, measure your own volume. make preflight prints the current volume sizes.

Air-gapped installs

An air-gapped install runs on a network with no internet connection at all. The air-gapped install guide on GitHub explains the steps:

1. On a machine with internet access, build an offline bundle with make offline-bundle.
2. Copy the bundle to the install site and check it.
3. Import it with scripts/release/import-bundle.sh. The script refuses a bundle whose checksum does not match, loads the images and renames them under one registry.
4. Deploy on Compose, Kubernetes or OpenShift.

The bundle holds the application, PostgreSQL, Valkey, versitygw and busybox images with their licence texts. It also holds software bills of materials (SBOMs) when the build machine has syft installed. MongoDB is not in the bundle, and the Ollama and ChromaDB images are added only with --with-llm.

Secrets and TLS

Supply the production secrets through your install mode’s own secret store: the application, sign-in token (JWT), webhook, database, file-storage, integration and AI provider secrets. Required secrets by mode below lists them.

Before you open TestLookup to other users, set up TLS, trusted proxies, public addresses, certificate checks and key rotation. If your company has its own certificate authority, set HTTP_CA_BUNDLE instead of turning certificate checks off. A secret setting can also point to an entry in Vault or OpenBao when the installation is set up for it.

Database updates (migrations)

Before you send traffic to a new version:

1. Back up PostgreSQL, MongoDB and file storage.
2. Run the database update job (migration) with the matching backend image.
3. Check that the database reached the expected migration version.

In this version, the latest migration is 0255. A ready API does not show that the migrations finished.

Upgrading an installation

Upgrade one release line at a time, to the latest patch of the target line. Read the upgrade notes for every version you pass through. On Compose, from a clone:

1. Run make backup.
2. Run make preflight TAG=vX.Y.Z. It checks the image tags, the pending migrations and the free disk space.
3. Run make upgrade TAG=vX.Y.Z. It pulls or rebuilds the images and runs alembic upgrade head before it replaces the application. It then restarts the services in order and runs a quick smoke check. It does not roll back on its own, so the backup from step 1 is your way back.

Release-image Compose sets TESTLOOKUP_VERSION, and its db-migrate service runs the migrations first. Kubernetes runs the migrations before the rollout, with scripts/run-k8s-migrations.sh or make k8s-migrate. On K3s, run this from the repository root (or make k8s-deploy-homelab-update):

homelabsetup/deploy-homelab.sh --skip-registry --skip-models

Finish by checking that /health/ready returns ok and that /health/version reports the new commit. On Compose the backend answers on port 8000:

curl http://localhost:8000/health/ready

Workers

Start a worker for every required queue and exactly one scheduler (beat). Check that the agent_children queue and every configured ingestion shard queue have a worker reading them. Count the API and worker processes in the PostgreSQL connection budget.

TestLookup enforces that budget. The shipped configuration assumes:

• PostgreSQL allows 400 connections (max_connections), with 50 kept in reserve for maintenance work.
• Each API process uses 2 pooled connections plus 1 extra when busy.
• Each worker process uses 1 pooled connection plus 1 extra.

With APP_ENV=production, the API refuses to start when the database offers fewer connections than PG_FLEET_MAX_CONNECTIONS (400 by default). Compose also runs a db-budget-check service before the application starts. Before you add copies of a service or raise how many jobs each worker runs, recalculate the budget with the connection-budget guide on GitHub.

AI model files

A local AI model’s needs for memory, graphics memory (VRAM), disk, context length and parallel requests depend on the models you choose. The upgrade notes put the default qwen2.5:7b-instruct at about 5 GB of disk and 6 to 8 GB of memory while it runs. Model files (weights) and embedding files are kept outside Git.

An offline installation needs its images, software packages and model files copied in and checked before you cut it off from the network. The offline bundle never includes model files. Build an offline model pack instead:

1. On a machine with internet access, pull the exact LLM_MODEL and EMBEDDING_MODEL versions.
2. Archive the Ollama model folder and write a SHA-256 checksum file.
3. After you copy the pack across, check it with sha256sum -c.
4. Restore it into the Compose volume, or into the K3s or OpenShift volume claim.

Check the installation

Check these health addresses:

• /health/live: the API process is running.
• /health/ready: the services it needs are ready.
• /health/version: the version that is running.
• /health/details: the state of each service. It can return HTTP 200 while one of them is only partly working, so read the response.

Then check that workers are taking jobs from the queues, that the database is at the expected migration, and that metrics are being collected. Finally, send one real report all the way through to a finished report, under your own review and integration settings.

Going back to an earlier version

Before an upgrade, keep the IDs of the previous images and a matching backup of PostgreSQL, MongoDB and file storage, plus the keys and a recovery plan. Valkey (or Redis) and model files are not in the backup. You rebuild or reload them instead; see Backups below. Going back to an older database version, or restoring a backup, can destroy data, and it is not safe after a migration that changed data.

Moving an installation from before 0.1.0 also needs the upgrade notes:

• Valkey cannot read the old Redis data.
• versitygw starts empty until you copy the stored files into it.
• ChromaDB collections must be rebuilt.

The same notes cover these changes:

• MongoDB is no longer in the offline bundle.
• There is a new default model.
• The manual_upload flag.
• Administrator accounts: there are no default passwords, and an existing administrator is never changed.
• A release install from before 0.1.0 that kept a development .env with passwordless sign-in.
• The homelab site configuration moved to TESTLOOKUP_HOMELAB_ENV.
• The new image locations.

The rest of this page is TestLookup’s own deployment guide. It has a table to help you pick a mode, the services every installation runs, the secrets each mode needs, TLS, start-up order, backups and health checks.

Full deployment guide

TestLookup ships one set of images (backend, frontend and MCP server) and several ways to run them. This guide helps you pick a mode and lists what each mode needs. It also covers the tasks that every mode shares: secrets, TLS, backups, start-up order, upgrades and health checks. Each mode has its own guide with the commands.

Whichever mode you choose, check your own installation with the health checks at the end of this guide.

Pick a deployment mode

Mode	What you run	Datastores	Typical use	Guide
Compose: demo and development	docker-compose.yml via make quickstart, make dev, make dev-llm	In Compose	Evaluation, local development	Compose
Compose: dev-lite	docker-compose.dev-lite.yml via make dev-lite	In Compose	Low-resource laptops	Compose
Compose: release images, no clone	install.sh downloads docker-compose.release.yml and pulls prebuilt images	In Compose	A single host running published images	Compose
Compose: air-gapped	docker-compose.release.yml + docker-compose.airgap.yml	In Compose	Hosts with no internet access	Compose, air-gapped install
Compose: monitoring add-on	docker-compose.monitoring.yml layered on the stack	n/a	Prometheus, Grafana and Jaeger next to Compose	Compose
Single VM (GCP)	docker-compose.yml + docker-compose.gcp-vm.yml	In Compose on the VM	A small team on one cloud VM	Single VM
Cloud Run templates	infra/cloudrun/*	Managed services you provide	Serverless API, web app and MCP server, with managed data services	Single VM and Cloud Run
Kubernetes: generic overlays	k8s/base + k8s/overlays/{dev,staging,prod,self-hosted}	Provided by you (managed or in-cluster)	Any conformant cluster	Kubernetes
Kubernetes: K3s (one node or more)	k8s/overlays/homelab + homelabsetup/deploy-homelab.sh	In-cluster	One machine, plain HTTP on a private network	Homelab guide
OpenShift	k8s/overlays/openshift	Provided by you (Valkey or Redis must save its data to disk)	OpenShift with Routes	OpenShift
OpenShift + Artifactory	k8s/overlays/openshift-artifactory + openshiftsetup/	In-cluster	Air-gapped OpenShift that pulls from one registry	OpenShift
Offline bundle	make offline-bundle, scripts/release/import-bundle.sh	Compose, Kubernetes or OpenShift	Sites with no internet connection at all	Air-gapped install
AWS EKS	infra/aws/ (Terraform) + k8s/overlays/aws-eks + scripts/deploy-k8s.sh	RDS, DocumentDB, ElastiCache, S3	AWS with managed data services	Managed clouds
Google GKE	k8s/overlays/gcp-gke + scripts/deploy-k8s.sh	Provided by you	GKE with Google-managed certificates	Managed clouds
Azure AKS	k8s/overlays/azure-aks + scripts/deploy-k8s.sh	Provided by you	AKS with NGINX ingress	Managed clouds

Rules of thumb:

• Trying TestLookup: run make quickstart from a clone, or use the one-line install.sh command shown in the quickstart if you do not want a clone.
• One team, one host: release images on Compose, with backups (make backup) on a schedule.
• Several teams, or you need several copies of each service and upgrades without downtime: Kubernetes with managed PostgreSQL, MongoDB, Valkey or Redis, and S3-compatible storage.
• No internet at the install site: the offline bundle. Models are prepared separately with the offline model pack.

The sizing guide gives the hardware for each profile (small, medium and large) and for the local AI model add-on.

What every installation runs

Component	Image or service	Notes
Frontend	testlookup/frontend	nginx serving the React app; proxies /api/, /ws/ and /webhooks/ to the backend
Backend API	testlookup/backend	FastAPI on port 8000; Swagger at /api-docs
Workers and scheduler	testlookup/backend running Celery	Queues critical, ingestion (plus shard queues), ai_analysis, default, and the isolated agent_children queue; one beat scheduler
MCP server	testlookup/mcp	SSE on port 8002 (bearer-authenticated) or stdio
PostgreSQL 16	postgres:16-alpine	The main database, which holds the official records
MongoDB 7	mongo:7	Evidence, summaries and decision documents. MongoDB is licensed SSPL-1.0 and is not included in the offline bundle; provide it yourself there
Valkey 8	valkey/valkey:8.1-alpine	Job queue and results for the background workers (Celery), live-event streams and caches. The settings keep their REDIS_* names. Any Redis-compatible server that saves its data as described below works
S3-compatible object store	ghcr.io/versity/versitygw (Apache-2.0)	Uploads, reports, exports, knowledge documents. Settings keep the MINIO_* names and point at any S3-compatible endpoint
Ollama (optional)	ollama/ollama	Local models. Default chat model qwen2.5:7b-instruct, embeddings nomic-embed-text:v1.5
ChromaDB (optional)	chromadb/chroma	Stores the number summaries (vectors) used for search by meaning and knowledge retrieval

deploy/images.manifest.txt is the one list of every image and tag. make images-check fails when Compose, Kubernetes or the mirror scripts no longer match it.

Required secrets by mode

The backend refuses to start with APP_ENV=production or staging when APP_SECRET_KEY, JWT_SECRET_KEY or WEBHOOK_SECRET is still a placeholder, when the password inside MONGO_URI is a placeholder, or when DEV_AUTO_LOGIN_ENABLED is true (config checks). Generate each secret with openssl rand -hex 32.

Mode	Where secrets live	Keys you must set
Compose (all variants)	.env next to the compose file. make quickstart generates it with scripts/gen-dev-env.sh (development mode); install.sh uses scripts/gen-dev-env.sh --release (production mode)	POSTGRES_PASSWORD, MONGO_PASSWORD, MINIO_ACCESS_KEY, MINIO_SECRET_KEY (Compose refuses to start without them), FLOWER_PASSWORD (source-built stack), APP_SECRET_KEY, JWT_SECRET_KEY, WEBHOOK_SECRET; REDIS_PASSWORD turns on broker authentication; INITIAL_ADMIN_PASSWORD for the first admin of a release install; GF_SECURITY_ADMIN_PASSWORD for the monitoring add-on
Single VM (GCP)	.env copied from .env.gcp-vm.example	The Compose keys above plus GCP_PROJECT_ID and IMAGE_TAG (required by docker-compose.gcp-vm.yml)
Kubernetes, all overlays	Secret testlookup-secrets in the target namespace	APP_SECRET_KEY, JWT_SECRET_KEY, DATABASE_URL, POSTGRES_PASSWORD, MINIO_ACCESS_KEY, MINIO_SECRET_KEY, MONGO_URI, WEBHOOK_SECRET; optional provider and integration tokens. See Kubernetes secrets
OpenShift + Artifactory	Created by deploy-openshift-artifactory.sh on first install	Generated; you supply only the Artifactory credentials in artifactory.env
EKS via Terraform	AWS Secrets Manager (written by Terraform)	Synced into testlookup-secrets before deploy

Never commit .env, openshiftsetup/artifactory.env or a filled-in Secret manifest. Losing APP_SECRET_KEY makes saved encrypted credentials (integration tokens) unreadable even after a database restore, so back it up with your secrets store.

The first administrator

A production-mode installation has no passwordless login. The script scripts/create_admin.py creates the first ADMIN user on a database that has none, and it never changes an existing admin. If you supply no password, it generates one and prints it once. The new admin must change that password at the first sign-in.

• Release install (install.sh): the installer prints the admin login once; the password is stored in .env as INITIAL_ADMIN_PASSWORD, and the db-migrate service creates the user admin with it on first start. See release images.
• Source-built Compose: make create-admin runs the script in the backend container. The local demo stack (make quickstart, make dev) runs in development mode and uses passwordless Quick Login instead.
• Kubernetes and OpenShift: scripts/deploy-k8s.sh and openshiftsetup/deploy-openshift-artifactory.sh create the admin at first install and print the generated password once; make create-admin-k8s runs the script by hand.

TLS and ingress

Mode	How traffic reaches TestLookup	TLS
Compose	Frontend on host port 3000 (source build) or 3000 mapped to nginx port 80 (release images); backend on 8000; MCP server on 8002, reachable only from the same machine	None built in. Put a reverse proxy (nginx, Caddy or Traefik) in front and let it handle HTTPS
Single VM	Frontend nginx on port 80	Add a reverse proxy or load balancer with a certificate before exposing it publicly
Kubernetes prod, self-hosted and azure-aks	NGINX Ingress testlookup-ingress (/api/, /api-docs, /ws/, /webhooks/, /health/, /metrics to the backend, / to the frontend) and testlookup-mcp-ingress on its own host	cert-manager annotation cert-manager.io/cluster-issuer: letsencrypt-prod, Secrets testlookup-tls-cert and testlookup-mcp-tls-cert
GKE	GCE Ingress	Google-managed certificate (managed-cert.yaml)
EKS	AWS Load Balancer Controller (ALB)	ACM certificate ARN annotation
OpenShift	Routes that handle HTTPS at the router and redirect HTTP to HTTPS	Router default wildcard certificate, or your own PEM files
K3s (homelab)	Traefik on plain HTTP	Private networks only; see the homelab guide

Behind any proxy, set CORS_ORIGINS and PUBLIC_BASE_URL to the address users open. Set TRUSTED_PROXY_IPS to the proxy’s address range only, and never to *. The API uses it to find the real client address for rate limits and the activity log. The overlays include commented-out defaults for each platform.

Startup and upgrade order

1. Choose an image tag or image ID that never changes, and fix your configuration. Supply the production secrets, addresses, TLS and trusted-proxy settings. Keep the web app, API and MCP server on matching versions.
2. Set up the services that keep data, and check they have enough capacity. The PostgreSQL connection budget must count the API and worker processes, the reserved connections and the connection the migration needs. See database connection budget.
3. Run the database update job (migration) up to the target version, with the matching backend image. On Compose this is the db-migrate service, and on Kubernetes it is scripts/run-k8s-migrations.sh. A ready API does not show that the migration finished.
4. Start the API, a worker for every required queue and exactly one scheduler (beat). Check that the agent_children queue and every configured ingestion shard queue have a worker reading them.
5. Check the ingress paths, /health/version, readiness and metrics. Then send one report all the way to a finished report, and check that integrations behave as expected under your real settings.

On Compose, run make backup, then make preflight TAG=..., then make upgrade TAG=.... Together they follow this order. Read the ops scripts first. upgrade.sh pulls the new images (release stack) or rebuilds them (source stack), and runs alembic upgrade head before it replaces the application. It then restarts the services in order and runs a quick smoke check. It does not roll back on its own. The backup you took before the upgrade is the way back. After a migration that changed data, going back one database version is not a safe way to undo a release. Keep your backups and the IDs of the previous images.

Version-specific steps, such as moving an existing install from Redis to Valkey, are in the upgrade notes (UPGRADING.md). To change LIVE_INGEST_SHARD_COUNT, change the services that send work and the workers at the same time, following the ingestion shard count change runbook.

Backups

Mode	Tooling	What it covers
Compose	make backup (QUIESCE=1 stops the app first, BACKUP_DIR sets the destination) and make restore FILE=... [FORCE=1] [CONFIRM=yes]	PostgreSQL dump, MongoDB archive, object-store data and a manifest with the Alembic head, in one .tar.gz
Kubernetes	The backup component: a daily CronJob to a PVC plus an on-demand restore Job	Same three stores; enabled in the K3s and OpenShift + Artifactory overlays, opt-in elsewhere
Managed services	Your provider’s snapshots and point-in-time recovery	Back up PostgreSQL, MongoDB and the bucket together

Valkey (or Redis), ChromaDB and Ollama are left out of the archive on purpose. They can be rebuilt, and replaying an out-of-date job queue after a restore is dangerous. Back up PostgreSQL, MongoDB and file storage together so that the links between them stay correct. Copy the archives to another machine, and test a restore. For step-by-step backup and restore, see storage and backups and the administration guide.

Persistence, capacity and models

Valkey (or Redis) is more than a cache. It is the job queue for the background workers (the Celery broker), and it holds live-event streams and temporary status, so a Valkey failure affects all of them. The Compose and in-cluster setups save its data to an append-only file every second (appendfsync everysec). They use the volatile-lru policy, which removes only keys that have an expiry time (TTL) when memory runs low. Even so, some data can still be lost. If you run your own Valkey or Redis, it must save its data to disk and must never remove keys that have no expiry time. The OpenShift page lists all the requirements. Give it enough memory for your event and queue volume, and watch the levels at which TestLookup starts refusing new work. Separate child workers and the database connection budget limit how much work runs at once. If you add copies of a service without recalculating database connections, TestLookup can refuse to start or use up all the database’s connections.

How much memory, graphics memory (VRAM) and disk a local model needs depends on the model, how much it is compressed (quantization), the context length and how many requests run at once. Measure it on the machine you will use. Model files are kept outside Git, in a Compose volume or the deployment’s storage. make dev-llm downloads the models, so it needs internet access. A site with no internet connection loads them from an offline model pack. Offline mode (AI_OFFLINE_MODE=true, the default) blocks hosted AI providers and other outgoing AI calls, but still allows local services such as Ollama. It does not replace cutting the server off from the network.

On K3s, local-path volumes keep data on one machine’s disk. Extra copies of the application do not protect you if that disk fails, so keep backups on another machine.

Observability and troubleshooting

Signal	What it tells you
/health/live	The API process can answer. Do not restart other services based on this alone
/health/ready	PostgreSQL, MongoDB and Valkey (or Redis) are ready. A 503 takes the pod out of traffic
/health/version	The version, commit, build and environment
/health/details	A short report on each service it depends on. It can return HTTP 200 while some of them are only partly working
/metrics	Counters and gauges for HTTP requests and TestLookup’s own work, when METRICS_ENABLED is on. Queue lengths are refreshed each time the metrics are read
Worker and beat logs, Flower	Which jobs were sent, picked up and retried. On Compose, Flower listens only on the local machine
Project activity and decision trail	What users did, and the reasons and sources behind each agent’s work
Upload, outbox, dead-letter queue and pipeline records	Which background step failed, beyond what HTTP health shows

Prometheus alert rules live in infra/monitoring/prometheus-rules/; Compose loads them through the monitoring add-on and Kubernetes through k8s/monitoring. Set alerts for pipelines that stop moving, and for leases or retries that run out. Also alert on jobs piling up in the outbox or the dead-letter queue (a holding area for messages that kept failing), uneven queues, database connections and the job queue’s memory. Installing an OpenTelemetry library does not on its own trace the Celery tasks. See the monitoring guide and the troubleshooting page.

Related

• Architecture overview and deployment topologies
• Local development for working on the code directly on your machine
• Configuration reference (built from the settings code) and worker operations

Next, read day-two operations for regular checks and recovery.