TestLookup is an early version
TestLookup hasn’t had its first release yet. The upcoming 0.1.0 will be the first, and there are no tagged releases so far. Until version 1.0, a new version may change how things work, so read the changelog before you upgrade.
- Expect some bugs. The project fixes problems as it finds them, and later releases will bring improvements and fixes. The roadmap shows what is planned, and the changelog shows what has changed.
- The one-command install uses the latest build. By default it downloads the newest images from the main line of development, not a tested release.
- Support is best effort. Volunteers maintain TestLookup, and one maintainer is listed. There is no paid support and no service-level agreement, as the support policy explains.
- No warranty. TestLookup is open source under the Apache 2.0 license and is provided as is, without warranties, as the README states.
- Security reports get a reply. The maintainers aim to acknowledge a report within 72 hours, send an update within 7 days and fix a confirmed critical problem within 30 days. See the security policy.
If you find a problem, please open an issue on GitHub.
Keeping data safe and consistent
- Accepted isn’t saved yet. TestLookup accepts a report right away and saves it in the background. Check the upload or the run before you rely on it.
- Each store is saved on its own. TestLookup keeps data in PostgreSQL, MongoDB, a job queue and file storage, and no single transaction covers all of them. A retried upload lands in the same run, but not every follow-up step is guaranteed to happen exactly once.
- The job queue can lose a moment of work. Its storage writes to disk every second, which is not a promise of zero data loss.
- Some limits stop working if the job queue fails. If the queue has an error, rate limits and some safety checks let work through rather than block it.
- Backups need planning. The included backup covers PostgreSQL, MongoDB and file storage, but not the job queue, the search index or AI models. On Docker Compose it isn’t scheduled, isn’t copied to another machine and isn’t encrypted. On Kubernetes a daily backup job is included.
- Single-node K3s keeps data on one disk. Extra copies of the app don’t protect you if that disk fails.
Growing to more teams
- Some parts run as a single copy. The scheduler, the MCP server for AI assistants and the process that handles live test results each run as one copy. Under heavy live traffic, the API copy that handles live results slows down.
- One Valkey or Redis. Redis Cluster mode isn’t supported.
- Database connections must be planned. If you add copies of the app or workers without checking the database’s connection limit, TestLookup can refuse to start or run out of connections.
- Very busy live projects keep fewer details. A project that sends at least 1,000 test events a minute for 3 minutes keeps only every second detailed test row. Totals stay exact.
- One copy per machine on Docker Compose. The names and ports are fixed, so a machine can run only one TestLookup.
- No published capacity figures. The maximum number of users or requests hasn’t been measured. See system requirements for what has been measured.
Security and your network
- Docker Compose has no HTTPS. Put a reverse proxy in front of it. The single-node K3s setup uses plain HTTP and is meant for private networks only.
- Some included services have no login. The local AI model server and the search index have no authentication, and the job queue doesn’t use TLS by default. Keep them on a private network.
- Hiding personal data is best effort. TestLookup tries to remove personal data before sending text to an AI model, but can’t catch everything.
- Overloading your own server is out of scope. The security policy treats running out of resources on a self-hosted install as your hardware to manage.
- Single sign-on has gaps. With SAML there is no single logout, and identity providers that require signed login requests aren’t supported.
Offline mode
Offline mode is on by default, and it keeps AI on your own machine or private network. It isn’t a firewall:
- it is a setting, so a wrong setting can switch it off;
- it doesn’t replace network isolation if you need a sealed system;
- AI model files must be downloaded before you seal a system, because the offline bundle doesn’t include them;
- MongoDB isn’t in the offline bundle either, for license reasons, so you bring it in yourself.
If an administrator turns on a hosted AI service, details of your test failures are sent to that provider.
What AI can and can’t tell you
- A confidence score is a guide, not a probability. Treat it as a setting you can tune, not a measured chance of being right.
- AI suggestions can be wrong. TestLookup shows them as suggested causes. By default, AI conclusions below a confidence of 80 out of 100 wait for a person to review them.
- Without a model, analysis is simpler. Rules always work, and the trained model needs 200 labelled failures and 80% accuracy before TestLookup uses it.
- No accuracy figures are published for AI analysis yet.
Features that are off by default
Some features ship switched off, and an administrator turns them on for an installation or a project. A feature marked experimental may change or be removed. Notable ones:
- the review gate, which holds back AI-written reports until a person reviews them;
- fine-tuning AI models on your own data;
- a spending limit for hosted AI. By default there is no limit, so set one if you use a paid provider.
Upgrades
- Upgrades only go forward. To go back, restore a backup made before the upgrade. The upgrade script doesn’t roll back on its own.
- Upgrade one release at a time, as the upgrade notes explain.
- Some parts of the API have no detailed description of their replies yet, so check the API reference and test before you depend on a field.
The full list of known limits and the deployment guide have more detail.