AI configuration guide

TestLookup’s AI features are optional. Which model analyses a run depends on several settings together: the project, the workflow, the provider and the policy. A single environment variable does not decide it on its own.

Modes and providers

TestLookup has four analysis modes:

• rules: fixed rules. The same failure always gets the same answer.
• ml: a classifier trained on failures your team has labelled (machine learning).
• llm: an AI language model.
• auto: TestLookup chooses. The older automatic choice uses a trained classifier if one is ready, then a model it can reach, then the rules.

Each analysis step can also pick a small or a large model that passed evaluation, and TestLookup records why it fell back to a simpler method when it did.

Set the mode and the provider on Settings › AI › AI configuration. QA leads can view this page, and only administrators can save it.

Settings › AI agents sets three things for each agent: its mode, its spending limits and the actions it may take in Act mode. There are three modes:

• Shadow (the default) only logs what the agent would do.
• Suggest proposes an action, and a person decides.
• Act runs approved actions.

The actions Act mode can run are:

• quarantine.apply sets a flaky test aside after someone approves it, so its failures stop blocking the build.
• failure.reassign assigns a failure to the owner of its suite.
• defect.promote files an internal defect.

The Fixer agent has its own settings on the same page. It can run only in Shadow or Suggest mode. It is turned off by default, and its runner is set to none by default. It produces nothing while offline mode (AI_OFFLINE_MODE) is on. The deep investigation overview describes both agents.

ML classifier

The ML classifier learns from failures your team has labelled. It needs no AI language model. These settings control it:

Setting	Default	What it does
ML_MIN_TRAINING_SAMPLES	200	The fewest labelled failures it needs before it trains.
ML_ACCURACY_THRESHOLD	0.80	The accuracy a new model must reach, on labelled failures kept back from training, before it replaces the old one.
ML_HUMAN_LABEL_FLOOR	50	Until people have confirmed or corrected this many results, the classifier shows Bootstrap (LLM-imitating).
ML_RETRAIN_ENABLED	true	Retrains the model every night at 03:10 UTC. An administrator can also press Train now on the ML classifier card.
ML_MODEL_SYNC_ENABLED	true	Shares trained models between servers through file storage. A single server with one shared model folder can set it to false.

Ask AI

Ask AI, the chat assistant, works only when the analysis mode is LLM or Auto. In Rules or ML mode, TestLookup refuses chat requests (HTTP 409 llm_disabled) and hides both the sidebar entry and the Ask AI about this run button. Every chat call counts against the pipeline’s spending limit and cost cap, goes through the circuit breaker that pauses calls to a failing provider, and is charged to the project.

These server settings limit each answer:

• CHAT_MAX_TOOL_ROUNDS (4): how many rounds of lookups one answer may make.
• CHAT_FIRST_TOKEN_TIMEOUT_SECONDS (45): how long to wait for the first words of the answer.
• CHAT_TURN_TIMEOUT_SECONDS (120): how long the whole answer may take.
• CHAT_HISTORY_MESSAGES (12): how many earlier messages are sent again with each question.
• CHAT_LLM_MODEL: a separate chat model on the same provider. When it is empty, chat uses LLM_MODEL.

Local model defaults

In this version, make dev-llm starts the local AI services (the local-llm profile) and runs make pull-llm. That command downloads three models:

• qwen2.5:7b-instruct, the small model.
• qwen2.5:14b-instruct-q5_K_M, the large model.
• nomic-embed-text:v1.5, which turns text into numbers so that TestLookup can search by meaning (embeddings).

TestLookup’s default LLM_MODEL is also qwen2.5:7b-instruct. The previous default, qwen2.5:3b-instruct-q5_K_M, is no longer recommended.

Search by meaning starts with ChromaDB’s built-in MiniLM model. The Ollama embedding model you configured takes over only after an administrator presses Re-embed now on Settings › AI › AI configuration. Download size, disk use and speed depend on your machine and the models you choose.

Feature flags

Feature flags are on/off settings. Deep investigation, knowledge retrieval, evaluation, training and single workflow steps can each have their own flag, for one project or for the whole installation. So a working AI provider does not mean that a feature is on for your project. Administrators manage the flags on Settings › System › Feature flags.

On a new installation:

• llm_cost_budget is on.
• knowledge_rag and rag_faithfulness_gate are off.
• The defect_commander step, which changes data, is off.
• ask_ai_chat is on. Turning it off hides the Ask AI entry, but the chat API keeps working. Ask AI also needs the LLM or Auto mode.

The Ragas evaluator works only on an image built with INSTALL_RAGAS=1, and only while offline mode is off.

The deep investigation has extra steps, and their flags differ between new and upgraded installations. Four of these steps only read data, and they are on for a new installation:

• contract_validation checks API contracts.
• log_intelligence gathers evidence from logs and traces.
• regression_watchman decides whether a failure is a new regression, a known flaky test failing again, or a problem with the environment.
• change_ownership looks at what changed since the build the run is compared against, and who owns it.

On an upgraded installation these four start off. The exception is contract_validation and change_ownership: each one starts on if its matching AIQ_* environment variable was on while the upgrade ran. Until an administrator turns the other steps on, an upgraded installation runs a shorter deep investigation.

Three more flags affect the pipeline:

• aiq_gap_refinement is on for every installation. Its rule-based steps find gaps and refine the report, and they add the “Analysis coverage” line to decision reports.
• async_decision_report_supersession is off. It replaces out-of-date decision reports in the background.
• flaky_auto_quarantine controls the whole quarantine workflow. It is on for new installations, and for upgrades where nobody changed it.

TestLookup reads step flags for each project and keeps them fixed for each pipeline run, even when the run resumes. log_intelligence also needs Splunk, which stays off while offline mode is on.

Spending limits and tool rules

For each run, TestLookup works out all its limits together: tokens, cost, running time, retries, child investigations, how many tasks may run at once, and which tools the AI may use. Settings › AI agents sets each agent’s mode, spending limits and Act actions. Settings › AI › LLM cost budget sets each project’s monthly cap.

TestLookup does not retry an invalid request, a policy refusal, a used-up budget, a rejected review or a cancellation. It can retry network failures and lost leases. A lease is a time-limited claim that a worker holds on a job.

The LLM cost budget limits how much a project spends on AI calls. It works only while the llm_cost_budget flag is on. Administrators edit a project’s budget, and other roles can view it. Its fields are Included per month, Hard cap, Overage rate, Soft-warn threshold (%), At-cap action and Budget enforcement enabled.

• The at-cap action is one of Soft warn, Auto-downgrade to ML (the default), Auto-downgrade to rules, or Hard block.
• No cap applies until the hard cap is above 0.
• Below the soft-warn threshold, nothing happens. Above it, the at-cap action applies. At 100% of the hard cap, calls are always blocked.
• Only the hard cap limits calls. The included and overage figures only feed the spending report on that page.
• Models you run yourself (ollama, lmstudio, localai, vllm) always count as zero cost.

Confidence gate

Automatic actions use an AI conclusion only when its confidence is at or above a threshold. The threshold is AI_CONFIDENCE_THRESHOLD (80 by default), or the Confidence Threshold saved on Settings › AI › AI configuration. A result with no confidence score never passes. A result below the threshold is held and marked “low confidence — needs human review”.

Each pipeline reads the threshold once, when it starts. If you change the value and then press Retry, TestLookup starts a new run. Each analysis method scores confidence in its own way, so treat the threshold as a setting you choose. It is not a measured probability.

Settings recorded per run

When an analysis run starts, TestLookup works out its settings and fixes them for that run: the provider, the model size, the workflow and prompt versions, the tool rules and the spending limits. It also records a hash, a short fingerprint of the input. An automatic retry reuses the same fixed settings. If you change a setting that affects the result and then retry by hand, TestLookup starts a new, linked run and does not reuse saved progress from the old one.

AI evaluation

Settings › AI › AI evaluation checks the quality of the AI itself against answers that people have confirmed. It has six tabs: Overview, Datasets, Gate, Agent stack, Tiers and Reviewer. Administrators run evaluations and quality gates, and QA leads can view every tab.

• A candidate replay sends up to 200 items from a dataset through the real analysis engine of one model size. It counts against the project’s spending limit.
• An item counts as an error if its engine fell back to a simpler method, timed out or failed. It never counts as correct.
• A score with no scored items shows Not measured.
• The gate shows PASS, FAIL or Not enough evidence for each task type.
• In offline mode, only a local provider can be a language-model candidate.
• Pairs of live and candidate results that nobody labelled are deleted after 30 days.

AI agents describes the Workflow editor and Pipeline runs pages.

Offline mode and simpler results

By default, offline mode is on (AI_OFFLINE_MODE=true) and the review gate is off (REVIEW_GATE_ENFORCED=false). They are separate settings, and operators can change either one.

• Offline mode still allows private addresses that pass TestLookup’s checks, such as a model server on your own network. It is not a firewall for the whole server.
• A value saved on Settings › AI can make offline mode stricter. It never loosens what the environment variable sets.
• When TestLookup falls back to a simpler method, the result keeps the same layout but can have less evidence and less depth.
• An installation with no network connection at all needs the images, software packages and model files copied in beforehand. Check that it starts and processes a report with the network blocked.

Once a run is saved, its follow-up work, including the AI analysis, goes into an outbox. The outbox is a list of jobs saved together with the run, so that none is lost. A scheduled task (Celery beat) sends the waiting jobs on every minute. When a live session closes, it sends its job at once. GET /health/ingestion shows how many jobs are waiting, as ai_pipeline.pending_dispatches.

The sections below come from TestLookup’s own pipeline guide. They explain how an analysis pipeline starts, how it picks a model, how it retries and resumes, and how repeatable its results are.

How the analysis pipeline runs

Entrypoints and configuration

A pipeline starts when a run finishes, or when someone starts it by hand. You can also call a single agent through the API. TestLookup then picks that agent’s capability from its catalog, adds the steps it depends on, and runs them in the usual workflow engine. In sync mode, the request still hands the work to a worker and waits for a limited time. There is no second engine for sync calls. An agent call works on something already stored in TestLookup, such as a run. The invoke endpoint does not run any raw data you send, even where the catalog shows an input format. To follow progress, check the invocation from time to time, or get a short-lived ticket that opens its event stream once. See the invocation API reference.

When a pipeline starts, TestLookup works out its settings and fixes them for that run. They include the project’s settings, the provider’s limits and the model size. They also include the tools it may use, its cost, token and time limits, and the workflow version in charge. Changes to settings and workflows go through review and evaluation, and each change gets a new version. To see which model a step used, read the settings recorded on the run and where each one came from. The .env file alone does not choose every step’s model.

Model routing and output

The four modes are described at the top of this page. On top of them, each project and each step can choose a small or a large model, or use a model that passed evaluation. It can also switch to the larger model, or fall back to the rules, and it records why. All modes produce analyses and summaries in the same layout, but they differ in how much evidence they use and where it comes from. Source code: analysis router and model router.

The pipeline reads cleaned-up data about the run, its tests, their history and the evidence. TestLookup records how each result was made, so that you can trace it. It keeps the prompt versions, how it chose the model and settings, and the tool calls. It also keeps the cost, the tokens used, a fingerprint of the input and the expected output of each step. Checking that the AI’s output has the right fields, and repairing broken JSON, only checks its form. It does not show that the content is correct. Two separate controls check the content: rule-based steps that assemble and criticise the report, and a person’s review. Knowledge retrieval (RAG, which looks up your own documents first) is a different workflow from failure classification. It syncs your registered sources, splits them into chunks, finds the relevant chunks, writes a draft based on them, and then evaluates the draft and sends it for review.

Retry, resume, cancellation and deadlines

Inside TestLookup, a pipeline moves through these states: pending, running, retry_wait, completed, passed and failed. The API shows the first three as in_progress. A pipeline that finished with some steps skipped records that in its details and has no status of its own. When a reviewer accepts its report, a completed pipeline becomes passed. A pipeline that produces no report records that a review does not apply. Source code: state machine.

By default, the shared retry policy tries a job at most five times in total. It waits 30 seconds before the first retry, and the wait grows each time up to 600 seconds. It then adds or removes up to 20% at random, so that retries do not all happen at once. TestLookup never retries an invalid request, a policy refusal, a used-up budget, a rejected review or a cancellation. Model, timeout, tool and lease failures can be retried under the policy. Some tasks and steps have their own retry limits, so five attempts is not a rule for everything. Source code: retry policy.

An automatic retry carries on with the settings that were fixed when the run started. A manual retry first compares the current settings that affect the result, and if they changed, it starts a new, linked run. Before TestLookup reuses saved progress from a step (a checkpoint), it checks that the step, its input, settings, version and evidence still match the current pipeline. Copying a step’s output by hand is unsafe. Leases, together with a check that each write comes from the current attempt, stop an older attempt from overwriting newer results. Cancelling asks the running steps to stop at a safe point. TestLookup records the run as failed or cancelled and adds no new status.

By default, a pipeline may run for 1,500 seconds (25 minutes). The last two steps, the decision report and the critic, are rule-based and still run after the time is up, so a run that was cut short can explain what it did not finish. The time until you see a result also includes the outbox relay (a scheduled task that runs every minute), waiting in the queue, gathering evidence, the model’s own work, waiting for child investigations, and retries.

Determinism and evaluation

Fixed plans, fingerprints, exact model versions and prompt versions make a run easier to repeat. They cannot make it give exactly the same answer every time. AI models have some randomness, outside evidence changes, some settings can change while TestLookup runs, and hardware differs. Judge these things separately: the outcome, the quality of each step, the model size, where the evidence came from, and whether a person accepted the result. Training and promoting a new model depend on labelled data, the configured gates and evaluations. Exporting training data on a schedule does not by itself mean that a better model was promoted. See the AI quality guide and testing.

Next, follow the local AI tutorial to set up a local model on a test machine and watch each step work.