Connect a CI job | TestLookup

Prerequisites

This tutorial connects a continuous integration (CI) job to TestLookup. CI is the service that builds and tests your code on every change, such as GitHub Actions or Jenkins. You need:

A TestLookup server that your CI machines can reach, and a test project on it. If you haven’t sent a report before, do Import your first report first.
A key for that project, and the project’s ID. The secrets step below shows where to keep them.
Python 3.11 or newer, and jq, a small tool for reading JSON data, on the CI machine.
A test framework that already writes a report file, for example JUnit XML.

This tutorial sends a finished report after the tests have run. To send results live while the tests run, you use separate reporters, which follow their own session rules. See the client reference.

Client choice

This tutorial uses the TestLookup command-line tool, called testlookup. TestLookup also has a Python reporter and clients for JavaScript, Java and Go, and they offer different commands.

Before you swap in another client, check the client reference. Not every client supports every command for uploads, live results, progress checks or pass and fail decisions. The integration map on GitHub lists what each one does.

The command-line tool, the code libraries (SDKs) and the MCP server aren’t published to a package registry. The MCP server lets AI assistants use TestLookup. Install the command-line tool, version 0.4.0, from a copy of the code, as its README on GitHub describes:

git clone https://github.com/test-intelligence/testlookup.git
python -m pip install ./testlookup/cli
testlookup --version

To stay on one exact version, check out a specific commit in your copy before you install. Then follow the README from that same commit.

If you’d rather send results live, use a reporter. Reporters exist for pytest, Robot Framework, Jest, Mocha, Playwright, Vitest, Cypress, Cucumber-JS, TestNG, JUnit 5 and Go’s built-in testing package. The Cypress reporter sends results for one spec file at a time.

Store the secrets

Store three values in your CI system’s protected secrets:

Secret	Value
`TESTLOOKUP_URL`	The server’s API address, for example `https://testlookup.example.com`
`TESTLOOKUP_API_KEY`	The key
`TESTLOOKUP_PROJECT_ID`	The project ID

The command-line tool reads the address and the key from the first two values. You pass the project ID yourself with an option.

Details: the tool sends the key in the X-API-Key header. You pass the project ID with --project.

Keep the key secret:

Don’t print it.
Don’t write it into the workflow file.
Don’t pass it on through a build output file.
Don’t let untrusted code from a pull request see it.

Use a key that’s bound to the project and has a filled-in list of permissions, called scopes. A list with entries limits what the key can do.

Older keys with an empty list can have wider access, so never treat them as having no access. Replace or review them before you use them in CI.

The key’s owner needs the QA engineer role or higher to send results.

Keep retries in one run

Give each submission a fixed identity. Use the build number, the commit ID, the branch and, where you can, the link to the CI run.

The tool picks up some of these details by itself from GitHub Actions, GitLab CI, Jenkins, Azure DevOps and CircleCI. It detects the CI system, the repository, the pull request, who started the job and the link to the run.

It can’t detect the build number, commit ID or branch, so pass those yourself. Anything you pass replaces what the tool detects.

When a job retries the same submission, reuse the same build number. Then every attempt ends up in the same run, instead of creating look-alike runs.

Use the commit ID, not only the branch name. Don’t make a new build number for each network retry.

By default, the upload also sends details about where the tests ran:

fingerprints of your dependency lock files and container images;
the runtimes;
environment values from an approved list.

One option turns those details off, and another skips the AI analysis of the run.

Details: pass the build number with --build, which is required, the commit with --commit and the branch with --branch. --no-run-context leaves out the extra details, and --no-ai skips the AI analysis.

Upload the report

After your tests have written their report, upload it and keep the run ID that comes back:

RUN_ID=$(testlookup upload file results.xml \
  --project "$TESTLOOKUP_PROJECT_ID" \
  --build "$BUILD_ID" \
  --commit "$COMMIT_SHA" \
  --wait -o json | jq -r '.run_id')

Fill in the build ID and the commit ID from your CI system’s own variables. The accepted upload also returns a task ID. Keep both IDs in the job’s log or details.

The workflow guide on GitHub explains why. An accepted upload starts work in the background. It isn’t a pass or fail decision.

While you set this up, use a report with made-up results. Stay within the format and upload limits. Let the job fail clearly when the tool or the server refuses the report.

Wait for the results

With the wait option, the tool keeps checking the upload until the server has saved the report. The command fails if the server refuses the report, for example because the file is empty or can’t be read. By default, it waits up to 600 seconds.

Without the wait option, the command returns as soon as TestLookup accepts the upload.

Details: --wait turns waiting on, and --wait-timeout sets the time limit in seconds. A refusal ends the command with a non-zero exit code.

TestLookup can save a run before its analysis or report is finished. If a later step depends on the analysis or the review, check for that too, with a time limit.

Report each problem as what it is: a failed task, rejected results, skipped analysis or a pending review. Don’t lump them together as one general job failure.

Pass or fail the job

Then judge the run:

testlookup ci-verdict --run "$RUN_ID" --project "$TESTLOOKUP_PROJECT_ID"

The ci-verdict command looks at the run’s failed and broken tests, and at the project’s quarantine list. Quarantine sets a flaky test aside so its failures don’t block the build. A flaky test is one that sometimes passes and sometimes fails with no change to the code.

The command ends with an exit code, a number that your CI system reads. 0 means OK, and anything else fails the job:

Exit code	Meaning
0	No real failures. Quarantined failures don’t block.
1	At least one real failure that isn’t quarantined, or any failure at all with `--strict`.
2	An API or network error, or the run was still in progress when `--timeout-seconds` ran out.

Exit code 2 fails the job unless you add --fail-open.

By default, TestLookup proposes flaky tests for quarantine automatically. But nothing is quarantined until a QA lead approves a proposal. So at first, every failure counts as real.

The ci-verdict command isn’t a release decision, and it doesn’t replace report review. To pass or fail a job on a release, use the release gate command. It reads the release’s saved decision:

testlookup release gate <release>

Exit code	Meaning
0	GO, or Conditional GO unless you pass `--fail-on CONDITIONAL_GO`
1	NO-GO
3	No decision to act on: Not evaluated, or never evaluated
2	An infrastructure error, or a release name that’s unknown or matches more than one release

The table includes Conditional GO because the command can handle it. At this version, though, TestLookup gives a whole release only GO, NO-GO or Not evaluated.

Exit codes differ between commands, so check each command’s own table.

Details: --evaluate records a fresh decision first, and needs a QA lead or a project API key. --json prints the release ID and the gate result.

Here’s a minimal GitHub Actions step. Place it after the steps that install the tool and run your tests:

- name: Send results to TestLookup
  env:
    TESTLOOKUP_URL: ${{ secrets.TESTLOOKUP_URL }}
    TESTLOOKUP_API_KEY: ${{ secrets.TESTLOOKUP_API_KEY }}
    TESTLOOKUP_PROJECT_ID: ${{ secrets.TESTLOOKUP_PROJECT_ID }}
  run: |
    RUN_ID=$(testlookup upload file results.xml --project "$TESTLOOKUP_PROJECT_ID" \
      --build "${{ github.run_id }}" --commit "${{ github.sha }}" --wait -o json | jq -r '.run_id')
    testlookup ci-verdict --run "$RUN_ID" --project "$TESTLOOKUP_PROJECT_ID"

For each result, write down which run, which quarantine state and which version of the tool produced it.

Retry limits

The tool’s upload already retries when the server is busy or has hit a limit (HTTP 429 or 503). It waits for the time the server gives in its Retry-After header.

If there isn’t one, it starts with half a second and waits longer each time, for up to five minutes in total. If you write your own client, do the same.

Limit retries for network errors and progress checks.
Keep the original build identity.
Retry a request only when its client or API documentation says that’s safe.

A fixed build identity doesn’t make every other request that sends data safe to repeat.

If TestLookup couldn’t read the file or refused your permissions, fix the cause before you retry. Keep a log of each attempt, without the key.

Cleanup

When you finish the tutorial, collect the run ID, the task ID, the result and any failure details. Then:

Revoke the temporary key.
Remove the test-only CI secrets and build output files.
Delete the test project if you no longer need it.

What a recorded run showed

This run was recorded on September 21, 2026, with an earlier development version of TestLookup, before the release gate command existed. Details may differ on later versions.

The tool ran in a throwaway container against a throwaway TestLookup on the same machine, not on a hosted CI service. The upload named the TestLookup repository, a commit and a build named after that commit. Checking the upload ended with 2 results accepted, 0 rejected, 1 passed and 1 failed.

The upload command succeeded, with exit code 0, because TestLookup saved the results. Then ci-verdict returned fail, with exit code 1: one real failure, none quarantined. When you adapt the workflow, write down your own CI system and machine details.

Read the client reference.