Reports and what they count

TestLookup uses the word “report” for several things, and each has its own inputs and rules. Before you read or share a report, check four things: which kind it is, which tests it counts, whether someone reviewed it, and which policy applies.

Report types and downloads

The run page brings together a run’s evidence and its analyses. AI summaries and decision reports record which model wrote them and whether they passed their checks. The project summary adds up a chosen set of tests.

HTML files, PDFs, evidence bundles, compliance packs and release decisions each have their own review, policy and delivery rules. Release decisions, below, lists the possible decisions.

TestLookup also produces these files from the same data:

Product	Where to get it	What to know
Summary PDF and Excel (XLSX)	The ⋯ menu on the Summary page; GET /api/v1/reports/summary/pdf or /xlsx; POST /api/v1/reports/summary/exports	Downloads straight away up to REPORT_EXPORT_SYNC_MAX_TESTS, 200,000 estimated tests by default. A larger report, or one sent to the background, becomes a job. The job is queued, running, completed or failed, and a failed job can be retried. TestLookup keeps the file for 7 days.
Analysis report (one HTML file)	The ⋯ menu on the Summary page; GET /api/v1/projects/{project_id}/reports/analysis with window=1d or 7d (QA engineer or higher)	Self-contained, so you can read it offline. The same file can be attached to a daily or weekly summary email.
Trends PDF and email	The ⋯ menu on the Trends page; GET /api/v1/reports/trends/pdf; POST /api/v1/reports/email-trends	Every role can download the PDF. The one-off email needs a QA lead or higher, goes to at most 10 people, and uses the configured mail server.
Chart exports	Each chart’s Export menu	One chart as a PNG or SVG image, or its data as CSV.
Run AI history	The ⋯ menu on the run page; GET /api/v1/runs/{run_id}/ai-history/export	JSON with every stored version of the run’s analysis and the differences between versions.
Run timing CSV	The Timing tab on the run page; GET /api/v1/runs/{run_id}/timing.csv	Every test in the run, with its timing.

How summary figures are calculated

The summary has two modes:

• window mode counts each test once, using its most recent status in the chosen dates. Only when TestLookup cannot identify the tests does it add up run totals instead.
• latest mode adds up the latest run of each suite (primary_suite_name). Runs without a suite name count separately.

The API defaults to mode=window, for GET /api/v1/reports/summary and the export routes. The Summary page defaults to latest. So pass mode yourself when you compare a figure from the API with the page.

The headline pass rate, pass_rate_pct, is passed ÷ total × 100. Its total includes skipped and unknown results. The weighted pass rate, weighted_pass_rate_pct, is passed ÷ (passed + failed + broken) × 100, which leaves skipped and unknown results out. The table under How the summary adds up results, below, has a worked example.

Filtering by release

When you filter the summary by release (release_id), every figure follows the filter, in both modes:

• the headline totals, with the run count, average duration and latest run;
• the per-suite breakdown;
• the last run before the chosen dates;
• the top failing tests;
• the step figures for each suite.

The one exception is the flaky-test count, which always covers the whole project. A test counts as flaky by its last executions across the project’s history, and the response says so on that field. A suite filter does narrow the flaky-test count.

Review lifecycle

These are separate steps: generating a report, checking it, a person’s review, and the distribution policy. A stored accepted report stays the accepted report, even when a later attempt to generate a new one fails.

Review enforcement is off by default. It starts in either of two ways:

• an administrator sets an enforcement date on Settings › AI › Review gate (/settings/review-gate);
• the server setting REVIEW_GATE_ENFORCED is true.

The Review gate page shows how many runs’ reports would have been held back in the last 30 days, for each channel. If TestLookup cannot read the audit log, the count reads “Not measured” instead of zero. A saved date reaches every server within 30 seconds. The review policy explains each mode.

Exports and share links

Downloads, share links, evidence bundles, compliance packs, notifications and CI results each apply their own rules about what they include and who can see them. A compliance pack exports evidence. It is not a legal or regulatory certification.

A share link works for 7 days by default. Through the API, expiry_days can set 1 to 30 days. You can revoke a link at any time, and TestLookup records every visit in the audit log.

While review is enforced, someone who opens a share link without signing in cannot see an unreviewed AI report. They see a “Report awaiting review” page instead, unless the project allows watermarked drafts. Summary exports check the requester’s project access again, both when the file is generated and when it is downloaded.

Delivery errors

An export can fail after the analysis succeeded. A delivery to another system can fail after the export succeeded. The Export and notification failures section below lists what to check.

Known problems with the pass-rate label

The summary always labels its pass rate as counted per unique test. The field pass_rate_basis is always unique_tests, and its label always reads “per unique test”. That label is right for the usual window mode. It is wrong for latest mode and for the fallback to run totals.

Because latest is the Summary page’s default, the default view shows the wrong label. With a suite filter, the window headline is the sum of the suite rows. It can include run totals for suites whose test rows never arrived. The page’s hint for window mode also says the rate is weighted across executions, although that mode counts each test once.

The flaky-test count uses each test’s ten most recent executions, not the chosen dates or release. Its rate divides by the chosen total, up to 100%. The response states the rule in flaky_criteria:

• the last 10 executions of the test;
• at least 5 of them;
• a share of failures between 10% and 90%;
• at least 2 changes from pass to fail or fail to pass, in run order (min_flips).

The Flaky tests page uses its own weighted score, so its list can differ from this count. When you present these figures, say which mode you used, and give both the number counted and the total. Do not rely on the label alone.

How reports are built and shared

How the summary adds up results

The project summary can count tests in two ways. In window mode, it counts each test once by its fingerprint, using the test’s most recent status in the window. Only when no test result has a fingerprint does it add up run totals instead. In latest mode, it adds up the totals of the latest run of each suite (primary_suite_name) in the window. Runs without a suite name count on their own. Neither mode averages the percentages of separate runs. The code is in the window query and the latest query.

Response field	Calculation	What it means
totals.pass_rate_pct	passed / total × 100	The Pass % headline in the web app. The total includes skipped and unknown results
totals.weighted_pass_rate_pct	passed / (passed + failed + broken) × 100	Counts only tests that passed, failed or broke. Skipped and unknown results are left out
totals.pass_rate_basis	Always unique_tests at this version	Right for the usual window mode, but misleading for latest mode and for the fallback to run totals

Both percentages are rounded to one decimal place, and they read zero when the total they divide by is zero. For 70 passed, 15 failed, 5 broken and 10 skipped, the headline is 70.0% and the weighted field is 77.8%. See the service, the tiles in the web app and the tests.

Release filter: filtering by release works as described under Filtering by release, above. The flaky-test count is the same figure that the dashboard reads, and that the release gate checks against its flaky-test limit. The pass-rate label still always reads unique_tests, as the table shows. See the route, the service and the test for the release filter.

The flaky-test count is neither an all-time count nor a rate for the selected dates. The section Pass-rate label limitations, above, explains the flaky calculation. When you present these figures, keep the field names, the mode and what was counted. The overview dashboard counts every execution, so its figures can rightly differ from the summary, which counts each test once.

While TestLookup finishes a run, the run’s totals can appear before its test rows. The per-suite breakdown reads test rows, while the headline can use run totals, so the two can briefly disagree.

How a report is reviewed and shared

A report goes through these steps:

1. TestLookup gathers the stored evidence for the run or project.
2. It writes or assembles the report.
3. It checks the report’s structure, and a separate rule-based checker reviews the report before it goes further.
4. It opens a review request for a person.
5. The report ends up accepted by a person, pending review, or rejected or replaced by a newer one.
6. The distribution policy decides what may leave TestLookup.
7. The report goes out as a download, a share link, a notification or a result for your CI system.

While enforcement is off, which is the default, TestLookup records in the audit log each report it would have refused, and still sends it. While enforcement is on, TestLookup sends accepted reports and output that is not AI-generated. A QA lead or administrator can still export a pending draft as a run PDF with include_unreviewed=true. That PDF carries a draft watermark, and the export is recorded in the audit log. Rejected or replaced reports cannot use this exception. HTTP responses can carry the X-TestLookup-AI-Generated and X-TestLookup-Review-State headers, so a download keeps its review state. See the distribution policy code.

Share links have their own token, access and expiry rules, and they still follow the report policy. A compliance pack, a signed bundle of evidence for auditors, is an export. Generating one is not a legal compliance certification.

Release decisions

A single run’s decision and the decision for a whole release, or for one of its phases, are related but separate. A run’s decision can be GO, Conditional GO or NO-GO. A release or phase decision is GO, NO-GO or Not evaluated (GO, NO_GO or NOT_EVALUATED in the API). The stored release decision also allows CONDITIONAL_GO, but nothing in this version records it. Not evaluated is a real result. It means TestLookup could not decide, for example because too few tests have results, and it does not count as a pass. While review is enforced, a decision shown outside TestLookup reads PENDING_REVIEW until its report is reviewed.

Each release, and each phase, has one current decision. TestLookup keeps the earlier decisions as history, each with the policy, the evidence, the number of tests counted and the reasons it used. A new evaluation adds a decision and keeps that history. Always check which release, phase, policy version, set of runs and environment a decision describes, and which earlier release it is compared with.

Export and notification failures

Export generation can fail separately from analysis, and delivery can fail separately from generation. Long exports run in the background. Check each of these on its own: storage, errors from the file renderer, whether the report exists, review refusals, the policy for the destination, and the retry and dead-letter records. Replaying an old webhook does not get around review, because every delivery checks the current policy again.

See an example run summary, with made-up data, on the dashboards and reports page.