Why the run is red when every test is green

Understand failuresLesson 6 of 9About 7 minutes

You will

• Read the gate row that a finding feeds
• Explain why the test list stays green while the run exits 1
• Tell an Error finding from a Warning finding

You need

• The previous lesson, findings
• The sample cloned for the failing teardown step. Reading the report alone also works.

The problem

Every test passed and the CI job is red. The runner summary says Passed: 1, and the process still exits 1.

The difference is a run gate. A run gate is a check over the whole run, not over one test. It runs once after the last test, and a failed gate fails the run.

Do it

1. Find the gate

The sample registers one gate over its findings:

Setup.cs3 notes

1.AddRunGate("no error findings", context => context1
2.ItemsOfKind(ProtoReportItemKinds.Finding)2
3.Any(item => item.Status == ProtoReportStatus.Error)
4? ProtoRunGateResult.Failed("The run recorded error findings.")3
5: ProtoRunGateResult.Passed("No error findings were recorded."));

1. 1
   A name and a delegate

   The name becomes the row in the report and the message in the failure; a class implementing IProtoRunGate works the same way.

2. 2
   It reads items, not tests

   ItemsOfKind selects the findings the run collected, wherever they came from. A teardown finding counts like any other.

3. 3
   Failed fails the run

   A Failed verdict makes the run teardown throw a ProtoRunGateException, so the process exits 1. Warning and Skipped are recorded without failing.

From samples/Northstar.ProtoTest/Setup.cs. Gates run once after the last test and before the reports are written, so a failed gate still produces its report.

2. Run the broken teardown

Add the failing teardown attribute from the findings lesson to one journey and run it alone. Open TestResults/Northstar.ProtoTest/report.json and look for these records:

Record	Reading
After · FailingTeardownAttribute, failed	the teardown step that threw
test.teardown, failed, error System.InvalidOperationException	the phase carries the exception type and message; the body's own result is untouched
finding finding-001, Error, category Teardown, message Teardown failed: the cleanup step failed on purpose	the report item, grouped under the test
gate no error findings, Error, message The run recorded error findings.	the verdict the gate returned
run event Gate · no error findings, outcome failed	the run-level trace entry, written once for the whole run
runner: Passed 1, exit code 1	the test passed; the gate failed the run

This is one recorded run. Timings differ per run, so compare names, states and messages.

3. See a finding that does not fail the run

Not every finding fails the run. Open l4-partial.prototrace. Its journey passed its checks but left four response fields unread, and the run recorded:

Record	Reading
test.execution, partial	the test called Assert.Warn, so the outcome is partial. The finding alone would not change it
finding finding-001, Warning, category Coverage, message The create response carried 4 fields no assertion mentioned: createdAtUtc, environmentCount, id, slug.	the report item
gate no error findings, passed	a Warning is recorded, and this gate only looks for Error

For this test, read the trace for the outcome, not the summary line. It says partial.

What happened

A test result and a run verdict are two different records. The test result comes from the runner. The run verdict comes from the gates, which read everything the run collected: findings, coverage, resources.

The teardown failure became an Error finding. The test body had passed, so the test result stayed Passed. The gate then read the finding after the last test and failed the run. The failure comes out of the run teardown, which is why the output says TearDown failed for test fixture.

A cleanup error never hides a failed assertion, and it never becomes one either. It is evidence that the run judges.

Check yourself

The runner prints Passed: 1 and the run exits 1. Which record does the exit code follow, and what did the finding do to the result the test itself reported?

Verify

Compare the runner summary with the finding and gate rows in report.json.

Remember

• A run gate judges the collected evidence after the last test. A failed gate fails the run, and the report is still written.
• The exit code follows the gate verdict, not the test list.
• A Warning finding leaves this gate passing, because it only looks for Error.

Go deeper

• Reporting: the JSON and HTML sinks, and the sections every report carries.
• l4-coverage.prototrace: a committed archive where the same gate passes with No error findings were recorded.