Why a passing test can still leave an error

Understand failuresLesson 5 of 9About 7 minutes

You will

• Tell an observation, a finding and an attachment apart
• Read the finding a failing teardown leaves
• Say why a finding never replaces the test result

You need

• The previous lesson, diagnose a flaky test
• The sample cloned, to break a teardown. Reading the report alone also works.

The problem

A cleanup step throws after the test body passed. The runner prints Passed: 1, but the run still fails. You need to know where that error went.

ProtoTest records it as a finding. A finding is something the run reports without changing the result the test reported.

Do it

1. Break a teardown on purpose

Add this file to the sample. It fails after one test:

namespace Northstar.ProtoTest;

using global::ProtoTest.Core;

public sealed class FailingTeardownAttribute : ProtoAttribute
{
    public override Task AfterTestAsync(ProtoExecutionContext context)
        => throw new InvalidOperationException("the cleanup step failed on purpose");
}

2. Run one journey with it

Apply [FailingTeardown] to one journey method and run it alone:

dotnet test samples/Northstar.ProtoTest --filter "FullyQualifiedName~ProjectsJourney.CreatingAProjectReturnsIt"

Look for a passing test and a failing run:

TearDown failed for test fixture Northstar.ProtoTest.Setup
TearDown : ProtoTest.Core.ProtoRunGateException : Run gate 'no error findings' failed: The run recorded error findings.

Passed!  - Failed:     0, Passed:     1, Skipped:     0, Total:     1

The Passed line is the test result. The TearDown lines come from the next lesson, the run gate.

3. Find the finding

Open TestResults/Northstar.ProtoTest/report.json under the sample. Look for an item of kind finding with category Teardown and the message Teardown failed: the cleanup step failed on purpose. It sits next to the test result, not in place of it.

What happened

The lifecycle collects teardown failures instead of throwing at the first one. Each collected failure becomes a finding:

ProtoTestLifecycle.cs3 notes

1// The teardown exception is evidence, not a replacement for the result the test reported:1
2// the original outcome stands, so a failed assertion is not hidden by a cleanup error.
3lifecycleOperation.Fail(exceptions[^1]);2
4foreach (var failure in exceptions.Skip(exceptionCountBeforeTeardown))
5{
6// The same path as any other finding, so a teardown failure reaches the sinks and run
7// gates instead of living only in the trace.
8context.AddFinding(
9$"Teardown failed: {failure.Message}",3
10ProtoReportStatus.Error,
11category: "Teardown",
12targetName: context.TestName,
13tags: [failure.GetType().Name]);
14}

1. 1
   The result stands

   The teardown exception is evidence, not a replacement: the passing body stays a passed test.

2. 2
   The phase records the failure

   The teardown operation fails with the exception, so the trace shows which step threw.

3. 3
   The finding carries the rest

   Status Error, category Teardown, the test as its target and the exception type as a tag. It reaches every sink and every run gate.

From src/ProtoTest.Core/Internal/ProtoTestLifecycle.cs. The collect-mode flow is why one failing step does not stop the others.

A report holds three kinds of evidence:

Kind	Reading	Who writes it
Observation	counts or lists what the run saw; it never passes or fails	clients and collectors, through RecordObservation
Finding	something worth reporting that is deliberately not the test result	a test or the lifecycle, through AddFinding
Attachment	the exact file the test read	tests and clients, through context.AddAttachment(...) and .CaptureAttachments()

Findings land as a report item of kind finding and as a finding event in the trace. Attachments land under resources/<test id>/artifact-N/ inside the archive, with the bytes.

Check yourself

A coverage gap arrives as a Warning with category Coverage. Which of the three kinds is it, and what does it change about the result the test reported?

Verify

With the failing teardown still in place, open report.json and compare the test result with the finding row beside it.

Remember

• An observation feeds coverage, a finding explains an outcome, an attachment carries bytes.
• A failing teardown becomes an Error finding. The test result stands.
• A Warning finding is recorded and does not fail anything by itself.

Go deeper

• Run gates: the next lesson, where the gate turns the finding into a failed run.
• Contract coverage: reads observations.
• Reporting: the JSON and HTML sinks, and the sections every report carries.