Skip to content

Structured assertion messages for reference and type assertions#8214

Merged
Evangelink merged 7 commits into
mainfrom
dev/amauryleve/structured-messages-reference-type
May 14, 2026
Merged

Structured assertion messages for reference and type assertions#8214
Evangelink merged 7 commits into
mainfrom
dev/amauryleve/structured-messages-reference-type

Conversation

@Evangelink
Copy link
Copy Markdown
Member

Apply the RFC 012 structured assertion message format to reference and type assertions:

  • Assert.AreSame / Assert.AreNotSame (reference equality)
  • Assert.IsInstanceOfType / Assert.IsNotInstanceOfType (type checking)
  • Assert.IsExactInstanceOfType / Assert.IsNotExactInstanceOfType (exact type checking)

Each method now produces structured output with summary, evidence block, optional user message, and a call-site expression per the RFC specification. Test expectations have been updated to match the new format.

Notable changes

  • New internal helpers Assert.FormatCallSiteExpression(...) (single-arg and two-arg overloads) used by all reference and type assertions to build the bottom call-site line. Multiline call-site expressions are replaced by caller-controlled placeholders (e.g. <value>, <expected>/<actual>, <notExpected>/<actual>).
  • IsInstanceOfType and IsExactInstanceOfType populate WithExpectedAndActual(...) so the IDE diff viewer gets a payload consistent with AreSame.
  • AssertionValueRenderer.RenderValue is hardened against user types whose ToString() throws — the failure now reports "<TypeName> (ToString threw <ExName>)" instead of letting the user exception escape and mask the assertion failure.

Validation

  • Resolved a pending merge from main (RFC 012 infrastructure landed via Add structured assertion message infrastructure (RFC 012) #8170 in main; conflicts in Assert.cs, AssertionValueRenderer.cs, EvidenceBlock.cs, StructuredAssertionMessage.cs, and the corresponding test files were resolved by taking main's polished version of the infrastructure and re-applying the new FormatCallSiteExpression helpers needed by this branch).
  • Build.cmd -c Debug succeeds with 0 warnings, 0 errors.
  • Iterated with the expert-reviewer agent and addressed actionable findings (caller-controlled placeholders, defensive ToString handling, WithExpectedAndActual for IDE diff, strengthened scope tests).

Public API

No public API surface added. Both FormatCallSiteExpression overloads are internal static, so no PublicAPI.Unshipped.txt update is required. No init accessors introduced.

Introduce the foundational types and helpers for structured multi-line
assertion failure messages as described in RFC 012:

- EvidenceLine: labeled line record struct for evidence blocks
- EvidenceBlock: collection of labeled lines with automatic alignment
- StructuredAssertionMessage: builder producing the new multi-line format
  (prefix + summary + user message + evidence block + call-site)
- AssertionValueRenderer: renders values per RFC 012 rules (null, quoted
  strings with escape sequences, booleans, collections as JSON arrays)
- AssertFailedException: add ExpectedText/ActualText public properties
- Assert: add ReportAssertFailed/ThrowAssertFailed overloads accepting
  StructuredAssertionMessage

No existing assertion methods are changed yet - this PR only introduces
the infrastructure that subsequent PRs will use to migrate each
assertion method to the new format.
Apply RFC 012 structured assertion message format to:
- AreSame / AreNotSame (reference equality)
- IsInstanceOfType / IsNotInstanceOfType (type checking)
- IsExactInstanceOfType / IsNotExactInstanceOfType (exact type checking)

Each method now produces structured output with summary, evidence block,
user message, and call-site expression per the RFC specification.

Update all corresponding test expectations to match the new format.
…-type

Resolved conflicts in TestFramework structured assertion files by taking main's polished version of EvidenceBlock (sealed class supporting multiple blocks), AssertionValueRenderer (concise switch expression), StructuredAssertionMessage (List<EvidenceBlock>), and AssertFailedException Data entries; re-applied FormatCallSiteExpression helpers from this branch needed by AreSame/IsInstanceOfType assertions.
- Make FormatCallSiteExpression placeholders caller-controlled (H1, M3) so AreNotSame, IsInstanceOfType, etc. show meaningful parameter names instead of misleading <expected>/<actual> defaults.- Guard AssertionValueRenderer against user ToString() exceptions (H3) so a faulty ToString in a tested type does not mask the original assertion failure.- Populate WithExpectedAndActual on IsInstanceOfType / IsExactInstanceOfType (M5) to surface the IDE diff payload consistently with AreSame.- Strengthen Scope tests for type assertions (M7) by additionally asserting evidence-block lines and call-site expression.- Strip spurious BOM accidentally added to Assert.IsInstanceOfType.cs and Assert.IsExactInstanceOfType.cs (N1).
IsInstanceOfType / IsExactInstanceOfType only capture the 'value' argument via CallerArgumentExpression, so the call-site shows only Assert.X(value), not Assert.X(value, typeof(...)).
Copilot AI review requested due to automatic review settings May 14, 2026 09:48
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR migrates reference and type assertion failures in MSTest to the RFC 012 structured assertion message format, adding structured summaries, evidence/call-site formatting, and safer value rendering.

Changes:

  • Updates AreSame / AreNotSame and type assertion implementations to build StructuredAssertionMessage instances.
  • Adds shared internal call-site expression formatting helpers.
  • Updates assertion message tests and scope tests to expect the new structured output.
Show a summary per file
File Description
src/TestFramework/TestFramework/Assertions/Assert.cs Adds shared call-site expression formatting helpers.
src/TestFramework/TestFramework/Assertions/Assert.AreSame.cs Migrates reference equality assertions to structured messages.
src/TestFramework/TestFramework/Assertions/Assert.IsInstanceOfType.cs Migrates instance-of type assertions to structured messages.
src/TestFramework/TestFramework/Assertions/Assert.IsExactInstanceOfType.cs Migrates exact type assertions to structured messages.
src/TestFramework/TestFramework/Assertions/AssertionValueRenderer.cs Adds defensive handling for objects whose ToString() throws.
test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.AreSame.cs Updates reference assertion message expectations.
test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.IsInstanceOfTypeTests.cs Updates instance-of type assertion expectations.
test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.IsExactInstanceOfTypeTests.cs Updates exact type assertion expectations.
test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.ScopeTests.cs Updates scoped assertion checks for structured messages.

Copilot's findings

Comments suppressed due to low confidence (11)

src/TestFramework/TestFramework/Assertions/Assert.IsInstanceOfType.cs:430

  • Calling AssertionValueRenderer.RenderValue here will enumerate any IEnumerable actual value. A failing negative type assertion against a large, lazy, throwing, or infinite sequence can hang or throw while formatting the failure message, masking the assertion; type assertions should avoid unbounded value rendering or use a bounded/safe renderer.
                .AddLine("actual value:", AssertionValueRenderer.RenderValue(value));

src/TestFramework/TestFramework/Assertions/Assert.IsExactInstanceOfType.cs:426

  • Calling AssertionValueRenderer.RenderValue here will enumerate any IEnumerable actual value. A failing negative exact type assertion against a large, lazy, throwing, or infinite sequence can hang or throw while formatting the failure message, masking the assertion; type assertions should avoid unbounded value rendering or use a bounded/safe renderer.
                .AddLine("actual value:", AssertionValueRenderer.RenderValue(value));

src/TestFramework/TestFramework/Assertions/Assert.IsInstanceOfType.cs:423

  • When wrongType is null this builds the summary "Expected value to not be of type null (or derived)", which is not a meaningful type assertion and gives users an unclear failure. Handle the null type case with a dedicated message (for example, indicating that the disallowed type argument was null).
        string typeName = wrongType?.Name ?? "null";
        StructuredAssertionMessage msg = new($"Expected value to not be of type {typeName} (or derived).");

src/TestFramework/TestFramework/Assertions/Assert.IsExactInstanceOfType.cs:419

  • When wrongType is null this builds the summary "Expected value to not be exactly of type null", which is not a meaningful type assertion and gives users an unclear failure. Handle the null type case with a dedicated message (for example, indicating that the disallowed type argument was null).
        string typeName = wrongType?.Name ?? "null";
        StructuredAssertionMessage msg = new($"Expected value to not be exactly of type {typeName}.");

src/TestFramework/TestFramework/Assertions/Assert.cs:209

  • The two-argument multiline-expression branch is new but has no regression coverage. Add a test with a multiline expected or actual expression to verify the caller-controlled placeholders are emitted instead of the raw multiline expression.
        string arg1 = expression1.Contains('\n') || expression1.Contains('\r') ? placeholder1 : expression1;
        string arg2 = expression2.Contains('\n') || expression2.Contains('\r') ? placeholder2 : expression2;

        return $"{assertionMethodName}({arg1}, {arg2})";

src/TestFramework/TestFramework/Assertions/Assert.IsInstanceOfType.cs:423

  • The new structured failure format for IsNotInstanceOfType when the value actually is of the disallowed type is not covered by the updated tests; current tests only assert that null wrongType throws. Add a message assertion for a non-null wrongType failure so this path is protected like the other migrated assertions.
        StructuredAssertionMessage msg = new($"Expected value to not be of type {typeName} (or derived).");

src/TestFramework/TestFramework/Assertions/Assert.IsInstanceOfType.cs:423

  • This user-facing assertion summary is now hardcoded in English, whereas the previous type assertion messages came from FrameworkMessages.resx (see Resources/FrameworkMessages.resx:215-222) and the project carries localized XLF resources. This drops localization for these failures; add resource entries for the structured summaries/details instead of embedding literals.
        string typeName = wrongType?.Name ?? "null";
        StructuredAssertionMessage msg = new($"Expected value to not be of type {typeName} (or derived).");

src/TestFramework/TestFramework/Assertions/Assert.IsInstanceOfType.cs:333

  • This user-facing assertion summary is now hardcoded in English, whereas the previous type assertion messages came from FrameworkMessages.resx (see Resources/FrameworkMessages.resx:215-222) and the project carries localized XLF resources. This drops localization for these failures; add resource entries for the structured summaries/details instead of embedding literals.
        string typeName = expectedType?.Name ?? "null";
        StructuredAssertionMessage msg = new($"Expected value to be of type {typeName} (or derived).");

src/TestFramework/TestFramework/Assertions/Assert.IsExactInstanceOfType.cs:419

  • This user-facing assertion summary is now hardcoded in English, whereas the previous exact type assertion messages came from FrameworkMessages.resx (see Resources/FrameworkMessages.resx:224-228) and the project carries localized XLF resources. This drops localization for these failures; add resource entries for the structured summaries/details instead of embedding literals.
        string typeName = wrongType?.Name ?? "null";
        StructuredAssertionMessage msg = new($"Expected value to not be exactly of type {typeName}.");

src/TestFramework/TestFramework/Assertions/Assert.AreSame.cs:271

  • These user-facing assertion strings are now hardcoded in English, whereas the previous AreNotSame message came from FrameworkMessages.resx (see Resources/FrameworkMessages.resx:174-175) and the project carries localized XLF resources. This drops localization for these failures; add resource entries for the structured summaries/details instead of embedding literals.
        msg.WithAdditionalSummaryLine(
            notExpected is null && actual is null
                ? "Both values are null."
                : "Both variables refer to the same object.");

src/TestFramework/TestFramework/Assertions/Assert.AreSame.cs:275

  • The PR description says every migrated assertion emits an evidence block, but this AreNotSame failure path only adds summary lines and the call-site expression. If that description is intended, add evidence for the notExpected/actual reference identity; otherwise update the PR/tests to avoid claiming an evidence block here.
        msg.WithUserMessage(userMessage);

        msg.WithCallSiteExpression(FormatCallSiteExpression("Assert.AreNotSame", notExpectedExpression, actualExpression, "<notExpected>", "<actual>"));
  • Files reviewed: 9/9 changed files
  • Comments generated: 15

Comment thread src/TestFramework/TestFramework/Assertions/Assert.IsInstanceOfType.cs Outdated
Comment thread src/TestFramework/TestFramework/Assertions/Assert.IsExactInstanceOfType.cs Outdated
Comment thread src/TestFramework/TestFramework/Assertions/Assert.IsInstanceOfType.cs Outdated
Comment thread src/TestFramework/TestFramework/Assertions/Assert.IsExactInstanceOfType.cs Outdated
Comment thread src/TestFramework/TestFramework/Assertions/Assert.IsExactInstanceOfType.cs Outdated
Comment thread src/TestFramework/TestFramework/Assertions/Assert.AreSame.cs
Comment thread src/TestFramework/TestFramework/Assertions/Assert.IsExactInstanceOfType.cs Outdated
Comment thread src/TestFramework/TestFramework/Assertions/Assert.AreSame.cs Outdated
Comment thread src/TestFramework/TestFramework/Assertions/Assert.AreSame.cs
Copy link
Copy Markdown
Member Author

@Evangelink Evangelink left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Summary

This PR continues the structured-assertion-message migration for AreSame/AreNotSame, IsInstanceOfType/IsNotInstanceOfType, and IsExactInstanceOfType/IsNotExactInstanceOfType. The overall direction is good — richer, aligned evidence blocks replace hard-to-parse concatenated strings — and the defensive RenderObject guard is a welcome improvement.

🔴 BLOCKING — Cross-TFM compilation failure

string.Contains(char) is unavailable on netstandard2.0 and net462.

TestFramework.csproj targets netstandard2.0;$(NetFrameworkMinimum);$(SupportedNetFrameworks). string.Contains(char) was added in .NET Standard 2.1 / .NET Core 2.1. There is no polyfill in src/Polyfills/ for this overload. The three new uses in FormatCallSiteExpression (Assert.cs lines 191, 206, 207) will produce compilation errors on the two older TFMs.

Fix: replace expression.Contains('\n') with expression.IndexOf('\n') >= 0 (and same for '\r'). string.IndexOf(char) is available on all TFMs.

🟡 MAJOR — Localization

New user-facing summary strings (e.g., "Expected both values to refer to the same object.", "Do not pass value types to AreSame ...", the type-mismatch sentences) are hardcoded English literals in C# rather than entries in FrameworkMessages.resx. The project's localization policy requires user-visible strings to live in .resx files. If the StructuredAssertionMessage pattern intentionally exempts its summaries from this requirement, that design decision should be explicitly documented.

🟡 MODERATE — Asymmetric call-site suppression

In the two-expression FormatCallSiteExpression overload, if either expression is null/whitespace the entire call-site line is suppressed. The single-expression overload only suppresses when its one expression is empty. A partial call-site (e.g., Assert.AreSame(<expected>, actual)) would be more informative than silence.

🔵 NIT — Missing test coverage

The new RenderObject catch-path (when ToString() throws) has no test. A single [DataRow] case with a type whose ToString() throws would cover this and give confidence in the fallback message format.

✅ Correctness

  • The expected is ValueType && actual is ValueType / expected is not ValueType || actual is not ValueType logic is De Morgan–correct and the evidence is only shown when at least one side is a reference type.
  • The old value!.GetType() null-forgiving call in IsNotExactInstanceOfType is replaced by AssertionValueRenderer.RenderValue(value), which handles null safely — an improvement.
  • RuntimeHelpers.GetHashCode is available on all supported TFMs. ✓
  • Test assertions use AwesomeAssertions / Should() consistent with the test project's assertion library. ✓

Generated by Expert Code Review (on open) for issue #8214 · ● 24M

Comment thread src/TestFramework/TestFramework/Assertions/Assert.cs Outdated
Comment thread src/TestFramework/TestFramework/Assertions/Assert.AreSame.cs
Comment thread src/TestFramework/TestFramework/Assertions/Assert.cs
…ing tests, full-equality assertions

- Add WithExpectedAndActual payload to IsNotInstanceOfType / IsNotExactInstanceOfType failures so ExpectedText/ActualText/Data are populated for the negative paths.

- Convert all type-assertion tests to use C# raw string literals instead of Environment.NewLine interpolations and update for new message format (null-type guard, dropped 'actual value:', added 'actual type:' on negative paths, 'Both values' wording).

- Replace .Should().Contain(...) checks in scope soft-failure tests with full-equality .Should().Be(rawString) assertions.

- Add ObjectWithThrowingToString test for AssertionValueRenderer.RenderObject ToString-throws fallback.

- Add new AssertTests.FormatCallSiteExpressionTests covering null/empty/multiline behavior on both single- and two-arg overloads (uses named arguments to disambiguate overloads).
@Evangelink
Copy link
Copy Markdown
Member Author

@copilot resolve the merge conflicts in this pull request

…ured-messages-reference-type

# Conflicts:
#	src/TestFramework/TestFramework/Assertions/Assert.cs

Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
Copilot AI review requested due to automatic review settings May 14, 2026 16:20
@Evangelink Evangelink review requested due to automatic review settings May 14, 2026 16:20
Copy link
Copy Markdown
Contributor

Copilot AI commented May 14, 2026

@copilot resolve the merge conflicts in this pull request

Resolved the merge conflicts and merged origin/main in edc0558.

@Evangelink Evangelink merged commit 1593058 into main May 14, 2026
17 checks passed
@Evangelink Evangelink deleted the dev/amauryleve/structured-messages-reference-type branch May 14, 2026 18:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants