Structured assertion messages for reference and type assertions#8214
Conversation
Introduce the foundational types and helpers for structured multi-line assertion failure messages as described in RFC 012: - EvidenceLine: labeled line record struct for evidence blocks - EvidenceBlock: collection of labeled lines with automatic alignment - StructuredAssertionMessage: builder producing the new multi-line format (prefix + summary + user message + evidence block + call-site) - AssertionValueRenderer: renders values per RFC 012 rules (null, quoted strings with escape sequences, booleans, collections as JSON arrays) - AssertFailedException: add ExpectedText/ActualText public properties - Assert: add ReportAssertFailed/ThrowAssertFailed overloads accepting StructuredAssertionMessage No existing assertion methods are changed yet - this PR only introduces the infrastructure that subsequent PRs will use to migrate each assertion method to the new format.
Apply RFC 012 structured assertion message format to: - AreSame / AreNotSame (reference equality) - IsInstanceOfType / IsNotInstanceOfType (type checking) - IsExactInstanceOfType / IsNotExactInstanceOfType (exact type checking) Each method now produces structured output with summary, evidence block, user message, and call-site expression per the RFC specification. Update all corresponding test expectations to match the new format.
…-type Resolved conflicts in TestFramework structured assertion files by taking main's polished version of EvidenceBlock (sealed class supporting multiple blocks), AssertionValueRenderer (concise switch expression), StructuredAssertionMessage (List<EvidenceBlock>), and AssertFailedException Data entries; re-applied FormatCallSiteExpression helpers from this branch needed by AreSame/IsInstanceOfType assertions.
- Make FormatCallSiteExpression placeholders caller-controlled (H1, M3) so AreNotSame, IsInstanceOfType, etc. show meaningful parameter names instead of misleading <expected>/<actual> defaults.- Guard AssertionValueRenderer against user ToString() exceptions (H3) so a faulty ToString in a tested type does not mask the original assertion failure.- Populate WithExpectedAndActual on IsInstanceOfType / IsExactInstanceOfType (M5) to surface the IDE diff payload consistently with AreSame.- Strengthen Scope tests for type assertions (M7) by additionally asserting evidence-block lines and call-site expression.- Strip spurious BOM accidentally added to Assert.IsInstanceOfType.cs and Assert.IsExactInstanceOfType.cs (N1).
IsInstanceOfType / IsExactInstanceOfType only capture the 'value' argument via CallerArgumentExpression, so the call-site shows only Assert.X(value), not Assert.X(value, typeof(...)).
There was a problem hiding this comment.
Pull request overview
This PR migrates reference and type assertion failures in MSTest to the RFC 012 structured assertion message format, adding structured summaries, evidence/call-site formatting, and safer value rendering.
Changes:
- Updates
AreSame/AreNotSameand type assertion implementations to buildStructuredAssertionMessageinstances. - Adds shared internal call-site expression formatting helpers.
- Updates assertion message tests and scope tests to expect the new structured output.
Show a summary per file
| File | Description |
|---|---|
src/TestFramework/TestFramework/Assertions/Assert.cs |
Adds shared call-site expression formatting helpers. |
src/TestFramework/TestFramework/Assertions/Assert.AreSame.cs |
Migrates reference equality assertions to structured messages. |
src/TestFramework/TestFramework/Assertions/Assert.IsInstanceOfType.cs |
Migrates instance-of type assertions to structured messages. |
src/TestFramework/TestFramework/Assertions/Assert.IsExactInstanceOfType.cs |
Migrates exact type assertions to structured messages. |
src/TestFramework/TestFramework/Assertions/AssertionValueRenderer.cs |
Adds defensive handling for objects whose ToString() throws. |
test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.AreSame.cs |
Updates reference assertion message expectations. |
test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.IsInstanceOfTypeTests.cs |
Updates instance-of type assertion expectations. |
test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.IsExactInstanceOfTypeTests.cs |
Updates exact type assertion expectations. |
test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.ScopeTests.cs |
Updates scoped assertion checks for structured messages. |
Copilot's findings
Comments suppressed due to low confidence (11)
src/TestFramework/TestFramework/Assertions/Assert.IsInstanceOfType.cs:430
- Calling AssertionValueRenderer.RenderValue here will enumerate any IEnumerable actual value. A failing negative type assertion against a large, lazy, throwing, or infinite sequence can hang or throw while formatting the failure message, masking the assertion; type assertions should avoid unbounded value rendering or use a bounded/safe renderer.
.AddLine("actual value:", AssertionValueRenderer.RenderValue(value));
src/TestFramework/TestFramework/Assertions/Assert.IsExactInstanceOfType.cs:426
- Calling AssertionValueRenderer.RenderValue here will enumerate any IEnumerable actual value. A failing negative exact type assertion against a large, lazy, throwing, or infinite sequence can hang or throw while formatting the failure message, masking the assertion; type assertions should avoid unbounded value rendering or use a bounded/safe renderer.
.AddLine("actual value:", AssertionValueRenderer.RenderValue(value));
src/TestFramework/TestFramework/Assertions/Assert.IsInstanceOfType.cs:423
- When wrongType is null this builds the summary "Expected value to not be of type null (or derived)", which is not a meaningful type assertion and gives users an unclear failure. Handle the null type case with a dedicated message (for example, indicating that the disallowed type argument was null).
string typeName = wrongType?.Name ?? "null";
StructuredAssertionMessage msg = new($"Expected value to not be of type {typeName} (or derived).");
src/TestFramework/TestFramework/Assertions/Assert.IsExactInstanceOfType.cs:419
- When wrongType is null this builds the summary "Expected value to not be exactly of type null", which is not a meaningful type assertion and gives users an unclear failure. Handle the null type case with a dedicated message (for example, indicating that the disallowed type argument was null).
string typeName = wrongType?.Name ?? "null";
StructuredAssertionMessage msg = new($"Expected value to not be exactly of type {typeName}.");
src/TestFramework/TestFramework/Assertions/Assert.cs:209
- The two-argument multiline-expression branch is new but has no regression coverage. Add a test with a multiline expected or actual expression to verify the caller-controlled placeholders are emitted instead of the raw multiline expression.
string arg1 = expression1.Contains('\n') || expression1.Contains('\r') ? placeholder1 : expression1;
string arg2 = expression2.Contains('\n') || expression2.Contains('\r') ? placeholder2 : expression2;
return $"{assertionMethodName}({arg1}, {arg2})";
src/TestFramework/TestFramework/Assertions/Assert.IsInstanceOfType.cs:423
- The new structured failure format for IsNotInstanceOfType when the value actually is of the disallowed type is not covered by the updated tests; current tests only assert that null wrongType throws. Add a message assertion for a non-null wrongType failure so this path is protected like the other migrated assertions.
StructuredAssertionMessage msg = new($"Expected value to not be of type {typeName} (or derived).");
src/TestFramework/TestFramework/Assertions/Assert.IsInstanceOfType.cs:423
- This user-facing assertion summary is now hardcoded in English, whereas the previous type assertion messages came from FrameworkMessages.resx (see Resources/FrameworkMessages.resx:215-222) and the project carries localized XLF resources. This drops localization for these failures; add resource entries for the structured summaries/details instead of embedding literals.
string typeName = wrongType?.Name ?? "null";
StructuredAssertionMessage msg = new($"Expected value to not be of type {typeName} (or derived).");
src/TestFramework/TestFramework/Assertions/Assert.IsInstanceOfType.cs:333
- This user-facing assertion summary is now hardcoded in English, whereas the previous type assertion messages came from FrameworkMessages.resx (see Resources/FrameworkMessages.resx:215-222) and the project carries localized XLF resources. This drops localization for these failures; add resource entries for the structured summaries/details instead of embedding literals.
string typeName = expectedType?.Name ?? "null";
StructuredAssertionMessage msg = new($"Expected value to be of type {typeName} (or derived).");
src/TestFramework/TestFramework/Assertions/Assert.IsExactInstanceOfType.cs:419
- This user-facing assertion summary is now hardcoded in English, whereas the previous exact type assertion messages came from FrameworkMessages.resx (see Resources/FrameworkMessages.resx:224-228) and the project carries localized XLF resources. This drops localization for these failures; add resource entries for the structured summaries/details instead of embedding literals.
string typeName = wrongType?.Name ?? "null";
StructuredAssertionMessage msg = new($"Expected value to not be exactly of type {typeName}.");
src/TestFramework/TestFramework/Assertions/Assert.AreSame.cs:271
- These user-facing assertion strings are now hardcoded in English, whereas the previous AreNotSame message came from FrameworkMessages.resx (see Resources/FrameworkMessages.resx:174-175) and the project carries localized XLF resources. This drops localization for these failures; add resource entries for the structured summaries/details instead of embedding literals.
msg.WithAdditionalSummaryLine(
notExpected is null && actual is null
? "Both values are null."
: "Both variables refer to the same object.");
src/TestFramework/TestFramework/Assertions/Assert.AreSame.cs:275
- The PR description says every migrated assertion emits an evidence block, but this AreNotSame failure path only adds summary lines and the call-site expression. If that description is intended, add evidence for the notExpected/actual reference identity; otherwise update the PR/tests to avoid claiming an evidence block here.
msg.WithUserMessage(userMessage);
msg.WithCallSiteExpression(FormatCallSiteExpression("Assert.AreNotSame", notExpectedExpression, actualExpression, "<notExpected>", "<actual>"));
- Files reviewed: 9/9 changed files
- Comments generated: 15
Evangelink
left a comment
There was a problem hiding this comment.
Review Summary
This PR continues the structured-assertion-message migration for AreSame/AreNotSame, IsInstanceOfType/IsNotInstanceOfType, and IsExactInstanceOfType/IsNotExactInstanceOfType. The overall direction is good — richer, aligned evidence blocks replace hard-to-parse concatenated strings — and the defensive RenderObject guard is a welcome improvement.
🔴 BLOCKING — Cross-TFM compilation failure
string.Contains(char) is unavailable on netstandard2.0 and net462.
TestFramework.csproj targets netstandard2.0;$(NetFrameworkMinimum);$(SupportedNetFrameworks). string.Contains(char) was added in .NET Standard 2.1 / .NET Core 2.1. There is no polyfill in src/Polyfills/ for this overload. The three new uses in FormatCallSiteExpression (Assert.cs lines 191, 206, 207) will produce compilation errors on the two older TFMs.
Fix: replace expression.Contains('\n') with expression.IndexOf('\n') >= 0 (and same for '\r'). string.IndexOf(char) is available on all TFMs.
🟡 MAJOR — Localization
New user-facing summary strings (e.g., "Expected both values to refer to the same object.", "Do not pass value types to AreSame ...", the type-mismatch sentences) are hardcoded English literals in C# rather than entries in FrameworkMessages.resx. The project's localization policy requires user-visible strings to live in .resx files. If the StructuredAssertionMessage pattern intentionally exempts its summaries from this requirement, that design decision should be explicitly documented.
🟡 MODERATE — Asymmetric call-site suppression
In the two-expression FormatCallSiteExpression overload, if either expression is null/whitespace the entire call-site line is suppressed. The single-expression overload only suppresses when its one expression is empty. A partial call-site (e.g., Assert.AreSame(<expected>, actual)) would be more informative than silence.
🔵 NIT — Missing test coverage
The new RenderObject catch-path (when ToString() throws) has no test. A single [DataRow] case with a type whose ToString() throws would cover this and give confidence in the fallback message format.
✅ Correctness
- The
expected is ValueType && actual is ValueType/expected is not ValueType || actual is not ValueTypelogic is De Morgan–correct and the evidence is only shown when at least one side is a reference type. - The old
value!.GetType()null-forgiving call inIsNotExactInstanceOfTypeis replaced byAssertionValueRenderer.RenderValue(value), which handlesnullsafely — an improvement. RuntimeHelpers.GetHashCodeis available on all supported TFMs. ✓- Test assertions use
AwesomeAssertions/Should()consistent with the test project's assertion library. ✓
Generated by Expert Code Review (on open) for issue #8214 · ● 24M
…ing tests, full-equality assertions - Add WithExpectedAndActual payload to IsNotInstanceOfType / IsNotExactInstanceOfType failures so ExpectedText/ActualText/Data are populated for the negative paths. - Convert all type-assertion tests to use C# raw string literals instead of Environment.NewLine interpolations and update for new message format (null-type guard, dropped 'actual value:', added 'actual type:' on negative paths, 'Both values' wording). - Replace .Should().Contain(...) checks in scope soft-failure tests with full-equality .Should().Be(rawString) assertions. - Add ObjectWithThrowingToString test for AssertionValueRenderer.RenderObject ToString-throws fallback. - Add new AssertTests.FormatCallSiteExpressionTests covering null/empty/multiline behavior on both single- and two-arg overloads (uses named arguments to disambiguate overloads).
|
@copilot resolve the merge conflicts in this pull request |
…ured-messages-reference-type # Conflicts: # src/TestFramework/TestFramework/Assertions/Assert.cs Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
Resolved the merge conflicts and merged |
Apply the RFC 012 structured assertion message format to reference and type assertions:
Assert.AreSame/Assert.AreNotSame(reference equality)Assert.IsInstanceOfType/Assert.IsNotInstanceOfType(type checking)Assert.IsExactInstanceOfType/Assert.IsNotExactInstanceOfType(exact type checking)Each method now produces structured output with summary, evidence block, optional user message, and a call-site expression per the RFC specification. Test expectations have been updated to match the new format.
Notable changes
Assert.FormatCallSiteExpression(...)(single-arg and two-arg overloads) used by all reference and type assertions to build the bottom call-site line. Multiline call-site expressions are replaced by caller-controlled placeholders (e.g.<value>,<expected>/<actual>,<notExpected>/<actual>).IsInstanceOfTypeandIsExactInstanceOfTypepopulateWithExpectedAndActual(...)so the IDE diff viewer gets a payload consistent withAreSame.AssertionValueRenderer.RenderValueis hardened against user types whoseToString()throws — the failure now reports"<TypeName> (ToString threw <ExName>)"instead of letting the user exception escape and mask the assertion failure.Validation
main(RFC 012 infrastructure landed via Add structured assertion message infrastructure (RFC 012) #8170 in main; conflicts inAssert.cs,AssertionValueRenderer.cs,EvidenceBlock.cs,StructuredAssertionMessage.cs, and the corresponding test files were resolved by taking main's polished version of the infrastructure and re-applying the newFormatCallSiteExpressionhelpers needed by this branch).Build.cmd -c Debugsucceeds with 0 warnings, 0 errors.ToStringhandling,WithExpectedAndActualfor IDE diff, strengthened scope tests).Public API
No public API surface added. Both
FormatCallSiteExpressionoverloads areinternal static, so noPublicAPI.Unshipped.txtupdate is required. Noinitaccessors introduced.