test: add a Hedgehog adapter for MSTest property tests by xperiandri · Pull Request #43 · fsprojects/FSharp.Azure.Cosmos

xperiandri · 2026-10-05T23:19:05Z

Proposed Changes

The query translator planned in ADR 0001 (#34) is meant to be checked with property-based tests: the SQL printer against generated syntax trees, and generated predicates against both F# and the emulator. Hedgehog, the property-testing library the plan uses, ships adapters for xUnit and NUnit but not for MSTest, which this repository uses. Without one, a property has to be a [<TestMethod>] that calls Property.check in its body: MSTest sees one opaque test, a failure is a single exception with no way to replay it, and [<TestInitialize>], [<TestCleanup>], data rows and timeouts do not apply to the individual generated cases.

This pull request adds that adapter, tests/Hedgehog.MSTest, and its own suite, tests/Hedgehog.MSTest.Tests.

A property is a test that must hold for every input Hedgehog generates. When an input fails, Hedgehog shrinks it: it searches for the smallest input that still fails, the counterexample. With the adapter, a property is an ordinary test method whose parameters Hedgehog generates:

[<TestClass>]
type ReverseTests () =

    [<Property(200<tests>)>]
    member _.``reversing a list twice gives the list back`` (xs : int list) =
        Assert.AreEqual (xs, List.rev (List.rev xs), "reversing twice must give the list back")

[<Property>] derives from MSTest's TestMethodAttribute. MSTest invokes the method once for every generated case and every shrink step, each time with a new test class instance, its [<TestInitialize>] and [<TestCleanup>], and its own timeout. The whole run is reported as one test result. A failure shows:

Hedgehog's report with the shrunk arguments;
the original exception as the inner exception;
the output of the failing invocation;
a line such as [<Property; Recheck("33_7178138842248401496_3704456211830240771_4-4-4")>].

That string is the recheck data: the size, seed and shrink path of the counterexample. Pasted onto the method, it replays exactly that one case.

The adapter is a port of Hedgehog.NUnit and Hedgehog.Xunit from hedgehogqa/fsharp-hedgehog at the Hedgehog 2.0.4 release commit a469772. The first commit copies the nine framework-neutral files unchanged except for the namespace, so the port can be compared with upstream line by line. The same code is meant to be proposed upstream as src/Hedgehog.MSTest, so both projects:

keep upstream's conventions (nullness checking off, no Fantomas);
reference no project of this repository;
carry an Apache-2.0 header in every copied file and are listed in the new THIRD-PARTY-NOTICES.md.

Once upstream publishes the package, it replaces this copy.

The adapter keeps upstream's attribute surface:

[<Property>] with its six constructor overloads and the AutoGenConfig, AutoGenConfigArgs, Tests, Shrinks and Size settings;
class-level [<Properties>] and [<Recheck>];
GenAttribute<'T> with its 18 built-in subclasses ([<Int>], [<Email>], …);
AutoGenConfig types: a type whose single static member returns Hedgehog's IAutoGenConfig, the generators to use per parameter type.

On top of that:

a Seed setting, so a suite can fix the cases it generates;
the values of a [<DataRow>] or [<DynamicData>] row are the leading arguments, and only the remaining parameters are generated;
Assert.Inconclusive discards a case (its precondition does not hold), and a give-up (Hedgehog stopping after 100 discards) fails the test;
settings come from the test class MSTest runs, so a property inherited from an abstract base gets each derived class's [<Properties>], and a derived class's settings win over its base's;
a GenAttribute is found through the whole inheritance chain; upstream silently ignores one derived from a built-in attribute;
a generic method, or a return type other than unit, Task or ValueTask, gives an error result before anything is generated;
each shrink step is invoked once; Hedgehog 2.0.4 alone would invoke the final counterexample a second time;
once the TestContext token is cancelled, by a [<Timeout>] (which applies to each invocation) or by the test run, the property stops without shrinking.

The build gains a DotnetListTests target that runs before DotnetTest. MSTest 4 fails the discovery of a whole assembly when one test method has a signature it cannot run, and a [<Property>] returning bool compiles without a warning. Listing the tests reports that before the emulator run starts. The agent instructions now explain how to write property tests.

What the suite covers

116 tests:

Upstream's F# xUnit and NUnit adapter tests, rewritten for MSTest:
- generation and shrinking, recheck data and Size, shrink limits;
- AutoGenConfig types and their error messages, AutoGenConfigArgs;
- class-level settings, attribute subclasses, tuples, IDisposable arguments;
- GenAttribute with the 18 built-in attributes.
Cases MSTest 4 cannot discover (bool, Async or Property returns, for example) are left out.
Every addition and departure above, each pinning the MSTest behaviour the adapter relies on:
- a new instance, TestContext, [<TestInitialize>], [<TestCleanup>] and Dispose for every case;
- data rows and inherited properties;
- the caller information of all six constructors;
- Task and ValueTask bodies;
- Assert.Inconclusive and give-ups;
- generic methods and unsupported return types;
- seed reproducibility and a single invocation of the counterexample;
- cancellation and per-invocation timeouts.
Failure reporting through the real MSTest pipeline: a failing property reports its counterexample, recheck data and output in exactly one result, and [<Recheck>] replays the counterexample and passes once the bug is fixed.

Properties that fail by design never fail the suite. They run in one of two ways:

through the adapter's entry point, with a harness that inspects the report;
through a [<FailingProperty>] attribute that passes when MSTest's single result has the expected outcome and message.

Twenty behaviours of the adapter were each reverted on their own, and every one makes at least one test fail.

Verification

Build: dotnet build FSharp.Azure.Cosmos.slnx, Debug and Release, gives 0 errors. The warnings are the same as on main: nullness warnings in existing test code, none from the adapter projects.
Adapter suite: 116 of 116 passed in Debug and in Release, three runs each. --list-tests discovers 116 tests.
Existing unit tests (--filter TestCategory=Unit, with COSMOS_EMULATOR_ENDPOINT=https://127.0.0.1:9): 47 of 47 passed. The emulator integration tests were not run.
Fantomas: clean.
Commits: every commit builds in Release on its own.
Listing: build.cmd DotnetListTests -s lists the tests of both test applications.
Example: the one above was compiled against the built adapter.

Types of changes

Bugfix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist

Build and tests pass locally
I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added necessary documentation (if appropriate)

Further comments

Nothing ships: both projects are test projects, so no package, public API or changelog entry changes.
Framework layer: MSTest has no counterpart to NUnit's test builder or xUnit's discoverer, so this layer is a single attribute that overrides TestMethodAttribute.ExecuteAsync.
MSTest behaviour the API does not promise. The adapter relies on:
- the per-invocation lifecycle of ITestMethod.InvokeAsync;
- the experimental TestContext.Current (warning FS0057, suppressed in one helper);
- the per-invocation timeout;
- the reflected type of an inherited test method;
- the unpacking of data rows.
Each is pinned by a test, so an MSTest update that changes one fails the suite.
Departures from upstream: every one is recorded in the header of the changed file.

🤖 Generated with Claude Code

Start the Hedgehog MSTest adapter, tests/Hedgehog.MSTest (ADR 0001, sections 5, 14 and 22.6), with the nine files that Hedgehog.NUnit and Hedgehog.Xunit of hedgehogqa/fsharp-hedgehog share and that do not depend on a test framework: Prelude, ReflectionHelpers, AutoGenConfig, IPropertyAttribute, RecheckAttribute, GenAttribute, GenAttribute.Prelude, PropertyContext and InternalLogic, copied from src/Hedgehog.NUnit at the Hedgehog 2.0.4 release commit a469772 in upstream's file layout and compile order. The only change is the namespace, Hedgehog.NUnit becomes Hedgehog.MSTest, so this commit can be compared with upstream line by line; the MSTest attribute and the changes it needs follow separately. The code is Apache-2.0: every copied file names its origin, the copyright and the license in a header comment, and the new THIRD-PARTY-NOTICES.md records the source, the commit, the files and the license text. The same code is proposed upstream as src/Hedgehog.MSTest, so the project keeps upstream's conventions rather than the repository's: nullness checking is off (upstream enables it nowhere, and the copied files give 23 nullness warnings under the repository's Nullable=enable) and .fantomasignore excludes the project, so the Fantomas run of the build leaves upstream's formatting alone. It sets IsTestProject=false and IsPackable=false, because tests/Directory.Build.props marks every project under tests/ as a test project, references MSTest.TestFramework rather than the MSTest metapackage, and is excluded from the test infrastructure reference of tests/Directory.Build.props, because the adapter must not depend on this repository. The project joins both FSharp.Azure.Cosmos.slnx and FSharp.Azure.Cosmos.slnf, which FAKE builds, so every build compiles it, and THIRD-PARTY-NOTICES.md joins the solution items. Directory.Packages.props gets Hedgehog 2.0.4 (Apache-2.0; FSharp.Core >= 8.0.403 and TypeShape 9.0.0 transitively, checked in the nuspec), which contains the former Hedgehog.Experimental auto-generation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Turn the ported sources into an MSTest adapter (ADR 0001, section 22.6). PropertyAttribute derives from TestMethodAttribute and, like it, applies to methods only and is not inherited. It keeps upstream's six constructor overloads and named settings (AutoGenConfig, AutoGenConfigArgs, Tests, Shrinks, Size), forwards the caller information of every constructor to the base constructor, as STATestMethodAttribute does, and overrides ExecuteAsync. MSTest has no counterpart to NUnit's test builder or xUnit's discoverer, so this one attribute is the framework layer; PropertiesAttribute is copied from upstream with the same settings. Each generated case and each shrink step is one call of ITestMethod.InvokeAsync, never MethodInfo.Invoke, so MSTest creates the test class, injects the TestContext and runs TestInitialize, TestCleanup and Dispose for every case, and applies a [<Timeout>] to each of them. The values of a [<DataRow>] or [<DynamicData>] row, which MSTest passes in ITestMethod.Arguments, are the leading arguments and also appear in the report; only the parameters after them are generated. Generated IDisposable arguments are disposed after their invocation. The run is folded into one TestResult, because several results from one ExecuteAsync appear as separate results of one test: a failure carries Hedgehog's report with the shrunk arguments, the exception of the counterexample as inner exception (unwrapped from MSTest's internal TestFailedException), the output of its invocation and a paste-ready [<Property; Recheck("...")>]. A run that gives up after Hedgehog's 100 discards fails, as in upstream. Additions beyond upstream: a uint64 Seed setting on both attributes; Assert.Inconclusive in an invocation discards the case, and fails it while rechecking, because a recheck cannot discard its only case; a run whose TestContext cancellation token is cancelled stops without shrinking, reading the token before every invocation, because MSTest gives the TestContext a new token source before each TestCleanup. The token comes from TestContext.Current, experimental in MSTest 4.2.3 (MSTESTEXP), in one helper with a scoped #nowarn "57". Departures from upstream, each recorded in the header of the changed file: class settings come from the reflected type of the method, so a property inherited from an abstract base gets the settings of each derived test class, and a derived class's [<Properties>] win over its base's; a GenAttribute is found through a new non-generic GenAttribute base wherever it sits in the inheritance chain (upstream checks only the direct base type and silently auto-generates otherwise); generic methods, which MSTest 4.2.3 discovers, and return types other than unit, Task and ValueTask, which MSTest 4 cannot discover, are rejected before any generator is built, so the return-value handling and the reflection-based invocation of upstream's InternalLogic and ReflectionHelpers are removed; every invocation is memoised per shrink-tree node, because Hedgehog 2.0.4 runs an asynchronous case again whenever it unwraps it and would invoke the counterexample a second time to read its journal; the recheck, which is synchronous in Hedgehog 2.0.4, runs on the thread pool; any exception, an invalid AutoGenConfig type or malformed recheck data included, becomes a result with the outcome Error instead of escaping ExecuteAsync. InternalLogic.executeAsync, a function from the method, the data row, an invoker and a function that reads the cancellation token to a TestResult, is the entry point of ExecuteAsync; InternalsVisibleTo exposes it to the adapter's suite, so the suite can run properties that fail by design without failing itself. The repository turns GenerateAssemblyInfo off for the FAKE-written AssemblyInfo.fs of the src projects, so this project turns it back on for the SDK to emit the InternalsVisibleTo attribute. Every public type and member is documented, and every reference to a type or member is a documentation ID that resolves in the documentation files of the referenced assemblies, including the primary constructors, whose comments sit before their parameters. THIRD-PARTY-NOTICES.md lists the two attribute files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Add tests/Hedgehog.MSTest.Tests, an MSTest runner executable (OutputType Exe, EnableMSTestRunner) with method-level parallelism in testconfig.json, that runs 116 tests against the adapter (ADR 0001, section 22.6). Upstream's F# xUnit and NUnit adapter tests at a469772 are rewritten as [<TestClass>] types: generating and shrinking, AutoGenConfig types and their error messages, AutoGenConfigArgs with generic, concrete and mixed arguments, class-level settings and their overrides, attribute subclasses, tuples, shrink limits, recheck data and Size, IDisposable arguments, GenAttribute and the 18 built-in generator attributes. The cases MSTest 4 cannot discover (bool, Result, Async, Task<'T> and Property returns, unresolved generics, module-level properties) and the xUnit-specific ones are left out; the ported files and the shared configurations carry upstream's Apache-2.0 attribution, and THIRD-PARTY-NOTICES.md lists them. The targets that must shrink exactly once fix their seed: with a random one, a run whose first failing value is exactly 2500 has nothing smaller to shrink to, which happened in 8 of 20,000 simulated runs. MSTest-specific tests cover what the adapter adds or does differently, each pinning MSTest behaviour that the adapter relies on: a new instance, TestContext, TestInitialize, TestCleanup and Dispose for every case, with property and constructor injection; DataRow, DynamicData and TestDataRow<struct (string * int)> rows as leading arguments; a property inherited from an abstract base reading the settings of each derived test class, and a derived class's settings and configuration winning over its base's; a GenAttribute derived from a built-in one; the caller information of each of the six PropertyAttribute constructors; Task and ValueTask bodies; Assert.Inconclusive as a discard and as a failure while rechecking; give-ups failing; generic methods and unsupported return types giving an error result; the same Seed giving the same cases; the counterexample invoked once; a token cancelled at a failing case that has smaller values to shrink to, and a timed-out invocation, stopping the run without shrinking, with a per-invocation [<Timeout>] that a whole property exceeds and a TestCleanup that makes MSTest replace the token source. Properties that fail by design never fail the suite. Targets in classes without [<TestClass>], which MSTest does not run, go through Harness and the adapter's entry point InternalLogic.executeAsync with an invoker that creates the class and awaits the method, so the tests inspect the report, the journal and the result. FailingPropertyAttribute runs a property through the real MSTest pipeline instead and passes when its single result has the expected outcome, message, output and inner exception: a failure reports its counterexample, its recheck data and the output of its invocation, [<Recheck>] replays the counterexample of the seed-42 run and passes once the bug is fixed, a data row appears among the parameters, malformed recheck data gives an error result and a give-up fails. Twenty behaviours of the adapter, from the reflected type and the memoised invocations to the single result and the cancellation check before each invocation, were each reverted in the adapter on its own: a test fails for every one. Like the adapter, the suite keeps upstream's conventions (Nullable disabled, listed in .fantomasignore) and references no project of this repository but the adapter, so tests/Directory.Build.props leaves the Cosmos test infrastructure out of it too. The suite joins both FSharp.Azure.Cosmos.slnx and FSharp.Azure.Cosmos.slnf; testsGlob, already narrowed to tests/**/*.Tests.??proj, picks it up and leaves the adapter library out, so DotnetTest runs Hedgehog.MSTest.Tests next to FSharp.Azure.Cosmos.Tests. The coverage report excludes the adapter and Hedgehog, which the suite loads and whose package embeds its PDB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… them A new DotnetListTests target, between DotnetBuild and DotnetTest in both build chains, runs `dotnet test --list-tests` for every test application (ADR 0001, section 22.1). MSTest 4 fails the discovery of a whole assembly when one test method has a signature it cannot run, such as a [<Property>] method that returns bool, while the F# build gives no warning; checked by adding such a method to the adapter's suite: the build succeeded without a warning, and the listing failed with UTA007 and exit code -532462766. Listing reports that before the run, which needs the emulator, starts. DotnetTest and DotnetListTests share dotnetTestApplications, which runs `dotnet test --no-build` in the configuration of the executing targets for every project that testsGlob matches, so the loop over the test applications is written once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The solution tree of the agent instructions lists tests/Hedgehog.MSTest and its suite, and the libraries list Hedgehog. The testing rules say how to write a property test: a method of a [<TestClass>] type marked [<Property>], with generated parameters, never Property.check inside a [<TestMethod>]; that it is an instance method that returns unit, Task or ValueTask and fails by throwing, because MSTest 4 rejects static methods and other return types and one such method fails the discovery of the whole assembly, which --list-tests shows before a run; that MSTest runs [<TestInitialize>] and [<TestCleanup>] for every generated case and every shrink step; and that a failure reports a [<Recheck>] that replays it. They also record that the adapter and its suite are the exception to the shared test infrastructure reference, the suite referencing the adapter alone, and that both keep upstream's conventions instead of the rules of the file, with Apache-2.0 headers and THIRD-PARTY-NOTICES.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot

Copilot review overview

🟡 Changes recommended

Generator edge cases, replay guidance, generic configuration inference, and cancellation cleanup contain correctness defects.

Review effort: Balanced
Findings: 4 · 1

Open (5)

Infer generic arguments from nested and repeated parameter types · New
Handle zero endpoints without creating empty ranges · New
Dispose generated arguments when cancellation skips invocation · New
Preserve Property settings in ready-to-paste recheck hints · New
Correct misspelling of overridden in target names and tests · New

What changed in this PR

Adds a Hedgehog-to-MSTest adapter and test suite to support replayable property tests for planned quotation/query translation work.

Changes:

Adds property generation, shrinking, replay, cancellation, and MSTest lifecycle integration.
Adds comprehensive adapter tests and third-party attribution.
Integrates test discovery into the build pipeline.

File	Description
`.fantomasignore`	Excludes upstream-ported code from formatting.
`.github/copilot-instructions.md`	Documents property-test conventions.
`Directory.Packages.props`	Adds Hedgehog dependency version.
`FSharp.Azure.Cosmos.slnf`	Adds adapter projects to the filter.
`FSharp.Azure.Cosmos.slnx`	Adds projects and notices.
`THIRD-PARTY-NOTICES.md`	Records copied code and license.
`build/build.fs`	Adds test-discovery target.
`tests/Directory.Build.props`	Excludes adapter projects from Cosmos infrastructure.
`tests/Hedgehog.MSTest/AutoGenConfig.fs`	Instantiates generator configurations.
`tests/Hedgehog.MSTest/GenAttribute.Prelude.fs`	Provides built-in generator attributes.
`tests/Hedgehog.MSTest/GenAttribute.fs`	Defines generator attribute abstractions.
`tests/Hedgehog.MSTest/Hedgehog.MSTest.fsproj`	Defines the adapter library.
`tests/Hedgehog.MSTest/InternalLogic.fs`	Implements generation, invocation, and reporting.
`tests/Hedgehog.MSTest/IPropertyAttribute.fs`	Defines shared property settings.
`tests/Hedgehog.MSTest/Prelude.fs`	Adds internal helpers.
`tests/Hedgehog.MSTest/PropertiesAttribute.fs`	Adds class-level defaults.
`tests/Hedgehog.MSTest/PropertyAttribute.fs`	Integrates Hedgehog with MSTest.
`tests/Hedgehog.MSTest/PropertyContext.fs`	Resolves effective property settings.
`tests/Hedgehog.MSTest/RecheckAttribute.fs`	Adds counterexample replay metadata.
`tests/Hedgehog.MSTest/ReflectionHelpers.fs`	Adds return-type inspection helpers.
`tests/Hedgehog.MSTest.Tests/Common.fs`	Defines shared test generators.
`tests/Hedgehog.MSTest.Tests/GenAttributePreludeTests.fs`	Tests built-in generator attributes.
`tests/Hedgehog.MSTest.Tests/Harness.fs`	Provides an adapter test harness.
`tests/Hedgehog.MSTest.Tests/Hedgehog.MSTest.Tests.fsproj`	Defines the adapter test application.
`tests/Hedgehog.MSTest.Tests/MSTestTests.fs`	Tests MSTest-specific behavior.
`tests/Hedgehog.MSTest.Tests/PropertyTests.fs`	Tests generation, shrinking, and settings.
`tests/Hedgehog.MSTest.Tests/testconfig.json`	Enables method-level parallel testing.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+                if methodInfo.IsGenericMethod then
+                    methodInfo.GetParameters()
+                    |> Array.map _.ParameterType.IsGenericParameter
+                    |> Array.zip configArgs
+                    |> Array.filter snd
+                    |> Array.map (fun (arg, _) -> arg.GetType())
+                    |> fun argTypes -> methodInfo.MakeGenericMethod argTypes


+        match min, max with
+        | _, m when m < 0 -> Gen.int32 (chooseRangeInt32 min max)  // Range entirely negative
+        | n, _ when n > 0 -> Gen.int32 (chooseRangeInt32 min max)  // Range entirely positive
+        | n, m ->                                                   // 0 is in range, split it
+            Gen.choice [
+                Gen.int32 (chooseRangeInt32 n -1)
+                Gen.int32 (chooseRangeInt32 1 m)
+            ]


+                if state.Cancelled || token.IsCancellationRequested then
+                    state.Cancelled <- true
+                    return Journal.singletonMessage "Not run: the TestContext cancellation token is cancelled.", Discard


+    match report.Status with
+    | Failed { RecheckInfo = Some info } ->
+        let data = RecheckData.serialize info.Data
+        $"%s{Environment.NewLine}Reproduce with: [<Property; Recheck(\"%s{data}\")>]"


+    member _.``Class Properties works`` (_: int) = ()
+
+    [<Property(300<tests>)>]
+    member _.``Class Properties tests (count) is overriden by Method Property`` (_: int) = ()


xperiandri and others added 5 commits October 6, 2026 00:45

Copilot AI balanced review requested due to automatic review settings October 5, 2026 23:19

Copilot started reviewing on behalf of xperiandri October 5, 2026 23:19 View session

Copilot AI reviewed Oct 5, 2026

View reviewed changes

Navigation Menu

Repository navigation

test: add a Hedgehog adapter for MSTest property tests - #43

test: add a Hedgehog adapter for MSTest property tests#43
xperiandri wants to merge 5 commits into
mainfrom
test/hedgehog-mstest-adapter

xperiandri commented Oct 5, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

xperiandri commented Oct 5, 2026

Proposed Changes

What the suite covers

Verification

Types of changes

Checklist

Further comments

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants