[xharness] Prototype parallel simulator variation builds

[mmitche]

This is an experimental change to measure whether MSBuild-controlled parallelism improves the monotouch_ios simulator test job.

The current XHarness flow schedules all variation builds together, but every MSBuildTask acquires the desktop resource exclusively, so the 19 builds execute sequentially. In build 14886909 this phase took 58.5 minutes, including 28.1 minutes in Mono AOT and 16.4 minutes in Roslyn compilation.

This prototype:

• generates one wrapper project per cloned simulator variation;
• serializes restore under the existing NuGet and desktop resources;
• invokes one parent dotnet msbuild /m build with BuildInParallel="true", allowing MSBuild to choose worker-node concurrency;
• keeps per-variation success, restore-failure, and build-failure markers;
• releases the desktop resource after building, then runs all simulator tests serially as before;
• emits aggregate restore and build binlogs for performance analysis.

The purpose of this draft is to compare the build phase, total job duration, CPU utilization, and memory pressure with the 58.5-minute baseline before deciding whether to refine or keep the approach.

[mmitche]

/azp run

[azure-pipelines]

Commenter does not have sufficient privileges for PR 26442 in repo dotnet/macios

[mmitche]

Performance validation

The full validation build completed successfully: Azure build 14955876. All 19 monotouch_ios variations built and ran successfully, and the pipeline reported all 205 test groups passing.

Measurement	Baseline build 14886909	Prototype build 14955876	Change
monotouch_ios job	136.59 min	102.56 min	-34.03 min (-24.9%)
Run tests step	126.29 min	93.32 min	-32.97 min (-26.1%)
Build 19 variations	58.48 min	25.67 min	-32.81 min (-56.1%)
Post-build simulator/setup/run	64.50 min	67.46 min	+2.96 min (+4.6%)

The aggregate build binlog itself was 25:22.7 wall-clock, preceded by a 16.9-second serialized restore. Cumulative work inside that parallel interval was dominated by:

• AOTCompile: 18:12.5 total; longest invocation 4:32.6
• Csc: 18:07.7 total; longest invocation 1:24.6
• ILLink: 5:27.8 total; longest invocation 1:29.7
• Exec: 2:48.7 total; longest invocation 0:31.2

Those totals overlap across MSBuild nodes and are not wall-clock durations. Process snapshots confirm that MSBuild concurrently drove AOT/LLVM, Csc, linker, and native compilation. During the build, the sampled load averages were 42.36 / 47.94 / 31.87 at about +10 minutes and 12.80 / 19.10 / 24.51 at about +20 minutes. The largest observed individual process RSS was approximately 1.36 GB (illink); there was no OOM, timeout, crash, or build/test failure.

The simulator portion is still serial. Its approximately three-minute increase is outside the changed build phase and is not enough from one sample to establish a regression. XHarness also spends roughly 25 seconds after the aggregate build running per-variation property evaluation before simulator selection; these are short evaluations, not rebuilds.

Recommendation: continue with this approach. The first full CI sample cut variation-build wall time by 56% and the complete job by 25%, while preserving the serial simulator execution and per-variation result handling. Before considering the performance result final, run a few additional samples to quantify agent/test variance and watch memory pressure under concurrent linker/AOT workloads.

[mmitche]

Additional samples: correctness regression reproduced

The two additional runs do not support merging the current prototype:

• Build 14961204: monotouch_ios crashed because the MonoTouch.Dialog AOT module did not match the managed assembly. linker_ios independently crashed with the same error for System.ComponentModel.
• Build 14961205: two monotouch_ios variations crashed with mismatches for MonoTouch.Dialog and monotouchtest; monotouch_tvos also reproduced the MonoTouch.Dialog mismatch.

That is five AOT/assembly consistency failures across the two added samples. These are startup-time native aborts, not simulator provisioning or test assertion failures. The original validation run passed, so the failure is nondeterministic.

The aggregate binlogs show the performance benefit remains real: the two new monotouch_ios aggregate builds took 20:43.6 and 25:21.9, versus 25:22.7 in the first prototype sample and 58:29 baseline. The three-sample average is 24:09.4, approximately 58.7% faster than baseline.

However, the one parent /m invocation parallelizes more than the 19 wrapper projects: it also exposes parallelism inside each variation's nested project/build graph. The repeated AOT mismatches for both the app assembly and referenced/framework assemblies make this the leading explanation. The next prototype should preserve outer variation concurrency while forcing each individual variation's nested build graph to execute serially (for example by passing BuildInParallel=false into the child project builds), then repeat the same sampling.

Updated recommendation: do not merge the current implementation. The earlier recommendation is withdrawn until build isolation is corrected and repeated runs are clean.

[mmitche]

Multi-sample summary correction

Correction to the arithmetic in the previous comment: the aggregate-build durations were:

Build	Aggregate build	Result
14955876	25:22.7	Passed
14961204	20:43.6	Failed with AOT/assembly mismatches
14961205	25:21.9	Failed with AOT/assembly mismatches
Prototype mean	23:49.4	1/3 clean runs

The range is 20:43.6–25:22.7 (4:39.1), and the mean is 59.3% faster than the 58:28.8 baseline—not 58.7% as stated previously.

The failed runs' total job and simulator durations are not valid performance comparisons because startup crashes shortened some variations. The aggregate build timing remains useful because all 19 builds completed before execution began. Binlog variability also indicates contention: the longest AOT invocation ranged from 3:58.6 to 6:34.6 across the added samples.

Final conclusion remains unchanged: the speedup is substantial, but a 33% clean-run rate is unacceptable. Preserve top-level variation parallelism while disabling nested per-variation graph parallelism, then resample.

[vs-mobiletools-engineering-service2]

🔥 [CI Build #1e9ff6f] Test results 🔥

Test results

❌ Tests failed on VSTS: test results

1 tests crashed, 2 tests failed, 200 tests passed.

Failures

❌ linker tests (iOS)

1 tests failed, 14 tests passed.

Failed tests

• trimmode link/iOS - simulator/Release: Crashed

Html Report (VSDrops) Download

❌ monotouch tests (iOS)

1 tests failed, 18 tests passed.

Failed tests

• monotouch-test/iOS - simulator/Debug (static registrar): Crashed

Html Report (VSDrops) Download

❌ windows tests

🔥 Failed catastrophically on VSTS: test results - windows (no summary found).

Html Report (VSDrops) Download

Successes

✅ assembly-processing: All 1 tests passed. Html Report (VSDrops) Download
✅ cecil: All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (iOS): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (MacCatalyst): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (macOS): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (Multiple platforms): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (tvOS): All 1 tests passed. Html Report (VSDrops) Download
✅ framework: All 2 tests passed. Html Report (VSDrops) Download
✅ fsharp: All 4 tests passed. Html Report (VSDrops) Download
✅ generator: All 5 tests passed. Html Report (VSDrops) Download
✅ interdependent-binding-projects: All 4 tests passed. Html Report (VSDrops) Download
✅ introspection: All 4 tests passed. Html Report (VSDrops) Download
✅ linker (MacCatalyst): All 15 tests passed. Html Report (VSDrops) Download
✅ linker (macOS): All 21 tests passed. Html Report (VSDrops) Download
✅ linker (tvOS): All 15 tests passed. Html Report (VSDrops) Download
✅ monotouch (MacCatalyst): All 18 tests passed. Html Report (VSDrops) Download
✅ monotouch (macOS): All 21 tests passed. Html Report (VSDrops) Download
✅ monotouch (tvOS): All 19 tests passed. Html Report (VSDrops) Download
✅ msbuild: All 2 tests passed. Html Report (VSDrops) Download
✅ sharpie: All 1 tests passed. Html Report (VSDrops) Download
✅ xcframework: All 4 tests passed. Html Report (VSDrops) Download
✅ xtro: All 1 tests passed. Html Report (VSDrops) Download

macOS tests

✅ Tests on macOS Monterey (12): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Ventura (13): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Sonoma (14): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Sequoia (15): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Tahoe (26): All 5 tests passed. Html Report (VSDrops) Download

Linux Build Verification

✅ Linux build succeeded

Pipeline on Agent
Hash: 1e9ff6fe4fcd2d248f310f4464304c429b413452 [PR build]

[mmitche]

Additional root-cause analysis of the AOT/managed assembly mismatches:

The aggregate build is the cause boundary, but not because projects share obj/bin paths or because an MSBuild node reused stale task state.

• Each failing variation has a distinct cloned project tree and distinct AOT input/output directories.
• The failed roots ran on different MSBuild worker nodes, and neither worker had run an earlier AOTCompile task. This rules out a process-static cache leaking from a previous variation for these failures.
• For each failed assembly, the file passed to mono-aot-cross is the same root-local linked/postprocessed file later copied into that root's .app bundle.
• Assembly production completed before AOTCompile, and the bundle copy occurred after it; the binlog shows no intervening copy/write to those managed assembly paths.
• The significant behavioral change is nested AOT concurrency. The baseline admitted one variation build at a time through DesktopResource; that variation could use AOT's internal parallelism. The prototype admits several variation graphs concurrently through /m, and every AOTCompile task independently launches up to its own AotCompileMaxDegreeOfParallelism compiler processes.
• In build 14961205, failed root monotouch-test560 overlapped 3 other variation-level AOTCompile tasks. Failed root monotouch-test561 overlapped 5. These tasks compile many identically named assemblies (MonoTouch.Dialog.dll, monotouchtest.dll, framework assemblies) concurrently in separate processes.

So the prototype multiplied two levels of parallelism that were never concurrent before: MSBuild project parallelism x per-project AOT parallelism. The resulting AOT/native artifacts are occasionally inconsistent with their managed inputs. The available artifacts prove that concurrency boundary, but do not identify whether the final corruption is a Mono AOT compiler defect under this load or resource-pressure-induced corruption inside the native pipeline.

I do not recommend trying to fix this with BuildInParallel=false; baseline builds already use MSBuild node parallelism internally. The next useful experiment is to retain parallel variation scheduling but set AotCompileMaxDegreeOfParallelism=1, eliminating the nested fan-out. If that is stable across repeated samples, we can then choose a total concurrency budget deliberately instead of allowing each concurrent variation to consume half the machine independently.