Skip to content

Add ECMA-335 EnC metadata delta writer#20019

Open
NatElkins wants to merge 5 commits into
dotnet:mainfrom
NatElkins:enc-metadata-delta-writer
Open

Add ECMA-335 EnC metadata delta writer#20019
NatElkins wants to merge 5 commits into
dotnet:mainfrom
NatElkins:enc-metadata-delta-writer

Conversation

@NatElkins

@NatElkins NatElkins commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds an internal, standalone ECMA-335 Edit-and-Continue metadata delta writer to AbstractIL, with focused System.Reflection.Metadata-readable metadata tests. It has no behavior change and no in-tree semantic consumer yet; the consumer is the F# hot reload work in #19941. This follows the same upstreaming pattern as #20017 and #20018: land isolated infrastructure first, with deterministic byte and layout evidence, then wire the feature in a later PR.

What

Ten new internal AbstractIL files (file names kept from the development branch for diffability):

  • ILDeltaHandles - offset/handle primitives, coded-index discriminated unions, delta tokens
  • ILMetadataHeaps - heap sizes and offset bookkeeping
  • IlxDeltaStreams - user-string and standalone-signature token calculators (continue numbering from a baseline)
  • DeltaMetadataEncoding - ECMA-335 II.24.2.6 coded-index encoding
  • DeltaMetadataTypes - the writer's input model: plain row-description records (method/parameter/property/event rows, method-semantics updates, custom attribute rows) and the definition key types
  • DeltaMetadataTables / DeltaTableLayout / DeltaIndexSizing / DeltaMetadataSerializer - delta #- table stream, heap construction, layout and index sizing, EncLog/EncMap emission, generation GUID chaining
  • FSharpDeltaMetadataWriter - the coordinating writer

The writer's inputs are row records (names, tokens, signature blobs, RVAs) plus baseline heap offsets and row counts. It imports nothing from any diffing, baseline, or session machinery; a grep gate for such imports is part of how the extraction was validated. One line of ilwrite.fsi exposes the pre-existing markerForUnicodeBytes so the delta writer reuses the full writer's exact string-marker logic.

Tests

130 tests in tests/FSharp.Compiler.Service.Tests/DeltaMetadata/:

  • coded-index encodings asserted against the production Encoding.CodedIndices definitions, with ECMA-335 II.24.2.6 table order cross-checked against System.Reflection.Metadata's TableIndex as an independent reference;
  • System.Reflection.Metadata reader parity over emitted deltas (SrmReaderParityTests);
  • EncLog/EncMap correctness, stream layout, heap and index sizing;
  • multi-generation chaining with generation-2 heap offsets asserted against computed expected values;
  • standalone-signature delta rows asserted at baseline + 1 from a real seeded baseline;
  • serializer failure paths.

All additions are internal; no public surface area change.

Sequencing

This PR is part of splitting the F# hot reload work (#19941) into small, independently reviewable PRs. The planned order:

  1. Wave 1 (independent of each other, no cross dependencies): Add ResetCompilerGeneratedNameState to compiler-generated name generators #20017 (generated-name counter reset), Add Roslyn-format EnC CustomDebugInformation codec and portable PDB method CDI emission #20018 (EnC CustomDebugInformation codec and method CDI emission), Add ECMA-335 EnC metadata delta writer #20019 (EnC metadata delta writer), plus two upcoming slices: stable synthesized-name replay and the typed-tree differ with rude-edit classification.
  2. Wave 2: baseline reading and recorded CDI state (depends on Add Roslyn-format EnC CustomDebugInformation codec and portable PDB method CDI emission #20018 and the naming slice).
  3. Wave 3: the delta emitter and symbol matcher.
  4. Wave 4: the hot reload session, FCS surface, and the --test:HotReloadDeltas capture hook (F# hot reload: Edit-and-Continue delta emission behind --test:HotReloadDeltas #19941 in its final, much smaller form).
  5. Last, explicitly experimental: Add an experimental flag-gated in-process compile path for hot reload sessions #20031 (the flag-gated in-process compile perf path).

Each slice is rebased on current main, carries its own tests, and is flag-off zero-cost. Everything stays draft until reviewed in order.
Role in the train: the hot reload session emits its metadata deltas through this writer.

Refresh status (2026-07-17)

  • Refreshed against dotnet/fsharp main at 5928e91; the current reviewed head is e7d79c0.
  • A dedicated review pass completed for this PR, its findings were fixed in the lowest owning slice, and the PR has no unresolved review threads.
  • The downstream session, in-process compiler, umbrella, and SDK branches were restacked after the fixes, so this slice remains part of the decomposed review train.
  • The complete compiler stack passed the 11-step hot reload verifier, 456 service tests, 243 component tests with 2 expected skips, and 1411 EmittedIL tests with 3 expected skips. Replacement CI passed on this exact head.

@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

❗ Release notes required

You can open this PR in browser to add release notes: open in github.dev


✅ Found changes and release notes in following paths:

Change path Release notes path Description
src/Compiler docs/release-notes/.FSharp.Compiler.Service/11.0.100.md

Adds an internal, standalone ECMA-335 Edit-and-Continue metadata delta writer to
AbstractIL: delta #- table stream and heap construction (DeltaMetadataTables,
DeltaMetadataSerializer, DeltaTableLayout, DeltaIndexSizing), ECMA-335 II.24.2.6
coded-index encoding (DeltaMetadataEncoding), EncLog/EncMap emission, generation GUID
chaining, user-string and standalone-signature token calculators (IlxDeltaStreams),
and the coordinating writer (FSharpDeltaMetadataWriter) over a plain row-description
input model (DeltaMetadataTypes, ILDeltaHandles, ILMetadataHeaps).

The writer's inputs are row records (names, tokens, signatures, RVAs) plus heap
offsets; it has no dependency on any semantic diffing or session machinery. It
compiles with no in-tree consumer by design: the consumer is the F# hot reload work
in dotnet#19941, following the same upstreaming pattern as dotnet#20017 and dotnet#20018
(land isolated, test-covered infrastructure first, wire the feature in a later PR).

One line of ilwrite.fsi is touched to expose the pre-existing markerForUnicodeBytes
so the delta writer reuses the exact string-marker logic of the full writer. No
behavior change for any existing code path.

Tests (130): coded-index encodings asserted against the production definitions and
ECMA-335 II.24.2.6 order, System.Reflection.Metadata reader parity over emitted
deltas, EncLog/EncMap correctness, stream layout, heap and index sizing,
multi-generation heap-offset chaining asserted against computed expected values,
standalone-signature rows asserted at baseline+1 from a real seeded baseline, and
serializer failure paths.
@NatElkins
NatElkins force-pushed the enc-metadata-delta-writer branch from 16962d4 to c4eb63a Compare July 17, 2026 23:11
Verified the repository-wide Fantomas check, the service test project build with zero warnings and errors, and all 134 DeltaMetadata tests.
@azure-pipelines

Copy link
Copy Markdown
Commenter does not have sufficient privileges for PR 20019 in repo dotnet/fsharp

@NatElkins
NatElkins marked this pull request as ready for review July 18, 2026 04:09
@NatElkins
NatElkins requested a review from a team as a code owner July 18, 2026 04:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: New

Development

Successfully merging this pull request may close these issues.

1 participant