Skip to content

Plan: mutation rules, stage 1 (#323) - #417

Open
berkgeveci wants to merge 4 commits into
Kitware:mainfrom
berkgeveci:docs/plan-mutation-rules
Open

berkgeveci wants to merge 4 commits into
Kitware:mainfrom
berkgeveci:docs/plan-mutation-rules

Conversation

@berkgeveci

Copy link
Copy Markdown
Collaborator

A plan for #323, for review before the code. Comment on any line of docs/design/plans/mutation-rules.md. I've said yes to every recommendation (D1–D6); a "yes", "no" or "other" per decision from each of you is what it needs. The "Open questions" section has three things the plan leaves open.

Stacked on #416, which adds docs/design/plans/ and its index: the first commit here is #416's. Merge #416 first; this PR then shows its own commit alone.

The plan in short

A dataset's structure (topology ids, space tables, and faces, edges and links derived from them) becomes read-only once built, for every array over that memory. Values stay writable, and every cache derived from them notices a write. An array sent through tack.communication becomes read-only to its sender. Nothing is copied.

Why: tack.dataset keeps about 40 derived caches, and none of those derived from structure checks for changes. Writing a connectivity after its faces were derived silently gives wrong answers.

  • D1: a one-way sealed flag on the memory (DeviceBuffer), checked on host writes and kernel stores.
  • D2: what is sealed: topologies' arrays, including the caller's (constructors don't copy), derived structure, space tables, an octree's tree. Breaking.
  • D3: structural attributes become read-only properties.
  • D4: value-derived caches key on revisions.
  • D5: sent arrays are read-only to the sender. Breaking.
  • D6: checking external writes is deferred to stage 2.

Value immutability, dataset generations and a reset are later stages. They wait on caches without reference cycles, a tested in-situ loop, and a rule for update_ghosts.

Prototype (in the plan's "Prototype" section)

A local branch, experiment/mutation-rules:

  • The whole suite passes with structure sealed (6859, no failures), as do the 164 guide examples.
  • The check costs about 18 ns per array a kernel stores to, under 1% of a warm call.
  • Zero-copy export to VTK shares memory as before.
  • It found Set flags going stale when a set's ids are written; keyed on the ids' revision, they follow the write.
  • Sent arrays pass the communication and ghost tests over threads and mpiexec -n 3.

Docs only. mkdocs build --strict passes.

🤖 Generated with Claude Code

Structure sealed at the buffer once built, value-derived caches on
revisions, sent arrays read-only to their sender; the prototype
(experiment/mutation-rules) passes the whole suite with structure sealed,
costs about 18 ns per written array a call, keeps zero-copy export, and
found Set flags going stale when a set's ids are written.
Comment thread docs/design/plans/mutation-rules.md Outdated
- **Implicit arrays** have no buffer; sealing one seals the stored arrays it
holds.

Constructors keep the caller's arrays, so **the caller's arrays are sealed

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In situ, a solver often keeps its mesh in its own Tack arrays and wraps them as a topology for analysis. With the seal on the buffer, the solver’s own kernels can no longer write that memory, for example to remesh or refine in place between steps. Copying every step costs GPU memory, which in situ can’t spare. What’s the intended pattern? Perhaps a new topology per step over memory the solver still owns, or a seal that applies only to buffers Tack allocated.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, and it was worse than one step: the seal is one-way and holds the memory, so a solver's buffer would stay sealed even after the topology was dropped. Revised D2 (539d964, 265d62f) to use the per-array flag instead, which marks who holds the memory rather than the memory itself:

  • Array.read_only(), public: a view over the same memory that cannot write, with no copy.
  • Giving an array to a constructor hands it over, as D5's sending does: that object becomes read-only to its caller, so a stray connectivity.fill(0) is still refused at the write.
  • In situ, the solver passes conn.read_only() and keeps conn, the only writable handle, to remesh in place. Everything Tack gives out from the topology is read-only, so analysis code cannot write the simulation's mesh by mistake.
  • The topology records the revision of the arrays it was given, and its structural caches check it on use, one check for the topology; when the owner has written, the derived structure is dropped and rebuilt.

Only memory Tack allocates for derived structure is sealed, so a seal never reaches a caller's buffer. D4 also shows the same read_only() pattern for a simulation's field values, so an adaptor can protect them from analysis code while the solver keeps writing.


### D4. Every cache derived from values tracks their revision

Values stay writable, so a cache derived from them must notice a write: an

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revisions only see writes Tack makes. A solver writing positions or fields through DLPack or wrap_ptr, as a Catalyst adaptor does every step, moves nothing, so a cached locator still serves() moved points. Could stage 1 add an explicit “these values changed” call (like VTK’s Modified()), with the in-situ docs saying to call it each step?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Added to D4 in 539d964: Array.modified(), VTK's Modified(), which counts a write to the array's buffer as Tack's own writes are counted, so the locators, the other value caches and D2's topology check all notice. The in-situ docs will say to call it on each array written outside Tack, each step; D6's checks stay the safety net for a forgotten call.

Comment thread docs/design/plans/mutation-rules.md Outdated
(`_writable`), not D1's buffer seal, which would lock the receiver too. Its
limit: views of the sent array made before sending can still write.

A sender that reuses one send buffer step after step now gets an error: a

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reusing one send buffer every step is the normal pattern for a solver’s halo exchange, so this breaks solver code that uses tack.communication, even if Tack itself doesn’t do it. Could exchange take an opt-out (the sender keeps writing at its own risk), or could the plan name the supported reuse pattern?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. The hazard turns out to be narrower: across ranks exchange waits for every request before returning, so the receiver already has its own copy and reusing the send buffer is safe; only a same-rank receiver (and the threaded test world) gets the sender's memory. Revised D5 in 539d964: exchange(..., keep=True) keeps the sender's arrays writable and gives a same-rank receiver a copy. That is the supported pattern for reused halo buffers, and without it the error names keep=True.

`DeviceBuffer` gets a one-way `sealed` flag: set once, never cleared, shared
by every array over the buffer. Host writes and kernel stores check it beside
`_writable` and refuse with a message that says what the array is and to copy
it. A copy is writable. DLPack exports of sealed memory are flagged read-only.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do the main consumers accept read-only DLPack? It’s worth checking PyTorch and CuPy on a sealed connectivity before this lands, since some refuse read-only capsules.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked PyTorch 2.15 against a read-only Tack array: it asks for DLPack 1.0, gets the read-only flag, accepts the capsule and ignores the flag, so the tensor is writable and a write through it reached the Tack array. NumPy 2.4 honours the flag. So nothing refuses sealed memory; the risk is the other way, a consumer writing it. A legacy (pre-1.0) consumer gets it as writable, since that protocol cannot carry the flag. D1 now says the seal stops at DLPack, exports stay zero-copy, and a consumer's writes are outside D1, for D6. CuPy is listed as unchecked (it needs a CUDA machine).

@jlee-kitware

Copy link
Copy Markdown
Collaborator

Yes to D1 through D4 and D6. No outside users means the break needs no warning first. The open points are our own in-situ patterns: solver-owned meshes written between steps (D2), values written outside Tack’s view (D4) and reused send buffers (D5). Each needs a named pattern before the code hits.

A caller's own structure is not sealed: the topology holds read-only views
of it and drops its derived structure when the arrays' revision moves (D2).
Array.modified() counts writes Tack cannot see (D4). exchange(keep=True)
lets a sender reuse its buffers, copying for a same-rank receiver only (D5).
The seal stops at DLPack: NumPy honours the read-only flag, PyTorch 2.15
ignores it (D1).
…flag

Array.read_only() gives a view that cannot write. Giving an array to a
constructor makes that object read-only to its caller, so a stray fill is
still refused; in situ the solver passes a read-only view and keeps the
only writable handle, and the topology notices its writes by revision.
The same pattern protects a simulation's field values (D4).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants