Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/design/plans/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,3 +28,4 @@ recommendation, so a review can answer "yes", "no" or "other" per decision.
|---|---|---|
| [Cut filters' memory](cut-filters-memory.md) | [#318](https://github.com/Kitware/tack/issues/318) | Accepted |
| [Inspecting device-class objects](inspect-template.md) | [#324](https://github.com/Kitware/tack/issues/324) | Done |
| [Mutation rules, stage 1](mutation-rules.md) | [#323](https://github.com/Kitware/tack/issues/323) | Proposed |
336 changes: 336 additions & 0 deletions docs/design/plans/mutation-rules.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,336 @@
# Mutation rules, stage 1

**Status: proposed.** Issue [#323](https://github.com/Kitware/tack/issues/323),
2026-11-10. Berk Geveci has said yes to every recommendation (D1–D6); this
is for review by the others. Revised 2026-11-16 after Jeff Lee's review, for
in-situ use: a caller's own structure is handed over, not sealed (D2), writes Tack
cannot see have an explicit call (D4), a sender may keep its buffer (D5), and
the seal's limit at DLPack is stated (D1).

## Summary

The structure Tack builds (a topology's derived faces, edges and links, its
spaces' tables) becomes read-only once it is built, for every array over that
memory, not just one. Arrays a caller passes to a constructor are handed
over: read-only to whoever holds them, unless the caller passed a read-only
view and kept the original, as a solver does in situ; the topology then
notices the owner's writes.
Values (positions, field values, ghost marks) stay writable, and every cache
derived from them notices when they change. An array sent through
`tack.communication` becomes read-only to its sender unless it asks to keep
it. Nothing is copied: sharing memory with other libraries stays as it is.

This is the first stage of a longer direction: a completed dataset immutable
until it is explicitly reset. Stage 1 does not make values immutable, does not
add dataset generations or a reset, and does not freeze field mappings. Those
wait for the prerequisites listed under [Later stages](#later-stages).

## Why

Tack derives and keeps a lot from a dataset: faces, edges, point links, cell
groups, cell points, boundary surfaces, cell locators, ghost exchange plans.
An inventory of `tack.dataset` (at `4756ae3d`) found about forty such caches.
Five check whether what they were derived from has changed, all of them by
the values' revision (`arrays._revision`): the cell and point locators
(`serves()`), the measured cell geometry (`_mesh_kept`), the boundary
surface's geometry, and the serendipity spaces' full values. None of the
caches derived from structure checks anything. So this runs and gives wrong
answers silently:

```python
topology = td.UnstructuredTopology(types, offsets, connectivity)
faces = topology.faces() # cached on the topology
connectivity.fill(0) # or a kernel writing it
td.extract_surface(data) # uses the faces of the old connectivity
```

The caller's `connectivity` is the topology's own array (constructors keep a
Tack array as given), and several derived arrays share its memory: a
single-shape group's rows are `connectivity.reshape(...)`, `point_links`'
keys are `connectivity`, an `L2` space's offsets are `topology.offsets`, a
polygon topology's faces' `side_face` is its `cell_faces`. A write to one
reaches all of them.

## What exists today

- **Read-only arrays.** `Array._writable` is false for an `array_from_ptr`
import (by default) and a read-only DLPack import. Host writes
(`Array._check_writable`: `fill`, `copyto`, uploads) and kernel stores
(dispatch's `check_writable_arrays`, on every call, over only the arrays
the kernel may store to) refuse them. The flag is on one `Array` object,
so another array over the same memory (a `reshape`, a scalar view) can
still write.
- **Write counts.** `DeviceBuffer.version` counts every write Tack makes to a
buffer, shared by every array over it, and `arrays._revision` turns it into
a token. Writes Tack does not make (through memory shared by DLPack or
`wrap_ptr`) are not counted.
- **Zero-copy imports are read-only already.** The VTK interop's `copy=False`
imports offsets, connectivity, types and points as read-only arrays.

## Decisions

### D1. Seal structure at the buffer

`DeviceBuffer` gets a one-way `sealed` flag: set once, never cleared, shared
by every array over the buffer. Host writes and kernel stores check it beside
`_writable` and refuse with a message that says what the array is and to copy
it. A copy is writable. DLPack exports of sealed memory are flagged read-only.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do the main consumers accept read-only DLPack? It’s worth checking PyTorch and CuPy on a sealed connectivity before this lands, since some refuse read-only capsules.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked PyTorch 2.15 against a read-only Tack array: it asks for DLPack 1.0, gets the read-only flag, accepts the capsule and ignores the flag, so the tensor is writable and a write through it reached the Tack array. NumPy 2.4 honours the flag. So nothing refuses sealed memory; the risk is the other way, a consumer writing it. A legacy (pre-1.0) consumer gets it as writable, since that protocol cannot carry the flag. D1 now says the seal stops at DLPack, exports stay zero-copy, and a consumer's writes are outside D1, for D6. CuPy is listed as unchecked (it needs a CUDA machine).

Writes Tack does not make (VTK writing memory it shares with Tack) are not
refused: they remain under the interop protocol, as today.

**The seal stops at DLPack.** The flag reaches only a consumer that asks for
DLPack 1.0 (`max_version`); the legacy protocol cannot carry it, so such a
consumer gets the memory as writable. Consumers that get the flag differ:
NumPy 2.4 makes a read-only array, while PyTorch 2.15 accepts the capsule
and ignores the flag, so a write through the tensor reaches the sealed memory
(checked 2026-11-16). No consumer refuses it. Exporting sealed structure stays
zero-copy; what a consumer then writes is its own violation, outside D1, for
D6's checks.

The cost is one attribute read per array a kernel may store to, per call, on
the dispatch fast path.

*Recommended.*

### D2. What is sealed, and when

Sealed when built, the structure Tack itself makes:

- **Topologies' stored arrays that Tack allocated**, at the end of their
constructors: a topology a filter or reader makes (types, offsets,
connectivity; polyhedra's face and cell arrays; polygons' loops; polylines'
lines), and arrays a constructor converts or copies.
- **Derived structural arrays**, after their last construction write: faces
and edges and their sides, point links, cell points, groups' rows, matched
polygon facets, polyline ends, contour's sides, the boundary surface's
cells and points.
- **Spaces' tables**: an `H1` space's checked DOF table and its order-2
numbering, an `L2` space's orders and offsets.
- **An octree's tree** (`tree`, `level`, `key`), whose host copies the
octree reads.
- **Implicit arrays** have no buffer; sealing one seals the stored arrays it
holds.

**A caller's arrays are handed over, not sealed.** Constructors keep the
arrays they are given, without a copy, and in situ those are often a
solver's own (its mesh in Tack arrays, wrapped as a topology for analysis and
remeshed or refined in place between steps). A seal is one-way and holds the
memory, so sealing them would lock the solver out of its own memory for
good, even after the topology is dropped, and copying would cost the GPU
memory in situ cannot spare. The per-array flag (`_writable`) is the right
tool instead: it belongs to one `Array` object, so it marks who holds the
memory, not the memory itself.

- **`Array.read_only()`**, public: a view over the same memory that cannot
write, nor can any view made from it. No copy. Today only
`array_from_ptr` and read-only DLPack imports make such arrays.
- **Giving an array to a constructor hands it over**, as D5's sending does:
the object given becomes read-only to its caller. So the bug above is
still caught at the write: `connectivity.fill(0)` after building the
topology is refused, with a message to copy the array, or to pass
`connectivity.read_only()` and keep the original.
- **In situ, the solver keeps the only writable handle.** It passes
`conn.read_only()` and keeps `conn`, writing it to remesh in place. Every
array Tack gives out (`topology.connectivity`, groups' rows, point links'
keys, which are views of it) is read-only, so analysis code cannot write
the simulation's mesh by mistake.
- **The topology notices the owner's writes.** It records the revision
(`arrays._revision`) of the arrays it was given, and every structural
cache it keeps checks it on use: one check for the topology, not one per
cache. When it has moved, the topology drops its derived structure and
rebuilds it on demand, as D4's value caches do. Writes the owner makes
outside Tack's view need D4's explicit call.

Derived arrays that are views of a given array (a single-shape group's rows,
`point_links`' keys, an `L2` space's offsets) take its read-only view; only
memory Tack allocated for them is sealed, so a seal never reaches a caller's
buffer. A store to derived structure Tack made is still refused at the
write.

The handing over limits itself as D5's does: other objects over the same
memory made before it (a `reshape` kept from earlier, a DLPack export) can
still write.

Both are **breaking changes**, for the changelog: writing an array after
building a topology from it, and writing derived structure. Nothing in Tack,
its tests or its guides does either (the prototype, which sealed the
caller's arrays outright).

*Recommended.*

### D3. Structural attributes cannot be reassigned

Today every structural array is a plain attribute (`topology.connectivity =
other` works, and leaves every cache stale), as are counts and dimensions
(`num_points`, `point_dims`). They become read-only properties over private
storage. `PolylineTopology.faces()` sets attributes after construction today
and moves to private storage behind the same properties.

*Recommended.*

### D4. Every cache derived from values tracks their revision

Values stay writable, so a cache derived from them must notice a write: an

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revisions only see writes Tack makes. A solver writing positions or fields through DLPack or wrap_ptr, as a Catalyst adaptor does every step, moves nothing, so a cached locator still serves() moved points. Could stage 1 add an explicit “these values changed” call (like VTK’s Modified()), with the in-situ docs saying to call it each step?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Added to D4 in 539d964: Array.modified(), VTK's Modified(), which counts a write to the array's buffer as Tack's own writes are counted, so the locators, the other value caches and D2's topology check all notice. The in-situ docs will say to call it on each array written outside Tack, each step; D6's checks stay the safety net for a forgotten call.

automatic cache rebuilds, and a locator passed in explicitly that no longer
serves its dataset is refused (as `CellLocator.serves` does today). Already
tracked: the locators, `_mesh_kept`, the boundary surface's geometry,
serendipity's full values. To add:

- `Set` flags, derived from `Set.ids` (a plain public attribute today):
writing a set's ids leaves its flags, and every filter that reads them,
on the old members (the prototype confirmed it);
- the subgroups cached on a topology by `(group, spaces or sets)`, which
depend on a set's membership;
- subdivision's crossings, carried by field identity (to check).

Not needed after all: the ghost exchange plans a partitioned dataset keeps
map ghost entries to their owners by index, so moving points or changing
values leaves them right; only a renumbering would not, and D2 rules that
out. An octree's host copies of its tree are structure, sealed under D2.

And one the other way: a locator's cell neighbours depend on structure only,
but are rebuilt whenever the values change; they move to the topology.

**Writes Tack cannot see** (a solver or a Catalyst adaptor writing positions
or fields through DLPack or `wrap_ptr`, every step) move no revision, so a
cached locator would serve the old points. Stage 1 adds an explicit call,
`Array.modified()` (VTK's `Modified()`), which counts a write to the array's
buffer as Tack's own writes are counted, so every revision-tracked cache, and
D2's topology check, notices. The in-situ documentation says to call it on
each array written outside Tack, each step. D6's checks are the safety net
for a forgotten call.

**Protecting a simulation's values.** Values stay writable in stage 1, but
D2's `read_only()` lets an in-situ adaptor protect them with no copy: it
wraps the solver's arrays as `Field(space, u.read_only())` (and positions
alike), so a filter or a script cannot `fill` the simulation's velocity by
mistake, while the solver writes `u` each step and the caches notice. The
in-situ documentation shows this as the pattern.

*Recommended.*

### D5. Sent arrays are read-only to the sender

`Blocks.exchange` documents that a message's arrays belong to the receiver
afterwards, but nothing enforces it, and a same-rank receiver gets the
sender's very object. Each sent array becomes read-only to its sender after
`exchange` returns, on every transport, and a same-rank receiver gets a new
array over the same memory with the sent array's writability. This is
ownership moving, not immutability, so it is the per-array flag
(`_writable`), not D1's buffer seal, which would lock the receiver too. Its
limit: views of the sent array made before sending can still write.

Reusing one send buffer step after step is the normal pattern for a
solver's halo exchange, and the hazard is narrower than the whole of
`exchange`: across ranks, `exchange` waits for every request before it
returns, so the receiver holds its own copy and the sender may write its
buffer again. Only a same-rank receiver (and the threaded test world) gets
the sender's memory. So `exchange` takes **`keep=True`**: the sender keeps
its arrays writable, and a same-rank receiver gets a copy instead of the
sender's memory. That is the supported pattern for reused send buffers, at
the cost of a copy on the same rank only.

Without `keep`, a sender that writes a sent array gets an error naming
`keep=True`: a **breaking change**. Nothing in Tack reuses one (`Plan.run`
gathers into a fresh array per send).

*Recommended.*

### D6. Checking external writes: later

Tack cannot see writes through memory it shares (VTK writing a connectivity
it handed Tack). A debug switch of its own (never `TACK_DEBUG`, whose suite
run must stay fast) could hash shared structural memory and check it at each
filter's entry. It catches violations of a protocol, not Tack's own writes,
which D1 refuses. Not needed for stage 1's safety.

*Recommended: defer to stage 2.*

## Not in stage 1

- Field and set mappings stay editable (`data.fields[name] = field`): 75 such
assignments in tack-vis and over 600 in tests, docs and examples.
- Values are not locked: no leases, no generations, no `initialize()`,
`snapshot()` or `with_fields`.
- `update_ghosts`, communication plans and AMR fills keep writing values in
place; their writes are counted, so value caches see them.

## Later stages

Value immutability (a dataset published as a generation, values locked while
it is held, an explicit reset) waits for three things, each designed and
tested first:

1. **Caches without reference cycles.** A topology's cached locator holds a
dataset, so a lock that ends when the dataset is freed would end when the
cycle collector runs.
2. **A first-class in-situ loop.** `advance(u)` must not fail because the
previous step's dataset still holds `u`.
3. **A rule for in-place distributed writes**: `update_ghosts`, `Plan.run`
and AMR fills, which write published data on purpose.

## Prototype

Branch `experiment/mutation-rules` (`a26518cd`, local): D1 and D2 (the seal,
applied to topologies' stored arrays, their derived structure and an
octree's tree), D4 for sets and the subgroups by them, and D5 (the October
experiment, `e86415e7`). Not prototyped: D3's properties and D6. On `main`
at `4756ae3d`, the M1 Max CPU.

1. **Sealing breaks nothing in Tack.** The whole suite passes with structure
sealed: 6859 passed, 372 skipped, no failures. So do the 164 guide
examples. Nothing in Tack, its tests or its guides writes a topology's
structure after building it; the refusal costs only code that does,
outside.
2. **It costs about 18 ns per array a kernel may store to**, in the check on
every call (364 against 419 ns for three such arrays, the best of nine
runs on a loaded machine). A warm call is 7 to 10 µs, so under 1%.
3. **Zero-copy export still shares memory.** A topology with 64-bit ids
exported to VTK with `copy=False` gives VTK the same memory as on `main`;
VTK's array over it is writable to VTK, as before, under the interop
protocol (D6). 32-bit ids are converted, on `main` too. A GPU export was
not tried: it copies an array that is not `_writable`, and a sealed array
still is, so it would share as today.
4. **Value caches.** Writing a set's ids left its flags on the old members
(confirmed); keyed on the ids' revision, the flags and the subgroups by
the set follow the write, and the reassigned ids too. The cell locator
already notices moved points (a probe after moving every point finds
nothing, as it should).
5. **Communication.** With sent arrays read-only to the sender, the
communication, ghost update, ghost cells and partitioned tests pass, over
threads and under `mpiexec -n 3`, staged and GPU-aware, but for the one
test that asserted the receiver got the sender's very object, the
intended change; it now checks that the receiver gets the same memory,
writable, and the sender cannot write it.

The motivating bug is caught: after `topology.faces()`, a host write to the
connectivity and a kernel store to the faces' rows are both refused, with a
message to copy the array. **Verdict: stage 1 is viable** as planned, with D4
smaller than the inventory suggested.

## Open questions

- **When.** D2 and D5 refuse what worked before, which no deprecation can
phase in. A release could first warn where it will later refuse, if the
break should be softer than a changelog entry.
- **Sets.** Membership is treated as a value here (writable, tracked). Sealing
a set's ids instead would be simpler, but forbids editing a set in place.
- **Geometry.** Stage 1 leaves positions writable, as every value. Whether
some value (geometry first) should be sealed before stage 3 is open.
- **CuPy.** Whether it honours, ignores or refuses DLPack's read-only flag is
unchecked (it needs a CUDA machine). D1 holds whichever it does.
- **Not prototyped:** D2's handing over and `Array.read_only()` (the
prototype sealed the caller's arrays outright), `Array.modified()` and
`exchange(keep=True)`.

## Order

Each step is its own pull request, linking this plan:

1. D1 and D2, the seal and where it applies, `Array.read_only()` and the
handing over, the topology's check of the arrays it was given, and D3's
properties.
2. D4, the value caches' revisions, with a test per cache, and
`Array.modified()`.
3. D5, sent arrays, with `keep=True`.
4. This plan's status: done, with what changed on the way.
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -251,6 +251,7 @@ nav:
- design/plans/index.md
- Cut filters' memory: design/plans/cut-filters-memory.md
- Inspecting device-class objects: design/plans/inspect-template.md
- Mutation rules, stage 1: design/plans/mutation-rules.md
- Contracts: contracts/index.md
- Dataset API design: vis-guide/design/dataset-api.md
- Polyhedra: vis-guide/design/polyhedra.md
Expand Down
Loading