fix(indexing): make every view executable with its source grid - #4349
Merged
d-v-b merged 18 commits intoSep 16, 2026
Merged
Conversation
Assisted-by: Codex:GPT-6
Assisted-by: Codex:GPT-6
Assisted-by: Codex:GPT-6
Correct mathematical API documentation to match supported coordinate, grid, and chunk projection contracts. Assisted-by: Codex:GPT-6
Assisted-by: Codex:GPT-6
Retain the existing unsigned selector fix and update the unsupported mixed-dependency error assertion for general intersection routing. Assisted-by: Codex:GPT-6
…oundaries Assisted-by: Codex:GPT-6
Use lexicographic tuple grouping when chunk indices contain negative values. Cover shared one-axis and two-axis array dependencies, repeated points, and extreme signed coordinates. Assisted-by: Codex:GPT-6
Assisted-by: Codex:GPT-6
Assisted-by: Codex:GPT-6
github-actions
Bot
added
the
needs release notes
Assisted-by: Codex:GPT-6
Assisted-by: Codex:GPT-6
Documentation build overview
24 files changed ·
|
Remove implementation history and unsupported historical claims from source and test docstrings. Distinguish immutable coordinate mappings from mutable source values. Assisted-by: Codex:GPT-6
Assisted-by: Codex:GPT-6
Preserve current wire bounds validation and Dask tokenization while retaining the partition execution feature. Assisted-by: Codex:GPT-6
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4349 +/- ##
=======================================
Coverage 94.34% 94.34%
=======================================
Files 92 92
Lines 12948 12948
=======================================
Hits 12216 12216
Misses 732 732 🚀 New features to boost your workflow:
|
Remove Partition.result and partition-local source windows. Preserve the source grid in derived views and supply projections for every LazyArray reader call, including unpartitioned reads. Assisted-by: Codex:GPT-6
d-v-b
changed the title
d-v-b
marked this pull request as ready for review
September 15, 2026 22:45
github-actions
Bot
added
the
zarr-indexing
d-v-b
added a commit
to d-v-b/zarr-python
that referenced
this pull request
Sep 16, 2026
Add synchronous writes through composed selections and an explicit eager adapter for array consumers. Cover deferred reads, source mutation, masks, aliases, and repeated destinations. Assisted-by: Codex:GPT-6 Rebased-onto: upstream/main after zarr-developers#4345-zarr-developers#4349 merged; conflicts resolved with ClaudeCode:claude-fable-5-1
d-v-b
added a commit
that referenced
this pull request
Sep 16, 2026
* feat(indexing): make LazyArray indexing lazy by default Add synchronous writes through composed selections and an explicit eager adapter for array consumers. Cover deferred reads, source mutation, masks, aliases, and repeated destinations. Assisted-by: Codex:GPT-6 Rebased-onto: upstream/main after #4345-#4349 merged; conflicts resolved with ClaudeCode:claude-fable-5-1 * fix(indexing): plan fancy writes against the source's write grid The non-affine write fallback assigned one element at a time, which on a chunked source is one chunk read-modify-write per element: a 50-row orthogonal write into a 1000x1000 zarr array with 100x100 chunks cost 49,999 chunk writes and 27 s against 100 writes and 10 ms natively. `write_into` now scatters in bulk. NumPy sources get one fancy assignment. A readable source with a write grid, which `LazyArray.write` discovers from `write_chunk_sizes` then `chunks`, is written one cell at a time: the cell's touched hull is read once, updated in memory with last-occurrence-wins semantics, and written back with one basic slice. The same write into zarr now costs 100 chunk writes. Sources with no grid, transforms the planner cannot factor, and sources that cannot be read keep the per-element path, which never reads. Also: order the rank check before the value copy while still validating values for an empty selection; keep the payload of a masked zero-rank affine write, which `np.flip` with no axes was turning into the `masked` singleton; note that writes bypass the reader and pin that with a test; note that iterated elements are views; add API pages for the writer and eager adapter; and describe the shipped indexing surface in the unreleased `LazyArray` changelog fragment instead of its removed `.lazy` accessor. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(indexing): keep literal domains on views, positional keys unchanged `LazyArray` re-zeroed the domain of every derived view, so a view forgot which coordinates it was cut from and placement needed a side channel. TensorStore keeps literal coordinates on views; NumPy users expect positional keys. These are separable: the key dialect says how a key is read, the domain says what the view remembers. Views now keep their literal domain: `a[10:20]` has domain `[10, 20)` and `a[10:20][2:5]` has `[12, 15)`, while `a[10:20][0]` is still the first element because positional keys are normalized against the domain's origin, which `normalize_positional_selection` already did. Two literal keys join the NumPy ones, as in TensorStore's `__getitem__`: an `IndexDomain` restricts the view to coordinates of its own domain (empty intervals outside it are refused too) and an `IndexTransform` composes onto it. Box partitions keep the request's coordinates, so a part view's domain is a sub-domain of its parent's; a part placed by index arrays keeps a fresh zero-origin domain, and `out_selection` is the placement in both cases. A reversed view shows the negative origin the algebra already produced. Two frames stay zero-origin by construction so no consumer has to subtract an origin: `ReadContext` re-bases its transform, and `parts()` re-bases each projection's `cell_transform` from the request's literal domain to positions in the view's result buffer, which is what `Partition.projection` has always documented. A table-driven test states the expected literal domain for every indexing form and nested chain, checked against the transform algebra. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(indexing): state that a key's type picks relative or absolute indexing Make the one rule about indexing a LazyArray prominent: NumPy keys are positions relative to the view, an `IndexDomain` key names absolute coordinates of the view's domain, an `IndexTransform` key composes onto it, no key type has two readings, and every view keeps its absolute domain whichever key produced it. A guide section with a runnable snippet carries the table and the pandas `ix` / TensorStore comparison; the module docstring, README, and API page point at it. Assisted-by: ClaudeCode:claude-fable-5-1 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AI-authored PR that removes some redundancy in subsets of lazyarrays
🤖 AI text below 🤖
Make every
LazyArrayview executable through projection-aware readers. Previously,part.view.result()and unpartitioned reads bypassed planning and passedprojection=None, so they failed with readers that needed chunk-local coordinates.LazyArray.result()now plans every nonempty read and supplies both the source-global transform and a chunk projection. Partition views retain their source grid and reader, so they can be read, indexed, or repartitioned like any other view. Callers schedulepart.view.result()and assemble its values atpart.out_selection; partitions describe the plan and placement rather than providing a second execution API.Remove the special partition-local source window. All views now use the full source shape as
base_shape, including partition views.with_partsandwith_parts_per_axistherefore consistently describe the source grid.unpartitioned()treats the source as one grid cell and still supplies a projection. Independently executed views plan for their own output domain; their placement transform need not equal the parent's placement transform. Prepared parent plans remain reusable throughview.result(parts=parts).Validation: 1,672 tests and doctests passed, no skips, including Dask and TensorStore coverage; strict documentation build passed; Pyright reports 0 errors and 90 warnings. Tests cover independent, nested, reversed, and repartitioned views, masked output, reader errors, scalars, repeated indices, and empty selections. Six reader-contract cases reproduced the missing-projection failure before the change. Applicable repository hooks passed; the unrelated root mypy hook was skipped (it previously reported four unused-ignore errors in files unchanged by this PR). Independent review found no actionable issues and validated 8,222 randomized nested and repartitioned reads.