Skip to content

[core][common]support row range read of a DataSplit - #9705

Open
weijietong wants to merge 3 commits into
apache:masterfrom
weijietong:range_read_1
Open

[core][common]support row range read of a DataSplit#9705
weijietong wants to merge 3 commits into
apache:masterfrom
weijietong:range_read_1

Conversation

@weijietong

Copy link
Copy Markdown
Contributor

Purpose

This is to solve issue.

Tests

tests for pk table,append table, DE table.

@JingsongLi

Copy link
Copy Markdown
Contributor

Why not just use IndexedSplit?

@weijietong

Copy link
Copy Markdown
Contributor Author

The critical RowRange use case is slicing the stream after a filter — e.g. "give me the 1000th–2000th matching rows for training." Even composed with a filter, IndexedSplit only ANDs two physical-position bitmaps; it never renumbers rows into a
post-filter effective sequence, so it cannot express "the N-th row after filtering."

This is exactly what AbstractDataTableRead.outerWrap encodes:

  • RowRange + filter (executeFilter): outerWrap=true — RowRange wraps outside the filter, slicing the filtered effective stream.
  • IndexedSplit + filter: both reduce to physical-position bitmaps ANDed together — no "post-filter renumbering" step.

The AI training-shard use case (why RowRange exists)

In AI / ML data loading, a common pattern is sharding a dataset by effective sample position:

  • A training job with worldSize=W, rank=R wants its local shard — the rows at effective positions [R*N, (R+1)*N - 1] of the dataset.
  • With filters active (e.g. quality filters, column predicates) or deletion vectors, the "effective row order" is the filtered/compacted sequence. Each worker must receive a deterministic, contiguous slice of that sequence, not of the raw file layout.
  • RowRange models exactly this: a single contiguous [start, end] in the 0-based effective-row space of a split. The reader either pushes it down as a selection bitmap (parquet, full-scan — no extra rows decoded) or wraps a single RangeSkipReader over
    the filtered output (filter/DV/ORC/merge-tree) to skip start and take count.

IndexedSplit cannot express this:

  • It cuts physical row-ids, which have no relation to the post-filter effective order — worker rank=R wouldn't get a contiguous slice of matching rows.
  • It requires firstRowId (so plain append tables are out) or a pre-built global index.
  • For merge-tree primary-key tables, physical position ≠ output order, so the shard would be scrambled.

Conclusion

IndexedSplit and RowRange are complementary, not interchangeable:

  • IndexedSplit: read a discrete set of physical row-ids (vector search / global-index hits), AND-composed with filters in the physical-position space, requires row-ids or a pre-built index, excludes append tables / post-filter slicing / merge-output
    slicing.
  • RowRange: read a contiguous slice in effective-row space (AI training shards / pagination / range queries), applied as skip+limit over the filtered effective stream, works across append / primary-key / DE tables, no pre-built index needed.

RowRange was introduced precisely to cover IndexedSplit's blind spots — append tables, post-filter effective-row slicing, and merge-output slicing — all validated by tests in this PR (testAppendOrcRowRange..., testRowRangeWithFilterWrapsOutsideFilter,
testPrimaryKeyRowRangeOverMergeRead). They cover behavior IndexedSplit cannot express, so RowRange cannot be replaced by it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants