[core][common]support row range read of a DataSplit - #9705
Conversation
|
Why not just use |
|
The critical RowRange use case is slicing the stream after a filter — e.g. "give me the 1000th–2000th matching rows for training." Even composed with a filter, IndexedSplit only ANDs two physical-position bitmaps; it never renumbers rows into a This is exactly what AbstractDataTableRead.outerWrap encodes:
The AI training-shard use case (why RowRange exists) In AI / ML data loading, a common pattern is sharding a dataset by effective sample position:
IndexedSplit cannot express this:
Conclusion IndexedSplit and RowRange are complementary, not interchangeable:
RowRange was introduced precisely to cover IndexedSplit's blind spots — append tables, post-filter effective-row slicing, and merge-output slicing — all validated by tests in this PR (testAppendOrcRowRange..., testRowRangeWithFilterWrapsOutsideFilter, |
Purpose
This is to solve issue.
Tests
tests for pk table,append table, DE table.