[VL] Support parquet.enable.page.index write option - #12760
Open
sezruby wants to merge 1 commit into
Open
Conversation
|
Run Gluten Clickhouse CI on x86 |
sezruby
force-pushed
the
pr-page-index-write
branch
from
August 12, 2026 18:25
3355a00 to
a5a5fcf
Compare
|
Run Gluten Clickhouse CI on x86 |
Member
|
@sezruby would you please do a rebase? The parquet index patch is already landed |
sezruby
force-pushed
the
pr-page-index-write
branch
from
August 14, 2026 15:58
a5a5fcf to
95a1ca3
Compare
|
Run Gluten Clickhouse CI on x86 |
Add a parquet.enable.page.index write option that maps to the Velox ParquetWriterOptions::enableWritePageIndex field (facebookincubator/velox#18325): - GlutenConfig (Scala/C++): define the parquet.enable.page.index key. - VeloxParquetWriterInjects: forward the option to native write config. - VeloxWriterUtils::makeParquetWriteOption: set enableWritePageIndex, defaulting to on when the option is unset. When enabled, the writer emits the Parquet column index and offset index (page index); when disabled, they are omitted. The option defaults to true to match Spark/parquet-mr, which write the page index by default (SPARK-26345); Velox's writer leaves it opt-in. Set parquet.enable.page.index=false to turn it off. Documented in the parquet write configuration table and covered by a VeloxParquetWriteSuite test that asserts the ColumnIndex/OffsetIndex presence tracks the option (enabled, disabled, and unset/default) and that data round-trips. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sezruby
force-pushed
the
pr-page-index-write
branch
from
August 14, 2026 16:12
95a1ca3 to
ce5a0b5
Compare
|
Run Gluten Clickhouse CI on x86 |
sezruby
marked this pull request as ready for review
August 14, 2026 16:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
Add a
parquet.enable.page.indexwrite option that maps to Velox'sParquetWriterOptions::enableWritePageIndex(facebookincubator/velox#18325).When enabled, the native Parquet writer emits the column index and offset index
(page index); when disabled, they are omitted.
The option defaults to
true, matching Spark/parquet-mr, which write the pageindex by default (SPARK-26345); Velox's writer leaves it opt-in. Set
parquet.enable.page.index=falseto turn it off.GlutenConfig(Scala and C++): define theparquet.enable.page.indexkey.VeloxParquetWriterInjects: forward the option into the native write config.VeloxWriterUtils::makeParquetWriteOption: setenableWritePageIndex,defaulting to on when the option is unset.
How was this patch tested?
Added a
VeloxParquetWriteSuitetest that writes withparquet.enable.page.indexset to
true, set tofalse, and left unset, and asserts (via parquet-mr footerinspection) that the Parquet
ColumnIndex/OffsetIndexreferences are presentwhen the option is on or defaulted and absent when it is off, and that the written
data round-trips.
The enabled and disabled write paths were validated end-to-end in a Docker build
against a Velox snapshot that includes the
enableWritePageIndexfield; thedefault (unset) path sets the same
enableWritePageIndex=trueas theexplicit-enable path.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)