Repository navigation
Optimize arrays top down as execution reaches them, and retry a parent when its children change - #11090
Conversation
02fa232 to
d889c87
Compare
d889c87 to
6f69042
Compare
6f69042 to
0cf0f95
Compare
0cf0f95 to
1d684a8
Compare
Merging this PR will degrade performance by 17.68%1 warning
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
4db3b2a to
13495e7
Compare
With a session, the optimizer now optimizes the children of a node that were never optimized before it tries the rules of the node, and it does the same for the nodes that each rewrite builds. Children already marked are left without visiting their subtrees. The optimizer marks every array it returns, and it remembers for one call the array it rewrote each shared node into. optimize_recursive is now the same as optimize_ctx. The session-less optimize still rewrites only the root and does not mark the result, so a later optimize with a session still tries its kernels. execute_until now optimizes the root before it starts, each child that execution focuses, the result of each Done step, and the array it returns. Single-step execution optimizes with the session instead of trying only the static reduce and parent-reduce rules. Signed-off-by: Robert Kruszewski <github@robertk.io>
trace_op! dropped its arguments when the trace harness was not compiled in. Thus a value that only a trace event reads, such as any_optimizations in the optimizer, was reported as unused in plain and CodSpeed builds of vortex-array. The macro now borrows its arguments in a branch that never runs, so they are type-checked but not evaluated. Signed-off-by: Robert Kruszewski <github@robertk.io>
Execution now optimizes each array that it reaches unless the array is already marked. Arrays that a reader decodes are marked, but benchmarks build their inputs with constructors. Thus the timed region also timed the optimizer, and CodSpeed reported about 400 regressions that production does not see. test_harness::mark_optimized optimizes an input during setup and panics if the optimizer rewrites it, so a benchmark cannot time a different array than the one it built. test_harness::mark_decoded marks each node without trying any rule, as decoding does. The cosine similarity benchmark uses it for an extension array over constant storage, which a file can hold but the optimizer folds into a constant. Lazy arrays that a benchmark builds inside the timed closure stay unmarked, because their optimization is part of the cost it measures. Signed-off-by: Robert Kruszewski <github@robertk.io>
execute_until now keeps one optimizer for the whole execution. Thus a shared node that one step rewrote is reused when a later step reaches it through another parent, and is not rewritten again. The optimizer keys its cache of rewritten shared nodes by ArrayByPtr, a key that compares arrays by identity. The key holds the array, so its address cannot be reused while the optimizer exists. Signed-off-by: Robert Kruszewski <github@robertk.io>
13495e7 to
60c1c8b
Compare
split_extreme and the widen_decimal benchmarks came in with the rebase on develop. They built their inputs with constructors and did not mark them, so the timed region also timed the optimizer. They now mark their inputs during setup, as the other benchmarks do. Signed-off-by: Robert Kruszewski <github@robertk.io>
The optimizer now tries the rules of a node before it optimizes the children of the node. Thus a rule that drops or replaces a child does not optimize the subtree of that child first. After the children are optimized, the optimizer tries the rules of the node again if a child changed. Thus a parent rule or kernel that is keyed on the final encoding of a child still applies, such as LIKE over a slice of an FSST array. When a rule rewrites a node, the optimizer still optimizes the nodes that the rule built before the next pass on the node. This stops two rules that are inverses of each other from rewriting a node back and forth. The trace snapshot of the stack parent kernel test now shows the parent optimized before its child. Signed-off-by: Robert Kruszewski <github@robertk.io>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
commented
Nov 4, 2026
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.052x ➖, 0↑ 2↓)
|
commented
Nov 4, 2026
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.972x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.010x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.993x ➖, 0↑ 0↓)
|
commented
Nov 4, 2026
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.016x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
|
commented
Nov 4, 2026
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
|
commented
Nov 4, 2026
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.990x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.990x ➖, 1↑ 0↓)
|
commented
Nov 4, 2026
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
|
commented
Nov 4, 2026
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.088x ➖, 1↑ 2↓)
datafusion / parquet / ns (1.029x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (1.074x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.012x ➖, 1↑ 0↓)
|
commented
Nov 4, 2026
Benchmarks: TPC-DS SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 1↑ 0↓)
|
With a session, the optimizer now works bottom up, and execution optimizes each node that it reaches. This prepares for lazy constructors in #11091.
Changes
optimize_ctxoptimizes the children of a node that were never optimized before it tries the rules of the node. It does the same for the nodes that each rewrite builds. It does not visit marked subtrees, and it marks every array that it returns. Thus inverse rules, such asslice(get_item)andget_item(slice), cannot loop.optimize_recursiveis now the same asoptimize_ctx.optimizestill rewrites only the root and does not mark the result. Thus a lateroptimize_ctxstill tries the session kernels.execute_untiloptimizes the root before it starts, each child that execution focuses, the result of eachDonestep, and the array that it returns. Single-step execution optimizes with the session, instead of trying only the static reduce and parent-reduce rules.Chunked::reducedoes not reduce an empty variant array, becauseVarianthas no empty canonical form.