Skip to content

Ask the compressor plugin which encoding it produces - #11307

Open
robert3005 wants to merge 3 commits into
developfrom
rk/dict-probe-may-compress-to
Open

Ask the compressor plugin which encoding it produces#11307
robert3005 wants to merge 3 commits into
developfrom
rk/dict-probe-may-compress-to

Conversation

@robert3005

@robert3005 robert3005 commented Nov 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Replace CompressorPlugin::may_compress_to(chunk, encoding) with encoding_for(chunk), which returns the encoding that compress_chunk gives. The dict probe compares the result with Dict.

CascadingCompressor::encoding_for runs the normal scheme selection. It skips the full compression when the winner declares a fixed root encoding through the new Scheme::root_encoding.

🤖 Generated with Claude Code

@robert3005
Skip the dict probe compression when no dict scheme can win
72e22e0 
Add `may_compress_to` to `CascadingCompressor` and `CompressorPlugin`. The
check runs only the immediate estimates of the schemes that produce the
requested encoding, and does not compress. `DictStrategy` uses it to skip the
full sample compression when the result cannot be a dictionary.

Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005
robert3005 force-pushed the rk/dict-probe-may-compress-to branch from f726498 to 72e22e0 Compare November 7, 2026 00:01
@robert3005 robert3005 added the changelog/feature A new feature label Nov 7, 2026
@codspeed

codspeed Bot commented Nov 7, 2026 •
edited
Loading

Copy link
Copy Markdown

Merging this PR will not alter performance

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 1 benchmark is dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

✅ 1916 untouched benchmarks
⏩ 48 skipped benchmarks1

Comparing rk/dict-probe-may-compress-to (97433ad) with develop (a1fbac6)

Open in CodSpeed

Footnotes

  1. 48 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@robert3005
Fix ambiguous Dict id and redundant doc link
ca200fe 
Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005 robert3005 added the action/bench-compress Run only the compression benchmark on this PR label Nov 7, 2026
@github-actions github-actions Bot removed the action/bench-compress Run only the compression benchmark on this PR label Nov 7, 2026
@github-actions

github-actions Bot commented Nov 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done ca200fe compress-bench 1 Explore Profiling Data

Powered by Polar Signals Cloud

@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Benchmarks: Compression 📖

Commits: PR ca200fe8 vs base 0fee7df8

vortex / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
compress time/arade 451560444 / 455749386 / -0.9% 452709594 / 457824324 / -1.1% 1.00 / 1.00 / +0.2%
compress time/bimbo 2284818920 / 2323995800 / -1.7% 2264723939 / 2362122084 / -4.1% 1.01 / 0.98 / +2.5%
compress time/cmsprovider 1059753158 / 1070247964 / -1.0% 1044651048 / 1056919803 / -1.2% 1.01 / 1.01 / +0.2%
compress time/euro2016 170301278 / 177432394 / -4.0% 171580659 / 178015360 / -3.6% 0.99 / 1.00 / -0.4%
compress time/food 206956412 / 208908904 / -0.9% 207607554 / 211197578 / -1.7% 1.00 / 0.99 / +0.8%
compress time/hashtags 395545998 / 399350600 / -1.0% 391890990 / 398095640 / -1.6% 1.01 / 1.00 / +0.6%
compress time/sparse-wide-flat-runs 95844538 / 95603820 / +0.3% 93548532 / 100348465 / -6.8% 1.02 / 0.95 / +7.5%
compress time/sparse-wide-lists-runs 649831404 / 688777480 / -5.7% 655726493 / 692760886 / -5.3% 0.99 / 0.99 / -0.3%
compress time/sparse-wide-lists-shuffled 310692274 / 307152626 / +1.2% 292239907 / 302281560 / -3.3% 1.06 / 1.02 / +4.6%
compress time/sparse-wide-nested-child-nulls 629560340 / 627930863 / +0.3% 619981499 / 626116615 / -1.0% 1.02 / 1.00 / +1.3%
compress time/sparse-wide-nested-runs 627806418 / 630121279 / -0.4% 622940170 / 626297784 / -0.5% 1.01 / 1.01 / +0.2%
compress time/statpopgen 430982710 / 421305305 / +2.3% 432069678 / 421396262 / +2.5% 1.00 / 1.00 / -0.2%
compress time/taxi 205963906 / 208571280 / -1.3% 207797838 / 211669489 / -1.8% 0.99 / 0.99 / +0.6%
compress time/tpc-h l_comment canonical 413978765 / 415780274 / -0.4% 413083537 / 414702570 / -0.4% 1.00 / 1.00 / -0.0%
compress time/tpc-h l_comment chunked 413165920 / 416542856 / -0.8% 412490734 / 415989885 / -0.8% 1.00 / 1.00 / +0.0%
compress time/wide table cols=100 chunks=1 rows=1000 5812996 / 5693632 / +2.1% 7851840 / 8050488 / -2.5% 0.74 / 0.71 / +4.7%
compress time/wide table cols=100 chunks=50 rows=1000 5792118 / 5827972 / -0.6% 7840435 / 7939696 / -1.3% 0.74 / 0.73 / +0.6%
compress time/wide table cols=1000 chunks=1 rows=1000 56924604 / 58047368 / -1.9% 59950788 / 60641061 / -1.1% 0.95 / 0.96 / -0.8%
compress time/wide table cols=1000 chunks=50 rows=1000 57225680 / 56378062 / +1.5% 58296753 / 60482255 / -3.6% 0.98 / 0.93 / +5.3%
compress time/wide table cols=10000 chunks=1 rows=1000 648397642 / 645154916 / +0.5% 657943588 / 659364796 / -0.2% 0.99 / 0.98 / +0.7%
compress time/wide table cols=10000 chunks=50 rows=1000 647957448 / 649369606 / -0.2% 638211825 / 655369973 / -2.6% 1.02 / 0.99 / +2.5%
decompress time/arade 14147862 / 14254787 / -0.8% 16174388 / 15146961 / +6.8% 0.87 / 0.94 / -7.1%
decompress time/bimbo 68486390 / 67234698 / +1.9% 76342588 / 74352611 / +2.7% 0.90 / 0.90 / -0.8%
decompress time/cmsprovider 38208366 / 38333994 / -0.3% 40454615 / 41888914 / -3.4% 0.94 / 0.92 / +3.2%
decompress time/euro2016 14207840 / 14301002 / -0.7% 15681145 / 15685820 / -0.0% 0.91 / 0.91 / -0.6%
decompress time/food 3839563 / 3853166 / -0.4% 5435886 / 4718330 / +15.2% 🔴 0.71 / 0.82 / -13.5%
decompress time/hashtags 40993952 / 40370265 / +1.5% 41414009 / 40305507 / +2.8% 0.99 / 1.00 / -1.2%
decompress time/sparse-wide-flat-runs 24557622 / 24815762 / -1.0% 24486036 / 24465978 / +0.1% 1.00 / 1.01 / -1.1%
decompress time/sparse-wide-lists-runs 15404721 / 14610008 / +5.4% 15486980 / 15726851 / -1.5% 0.99 / 0.93 / +7.1%
decompress time/sparse-wide-lists-shuffled 57381261 / 58063404 / -1.2% 58172115 / 59026562 / -1.4% 0.99 / 0.98 / +0.3%
decompress time/sparse-wide-nested-child-nulls 27941414 / 27384628 / +2.0% 27080455 / 27617002 / -1.9% 1.03 / 0.99 / +4.1%
decompress time/sparse-wide-nested-runs 27712212 / 27708722 / +0.0% 27897627 / 28321471 / -1.5% 0.99 / 0.98 / +1.5%
decompress time/statpopgen 317429318 / 316331566 / +0.3% 320695445 / 327973008 / -2.2% 0.99 / 0.96 / +2.6%
decompress time/taxi 9838819 / 9780972 / +0.6% 12208743 / 11865535 / +2.9% 0.81 / 0.82 / -2.2%
decompress time/tpc-h l_comment canonical 15010306 / 15157176 / -1.0% 15185329 / 15495629 / -2.0% 0.99 / 0.98 / +1.1%
decompress time/tpc-h l_comment chunked 15037082 / 15195176 / -1.0% 15272957 / 16240137 / -6.0% 0.98 / 0.94 / +5.2%
decompress time/wide table cols=100 chunks=1 rows=1000 1574508 / 1584389 / -0.6% 1667797 / 1702606 / -2.0% 0.94 / 0.93 / +1.5%
decompress time/wide table cols=100 chunks=50 rows=1000 1564244 / 1586252 / -1.4% 1690619 / 1665660 / +1.5% 0.93 / 0.95 / -2.8%
decompress time/wide table cols=1000 chunks=1 rows=1000 14260230 / 14163566 / +0.7% 14699062 / 14674298 / +0.2% 0.97 / 0.97 / +0.5%
decompress time/wide table cols=1000 chunks=50 rows=1000 14252168 / 14117692 / +1.0% 14857922 / 14575630 / +1.9% 0.96 / 0.97 / -1.0%
decompress time/wide table cols=10000 chunks=1 rows=1000 148310773 / 146713102 / +1.1% 150478521 / 146346860 / +2.8% 0.99 / 1.00 / -1.7%
decompress time/wide table cols=10000 chunks=50 rows=1000 149479445 / 146011660 / +2.4% 151806081 / 147237924 / +3.1% 0.98 / 0.99 / -0.7%
vortex / vortex-file-compressed / bytes (1.000x ➖, 0↑ 0↓)
name bytes (PR / base / %diff)
vortex-file-compressed size/arade 142213732 / 142213732 / +0.0%
vortex-file-compressed size/bimbo 458581076 / 458581076 / +0.0%
vortex-file-compressed size/cmsprovider 420808044 / 420808044 / +0.0%
vortex-file-compressed size/euro2016 164566436 / 164572116 / -0.0%
vortex-file-compressed size/food 38951000 / 38951000 / +0.0%
vortex-file-compressed size/hashtags 195463900 / 195463900 / +0.0%
vortex-file-compressed size/sparse-wide-flat-runs 6872768 / 6872768 / +0.0%
vortex-file-compressed size/sparse-wide-lists-runs 10900860 / 10900860 / +0.0%
vortex-file-compressed size/sparse-wide-lists-shuffled 13447164 / 13447164 / +0.0%
vortex-file-compressed size/sparse-wide-nested-child-nulls 11303724 / 11303724 / +0.0%
vortex-file-compressed size/sparse-wide-nested-runs 11504220 / 11504220 / +0.0%
vortex-file-compressed size/statpopgen 176342196 / 176342196 / +0.0%
vortex-file-compressed size/taxi 49997860 / 49997860 / +0.0%
vortex-file-compressed size/tpc-h l_comment canonical 178708640 / 178708640 / +0.0%
vortex-file-compressed size/tpc-h l_comment chunked 178708640 / 178708640 / +0.0%
vortex-file-compressed size/wide table cols=100 chunks=1 rows=1000 934856 / 934856 / +0.0%
vortex-file-compressed size/wide table cols=100 chunks=50 rows=1000 934856 / 934856 / +0.0%
vortex-file-compressed size/wide table cols=1000 chunks=1 rows=1000 9333656 / 9333656 / +0.0%
vortex-file-compressed size/wide table cols=1000 chunks=50 rows=1000 9333656 / 9333656 / +0.0%
vortex-file-compressed size/wide table cols=10000 chunks=1 rows=1000 93357656 / 93357656 / +0.0%
vortex-file-compressed size/wide table cols=10000 chunks=50 rows=1000 93357656 / 93357656 / +0.0%
vortex / parquet / ns (0.997x ➖, 0↑ 0↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
compress time/arade 1162179820 / 1153590090 / +0.7% 1165594209 / 1156427880 / +0.8% 1.00 / 1.00 / -0.0%
compress time/bimbo 2124565710 / 2135483488 / -0.5% 2119417934 / 2146687689 / -1.3% 1.00 / 0.99 / +0.8%
compress time/cmsprovider 1666455028 / 1665245044 / +0.1% 1662867478 / 1668887790 / -0.4% 1.00 / 1.00 / +0.4%
compress time/euro2016 996885136 / 994748905 / +0.2% 1023557378 / 998015221 / +2.6% 0.97 / 1.00 / -2.3%
compress time/food 405531662 / 406120166 / -0.1% 408937008 / 406728480 / +0.5% 0.99 / 1.00 / -0.7%
compress time/hashtags 537971138 / 537074092 / +0.2% 553910834 / 543868857 / +1.8% 0.97 / 0.99 / -1.6%
compress time/sparse-wide-flat-runs 41503982 / 41663262 / -0.4% 41933131 / 42556242 / -1.5% 0.99 / 0.98 / +1.1%
compress time/sparse-wide-lists-runs 80808716 / 81915744 / -1.4% 81216724 / 81660203 / -0.5% 0.99 / 1.00 / -0.8%
compress time/sparse-wide-lists-shuffled 154841474 / 154785301 / +0.0% 154542437 / 155169261 / -0.4% 1.00 / 1.00 / +0.4%
compress time/sparse-wide-nested-child-nulls 53774994 / 54511790 / -1.4% 54577013 / 55667815 / -2.0% 0.99 / 0.98 / +0.6%
compress time/sparse-wide-nested-runs 55006347 / 55916782 / -1.6% 55116717 / 56496451 / -2.4% 1.00 / 0.99 / +0.8%
compress time/statpopgen 589825406 / 587268376 / +0.4% 588353344 / 589987722 / -0.3% 1.00 / 1.00 / +0.7%
compress time/taxi 238868067 / 239432444 / -0.2% 243637048 / 243460581 / +0.1% 0.98 / 0.98 / -0.3%
compress time/tpc-h l_comment canonical 1202239159 / 1198045741 / +0.4% 1213799029 / 1198189325 / +1.3% 0.99 / 1.00 / -0.9%
compress time/tpc-h l_comment chunked 1198049776 / 1200858122 / -0.2% 1208347433 / 1210402126 / -0.2% 0.99 / 0.99 / -0.1%
compress time/wide table cols=100 chunks=1 rows=1000 3868842 / 3783532 / +2.3% 4516421 / 4163514 / +8.5% 0.86 / 0.91 / -5.7%
compress time/wide table cols=100 chunks=50 rows=1000 3702701 / 3888115 / -4.8% 4499072 / 4423652 / +1.7% 0.82 / 0.88 / -6.4%
compress time/wide table cols=1000 chunks=1 rows=1000 36767800 / 37892719 / -3.0% 38889386 / 39132103 / -0.6% 0.95 / 0.97 / -2.4%
compress time/wide table cols=1000 chunks=50 rows=1000 37514184 / 37746703 / -0.6% 38833246 / 39007399 / -0.4% 0.97 / 0.97 / -0.2%
compress time/wide table cols=10000 chunks=1 rows=1000 377597902 / 376943169 / +0.2% 384099469 / 385148031 / -0.3% 0.98 / 0.98 / +0.4%
compress time/wide table cols=10000 chunks=50 rows=1000 374844730 / 374966844 / -0.0% 382284212 / 385834236 / -0.9% 0.98 / 0.97 / +0.9%
decompress time/arade 54567960 / 54623192 / -0.1% 54748264 / 55183590 / -0.8% 1.00 / 0.99 / +0.7%
decompress time/bimbo 92198147 / 92253078 / -0.1% 92543192 / 92813873 / -0.3% 1.00 / 0.99 / +0.2%
decompress time/cmsprovider 103050084 / 103102999 / -0.1% 102511158 / 103638444 / -1.1% 1.01 / 0.99 / +1.0%
decompress time/euro2016 159076384 / 159447093 / -0.2% 159699934 / 159967291 / -0.2% 1.00 / 1.00 / -0.1%
decompress time/food 26534142 / 26560906 / -0.1% 26748621 / 27161148 / -1.5% 0.99 / 0.98 / +1.4%
decompress time/hashtags 446029072 / 445340412 / +0.2% 446390259 / 445979178 / +0.1% 1.00 / 1.00 / +0.1%
decompress time/sparse-wide-flat-runs 15024788 / 15208630 / -1.2% 15718149 / 15855952 / -0.9% 0.96 / 0.96 / -0.3%
decompress time/sparse-wide-lists-runs 277053346 / 277215203 / -0.1% 276975339 / 277777179 / -0.3% 1.00 / 1.00 / +0.2%
decompress time/sparse-wide-lists-shuffled 318765572 / 318772248 / -0.0% 319516469 / 319358349 / +0.0% 1.00 / 1.00 / -0.1%
decompress time/sparse-wide-nested-child-nulls 53241156 / 52826421 / +0.8% 53515883 / 53401028 / +0.2% 0.99 / 0.99 / +0.6%
decompress time/sparse-wide-nested-runs 52607202 / 53280926 / -1.3% 52787657 / 53412398 / -1.2% 1.00 / 1.00 / -0.1%
decompress time/statpopgen 1636938470 / 1648870973 / -0.7% 1653974013 / 1661210891 / -0.4% 0.99 / 0.99 / -0.3%
decompress time/taxi 60542742 / 60650714 / -0.2% 60778058 / 60967141 / -0.3% 1.00 / 0.99 / +0.1%
decompress time/tpc-h l_comment canonical 82017582 / 82161374 / -0.2% 82343243 / 83449777 / -1.3% 1.00 / 0.98 / +1.2%
decompress time/tpc-h l_comment chunked 81907006 / 82006520 / -0.1% 82318786 / 83041294 / -0.9% 0.99 / 0.99 / +0.8%
decompress time/wide table cols=100 chunks=1 rows=1000 1957616 / 1955170 / +0.1% 2200164 / 2158483 / +1.9% 0.89 / 0.91 / -1.8%
decompress time/wide table cols=100 chunks=50 rows=1000 1940024 / 1960981 / -1.1% 2087372 / 2242974 / -6.9% 0.93 / 0.87 / +6.3%
decompress time/wide table cols=1000 chunks=1 rows=1000 22834416 / 22610316 / +1.0% 23732435 / 22991551 / +3.2% 0.96 / 0.98 / -2.2%
decompress time/wide table cols=1000 chunks=50 rows=1000 22844972 / 22581794 / +1.2% 23470186 / 23287472 / +0.8% 0.97 / 0.97 / +0.4%
decompress time/wide table cols=10000 chunks=1 rows=1000 234181164 / 232462583 / +0.7% 235596661 / 234222591 / +0.6% 0.99 / 0.99 / +0.2%
decompress time/wide table cols=10000 chunks=50 rows=1000 231424664 / 230756468 / +0.3% 233646150 / 232773229 / +0.4% 0.99 / 0.99 / -0.1%
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
name bytes (PR / base / %diff)
parquet size/arade 210020851 / 210020851 / +0.0%
parquet size/bimbo 376187078 / 376187078 / +0.0%
parquet size/cmsprovider 364268159 / 364268159 / +0.0%
parquet size/euro2016 111035158 / 111035158 / +0.0%
parquet size/food 34903783 / 34903783 / +0.0%
parquet size/hashtags 126189203 / 126189203 / +0.0%
parquet size/sparse-wide-flat-runs 1388944 / 1388944 / +0.0%
parquet size/sparse-wide-lists-runs 4327813 / 4327813 / +0.0%
parquet size/sparse-wide-lists-shuffled 5710569 / 5710569 / +0.0%
parquet size/sparse-wide-nested-child-nulls 1266042 / 1266042 / +0.0%
parquet size/sparse-wide-nested-runs 1388944 / 1388944 / +0.0%
parquet size/statpopgen 73385513 / 73385513 / +0.0%
parquet size/taxi 52142534 / 52142534 / +0.0%
parquet size/tpc-h l_comment canonical 153403836 / 153403836 / +0.0%
parquet size/tpc-h l_comment chunked 153403836 / 153403836 / +0.0%
parquet size/wide table cols=100 chunks=1 rows=1000 932404 / 932404 / +0.0%
parquet size/wide table cols=100 chunks=50 rows=1000 932404 / 932404 / +0.0%
parquet size/wide table cols=1000 chunks=1 rows=1000 9324004 / 9324004 / +0.0%
parquet size/wide table cols=1000 chunks=50 rows=1000 9324004 / 9324004 / +0.0%
parquet size/wide table cols=10000 chunks=1 rows=1000 93240004 / 93240004 / +0.0%
parquet size/wide table cols=10000 chunks=50 rows=1000 93240004 / 93240004 / +0.0%
vortex / arrow-ipc / ns (0.993x ➖, 0↑ 1↓)
name hot ns (PR / base / %diff) cold ns (PR / base / %diff) hot/cold (PR / base / %diff)
compress time/arade 94079423 / 95825934 / -1.8% 117049671 / 121171211 / -3.4% 0.80 / 0.79 / +1.6%
compress time/bimbo 594202730 / 604640512 / -1.7% 654937198 / 670412950 / -2.3% 0.91 / 0.90 / +0.6%
compress time/cmsprovider 374321516 / 378602904 / -1.1% 436360686 / 443675088 / -1.6% 0.86 / 0.85 / +0.5%
compress time/euro2016 37942302 / 38259567 / -0.8% 51645991 / 52968358 / -2.5% 0.73 / 0.72 / +1.7%
compress time/food 39450635 / 40146255 / -1.7% 52847557 / 54294951 / -2.7% 0.75 / 0.74 / +1.0%
compress time/hashtags 77475272 / 78165058 / -0.9% 105552948 / 106768614 / -1.1% 0.73 / 0.73 / +0.3%
compress time/sparse-wide-flat-runs 27911163 / 27825162 / +0.3% 36791008 / 37173448 / -1.0% 0.76 / 0.75 / +1.4%
compress time/sparse-wide-lists-runs 44400592 / 45054051 / -1.5% 51055758 / 51848907 / -1.5% 0.87 / 0.87 / +0.1%
compress time/sparse-wide-lists-shuffled 46006048 / 45225962 / +1.7% 50020351 / 50243747 / -0.4% 0.92 / 0.90 / +2.2%
compress time/sparse-wide-nested-child-nulls 40253234 / 40519833 / -0.7% 51644122 / 50627794 / +2.0% 0.78 / 0.80 / -2.6%
compress time/sparse-wide-nested-runs 40253209 / 40705014 / -1.1% 49943976 / 50823278 / -1.7% 0.81 / 0.80 / +0.6%
compress time/statpopgen 101264034 / 101407071 / -0.1% 143624002 / 144006351 / -0.3% 0.71 / 0.70 / +0.1%
compress time/taxi 58344585 / 58986924 / -1.1% 74518068 / 75871507 / -1.8% 0.78 / 0.78 / +0.7%
compress time/tpc-h l_comment canonical 126088066 / 128459474 / -1.8% 161099177 / 165037159 / -2.4% 0.78 / 0.78 / +0.6%
compress time/tpc-h l_comment chunked 128183052 / 128121561 / +0.0% 169273321 / 165377353 / +2.4% 0.76 / 0.77 / -2.3%
compress time/wide table cols=100 chunks=1 rows=1000 194404.5 / 200287.5 / -2.9% 416513 / 694066 / -40.0% 🟢 0.47 / 0.29 / +61.7%
compress time/wide table cols=100 chunks=50 rows=1000 194450 / 215635 / -9.8% 371539 / 642041 / -42.1% 🟢 0.52 / 0.34 / +55.8%
compress time/wide table cols=1000 chunks=1 rows=1000 2311052 / 2384668 / -3.1% 2786299 / 3283434 / -15.1% 🟢 0.83 / 0.73 / +14.2%
compress time/wide table cols=1000 chunks=50 rows=1000 2299814 / 2323985 / -1.0% 2797891 / 3107330 / -10.0% 0.82 / 0.75 / +9.9%
compress time/wide table cols=10000 chunks=1 rows=1000 24770062 / 24974274 / -0.8% 31060006 / 31449371 / -1.2% 0.80 / 0.79 / +0.4%
compress time/wide table cols=10000 chunks=50 rows=1000 24871316 / 25140836 / -1.1% 31078109 / 31570993 / -1.6% 0.80 / 0.80 / +0.5%
decompress time/arade 217579968 / 210873762 / +3.2% 217801095 / 211223571 / +3.1% 1.00 / 1.00 / +0.1%
decompress time/bimbo 409801252 / 425769866 / -3.8% 410209052 / 425258168 / -3.5% 1.00 / 1.00 / -0.2%
decompress time/cmsprovider 673803282 / 695957550 / -3.2% 674308495 / 696731743 / -3.2% 1.00 / 1.00 / +0.0%
decompress time/euro2016 83330242 / 83696014 / -0.4% 83573119 / 84053857 / -0.6% 1.00 / 1.00 / +0.1%
decompress time/food 67379226 / 65563486 / +2.8% 67482992 / 65728315 / +2.7% 1.00 / 1.00 / +0.1%
decompress time/hashtags 159528606 / 160107943 / -0.4% 159910994 / 160539716 / -0.4% 1.00 / 1.00 / +0.0%
decompress time/sparse-wide-flat-runs 18716908 / 18773532 / -0.3% 21568866 / 19416913 / +11.1% 🔴 0.87 / 0.97 / -10.2%
decompress time/sparse-wide-lists-runs 81069929 / 81420714 / -0.4% 81179807 / 81538928 / -0.4% 1.00 / 1.00 / +0.0%
decompress time/sparse-wide-lists-shuffled 81374430 / 81867726 / -0.6% 81610892 / 82005095 / -0.5% 1.00 / 1.00 / -0.1%
decompress time/sparse-wide-nested-child-nulls 29631182 / 26828010 / +10.4% 🔴 41564308 / 27341480 / +52.0% 🔴 0.71 / 0.98 / -27.3%
decompress time/sparse-wide-nested-runs 26309176 / 27090756 / -2.9% 27048246 / 27542668 / -1.8% 0.97 / 0.98 / -1.1%
decompress time/statpopgen 156938066 / 156280848 / +0.4% 157037637 / 157904997 / -0.5% 1.00 / 0.99 / +1.0%
decompress time/taxi 50691426 / 51128404 / -0.9% 50882467 / 51393825 / -1.0% 1.00 / 0.99 / +0.1%
decompress time/tpc-h l_comment canonical 191322998 / 189161350 / +1.1% 191515617 / 189753792 / +0.9% 1.00 / 1.00 / +0.2%
decompress time/tpc-h l_comment chunked 191396088 / 191771703 / -0.2% 191374582 / 191867498 / -0.3% 1.00 / 1.00 / +0.1%
decompress time/wide table cols=100 chunks=1 rows=1000 317831 / 319840 / -0.6% 347593 / 343025 / +1.3% 0.91 / 0.93 / -1.9%
decompress time/wide table cols=100 chunks=50 rows=1000 321291 / 318295 / +0.9% 345325 / 343827 / +0.4% 0.93 / 0.93 / +0.5%
decompress time/wide table cols=1000 chunks=1 rows=1000 3735306 / 3737092 / -0.0% 3667030 / 3735877 / -1.8% 1.02 / 1.00 / +1.8%
decompress time/wide table cols=1000 chunks=50 rows=1000 3579752 / 3677224 / -2.7% 3791860 / 3771140 / +0.5% 0.94 / 0.98 / -3.2%
decompress time/wide table cols=10000 chunks=1 rows=1000 36015304 / 35841121 / +0.5% 36711348 / 36076862 / +1.8% 0.98 / 0.99 / -1.3%
decompress time/wide table cols=10000 chunks=50 rows=1000 35841350 / 36556312 / -2.0% 36720501 / 37247917 / -1.4% 0.98 / 0.98 / -0.5%
vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)
name bytes (PR / base / %diff)
arrow-ipc size/arade 680639722 / 680639722 / +0.0%
arrow-ipc size/bimbo 3168319202 / 3168319202 / +0.0%
arrow-ipc size/cmsprovider 2394412322 / 2394412322 / +0.0%
arrow-ipc size/euro2016 359300042 / 359300042 / +0.0%
arrow-ipc size/food 252069442 / 252069442 / +0.0%
arrow-ipc size/hashtags 700364562 / 700364562 / +0.0%
arrow-ipc size/sparse-wide-flat-runs 245220370 / 245220370 / +0.0%
arrow-ipc size/sparse-wide-lists-runs 142046306 / 142046306 / +0.0%
arrow-ipc size/sparse-wide-lists-shuffled 142870882 / 142870882 / +0.0%
arrow-ipc size/sparse-wide-nested-child-nulls 251527506 / 251527506 / +0.0%
arrow-ipc size/sparse-wide-nested-runs 251527122 / 251527122 / +0.0%
arrow-ipc size/statpopgen 1094018226 / 1094018226 / +0.0%
arrow-ipc size/taxi 474481242 / 474481242 / +0.0%
arrow-ipc size/tpc-h l_comment canonical 1021072290 / 1021072290 / +0.0%
arrow-ipc size/tpc-h l_comment chunked 1021072290 / 1021072290 / +0.0%
arrow-ipc size/wide table cols=100 chunks=1 rows=1000 1231690 / 1231690 / +0.0%
arrow-ipc size/wide table cols=100 chunks=50 rows=1000 1231690 / 1231690 / +0.0%
arrow-ipc size/wide table cols=1000 chunks=1 rows=1000 12312506 / 12312506 / +0.0%
arrow-ipc size/wide table cols=1000 chunks=50 rows=1000 12312506 / 12312506 / +0.0%
arrow-ipc size/wide table cols=10000 chunks=1 rows=1000 123192506 / 123192506 / +0.0%
arrow-ipc size/wide table cols=10000 chunks=50 rows=1000 123192506 / 123192506 / +0.0%

connortsui20
connortsui20 requested changes Nov 9, 2026
View reviewed changes

@connortsui20 connortsui20 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is now doing extra work in the case that dict is the thing that we want. We might want to design this a bit more carefully, but the idea is right

@robert3005
Ask the compressor plugin which encoding it produces
97433ad 
Replace `CompressorPlugin::may_compress_to(chunk, encoding)` with
`encoding_for(chunk)`, which returns the encoding that `compress_chunk`
gives. The default compresses the chunk. The dict writer compares the
answer with `Dict`.

`CascadingCompressor` splits scheme selection from the winner's
compression, so `compress` and `encoding_for` share one selection. A
new `Scheme::root_encoding` lets `encoding_for` skip the full
compression when the winner always produces the same root encoding.

Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005 robert3005 changed the title Skip the dict probe compression when no dict scheme can win Ask the compressor plugin which encoding it produces Nov 9, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@connortsui20 connortsui20 connortsui20 requested changes

Requested changes must be addressed to merge this pull request.

Assignees

No one assigned

Labels

changelog/feature A new feature

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@robert3005 @connortsui20