[rapour]

Summary

• Tracking Issue: Tracking Issue: Elias-Fano Encoding #10605

This adds a naive EliasFano encoding with two children: upper, the number of values in each bucket of high bits, and lower, the low bits of each value. There are no kernels, no reduce rules and no slicing, and it is in no file edition. It also adds an EliasFanoScheme to vortex-btrblocks. Fused kernels, including unpacking a BitPacked lower in the same pass, are left to the later steps.

Changes

• elias_fano_encode splits a sorted, non-nullable unsigned array into the two children and leaves both to the compressor, as FoR::encode does. upper has a fixed u64 dtype, and the encoder holds its counts in a Widen over the narrowest type that fits them. Widen has no wire form yet, so the encoder's output can be written only once upper is compressed.
• The metadata holds lower_bits, which is floor(log2(max / len)) or 0 when max < len, and the length of upper.
• validate checks the dtype and the children's shapes. Decoding returns an error when the counts do not add up to the length, or a low value does not fit in lower_bits.
• execute rebuilds each value as (bucket << lower_bits) | low. When upper is Unary, it reads the buckets straight from the bits, as FoR does with a BitPacked child, so decoding never builds the u64 counts. Any other upper, including the encoder's Widen, is executed to a Primitive first.
• scalar_at combines a scalar from each child: the bucket is a select on the bits when upper is Unary, and a scan of the counts otherwise.
• EliasFanoScheme wraps the values in FoR, splits FoR's child with elias_fano_encode, encodes upper with unary_encode and bit-packs lower, as FoRScheme does. Its estimate comes from the min and max stats, and it checks is_sorted only when EF could beat the best scheme so far. It is not registered in the default CompressionSession, so builders made with unrestricted() don't start choosing it.
• Tests cover:
  • round trips, including through serde, with a Widen upper and with the Unary upper the scheme writes;
  • that a Unary upper is decoded without being executed first;
  • the rejected inputs, with both a Primitive and a Unary upper;
  • test_array_consistency, and a metadata golden file.

[connortsui20]

Hi @rapour, thanks for the contribution!

Could you let us know how this is different from other Rust implementations (the one that comes to my mind is the vers_vecs one https://docs.rs/vers-vecs/latest/vers_vecs/elias_fano/struct.EliasFanoVec.html)? And I am somewhat wondering why we can't rely on it (I bet that there are good reasons, though we should make sure we enumerate them)

[codspeed]

Merging this PR will not alter performance

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

✅ 2307 untouched benchmarks
⏩ 38 skipped benchmarks¹

---

_{Comparing rapour:feat/elias-fano-codec (d1afcb8) with develop (7d00329)}

Footnotes

1. 38 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[rapour]

Hi @connortsui20, Thanks a lot for looking into this PR

I have consulted two reference implementations:

1. https://github.com/Cydhra/vers/tree/master
2. https://github.com/AngeloSav/rise-rs/tree/main/src/elias_fano

Here is my initial reasoning:

• File format. Using either crate would make its internal layout our on-disk format. Neither commits to keeping that stable, so a dependency bump could break files already written.
• Low bits as a child. Both store the low bits inside their own buffers. We store them as a child array, so they're bit-packed with FastLanes and decoded by existing kernels.
• Zero-copy reads. vers's structures own their memory, and its only constructors re-encode from values or deserialise into new vectors. Every load would copy the whole array (or rebuild it from decoded values) before reading a single element, where ours reads the file buffers in place.
• What we need that they lack. The codec has helpers that matter to us but aren't in either library, such as O(1) slicing by element range.
• Room to combine them. Each has its merits. rise-rs supports partitioned EF, and vers has a multi-level rank/select directory that might be faster for select-heavy access. We could take ideas from both in later phases, which is easier when we own the layout.
• Maintenance. The codec is standard EF, small, and depends only on std, so keeping it in-house does not seem a heavy burden.

The design follows rise-rs (the sampling scheme and its parameters). Even if some of these could be worked around, I think keeping it in-house is simpler than maintaining a wrapper plus an external dependency. That being said, I'll follow your lead.

[connortsui20]

I unfortunately will not be able to review this until next week, but at a high level the implementation seems fine to me. Will have to spend a bit more time in the details though since this is a non-trivial PR

[rapour]

I totally understand, Thanks a lot!

I'll try to focus more on the vortex API part of the work in the meantime.

[connortsui20]

Ok I unfortunately cannot review this this week but I promise I will next Monday 🙏

[connortsui20]

ok so I wasn't aware of #9935, can you rebase things so that the changes are mostly disjoint (changes in #9935 don't change stuff here)?

[rapour]

Done, this PR now carries the latest codec, slightly different now that we have switched to a kernel for slicing. The #9935 is the follow up work stacked on top of these changes.

[connortsui20]

Ok so I've been thinking about this and I think it would be more ideal to have an Elias Fano encoding that has 2 children arrays, where the low bits are fastlanes bitpacked (we still get constant time random access which is all the EF encoding needs, plus every other benefit of fastlanes bitpacking), and the high bits are Unary encoded.

This means we should add a Unary encoding! Or DeltaUnary or UnaryDelta or maybe named something else, TBD.

I think this works nicely, and it should make the implementation a bit easier to understand and review as well. The only slightly more complex thing is that we need to match on the array children to make sure they have the correct encodings (unary + bitpacked) to get the fast asm output for the special EF operations. But if you look at FoR + BitPacked we already do this and I think it's quite easy.

So I think the first step is to create a Unary encoding. That in itself has a few design questions (like how do we store it, should we chunk it, have blocks or samples to help lookup, etc).

And it would be great if you could create an Epic and Tracking Issue for this (epic about adding EF, tracking issues about unary, fused kernels encoding, serialization, etc). Just follow the templates (and note that we have an loose rule of not LLM-generating tracking issues and epics). I'm also happy to write a tracking issue if you're not comfortable.

Also, please reach out on slack via #ext-vortex or messaging me! Also @joseph-isaacs I'm happy to take this over.

[connortsui20]

See comment above

[connortsui20]

overall the structure looks good. I left some review comments

[connortsui20]

shouldn't this be reusing the unary encode logic?

[rapour]

After a follow up discussion on Slack, elias_fano_encode keeps both children Primitive, as FoR::encode does, and I've documented that the compressor encodes them. The new EliasFanoScheme in vortex-btrblocks encodes upper with unary_encode and bit-packs lower. It isn't registered yet, so builders made with unrestricted() don't start choosing it. Happy to add it if you would prefer.

[connortsui20]

Is there a reason that this is hardcoded? I'm not against it (especially since we just merged the Widen encoding) but we should have some justification.

[rapour]

One bucket can hold every value, so a count can reach the array's length. The EF scheme stores upper as Unary, whose bits are the same whatever the dtype. If you think using Widen is in order, please let me know and I'll try to make the switch.

[connortsui20]

I should clarify: if we hardcode the dtype to u64 then whenever we canonicalize this array it will blow up to 64 bits per value, even if most of the values are small. And while I think there is no theoretical guarantee of the size of the buckets, I'm guessing that mostly uniform data is going to have smaller-sized buckets (especially if we remove duplicates and runs with encodings above this one).

I just added a Widen encoding that allows us to store a smaller width dtype underneath a larger logical dtype, which means we definitely want to use that if we hardcode this as u64.

This is also a more general question for Vortex in general: with the addition of the Widen encoding, should we start enforcing larger widths for integer child arrays that can be compressed into smaller widths rather than forcing a different dtype?

[connortsui20]

Ok we talked about this offline, I think we all agree that can fix this to u64 and then use Widen. Doing this everywhere in Vortex is beyond the scope of this PR