zstd array v2 #6341
onursatici
started this conversation in
Feature Requests
zstd array v2
#6341
Replies: 1 comment
-
|
filed #6362 |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
The current ZstdArray interleaves string lengths with content bytes
[u32 len][bytes]...and compresses them as a single blob. This forces a sequential parse of the decompressed data to reconstruct BinaryView entries, and because this is tricky to do in gpu's, we make a round trip to the host for varbinview conversion.I think it would be useful to have another zstd array encoding, that acts on a varbinview array, compresses the views and the data buffers. Once uncompressed, this would be immediately usable for the upstream encodings as a varbinview array
Beta Was this translation helpful? Give feedback.
All reactions