Skip to content

Conversation

@tigrannajaryan
Copy link
Collaborator

Resolves #341

Emptry root struct cannot work since decoding it is a noop and never reaches the end of column, thus potentially allowing a very large number of such empty records to be "decoded" by Reader without making any progress. This makes protecting from invalid inputs hard.

We now disallow this, since there is no plausible use case for a schema like this.

We still allow empty non-root structs, which is a valid way to have a schema that can evolve by adding fields to empty struct. It does not suffer from the same problem since Reader will have to make some progress for non-empty root struct and will eventually reach end of column for root struct.

We also disallow this in stefc schema parser.

Resolves #341

Emptry root struct cannot work since decoding it is a noop
and never reaches the end of column, thus potentially allowing
a very large number of such empty records to be "decoded" by
Reader without making any progress. This makes protecting from
invalid inputs hard.

We now disallow this, since there is no plausible use case
for a schema like this.

We still allow empty non-root structs, which is a valid way to
have a schema that can evolve by adding fields to empty struct.
It does not suffer from the same problem since Reader will have
to make some progress for non-empty root struct and will eventually
reach end of column for root struct.

We also disallow this in stefc schema parser.
@github-actions
Copy link

Benchmark Result

Benchmark diff with base branch
goos: linux
goarch: amd64
pkg: github.com/splunk/stef/benchmarks
cpu: AMD EPYC 7763 64-Core Processor                
                                                 │ bench-main.txt │            bench-new.txt            │
                                                 │     sec/op     │    sec/op     vs base               │
SerializeNative/STEF/serialize-4                     12.50m ±  6%   11.16m ± 15%  -10.74% (p=0.026 n=6)
SerializeNative/STEFU/serialize-4                    35.93m ±  3%   35.96m ±  2%        ~ (p=0.818 n=6)
DeserializeNative/STEF/deser-4                       2.455m ±  1%   2.475m ±  1%   +0.80% (p=0.009 n=6)
DeserializeNative/STEFU/deser-4                      7.332m ±  1%   7.347m ±  1%        ~ (p=0.937 n=6)
SerializeFromPdata/STEF/serialize-4                  149.5m ±  4%   142.7m ±  2%   -4.57% (p=0.004 n=6)
SerializeFromPdata/STEFU/serialize-4                 36.19m ±  2%   35.46m ±  1%   -2.03% (p=0.009 n=6)
DeserializeToPdata/STEF/deserialize-4                47.19m ±  2%   46.71m ±  2%        ~ (p=0.240 n=6)
DeserializeToPdata/STEFU/deserialize-4               64.44m ±  2%   63.83m ±  1%   -0.94% (p=0.002 n=6)
STEFReaderRead-4                                     2.533m ±  2%   2.557m ±  1%        ~ (p=0.132 n=6)
STEFSerializeMultipart/astronomy-otelmetrics-4        3.556 ± 22%    3.494 ± 21%        ~ (p=0.310 n=6)
STEFDeserializeMultipart/astronomy-otelmetrics-4     70.81m ± 17%   74.55m ± 12%        ~ (p=0.699 n=6)
ReadSTEF-4                                           2.602m ±  4%   2.662m ±  5%        ~ (p=0.065 n=6)
ReadSTEFZ-4                                          3.353m ±  2%   3.350m ±  1%        ~ (p=0.589 n=6)
ReadSTEFZWriteSTEF-4                                 7.455m ±  3%   7.530m ±  1%        ~ (p=0.485 n=6)
geomean                                              21.81m         21.64m         -0.81%

                                                 │ bench-main.txt │            bench-new.txt            │
                                                 │   sec/point    │  sec/point    vs base               │
SerializeNative/STEF/serialize-4                     186.9n ±  6%   166.8n ± 15%  -10.73% (p=0.026 n=6)
SerializeNative/STEFU/serialize-4                    537.3n ±  3%   537.9n ±  2%        ~ (p=0.818 n=6)
DeserializeNative/STEF/deser-4                       36.72n ±  1%   37.01n ±  1%   +0.79% (p=0.009 n=6)
DeserializeNative/STEFU/deser-4                      109.7n ±  1%   109.9n ±  1%        ~ (p=0.820 n=6)
SerializeFromPdata/STEF/serialize-4                  2.237µ ±  4%   2.135µ ±  2%   -4.56% (p=0.004 n=6)
SerializeFromPdata/STEFU/serialize-4                 541.3n ±  2%   530.4n ±  1%   -2.01% (p=0.009 n=6)
DeserializeToPdata/STEF/deserialize-4                705.9n ±  2%   698.7n ±  2%        ~ (p=0.255 n=6)
DeserializeToPdata/STEFU/deserialize-4               964.0n ±  2%   954.7n ±  1%   -0.96% (p=0.002 n=6)
STEFReaderRead-4                                     37.89n ±  2%   38.24n ±  1%        ~ (p=0.132 n=6)
STEFSerializeMultipart/astronomy-otelmetrics-4       4.519µ ± 22%   4.442µ ± 21%        ~ (p=0.310 n=6)
STEFDeserializeMultipart/astronomy-otelmetrics-4     90.00n ± 17%   94.75n ± 11%        ~ (p=0.699 n=6)
ReadSTEF-4                                           38.95n ±  4%   39.83n ±  5%        ~ (p=0.061 n=6)
ReadSTEFZ-4                                          50.18n ±  2%   50.15n ±  1%        ~ (p=0.589 n=6)
ReadSTEFZWriteSTEF-4                                 111.6n ±  3%   112.7n ±  1%        ~ (p=0.442 n=6)
geomean                                              229.4n         227.6n         -0.81%

                                                 │ bench-main.txt │            bench-new.txt             │
                                                 │      B/op      │     B/op      vs base                │
SerializeNative/STEF/serialize-4                     3.345Mi ± 0%   3.338Mi ± 0%  -0.21% (p=0.026 n=6)
SerializeNative/STEFU/serialize-4                    7.557Mi ± 0%   7.557Mi ± 0%  +0.00% (p=0.015 n=6)
DeserializeNative/STEF/deser-4                       951.4Ki ± 0%   951.4Ki ± 0%       ~ (p=1.000 n=6) ¹
DeserializeNative/STEFU/deser-4                      1.715Mi ± 0%   1.715Mi ± 0%       ~ (p=0.061 n=6)
SerializeFromPdata/STEF/serialize-4                  76.55Mi ± 0%   76.55Mi ± 0%       ~ (p=0.361 n=6)
SerializeFromPdata/STEFU/serialize-4                 7.557Mi ± 0%   7.557Mi ± 0%       ~ (p=0.195 n=6)
DeserializeToPdata/STEF/deserialize-4                31.99Mi ± 0%   31.99Mi ± 0%  +0.00% (p=0.050 n=6)
DeserializeToPdata/STEFU/deserialize-4               38.88Mi ± 0%   38.88Mi ± 0%       ~ (p=0.385 n=6)
STEFReaderRead-4                                     953.1Ki ± 0%   953.1Ki ± 0%       ~ (p=1.000 n=6)
STEFSerializeMultipart/astronomy-otelmetrics-4       3.384Gi ± 0%   3.383Gi ± 0%       ~ (p=0.589 n=6)
STEFDeserializeMultipart/astronomy-otelmetrics-4     20.30Mi ± 0%   20.30Mi ± 0%       ~ (p=0.394 n=6)
ReadSTEF-4                                           953.1Ki ± 0%   953.2Ki ± 0%       ~ (p=0.675 n=6)
ReadSTEFZ-4                                          10.29Mi ± 0%   10.29Mi ± 0%       ~ (p=0.329 n=6)
ReadSTEFZWriteSTEF-4                                 13.44Mi ± 0%   13.44Mi ± 0%       ~ (p=0.229 n=6)
geomean                                              10.66Mi        10.66Mi       -0.02%
¹ all samples are equal

                                                 │ bench-main.txt │            bench-new.txt            │
                                                 │   allocs/op    │  allocs/op   vs base                │
SerializeNative/STEF/serialize-4                      2.666k ± 0%   2.652k ± 1%  -0.51% (p=0.024 n=6)
SerializeNative/STEFU/serialize-4                      884.0 ± 0%    885.0 ± 0%       ~ (p=0.061 n=6)
DeserializeNative/STEF/deser-4                         463.0 ± 0%    463.0 ± 0%       ~ (p=1.000 n=6) ¹
DeserializeNative/STEFU/deser-4                        496.0 ± 0%    496.0 ± 0%       ~ (p=1.000 n=6) ¹
SerializeFromPdata/STEF/serialize-4                   134.7k ± 0%   134.7k ± 0%       ~ (p=0.535 n=6)
SerializeFromPdata/STEFU/serialize-4                   886.0 ± 0%    886.0 ± 0%       ~ (p=1.000 n=6) ¹
DeserializeToPdata/STEF/deserialize-4                 756.2k ± 0%   756.2k ± 0%       ~ (p=0.455 n=6)
DeserializeToPdata/STEFU/deserialize-4                944.9k ± 0%   944.9k ± 0%       ~ (p=1.000 n=6) ¹
STEFReaderRead-4                                       463.0 ± 0%    463.0 ± 0%       ~ (p=1.000 n=6) ¹
STEFSerializeMultipart/astronomy-otelmetrics-4        13.15M ± 0%   13.15M ± 0%       ~ (p=0.937 n=6)
STEFDeserializeMultipart/astronomy-otelmetrics-4      1.956k ± 0%   1.956k ± 0%       ~ (p=0.455 n=6)
ReadSTEF-4                                             464.0 ± 0%    464.0 ± 0%       ~ (p=1.000 n=6) ¹
ReadSTEFZ-4                                            502.0 ± 0%    502.0 ± 0%       ~ (p=1.000 n=6) ¹
ReadSTEFZWriteSTEF-4                                  1.231k ± 0%   1.231k ± 0%       ~ (p=1.000 n=6) ¹
geomean                                               6.307k        6.305k       -0.03%
¹ all samples are equal
Benchmark result
benchstat bench-new.txt
goos: linux
goarch: amd64
pkg: github.com/splunk/stef/benchmarks
cpu: AMD EPYC 7763 64-Core Processor                
                                                 │ bench-new.txt │
                                                 │    sec/op     │
SerializeNative/STEF/serialize-4                    11.16m ± 15%
SerializeNative/STEFU/serialize-4                   35.96m ±  2%
DeserializeNative/STEF/deser-4                      2.475m ±  1%
DeserializeNative/STEFU/deser-4                     7.347m ±  1%
SerializeFromPdata/STEF/serialize-4                 142.7m ±  2%
SerializeFromPdata/STEFU/serialize-4                35.46m ±  1%
DeserializeToPdata/STEF/deserialize-4               46.71m ±  2%
DeserializeToPdata/STEFU/deserialize-4              63.83m ±  1%
STEFReaderRead-4                                    2.557m ±  1%
STEFSerializeMultipart/astronomy-otelmetrics-4       3.494 ± 21%
STEFDeserializeMultipart/astronomy-otelmetrics-4    74.55m ± 12%
ReadSTEF-4                                          2.662m ±  5%
ReadSTEFZ-4                                         3.350m ±  1%
ReadSTEFZWriteSTEF-4                                7.530m ±  1%
geomean                                             21.64m

                                                 │ bench-new.txt │
                                                 │   sec/point   │
SerializeNative/STEF/serialize-4                    166.8n ± 15%
SerializeNative/STEFU/serialize-4                   537.9n ±  2%
DeserializeNative/STEF/deser-4                      37.01n ±  1%
DeserializeNative/STEFU/deser-4                     109.9n ±  1%
SerializeFromPdata/STEF/serialize-4                 2.135µ ±  2%
SerializeFromPdata/STEFU/serialize-4                530.4n ±  1%
DeserializeToPdata/STEF/deserialize-4               698.7n ±  2%
DeserializeToPdata/STEFU/deserialize-4              954.7n ±  1%
STEFReaderRead-4                                    38.24n ±  1%
STEFSerializeMultipart/astronomy-otelmetrics-4      4.442µ ± 21%
STEFDeserializeMultipart/astronomy-otelmetrics-4    94.75n ± 11%
ReadSTEF-4                                          39.83n ±  5%
ReadSTEFZ-4                                         50.15n ±  1%
ReadSTEFZWriteSTEF-4                                112.7n ±  1%
geomean                                             227.6n

                                                 │ bench-new.txt │
                                                 │     B/op      │
SerializeNative/STEF/serialize-4                    3.338Mi ± 0%
SerializeNative/STEFU/serialize-4                   7.557Mi ± 0%
DeserializeNative/STEF/deser-4                      951.4Ki ± 0%
DeserializeNative/STEFU/deser-4                     1.715Mi ± 0%
SerializeFromPdata/STEF/serialize-4                 76.55Mi ± 0%
SerializeFromPdata/STEFU/serialize-4                7.557Mi ± 0%
DeserializeToPdata/STEF/deserialize-4               31.99Mi ± 0%
DeserializeToPdata/STEFU/deserialize-4              38.88Mi ± 0%
STEFReaderRead-4                                    953.1Ki ± 0%
STEFSerializeMultipart/astronomy-otelmetrics-4      3.383Gi ± 0%
STEFDeserializeMultipart/astronomy-otelmetrics-4    20.30Mi ± 0%
ReadSTEF-4                                          953.2Ki ± 0%
ReadSTEFZ-4                                         10.29Mi ± 0%
ReadSTEFZWriteSTEF-4                                13.44Mi ± 0%
geomean                                             10.66Mi

                                                 │ bench-new.txt │
                                                 │   allocs/op   │
SerializeNative/STEF/serialize-4                     2.652k ± 1%
SerializeNative/STEFU/serialize-4                     885.0 ± 0%
DeserializeNative/STEF/deser-4                        463.0 ± 0%
DeserializeNative/STEFU/deser-4                       496.0 ± 0%
SerializeFromPdata/STEF/serialize-4                  134.7k ± 0%
SerializeFromPdata/STEFU/serialize-4                  886.0 ± 0%
DeserializeToPdata/STEF/deserialize-4                756.2k ± 0%
DeserializeToPdata/STEFU/deserialize-4               944.9k ± 0%
STEFReaderRead-4                                      463.0 ± 0%
STEFSerializeMultipart/astronomy-otelmetrics-4       13.15M ± 0%
STEFDeserializeMultipart/astronomy-otelmetrics-4     1.956k ± 0%
ReadSTEF-4                                            464.0 ± 0%
ReadSTEFZ-4                                           502.0 ± 0%
ReadSTEFZWriteSTEF-4                                 1.231k ± 0%
geomean                                              6.305k

@tigrannajaryan tigrannajaryan marked this pull request as ready for review January 26, 2026 20:11
@tigrannajaryan
Copy link
Collaborator Author

@abhisheksplunk this is stacked on top of #340
Please review that one, that needs to be merged first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants