Repository navigation
[common] Handle empty null counts in statistics filtering - #10852
Conversation
|
Requirement fit: NEEDS-EVIDENCE. Implementation: no confirmed regression in the local guard. Normal JDK 8 Maven verify passes all 15 LeafPredicateTest cases (including the added case), with the standard formatting/checkstyle/enforcer/license checks. Before treating this as a production fix, could you identify the supported writer/import path that emits nonempty min/max rows with an empty null-count array, and add a regression through its serialized manifest and an ordinary filtered table scan? The Java SimpleStatsConverter paths I traced emit one null-count slot per emitted stats column (unknown counts use null), the Python DataWriter does the same, and SimpleStats.EMPTY_STATS is documented as zero-column statistics. The added unit test constructs the mismatched shape directly, so it demonstrates the local guard but does not yet establish a reachable table-read failure or verify schema/dense-stats adaptation. A concrete producer or existing affected table plus the normal scan reproduction would resolve the end-to-end applicability question. |
Purpose
Manifest entries may contain complete min/max statistics but an empty
null_countsarray. Accessing this array inLeafPredicatecauses anArrayIndexOutOfBoundsExceptionduring statistics filtering.Treat an empty array as an unknown null count, allowing the existing
predicate logic to continue using available min/max bounds.
Tests
LeafPredicateTest#testEmptyNullCountsWithCompleteBounds