Skip to content

[common] Handle empty null counts in statistics filtering - #10852

Merged
JingsongLi merged 1 commit into
apache:masterfrom
yuzelin:fix-empty-null-counts
Oct 29, 2026
Merged

JingsongLi merged 1 commit into
apache:masterfrom
yuzelin:fix-empty-null-counts

Conversation

@yuzelin

@yuzelin yuzelin commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Purpose

Manifest entries may contain complete min/max statistics but an empty
null_counts array. Accessing this array in LeafPredicate causes an
ArrayIndexOutOfBoundsException during statistics filtering.

Treat an empty array as an unknown null count, allowing the existing
predicate logic to continue using available min/max bounds.

Tests

LeafPredicateTest#testEmptyNullCountsWithCompleteBounds

@JingsongLi

Copy link
Copy Markdown
Contributor

Requirement fit: NEEDS-EVIDENCE. Implementation: no confirmed regression in the local guard.

Normal JDK 8 Maven verify passes all 15 LeafPredicateTest cases (including the added case), with the standard formatting/checkstyle/enforcer/license checks. Before treating this as a production fix, could you identify the supported writer/import path that emits nonempty min/max rows with an empty null-count array, and add a regression through its serialized manifest and an ordinary filtered table scan?

The Java SimpleStatsConverter paths I traced emit one null-count slot per emitted stats column (unknown counts use null), the Python DataWriter does the same, and SimpleStats.EMPTY_STATS is documented as zero-column statistics. The added unit test constructs the mismatched shape directly, so it demonstrates the local guard but does not yet establish a reachable table-read failure or verify schema/dense-stats adaptation. A concrete producer or existing affected table plus the normal scan reproduction would resolve the end-to-end applicability question.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit f59549f into apache:master Oct 29, 2026
34 of 36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants