Skip to content

Fix time partition boundary overflows and splitter partition order#17893

Open
Caideyipi wants to merge 10 commits into
masterfrom
time-partition-boundary-overflow-fix
Open

Fix time partition boundary overflows and splitter partition order#17893
Caideyipi wants to merge 10 commits into
masterfrom
time-partition-boundary-overflow-fix

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator

Description

This patch fixes overflow-prone time partition boundary handling by making partition boundary calculations origin-aware and saturation-safe. It also preserves TsFile splitter output order for aligned chunks that are split across time partitions, which prevents Long.MAX_VALUE boundary chunks from being emitted in reverse partition order.

The main changes are:

  • keep time partition origin/interval boundary state in TimePartitionUtils
  • make partition start-time lookup and partition-start checks overflow-safe
  • cap partition estimate calculations instead of overflowing
  • use saturated arithmetic in TTL auto-clean partition boundary checks
  • refresh partition boundary origin from storage engine configuration
  • update related query and pipe realtime paths to use the shared boundary helper
  • preserve aligned TsFile splitter partition emission order when decoded chunks are split across pages or time partitions

Tests

  • mvn -pl iotdb-core/node-commons -DskipTests compile
  • mvn -pl iotdb-core/node-commons test currently fails during testCompile because CommitProgressKeeperTest references missing RegionProgress, WriterId, and WriterProgress symbols, which is unrelated to this change.
  • mvn -pl iotdb-core/datanode spotless:apply
  • mvn -pl iotdb-core/datanode -am -Dtest=TsFileSplitterTest#testSplitTableTimeOnlyAlignedChunkAtLongMaxPartitionEnd,testSplitTableTimeOnlyAlignedChunkAtExactLongMaxUpperBound -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false test currently fails during datanode compile because generated ModeAccumulator sources reference missing org.apache.commons.collections4.comparators.ComparatorChain and IoTDBConfig#getModeMapSizeThreshold(), which is unrelated to this splitter change.

@codecov

codecov Bot commented Jun 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 73.03704% with 182 lines in your changes missing coverage. Please review.
✅ Project coverage is 41.36%. Comparing base (f971de2) to head (dacc9ca).

Files with missing lines Patch % Lines
.../udf/builtin/UDTFEqualSizeBucketOutlierSample.java 0.00% 36 Missing ⚠️
...yengine/plan/optimization/LimitOffsetPushDown.java 75.38% 16 Missing ⚠️
...unary/scalar/DateBinFunctionColumnTransformer.java 55.55% 12 Missing ⚠️
.../exception/query/QueryTimeoutRuntimeException.java 0.00% 10 Missing ⚠️
...e/iotdb/db/queryengine/plan/parser/ASTVisitor.java 54.54% 10 Missing ⚠️
.../iotdb/commons/udf/builtin/UDTFTimeDifference.java 0.00% 7 Missing ⚠️
...yengine/plan/relational/sql/parser/AstBuilder.java 0.00% 6 Missing ⚠️
...sformation/dag/intermediate/IntermediateLayer.java 0.00% 6 Missing ⚠️
...validate/TsFileOverlapValidationAndRepairTool.java 0.00% 6 Missing ⚠️
...relational/function/tvf/ForecastTableFunction.java 0.00% 5 Missing ⚠️
... and 29 more
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #17893      +/-   ##
============================================
+ Coverage     41.21%   41.36%   +0.14%     
  Complexity      318      318              
============================================
  Files          5258     5261       +3     
  Lines        366348   366677     +329     
  Branches      47389    47434      +45     
============================================
+ Hits         151004   151666     +662     
+ Misses       215344   215011     -333     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Caideyipi Caideyipi changed the title Fix time partition boundary overflow handling Fix time partition boundary overflows and splitter partition order Jun 11, 2026
@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
B Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant