Move type dispatch outside batch read and write loops by jt2594838 · Pull Request #963 · apache/tsfile

jt2594838 · 2026-09-17T04:21:07Z

Move type selection out of hot per-value loops in TsFile's read, write, and import paths. Pages and non-aligned Tablet columns use generated type-specific batch loops; aligned writes and query projections reuse resolved types. Arrow, Parquet, and TabletBuilder populate column arrays directly and resolve conversion metadata once per column.

The implementation preserves decoder consumption for filtered/deleted rows, pagination accounting, null bitmaps, late-column page alignment, scalar encoding/SDT behavior, and the successfully written timestamp prefix when an out-of-order row is rejected. Regression tests cover these boundaries, supported data types, and DATE representations.

Validation

93 distinct tests passed across PageDataBatchReaderTest, AlignedNullWriteTest, TabletColumnWriteTest, TabletBuilderTest, ValueConverterTest, ArrowSourceReaderTest, and ParquetSourceReaderTest; the configured Maven lifecycle executed the selected suites twice, with no failures, errors, or skips.
Java Spotless checks and git diff --check passed.
Both compared revisions compiled independently; 114 benchmark output hashes matched. The full repository test suite was not run locally.

Benchmarks

Compared this branch at 5d15440e831dfeb00f025d8c78c716945cc250c2 with develop at 8fdbc49bc80eb4a4d6cf6e46cdf0cabd1a1c0837: JMH 1.37, Temurin 17.0.20.1, Windows 11, Intel i9-12900, one thread pinned to logical CPU 2, 512 MiB heap, 4096 rows, and 3 forks. Initial runs used 5 × 500 ms warmup and measurement iterations; six methods were repeated with 10 × 1 s warmup and 7 × 1 s measurement iterations, retaining both runs.

Path	Observed time reduction in cases with separated 99.9% intervals
Page reading / column construction	12.9%–30.2%
Null writing / catch-up	23.8%–33.0%
TabletBuilder import	40.9%–42.1%
Arrow reading	57.8%–60.5%
Tablet writing	16.2%–25.5%
Query projection	3.0%–13.3%

Of 30 paired scenarios, 25 had lower measured time with separated intervals; five remain inconclusive. Parquet and single-destination scalar projection had overlapping intervals. Both columnSelected reruns were affected by unequal background load and are explicitly inconclusive, despite higher measured means. Other IoTDB workloads were active on the machine during extended runs, so these are shared-machine microbenchmark observations, not isolated causal estimates or end-to-end throughput claims. Interval separation is a conservative screening rule, not a paired statistical test.

Benchmark sources, raw results, the detailed report, and reproduction commands remain local under target/branch-vs-develop/, as requested, and are not part of this PR. The benchmark report's validation section describes the earlier benchmark phase; the focused unit tests above were run subsequently for PR submission.

jt2594838

Implementation rationale and compatibility notes for the submitted changes. Focused tests and benchmark limitations are summarized in the PR description.

jt2594838 · 2026-09-17T04:23:37Z

-          } else {
-            row[c] = extractValue(vec, r);
-          }
+      Object[][] columns = new Object[numCols][rowCount];


Build SourceBatch columns directly because both Arrow and the destination are columnar. Resolving each vector once removes repeated name lookups and the intermediate row arrays/transpose; absent vectors remain null. Existing Arrow reader tests and the cross-version output checks passed.

jt2594838 · 2026-09-17T04:23:37Z

-    } else if (vec instanceof VarBinaryVector) {
-      return ((VarBinaryVector) vec).get(row);
+  /** Select the physical vector type once for the entire column. */
+  private void readColumn(FieldVector vec, Object[] output) {


Bind the physical vector type before iterating through its values. Each typed loop retains null checks, native timestamp precision, date conversion, UTF-8 decoding, and the generic fallback, while avoiding an instanceof chain for every cell.

jt2594838 · 2026-09-17T04:23:37Z

-
-      for (long r = 0; r < rowCount; r++) {
+      int count = Math.toIntExact(rowCount);
+      Object[][] columns = new Object[numCols][count];


Keep the Parquet record reader row-oriented but write directly to the final column arrays. Column indices and extractors are resolved once per batch; missing or malformed fields retain the existing null behavior. This removes the row transpose without changing the input reader protocol.

jt2594838 · 2026-09-17T04:23:37Z

  }

-  private Object extractValue(Group group, int fieldIndex) {
+  private Function<Group, Object> valueExtractor(int fieldIndex) {


Choose physical/logical-type conversion once per column, including the dedicated INT96 accessor, rather than inspecting schema metadata for every value. The wrapper still checks field presence per row. Local benchmarks showed lower allocation but inconclusive elapsed-time changes for Parquet.

jt2594838 · 2026-09-17T04:23:37Z

+    for (int col = 0; col < tableSchema.getColumnSchemas().size(); col++) {
+      IMeasurementSchema colSchema = tableSchema.getColumnSchemas().get(col);
+      String colName = colSchema.getMeasurementName();
+      Type type = Type.fromTsDataType(colSchema.getType());


Traverse columns after establishing sorted timestamps so schema lookup, the destination array, and type resolution are reused across rows. Direct insertion preserves Tablet bitmap semantics by unmarking only non-null converted values; tag defaults and sorted source indices retain their original behavior. Conversion is initialized lazily so empty/all-null columns do not invoke converters. The new tests compare this path with the public Tablet insertion API.

jt2594838 · 2026-09-17T04:23:38Z

+  }
+
+  @Test
+  public void testColumnConversionKeepsSortedRowsAndNulls() {


Mix strings and native values in a column with unsorted timestamps and null-format inputs. This specifically checks that a cached converter still chooses per-cell input handling and keeps tag defaults aligned with sorted rows.

jt2594838 · 2026-09-17T04:23:38Z

+
+public class AlignedNullWriteTest {
+  @Test
+  public void tabletWriterCacheKeepsLateColumnPagesAndFailureOrder() throws Exception {


Compare cached Tablet writing with scalar writes while adding a column after earlier rows. Byte/page/statistics comparisons and a rejected timestamp guard both late-column catch-up and the order of validation versus writer creation.

jt2594838 · 2026-09-17T04:23:38Z

+  }
+
+  @Test
+  public void lateColumnsAndMissingRowsKeepPageBoundaries() throws Exception {


Force short pages, add every supported type after a sealed page plus a partial page, then write missing and present rows. Verify synchronized boundaries, null bitmaps, and value statistics so direct null appends cannot silently disturb alignment.

jt2594838 · 2026-09-17T04:23:39Z

+
+public class TabletColumnWriteTest {
+  @Test
+  public void testEncodedPagesMatchScalarForAllTypes() throws Exception {


Compare sealed bytes, page counts, statistics, and accepted-point counts with scalar writing. The test covers all supported types, null masks, a nonzero source offset, both DATE arrays, and SDT settings for numeric types to guard encoding compatibility.

jt2594838 · 2026-09-17T04:23:39Z

+  }
+
+  @Test
+  public void testOutOfOrderKeepsWrittenPrefixAndSkipsNulls() throws Exception {


Verify that null rows do not trigger timestamp validation and that a later out-of-order row preserves the accepted prefix for a subsequent call. Long.MIN_VALUE, all-null ranges, null value arrays, and empty ranges exercise the context and early-exit boundaries.

codecov-commenter · 2026-09-17T04:46:54Z

Codecov Report

❌ Patch coverage is 93.48837% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.44%. Comparing base (8fdbc49) to head (5d15440).

Files with missing lines	Patch %	Lines
...ava/org/apache/tsfile/tools/ArrowSourceReader.java	84.00%	8 Missing ⚠️
...a/org/apache/tsfile/tools/ParquetSourceReader.java	86.66%	4 Missing ⚠️
...in/java/org/apache/tsfile/tools/TabletBuilder.java	96.55%	1 Missing ⚠️
...n/java/org/apache/tsfile/tools/ValueConverter.java	88.88%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #963      +/-   ##
===========================================
+ Coverage    63.95%   64.44%   +0.48%     
===========================================
  Files          758      759       +1     
  Lines        53253    53299      +46     
  Branches      8445     8451       +6     
===========================================
+ Hits         34060    34349     +289     
+ Misses       17530    17288     -242     
+ Partials      1663     1662       -1

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

jt2594838 added 4 commits September 16, 2026 14:46

optimze

dc3662b

tmp

9d2e028

tmp

bf46f2c

tmp

5d15440

jt2594838 commented Sep 17, 2026

View reviewed changes

hongzhi-gao approved these changes Sep 17, 2026

View reviewed changes

Navigation Menu

Move type dispatch outside batch read and write loops - #963

Move type dispatch outside batch read and write loops#963
jt2594838 wants to merge 4 commits into
developfrom
optimize_loops_with_datatype

jt2594838 commented Sep 17, 2026

Uh oh!

jt2594838 left a comment

Uh oh!

jt2594838 Sep 17, 2026

Uh oh!

jt2594838 Sep 17, 2026

Uh oh!

jt2594838 Sep 17, 2026

Uh oh!

jt2594838 Sep 17, 2026

Uh oh!

jt2594838 Sep 17, 2026

Uh oh!

jt2594838 Sep 17, 2026

Uh oh!

jt2594838 Sep 17, 2026

Uh oh!

jt2594838 Sep 17, 2026

Uh oh!

jt2594838 Sep 17, 2026

Uh oh!

jt2594838 Sep 17, 2026

Uh oh!

codecov-commenter commented Sep 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

jt2594838 commented Sep 17, 2026

Validation

Benchmarks

Uh oh!

jt2594838 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Sep 17, 2026

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants