Skip to content

[C++] Keep scanning after a fully filtered aligned page (#1038) - #1044

Merged
ColinLeeo merged 2 commits into
developfrom
claude/unruffled-darwin-2c5acd
Nov 2, 2026
Merged

ColinLeeo merged 2 commits into
developfrom
claude/unruffled-darwin-2c5acd

Conversation

@ColinLeeo

Copy link
Copy Markdown
Contributor

Problem

Fixes #1038.

A C++ aligned table query using TimeIn could silently drop matching rows that follow a page whose rows were all rejected by the filter. The API still returned E_OK and reported a normal end of results.

Root cause

AlignedChunkReader::decode_tv_buf_into_tsblock_by_datatype converted a successful decode that produced no rows into E_NO_MORE_DATA:

if (ret_tsblock->get_row_count() == 0 && ret == E_OK) {
    ret = E_NO_MORE_DATA;
}

That conflates "this page had no matching rows" with "the data stream is exhausted". TsFileSeriesScanIterator::get_next reacts to E_NO_MORE_DATA by advancing to the next chunk, or by ending the series when the current chunk is the last one — so the remaining pages of the chunk were never read. The iterator already has the correct handling for an empty successful block; the rewrite made it unreachable on this path.

The multi-value aligned path and the offset-aware non-aligned path do not have this rewrite, which is why the bug only surfaced on the per-column aligned read path (e.g. when a requested measurement is never written and the query falls back to one SSI per column).

Fix

Return E_OK with zero rows for a fully filtered page and let the series iterator continue within the chunk. True end of stream is still signalled by has_more_data().

Regression test

TimeInContinuesAfterFullyFilteredAlignedPage in cpp/test/reader/table_view/tsfile_reader_table_test.cc reproduces the reported layout: three devices, three chunks of seven rows with an explicit flush, page_writer_max_point_num_ = 5, a ghost measurement that is never written, v missing entirely on one device, sparse NULLs in v/w, serial and parallel read, and a 4096-byte internal TsBlock budget. It covers:

  • the reported TimeIn({0, 2, 3, 6, 7, 10, 13, 19, 20}) query, which lost timestamps 19 and 20 per device before the fix;
  • a filter matching only the page after a fully filtered page with no successor chunk;
  • a filter matching only the first chunk, leaving later empty pages with successor chunks;
  • all-matching and no-matching controls.

Verification

  • New test fails on the unmodified source with 7 timestamps per device and passes with the fix (serial and parallel).
  • Full C++ suite: 1414 tests pass, 0 failures.
  • The standalone reproducer from the issue now prints after scan ret=0 count=27 expected=27 and exits 0.

This is separate from the small-capacity hang tracked in #1035.

The single-column aligned reader turned a successful decode that produced
no rows into E_NO_MORE_DATA. The series scan iterator reads that as "chunk
exhausted": it jumps to the next chunk, or ends the series when the current
chunk is the last one, so matching pages after a fully filtered page were
never read. A TimeIn query over a table with a missing measurement could
therefore silently drop rows and still report E_OK.

An empty result only means every row of the page was filtered out, so
return E_OK and let the iterator's existing empty-block handling continue
within the chunk. True end of stream is still signalled by has_more_data().

Add a table-level regression test that reproduces the reported layout
(three devices, three chunks, page size 5, a never-written measurement and
a missing column on one device) and covers empty pages with and without a
successor chunk, plus all-matching and no-matching controls.
TsFile_Test is compiled with -fprofile-arcs/-ftest-coverage, but #1016
changed libtsfile's coverage runtime dependency to PRIVATE, so it no longer
propagated to consumers. The Release LTO link then failed with undefined
__gcov_init/__gcov_exit/__gcov_merge_add, which has kept the codecov job red
since 2026-10-16. Link the coverage runtime into TsFile_Test itself.
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 66.46%. Comparing base (d22008b) to head (7526b00).
⚠️ Report is 8 commits behind head on develop.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1044      +/-   ##
===========================================
+ Coverage    65.26%   66.46%   +1.20%     
===========================================
  Files          761      773      +12     
  Lines        53610    55796    +2186     
  Branches      8547     8945     +398     
===========================================
+ Hits         34986    37086    +2100     
- Misses       16978    17055      +77     
- Partials      1646     1655       +9     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ColinLeeo
ColinLeeo merged commit 2885cea into develop Nov 2, 2026
40 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][C++] TimeIn query drops rows after a fully filtered aligned page

2 participants