Skip to content

[Feature] Support appending to a normally closed TsFile in C++ and C APIs #923

Description

@ColinLeeo

Motivation

The Java implementation can reopen a normally closed (complete) TsFile for appending through:

RestorableTsFileIOWriter.getWriterForAppendingDataOnCompletedTsFile(File)

This method detects a complete file, locates the start of FileMetadata from the footer, truncates the separator marker and all tail metadata, recovers the existing schema/chunk metadata, and then allows new data to be appended. Closing the writer generates a new metadata section and tail magic.

The C++ implementation currently supports recovery and continued writing only when the file is incomplete or its tail is damaged. When the tail magic is valid, RestorableTsFileIOWriter::self_check() treats the file as complete, sets can_write_ = false, and closes the write handle.

As a result, users of the C++ API and the C wrapper cannot append to a TsFile that was closed normally, even though the equivalent workflow is available in Java.

Expected behavior

Provide a supported C++ API equivalent to the Java completed-file append workflow, and expose it through the C API as well.

The workflow should:

  1. Validate that the input is a compatible TsFile.
  2. Detect whether the file is complete by checking the tail magic.
  3. For a complete file, read the footer metadata size and locate the start of FileMetadata.
  4. Safely truncate the separator marker, metadata/index section, footer size, and tail magic while retaining all completed chunk groups.
  5. Recover the existing tree/table schema, alignment information, chunk metadata, statistics, and per-device last timestamps.
  6. Allow users to append data using the recovered schema.
  7. Allow registration of supported new schemas after recovery, with clear behavior for:
    • a new device or measurement in the tree model;
    • a new table;
    • attempts to replace or evolve an existing table/aligned schema.
  8. On close, write a new metadata section, footer, and tail magic so that the resulting file is complete and readable.
  9. Reject out-of-order timestamps relative to recovered data.

The operation should preserve the original file when validation or recovery fails before truncation. The API documentation should make the destructive/in-place nature of a successful append-open explicit.

Current C++ behavior

For an incomplete or damaged-tail file, the following path already works:

RestorableTsFileIOWriter::open(path, true)
→ recover schema and chunk metadata
TsFileWriter::init(&restorable_writer)
→ append data
→ close and rebuild metadata

For a normally closed file, open() returns successfully but can_write() is false, so it cannot be passed to TsFileWriter::init().

Suggested tests

  • Append to a normally closed tree-model TsFile and verify old and new rows.
  • Append to a normally closed table-model TsFile.
  • Preserve recovered encoding/compression/data types and alignment.
  • Register a new device and a new non-aligned measurement after recovery.
  • Verify documented behavior for aligned-series and existing-table schema evolution.
  • Reject duplicate or incompatible schemas.
  • Reject timestamps not newer than the recovered last timestamp.
  • Handle empty, incomplete, incompatible, encrypted, and large-metadata files.
  • Verify that a no-op append followed by close produces a valid file without losing data.
  • Add coverage for the C wrapper API.

Related implementation

  • Java: RestorableTsFileIOWriter.getWriterForAppendingDataOnCompletedTsFile(File)
  • C++: RestorableTsFileIOWriter::self_check()
  • C++ recovery integration: TsFileWriter::init(RestorableTsFileIOWriter*)

Metadata

Metadata

Assignees

No one assigned

    Labels

    CPull requests that update C codec++Pull requests that update c++ codefeatureNew featurehelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions