Support readTsFile table function for external TsFiles#17951
Support readTsFile table function for external TsFiles#17951shuwenwei wants to merge 34 commits into
Conversation
Caideyipi
left a comment
There was a problem hiding this comment.
I found two issues that should be fixed before merging:
- External TsFile resources can leak if planning fails after the resource is created.
RelationPlanner.java:1647 creates the ExternalTsFileQueryResource, and ExternalTsFileQueryResource.java:106-108 increments external file-reader references. Those resources are only released through QueryExecution.stopAndCleanup() at QueryExecution.java:395. However, if execution.start() throws during logical planning, distribution planning, or scheduling, Coordinator.java:340-343 only releases frontend memory and schema locks. A timeout or optimizer/distribution exception after planExternalTsFileScan() can therefore leave reader references and temporary directories alive. Please either make QueryExecution.start() transition to failed and clean up on thrown planning exceptions, or have Coordinator.execution() release queryContext.releaseExternalTsFileQueryResources() when execution.start() fails before normal cleanup can run.
read_tsfilesilently drops ATTRIBUTE columns.
TsFileSchemaCollector.java:218-229 only collects TAG and FIELD columns, and build() at TsFileSchemaCollector.java:292-307 only emits time, tags, and fields. If an external TsFile table schema contains ColumnCategory.ATTRIBUTE, those columns disappear from the TVF output schema. The execution path also creates AlignedDeviceEntry(deviceID, new Binary[0]) in ExternalTsFileQueryResource.java:125, so attribute values are not available later either. Since table model supports ATTRIBUTE columns, this should either be implemented end to end or rejected explicitly instead of returning an incomplete schema.
|




Description
This PR adds relational
readTsFiletable function support for querying external TsFiles.Main changes:
readTsFileTVF planning/analyze support and schema collection throughTsFileSchemaCollector..tsfilefiles and skip invalid TsFile contents by magic-number validation.ExternalTsFileQueryResourceto manage external TsFile readers, task partitioning, run-file merge reads, temporary files, and execution-lifetime memory reservation cleanup.readTsFiletable function path.Tests
Not run in this turn.