FileIO parameter should be arrow::fs::FileSystem ?

[rip-nsk]

The ManifestWriter/ManifestListWriter API uses the std::shared_ptr<FileIO> parameter for the IO interface,
which actually should be an internal arrow::ArrowFileSystemFileIO (not publicly accessible).
It appears that only the fs() is actually utilized.

  auto io = internal::checked_pointer_cast<arrow::ArrowFileSystemFileIO>(options.io);
  ICEBERG_ARROW_ASSIGN_OR_RETURN(auto output, io->fs()->OpenOutputStream(options.path));

[wgtmac]

This is by design because arrow::fs::FileSystem is just one approach to implement a FileIO. libiceberg does not have any Arrow dependency and we don't expose symbols like Arrow and Avro.

[rip-nsk]

The current API or implementation need to be revised, as they are currently unusable due to the unavailability of ArrowFileSystemFileIO.
Additionally, the FileIO interface may not be optimal for handling large metadata, as it requires full read/write operations at once.

[wgtmac]

We have two libraries: libiceberg and libiceberg-bundle. libiceberg only uses interfaces of FileIO, Reader and Writer and downstream projects are responsible for implementing these. ArrowFileSystemFileIO (from libiceberg-bundle) is just an example to demonstrate how we can do that with the Arrow library. Since libiceberg only deals with metadata json file so FileIO simply depends on simple read and write operations. libiceberg-bundle itself knows that the FileIO implementation is actually the ArrowFileSystemFileIO so the best practice is to down cast FileIO to ArrowFileSystemFileIO so we can have access to arrow::fs::FileSystem which is required by the Parquet reader and writers without conversion.

[wgtmac]

Of course we can enhance and improve the FileIO APIs for any reasonable requirement. We just don't want to over-design it at the beginning and avoid exposing its implementation internals.

[rip-nsk]

I'm referring to the process of writing a manifest in Avro format, which is a basic metadata operation (within libicebrg):
iceberg::ManifestWriter::Make*Writer
-> iceberg::OpenFileWriter
-> iceberg::WriterFactoryRegistry::Open(FileFormatType::kAvro
-> iceberg::avro::CreateOutputStream
The manifest can be large, potentially reaching hundreds of MBs or even a few GBs, due to the included statistics.

[wgtmac]

Yes, manifest can be large. It can be handled by https://github.com/apache/iceberg-cpp/blob/main/src/iceberg/avro/avro_stream_internal.h#L65 via a buffered write and the buffer size is configurable.

[chongchongxiao]

@wgtmac I don't quite understand， if i wan write to other remote object storage，should i to extend libarrow instead of libiceberg ?
This seems a bit unreasonable.

[wgtmac]

::arrow::fs::FileSystem is pluggable so you can extend it to implement OSS support in your application. Arrow-cpp has builtin implementations for AWS S3 and Azure blob store.

[chongchongxiao]

The documentation is missing. Could you provide more detailed documentation or examples of writing or reading data? The only example shown only reading data from an existing table.

[wgtmac]

We just supported basic reading. Writing support is still under development. Theoretically it should work out of box if you build Arrow with object store support by enabling ARROW_S3 and other similar components: https://arrow.apache.org/docs/developers/cpp/building.html#optional-components.

Just a kindly reminder that iceberg-cpp only focuses on the metadata layer of Iceberg spec. The heavy-lifting I/O implementation has been delegated to ::arrow::fs::FileSystem to avoid reinventing the wheels.

[chongchongxiao]

Thank you, I understand now. Data io will all reuse the arrrow api. By the way, besides RestCatalog, which type of catalog will be supported first, and is there a general deadline?

[wgtmac]

There is an issue discussing SqlCatalog: #135. Due to developer bandwidth, we don't have any deadline yet. Our next milestone is to release 0.2.0: #226

[chongchongxiao]

I noticed that InMemoryCatalog is also incomplete. Is there a plan to complete it?
Is there a release date for version 0.2.0?

[wgtmac]

InMemoryCatalog just reflects the current state of all supported operations. We will improve it once update actions are supported. We rely on it heavily for testing.

Is there a release date for version 0.2.0?

No, we will release it once we complete all planned tasks.