Skip to content

Reduce GCS QPS on ArrayRecordReader open via ReadLast. - #212

Open
copybara-service[bot] wants to merge 1 commit into
mainfrom
test_992624347
Open

copybara-service[bot] wants to merge 1 commit into
mainfrom
test_992624347

Conversation

@copybara-service

@copybara-service copybara-service Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

Reduce GCS QPS on ArrayRecordReader open via ReadLast.

Previously, opening a gs:// ArrayRecord shard constructed a new
google::cloud::storage::Client instance and issued 3 separate GCS HTTP GET
requests:

  1. An unbounded GET at offset 0 during GcsReader construction solely to obtain
    the object size (and then aborted mid-stream).
  2. A GET via reader->NewReader(size - 64KB) to read the Riegeli postscript.
  3. A GET via reader->NewReader(footer_offset) to read the ArrayRecord footer
    (and in OffloadedChunkOffset::operator[] when index_storage_option is
    offloaded).

This change:

  1. Shares a process-wide google::cloud::storage::Client singleton across
    ArrayRecordReader instances in array_record_module.cc.
  2. Opens riegeli::GcsReader with google::cloud::storage::ReadLast(1 MiB) so
    the initial HTTP GET retrieves the total object size (Content-Range) and
    prefetches the last 1 MiB of the shard.
  3. In ArrayRecordReaderBase::Initialize(), pulls the prefetched tail into
    reader's buffer (resetting the cursor to limit() so available() == 0 on
    close) and reads the postscript and footer (including offloaded chunk index
    lookups in OffloadedChunkOffset::operator[]) directly from reader's
    buffered tail via ReadChunk when footer_offset >= reader->start_pos().

@copybara-service
copybara-service Bot force-pushed the test_992624347 branch 3 times, most recently from 96b3852 to 1c4bdee Compare October 5, 2026 19:57
Previously, opening a `gs://` ArrayRecord shard constructed a new
`google::cloud::storage::Client` instance and issued 3 separate GCS HTTP GET
requests:
1. An unbounded GET at offset 0 during `GcsReader` construction solely to obtain
   the object size (and then aborted mid-stream).
2. A GET via `reader->NewReader(size - 64KB)` to read the Riegeli postscript.
3. A GET via `reader->NewReader(footer_offset)` to read the ArrayRecord footer
   (and in `OffloadedChunkOffset::operator[]` when `index_storage_option` is
   `offloaded`).

This change:
1. Shares a process-wide `google::cloud::storage::Client` singleton across
   `ArrayRecordReader` instances in `array_record_module.cc`.
2. Opens `riegeli::GcsReader` with `google::cloud::storage::ReadLast(1 MiB)` so
   the initial HTTP GET retrieves the total object size (`Content-Range`) and
   prefetches the last 1 MiB of the shard.
3. In `ArrayRecordReaderBase::Initialize()`, pulls the prefetched tail into
   `reader`'s buffer (resetting the cursor to `limit()` so `available() == 0` on
   close) and reads the postscript and footer (including offloaded chunk index
   lookups in `OffloadedChunkOffset::operator[]`) directly from `reader`'s
   buffered tail via `ReadChunk` when `footer_offset >= reader->start_pos()`.

PiperOrigin-RevId: 992624347

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant