Skip to content

Conversation

@kou
Copy link
Member

@kou kou commented Dec 21, 2025

Rationale for this change

It's a streaming variant of IPC format.

What changes are included in this PR?

  • Add ArrowFormat::MessagePullReader
  • Add ArrowFormat::StreamingPullReader
  • Add ArrowFormat::StreamingReader
  • Extract read schema and record batch features as ArrowFormat::Readable and reuse it in ArrowFormat::StreamingPullParser and ArrowFormat::FileReader

Are these changes tested?

Yes.

We should add more error case tests later.

Are there any user-facing changes?

Yes.

@github-actions
Copy link

⚠️ GitHub issue #48612 has been automatically assigned in GitHub to PR creator.

@kou kou merged commit 42043c8 into apache:main Dec 22, 2025
9 checks passed
@kou kou removed the awaiting committer review Awaiting committer review label Dec 22, 2025
@kou kou deleted the ruby-streaming-decoder branch December 22, 2025 01:17
@hiroyuki-sato
Copy link
Collaborator

+1 (When I reviewed it and tried to LGTM it, it had already been merged. 😄 )

LD_LIBRARY_PATH=/tmp/local/lib GI_TYPELIB_PATH=/tmp/local/lib/girepository-1.0 rake test
cd /path/to/arrow/ruby/red-arrow-format
/path/to/.rbenv/versions/3.4.7/bin/ruby test/run.rb
Loaded suite test
Started
/path/to/arrow/ruby/red-arrow-format/lib/arrow-format/file-reader.rb:43: warning: IO::Buffer is experimental and both the Ruby and C interface may change in the future!
Finished in 1.422469194 seconds.
--------------------------------------------------------------------------------------------------------------------------------
98 tests, 98 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
100% passed
--------------------------------------------------------------------------------------------------------------------------------
68.89 tests/s, 68.89 assertions/s
cd -

Jonahkel pushed a commit to Jonahkel/arrow that referenced this pull request Dec 22, 2025
…he#48613)

### Rationale for this change

It's a streaming variant of IPC format.

### What changes are included in this PR?

* Add `ArrowFormat::MessagePullReader`
* Add `ArrowFormat::StreamingPullReader`
* Add `ArrowFormat::StreamingReader`
* Extract read schema and record batch features as `ArrowFormat::Readable` and reuse it in `ArrowFormat::StreamingPullParser` and `ArrowFormat::FileReader`

### Are these changes tested?

Yes.

We should add more error case tests later.

### Are there any user-facing changes?

Yes.
* GitHub Issue: apache#48612

Authored-by: Sutou Kouhei <kou@clear-code.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
@conbench-apache-arrow
Copy link

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 42043c8.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants