Skip to content

pr: fix OOM on infinite streams and optimize merge mode - #13540

Open
0xScodyx wants to merge 5 commits into
uutils:mainfrom
0xScodyx:fix-pr-oom-and-optimize
Open

pr: fix OOM on infinite streams and optimize merge mode#13540
0xScodyx wants to merge 5 commits into
uutils:mainfrom
0xScodyx:fix-pr-oom-and-optimize

Conversation

@0xScodyx

Copy link
Copy Markdown

Summary

Fix OOM on infinite streams (/dev/zero, /dev/random) in both single-file and -m merge mode. Additionally optimized merge mode to be ~2x faster than GNU pr.

Changes

  • Streaming: read one page at a time, print, discard — never hold more than one page in memory
  • MAX_CHUNK_SIZE = 256KB: artificial newline when input has no line/FF separators
  • read_one_page(track_content=false): skip pages outside --pages range without allocation
  • write_merge_page(): specialized for -m mode with pre-allocated buffers and direct byte writes (no intermediate String allocations)
  • BufWriter: batch stdout writes, reduced syscall overhead significantly
  • Removed dead code: get_pages(), get_file_line_groups(), to_table_merged(), etc.

Benchmarks (200MB × 2 files, -m merge mode)

Version Wall time vs GNU
Rust (before) 1.30s ~20% faster
Rust (after) 0.79s ~2x faster
GNU pr 1.57s baseline

Tests

  • All 230 test_pr tests pass (including memory-limit test)
  • /dev/zero under ulimit -v 300000 runs without OOM in both modes

Closes #13479

@0xScodyx
0xScodyx force-pushed the fix-pr-oom-and-optimize branch from 814c9da to 2d256b0 Compare July 23, 2026 15:14
@0xScodyx
0xScodyx force-pushed the fix-pr-oom-and-optimize branch from 063df07 to af76813 Compare July 23, 2026 15:52
@github-actions

Copy link
Copy Markdown

GNU testsuite comparison:

Skip an intermittent issue tests/date/date-locale-hour (fails in this run but passes in the 'main' branch)
Skip an intermittent issue tests/misc/io-errors (fails in this run but passes in the 'main' branch)
Skip an intermittent issue tests/tail/retry (fails in this run but passes in the 'main' branch)
Skip an intermittent issue tests/tail/tail-n0f (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/pr/bounded-memory (passes in this run but fails in the 'main' branch)
Congrats! The gnu test tests/seq/seq-epipe is now passing!

Comment thread src/uu/pr/src/pr.rs Outdated
loop {
// Skip pages before start_page without allocating FileLine objects.
// This avoids wasting memory when piping infinite output through
// `pr --start-page=N --end-page=M`.

@Devel08 Devel08 Jul 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there's no such flag as --start-page=N, neither in GNU nor in uutils

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it was just a personal note in the code i forgot to remove

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pr: memory leak on infinite data streams

2 participants