Repository navigation
feat(acfs)!: bound ListEach memory by an entry limit - #29
Merged
kmendell merged 2 commits intoNov 11, 2026
Merged
Conversation
ListEach read and sorted every child name before list --limit applied,
so listing a folder with a million children held all of its names in
memory. ListEach now takes ListOptions{MaxEntries} and returns a
ListResult. With a limit, it keeps only the smallest names while
reading and trims back to the limit whenever the buffer reaches twice
that size. Output stays in name order. list --limit and Walk's depth
probe use the new option.
Closes getarcaneapp#28
BREAKING CHANGE: Root.ListEach, Host.ListEach, and FS.ListEach take an
acfstypes.ListOptions argument and return (acfstypes.ListResult, error).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The MaxDepth probe lists at most one child. If that child is deleted between the directory read and its lstat, the callback never runs and the walk is not marked truncated, although the directory still has other children. The probe now also uses the truncation flag from the ListResult. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
🔍 Code ReviewPR #29 · Low and informational (2)
🤖 Claude Opus 5.5 (High) · T3 Code (Claude Code) · Round 1 of 3 |
neurekadev
marked this pull request as ready for review
November 10, 2026 20:42
kmendell
approved these changes
Nov 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
list --limit(#26) caps the entriesacfs listemits. ButListEachstill read every child name into one slice and sorted it before the limit applied, so a folder with a million children held a million names in the helper's memory. This PR makes the limit bound that too.Changes
ListEachonRoot,Host, andFStakesacfstypes.ListOptions{MaxEntries}and returnsacfstypes.ListResult{Count, Truncated}. This mirrorsWalkOptionsandWalkResult. Zero means unlimited, and a negative value is an error.readDirectorykeeps only the smallest names while reading and trims back to the limit whenever its buffer reaches twice that size. Memory stays within 2× the limit, sorting is O(n log limit), and output stays in name order. Reserved.acfs-write-names are dropped while reading, so they never take a slot.acfs list --limitpasses the limit through and takestruncatedfrom the result instead of stopping early with a sentinel error. The output format is unchanged.Walk'sMaxDepthprobe only needs to know whether a directory has children, so it now passesMaxEntries: 1.Listpasses the zero value. Its behavior is unchanged.Reading the names (
getdents) is still O(n). Name-ordered output needs every name, so that part can't be avoided.Breaking change
The
ListEachsignature changes. Existing callers addacfstypes.ListOptions{}and drop or use theListResult. The only caller outside this repository is Arcane'sbackend/pkg/workspace/tree.goListFolder, which needs a one-line update at its next acfs bump. I chose this over separate bounded methods like the ones #27 added, so there's still oneListEachand it has the same shape asWalk. If changing the signature isn't worth it, closing this PR is a reasonable outcome.list --limitalready removed the per-entry checks and the large temporary file. What's left is about 50 MB per million names, held only while a listing runs.Measurements
One folder with 1,000,000 empty files,
acfs list --limit 10000, localCGO_ENABLED=0builds ofmain(bd17fe8) and this branch, two runs each:mainAt 300,000 files, peak RSS was 21–24 MB on
mainand 13 MB here. Both builds produced byte-identical output. Wall time doesn't change, because reading the directory dominates.Checks
just format --check,just lint(0 issues), andjust vetpass.just testpasses. Theupdatersuite needsDOCKER_HOST,DOCKER_TLS_VERIFY, andDOCKER_CERT_PATHunset in my shell, because they point at a TLS daemon. That's unrelated to this change.TestListEachMaxEntriescovers no limit, trimming during the read, trimming after the read, a limit equal to the count, a reserved name that sorts first, and a negative limit.Closes #28
🤖 Generated with Claude Code