Skip to content

feat(acfs)!: bound ListEach memory by an entry limit - #29

Merged
kmendell merged 2 commits into
getarcaneapp:mainfrom
neurekadev:feat/28-acfs-listeach-limit
Nov 11, 2026
Merged

kmendell merged 2 commits into
getarcaneapp:mainfrom
neurekadev:feat/28-acfs-listeach-limit

Conversation

@neurekadev

@neurekadev neurekadev commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

list --limit (#26) caps the entries acfs list emits. But ListEach still read every child name into one slice and sorted it before the limit applied, so a folder with a million children held a million names in the helper's memory. This PR makes the limit bound that too.

Changes

  • ListEach on Root, Host, and FS takes acfstypes.ListOptions{MaxEntries} and returns acfstypes.ListResult{Count, Truncated}. This mirrors WalkOptions and WalkResult. Zero means unlimited, and a negative value is an error.
  • With a limit, readDirectory keeps only the smallest names while reading and trims back to the limit whenever its buffer reaches twice that size. Memory stays within 2× the limit, sorting is O(n log limit), and output stays in name order. Reserved .acfs-write- names are dropped while reading, so they never take a slot.
  • acfs list --limit passes the limit through and takes truncated from the result instead of stopping early with a sentinel error. The output format is unchanged.
  • Walk's MaxDepth probe only needs to know whether a directory has children, so it now passes MaxEntries: 1.
  • List passes the zero value. Its behavior is unchanged.

Reading the names (getdents) is still O(n). Name-ordered output needs every name, so that part can't be avoided.

Breaking change

The ListEach signature changes. Existing callers add acfstypes.ListOptions{} and drop or use the ListResult. The only caller outside this repository is Arcane's backend/pkg/workspace/tree.go ListFolder, which needs a one-line update at its next acfs bump. I chose this over separate bounded methods like the ones #27 added, so there's still one ListEach and it has the same shape as Walk. If changing the signature isn't worth it, closing this PR is a reasonable outcome. list --limit already removed the per-entry checks and the large temporary file. What's left is about 50 MB per million names, held only while a listing runs.

Measurements

One folder with 1,000,000 empty files, acfs list --limit 10000, local CGO_ENABLED=0 builds of main (bd17fe8) and this branch, two runs each:

Peak RSS Wall
main 62–63 MB ~0.47 s
This PR 13 MB ~0.41–0.49 s

At 300,000 files, peak RSS was 21–24 MB on main and 13 MB here. Both builds produced byte-identical output. Wall time doesn't change, because reading the directory dominates.

Checks

  • just format --check, just lint (0 issues), and just vet pass.
  • just test passes. The updater suite needs DOCKER_HOST, DOCKER_TLS_VERIFY, and DOCKER_CERT_PATH unset in my shell, because they point at a TLS daemon. That's unrelated to this change.
  • New TestListEachMaxEntries covers no limit, trimming during the read, trimming after the read, a limit equal to the count, a reserved name that sorts first, and a negative limit.

Closes #28

🤖 Generated with Claude Code

neurekadev and others added 2 commits November 10, 2026 12:27
ListEach read and sorted every child name before list --limit applied,
so listing a folder with a million children held all of its names in
memory. ListEach now takes ListOptions{MaxEntries} and returns a
ListResult. With a limit, it keeps only the smallest names while
reading and trims back to the limit whenever the buffer reaches twice
that size. Output stays in name order. list --limit and Walk's depth
probe use the new option.

Closes getarcaneapp#28

BREAKING CHANGE: Root.ListEach, Host.ListEach, and FS.ListEach take an
acfstypes.ListOptions argument and return (acfstypes.ListResult, error).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The MaxDepth probe lists at most one child. If that child is deleted
between the directory read and its lstat, the callback never runs and
the walk is not marked truncated, although the directory still has
other children. The probe now also uses the truncation flag from the
ListResult.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@neurekadev

Copy link
Copy Markdown
Contributor Author

🔍 Code Review

PR #29 · bd17fe8..6972548 · 7 files · ✅ No blocking issues remain

Critical: 0 High: 0 Medium: 0 Low: 2 Informational: 0

# Severity Finding Status
1 Low Walk depth probe misses truncation when its kept child vanishes
acfs/acfs.go:290 · correctness
✅ Fixed 6972548
2 Low list --limit can return fewer entries when kept names vanish
acfs/acfs.go:137 · correctness
📝 Reported
Low and informational (2)
  • 1 · Walk depth probe misses truncation when its kept child vanishes (acfs/acfs.go:290)
    • Scenario: Walk with MaxDepth reaches a directory with children x and y. x is deleted between the probe's directory read and its lstat. The MaxEntries: 1 probe keeps only x, skips it, and never calls its callback, so WalkResult.Truncated stays false even though y exists.
    • Evidence: readDirectory returns at most MaxEntries names. visitChildren skips names whose lstat returns ErrNotExist. The probe set Truncated only inside its callback. Reproduced with a temporary hook that deletes x right after the directory read: {Count:1 Truncated:false}.
    • Fix: The probe also sets Truncated from the returned ListResult. With MaxEntries: 1, that flag means the directory held a second name when it was read. (6972548)
    • Fix check: Same reproduction now gives {Count:1 Truncated:true}. A single vanished child still leaves Truncated false. Root.ListEach fills in ListResult on the fs.SkipAll path. acfs tests pass.
  • 2 · list --limit can return fewer entries when kept names vanish (acfs/acfs.go:137)
    • Scenario: A directory holds a, b, c, d, e and acfs list --limit 2 runs. Another process deletes a between the directory read and its lstat. The output is [b] with truncated:true, where main returned [b, c].
    • Evidence: Names are trimmed to the limit before visitChildren skips vanished entries, so nothing refills the slot. Reproduced with a temporary program: deleting b during the visit of a gave [a] with {Count:1 Truncated:true}.
    • Suggested fix: Keep spare names past the limit, or re-read the directory after a skip.
    • 📝 Reported: Not fixed. Spare names would break the memory bound, and a re-read is not a small local change. The effect needs a concurrent delete in the gap between the read and the lstat. truncated:true still tells the caller the list is incomplete, and Count reports how many entries were visited.

🤖 Claude Opus 5.5 (High) · T3 Code (Claude Code) · Round 1 of 3

@neurekadev
neurekadev marked this pull request as ready for review November 10, 2026 20:42
@neurekadev
neurekadev requested a review from a team as a code owner November 10, 2026 20:42
@kmendell
kmendell merged commit ac646a7 into getarcaneapp:main Nov 11, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

acfs: bound ListEach memory and sorting by the list limit

2 participants