Skip to content

feat: add configurable io rate limiting for snapshot writes#2802

Open
blindchaser wants to merge 1 commit intorelease/v6.3from
yiren/v631-sp-limit
Open

feat: add configurable io rate limiting for snapshot writes#2802
blindchaser wants to merge 1 commit intorelease/v6.3from
yiren/v631-sp-limit

Conversation

@blindchaser
Copy link
Contributor

@blindchaser blindchaser commented Feb 4, 2026

Describe your changes and provide context

Background

v6.3 optimized snapshot creation time from >3 hours to ~20 minutes using aggressive parallel writes. While this improves performance on high-end machines, it can cause high I/O bursts that may impact page cache efficiency on systems with limited RAM.

Solution

Add optional I/O rate limiting for snapshot writes:

  • New config: sc-snapshot-write-rate-mbps (default: 0 = unlimited)
  • Global limiter shared across all trees/files using token bucket algorithm
  • Allows operators to trade snapshot speed for more stable I/O patterns

Configuration Example

[state-commit]
sc-snapshot-write-rate-mbps = 300

@github-actions
Copy link

github-actions bot commented Feb 4, 2026

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedFeb 4, 2026, 7:49 PM

@codecov
Copy link

codecov bot commented Feb 4, 2026

Codecov Report

❌ Patch coverage is 81.66667% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 45.95%. Comparing base (c68f987) to head (979adf7).

Files with missing lines Patch % Lines
sei-db/sc/memiavl/snapshot.go 84.61% 3 Missing and 3 partials ⚠️
sei-db/sc/memiavl/multitree.go 70.00% 1 Missing and 2 partials ⚠️
sei-db/config/config.go 0.00% 1 Missing ⚠️
sei-db/sc/store.go 0.00% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@               Coverage Diff                @@
##           release/v6.3    #2802      +/-   ##
================================================
+ Coverage         45.17%   45.95%   +0.78%     
================================================
  Files               853     1199     +346     
  Lines             70076   104465   +34389     
================================================
+ Hits              31654    48011   +16357     
- Misses            35760    52224   +16464     
- Partials           2662     4230    +1568     
Flag Coverage Δ
sei-chain 42.53% <100.00%> (+<0.01%) ⬆️
sei-db 45.15% <81.35%> (+0.29%) ⬆️
sei-ibc-go 55.96% <ø> (ø)
sei-tendermint 47.50% <ø> (?)
sei-wasmd 41.56% <ø> (ø)
sei-wasmvm 39.88% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
app/seidb.go 83.01% <100.00%> (+0.32%) ⬆️
sei-db/sc/memiavl/db.go 63.33% <100.00%> (+0.12%) ⬆️
sei-db/sc/memiavl/opts.go 100.00% <ø> (ø)
sei-db/config/config.go 0.00% <0.00%> (ø)
sei-db/sc/store.go 0.00% <0.00%> (ø)
sei-db/sc/memiavl/multitree.go 79.09% <70.00%> (-0.32%) ⬇️
sei-db/sc/memiavl/snapshot.go 58.44% <84.61%> (+1.62%) ⬆️

... and 348 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

cursor bot pushed a commit that referenced this pull request Feb 5, 2026
- Analyzed snapshot write rate limiting implementation
- Validated design decisions and code quality
- Identified pre-existing test failure in storev2/rootmulti (unrelated)
- Recommendation: APPROVE with minor notes
- All tests passing, well-designed global rate limiter approach

Co-authored-by: Steven Landers <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants