diff --git a/.claude/skills/devnet-log-review/SKILL.md b/.claude/skills/devnet-log-review/SKILL.md index f3322a00..85b1d7ee 100644 --- a/.claude/skills/devnet-log-review/SKILL.md +++ b/.claude/skills/devnet-log-review/SKILL.md @@ -145,7 +145,7 @@ grep "signature verification failed" lantern_0.log ### Finalization Debugging -Finalization should advance every 6-12 slots. If it stalls, investigate: +On a healthy devnet finalization advances every few slots. If it stalls, investigate: ```bash # Check finalization progress @@ -154,9 +154,11 @@ grep "finalized_slot=" ethlambda_0.log | tail -20 # If finalized_slot stays same for 50+ slots → finalization stalled ``` -**Finalization requires >2/3 supermajority:** -- 6 validators → need 5 votes minimum -- 9 validators → need 7 votes minimum +**Justification requires a ≥2/3 supermajority** (`3 * votes >= 2 * validator_count`, +i.e. `ceil(2N/3)` — see `crates/blockchain/state_transition/src/lib.rs`): +- 6 validators → need 4 votes minimum +- 9 validators → need 6 votes minimum +- 16 validators → need 11 votes minimum **See [references/FINALIZATION_DEBUG.md](references/FINALIZATION_DEBUG.md) for:** - Common causes of finalization stalls @@ -184,15 +186,15 @@ Different clients have different log formats and key patterns. ## Block Proposal Flow (ethlambda) -A healthy block proposal follows this sequence: +Since the pre-build change (#445), the proposer builds at the *previous* slot's +interval 4 and publishes aligned to the slot boundary. A healthy block proposal +follows this sequence: -1. `We are the proposer for this slot` - Node detects it's the proposer -2. `TODO precompute poseidons in parallel + SIMD` - XMSS aggregate proof starts -3. `packed_pcs_commit` - Proof commitment -4. `Logup data` - Logup protocol data -5. `AIR proof{table=poseidon16}` / `AIR proof{table=poseidon24}` - AIR proofs -6. `Published block` - Block successfully built and published -7. `Published block to gossipsub` - Block broadcast to network +1. `We are the proposer for this slot` - Node detects it's the proposer (fires one slot early) +2. leanVM proving output (`packed_pcs_commit`, `Logup data`, `AIR proof{table=poseidon16|poseidon24}`) - XMSS aggregate proof +3. `Finished building block` - Build complete (#474; this is where the build-time metric stops) +4. `Published block` - Published at the slot boundary (the gap to step 3 is idle wait, not build cost) +5. `Published block to gossipsub` - Block broadcast to network ## Summary Report Format diff --git a/.claude/skills/test-pr-devnet/SKILL.md b/.claude/skills/test-pr-devnet/SKILL.md index 4f3829bc..53f851c2 100644 --- a/.claude/skills/test-pr-devnet/SKILL.md +++ b/.claude/skills/test-pr-devnet/SKILL.md @@ -53,7 +53,7 @@ Test ethlambda branch changes in a multi-client local devnet with zeam (Zig), re **Success criteria:** - ✅ No errors in ethlambda logs - ✅ All 4 nodes at same head slot -- ✅ Finalization advancing (every 6-12 slots) +- ✅ Finalization advancing (every few slots on a healthy devnet) - ✅ Each validator produces blocks for their slots ### Sync Recovery (~90-120s) @@ -139,25 +139,13 @@ sleep 10 # Wait for sync # Quick status .claude/skills/test-pr-devnet/scripts/check-status.sh -# Detailed analysis (use devnet-log-review skill in lean-quickstart) -cd $LEAN_QUICKSTART +# Detailed analysis (use this repo's devnet-log-review skill; dump logs first) +for node in zeam_0 ream_0 qlean_0 ethlambda_0; do + docker logs "$node" > "${node}.log" 2>&1 +done .claude/skills/devnet-log-review/scripts/analyze-logs.sh ``` -## Protocol Compatibility - -| Client | Status | Gossipsub | BlocksByRoot | -|--------|--------|-----------|--------------| -| ream | ✅ Full | ✅ Full | ✅ Full | -| zeam | ✅ Full | ✅ Full | ⚠️ Limited | -| qlean | ✅ Full | ✅ Full | ⚠️ Limited | -| ethlambda | ✅ Full | ✅ Full | ✅ Full | - -**Notes:** -- zeam/qlean BlocksByRoot errors are expected (not a blocker) -- ream ↔ ethlambda BlocksByRoot should work perfectly -- All clients use Gossipsub for block propagation - ## Verification Checklist | Check | Command | Expected | @@ -256,5 +244,5 @@ docker logs ethlambda_0 2>&1 | grep -i "peer\|connection" | head -20 ## References -- **[ethlambda CLAUDE.md](../../CLAUDE.md)** - Development workflow, detailed debugging commands -- **[lean-quickstart devnet-log-review](../../../lean-quickstart/.claude/skills/devnet-log-review/SKILL.md)** - Comprehensive log analysis +- **[ethlambda CLAUDE.md](../../../CLAUDE.md)** - Development workflow, detailed debugging commands +- **[devnet-log-review](../devnet-log-review/SKILL.md)** - Comprehensive log analysis (in this repo) diff --git a/CLAUDE.md b/CLAUDE.md index 17ac1b34..74bb4f17 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -9,7 +9,7 @@ Not to be confused with Ethereum consensus clients AKA Beacon Chain clients AKA **Rust version:** 1.92.0 (edition 2024) **Test fixtures release:** Download latest production fixtures from leanSpec releases -## Codebase Structure (10 crates) +## Codebase Structure (12 workspace crates) ``` bin/ethlambda/ # Entry point, CLI, orchestration @@ -18,17 +18,24 @@ crates/ blockchain/ # State machine actor (GenServer pattern) ├─ src/lib.rs # BlockChain actor, tick events, validator duties ├─ src/store.rs # Fork choice store, block/attestation processing + ├─ src/block_builder.rs # Block assembly (pre-built at previous slot's interval 4) + ├─ src/aggregation.rs # Interval-2 signature aggregation worker + ├─ src/reaggregate.rs # Re-aggregation of block-borne votes on import + ├─ src/sync_status.rs # Sync-gate tracker (suppresses duties while syncing) ├─ src/key_manager.rs # Validator key management and signing ├─ src/metrics.rs # Blockchain-level Prometheus metrics - ├─ fork_choice/ # LMD GHOST implementation (3SF-mini) - └─ state_transition/ # STF: process_slots, process_block, attestations + ├─ fork_choice/ # [crate] LMD GHOST implementation (3SF-mini) + └─ state_transition/ # [crate] STF: process_slots, process_block, attestations + ├─ src/justified_slots_ops.rs # Relative-index helpers for justified_slots └─ src/metrics.rs # State transition timing + counters common/ ├─ types/ # Core types (State, Block, Attestation, Checkpoint) ├─ crypto/ # XMSS aggregation (leansig wrapper) - └─ metrics/ # Prometheus re-exports, TimingGuard, gather utilities + ├─ metrics/ # Prometheus re-exports, TimingGuard, gather utilities + └─ test-fixtures/ # Spec-fixture loading (prod dep of rpc's Hive test driver) net/ - ├─ p2p/ # libp2p: gossipsub + req-resp (Status, BlocksByRoot) + ├─ api/ # Actor protocol traits wiring BlockChain ↔ P2P + ├─ p2p/ # libp2p: gossipsub + req-resp (Status, BlocksByRoot, BlocksByRange) │ ├─ src/gossipsub/ # Topic encoding, message handling │ ├─ src/req_resp/ # Request/response codec and handlers │ └─ src/metrics.rs # Peer connection/disconnection tracking @@ -56,12 +63,14 @@ Interval 4: Accept accumulated attestations; build the NEXT slot's block and pub ### Attestation Pipeline ``` -Gossip → Signature verification → new_attestations (pending) - ↓ (interval 4) -promote → known_attestations (fork choice active) +Gossip → Signature verification → new_payloads (pending) + ↓ (intervals 0/4) +promote → known_payloads (fork choice active) ↓ Fork choice head update ``` +(Store buffer fields are `new_payloads`/`known_payloads`; the accessors are named +`extract_latest_new_attestations`/`extract_latest_known_attestations`.) ### State Transition Phases 1. **process_slots()**: Advance through empty slots, update historical roots @@ -105,7 +114,7 @@ let byte: u8 = code.into(); ### Ownership for Large Structures ```rust -// Prefer taking ownership to avoid cloning large data (signatures ~3KB) +// Prefer taking ownership to avoid cloning large data (signatures ~2.5KB) pub fn insert_signed_block(&mut self, root: H256, signed_block: SignedBlock) { ... } // Add .clone() at call site if needed - makes cost explicit @@ -252,7 +261,7 @@ actual_slot = finalized_slot + 1 + relative_index **XMSS (eXtended Merkle Signature Scheme):** - Post-quantum signature scheme -- 52-byte public keys, 3112-byte signatures +- 52-byte public keys, 2536-byte signatures (`SIGNATURE_SIZE` in `common/types/src/signature.rs`) - Epoch-based to prevent reuse - Aggregation via leanVM (previously leanMultisig) for efficiency @@ -268,11 +277,11 @@ actual_slot = finalized_slot + 1 + relative_index - Topic: `/leanconsensus/{fork_digest}/{block|aggregation|attestation_N}/ssz_snappy` - `fork_digest` is a 4-byte hex string (no `0x` prefix); currently the dummy `12345678` agreed across clients - Mesh size: 8 (6-12 bounds), heartbeat: 700ms -- **Req/Resp**: Status, BlocksByRoot (snappy frame compression + varint length) +- **Req/Resp**: Status, BlocksByRoot, BlocksByRange (snappy frame compression + varint length) ### Retry Strategy on Block Requests -- Exponential backoff: 10ms, 40ms, 160ms, 640ms, 2560ms -- Max 5 attempts, random peer selection on retry +- Exponential backoff: doubling from `INITIAL_BACKOFF_MS` (5ms → 2560ms) +- Max `MAX_FETCH_RETRIES` (10) attempts, random peer selection on retry ### Message IDs - 20-byte truncated SHA256 of: domain (valid/invalid snappy) + topic + data @@ -301,9 +310,9 @@ GENESIS_VALIDATORS: ### Test Categories 1. **Unit tests**: Embedded in source files 2. **Spec tests**: From `leanSpec/fixtures/consensus/` - - `forkchoice_spectests.rs` (uses `on_block_without_verification`) - - `signature_spectests.rs` - - `stf_spectests.rs` (state transition) + - `crates/blockchain/tests/forkchoice_spectests.rs` (uses `on_block_without_verification` via `spec_test_runner`) + - `crates/blockchain/tests/signature_spectests.rs` + - `crates/blockchain/state_transition/tests/stf_spectests.rs` (state transition) ### Running Tests ```bash @@ -316,7 +325,7 @@ cargo test -p ethlambda-blockchain --test forkchoice_spectests -- --test-threads ### Aggregator Flag Required for Finalization - At least one node **must** be started with `--is-aggregator` to finalize blocks -- Without this flag, attestations pass signature verification and are logged as "Attestation processed", but the signature is never stored for aggregation (`store.rs:368`), so blocks are always built with `attestation_count=0` +- Without this flag, attestations pass signature verification and are logged as "Attestation processed", but the signature is never stored for aggregation (the `is_aggregator` gate in `on_gossip_attestation`, `store.rs`), so blocks are always built with `attestation_count=0` - The attestation pipeline: gossip → verify signature → store gossip signature (only if `is_aggregator`) → aggregate at interval 2 → promote to known → pack into blocks - **Symptom**: `justified_slot=0` and `finalized_slot=0` indefinitely despite healthy block production and attestation gossip @@ -368,7 +377,7 @@ and are consumed during the tick pipeline (promotion at intervals 0/4, aggregation at interval 2). ### State Root Computation -- Always computed via `tree_hash_root()` after full state transition +- Always computed via `hash_tree_root()` after full state transition - Must match proposer's pre-computed `block.state_root` ### Finalization Checks @@ -383,8 +392,8 @@ aggregation at interval 2). **Critical:** - `leansig`: XMSS signatures (leanEthereum project) -- `ethereum_ssz`: SSZ serialization -- `tree_hash`: Merkle tree hashing +- `libssz` / `libssz-derive` / `libssz-types`: SSZ serialization +- `libssz-merkle`: Merkle tree hashing (`hash_tree_root()`) - `spawned-concurrency`: Actor model - `libp2p`: P2P networking (custom LambdaClass fork) - `vergen-git2`: Build-time git commit/branch info embedded in binary @@ -397,6 +406,7 @@ aggregation at interval 2). **Specs:** `leanSpec/src/lean_spec/` (Python reference implementation) **Devnet:** `lean-quickstart` (github.com/blockblaz/lean-quickstart) +**Docs:** `docs/` — `rpc.md`, `metrics.md`, `checkpoint_sync.md`, `3sf_mini.md`, `lmd_ghost.md` (mdbook via `make docs`) **Releases:** See `RELEASE.md` for release process documentation ## Other implementations diff --git a/crates/storage/src/api/tables.rs b/crates/storage/src/api/tables.rs index dcda1cbf..163bbe0f 100644 --- a/crates/storage/src/api/tables.rs +++ b/crates/storage/src/api/tables.rs @@ -5,15 +5,18 @@ pub enum Table { BlockHeaders, /// Block body storage: H256 -> BlockBody BlockBodies, - /// Block signatures storage: H256 -> BlockSignatures + /// Block signatures storage: (slot || root) -> BlockSignatures /// /// Stored separately from blocks because the genesis block has no signatures. - /// All other blocks must have an entry in this table. + /// Keyed by slot || root so pruning can scan in slot order and stop early. + /// Non-genesis blocks have an entry until finalized: signatures below the + /// finalized boundary are pruned (`prune_old_block_signatures`), while + /// headers and bodies are kept forever. BlockSignatures, /// State storage: H256 -> State /// - /// Holds full-state snapshots only: the bootstrap anchor plus one anchor per - /// 1024-slot window. Never pruned. Non-anchor states live in `StateDiffs` and + /// Holds full-state snapshots only: the bootstrap anchor plus one anchor + /// every `SNAPSHOT_ANCHOR_INTERVAL` slots. Never pruned. Non-anchor states live in `StateDiffs` and /// are reconstructed on demand (memoized by an in-memory cache). States, /// State diffs: H256 -> StateDiff