Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 14 additions & 12 deletions .claude/skills/devnet-log-review/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ grep "signature verification failed" lantern_0.log

### Finalization Debugging

Finalization should advance every 6-12 slots. If it stalls, investigate:
On a healthy devnet finalization advances every few slots. If it stalls, investigate:

```bash
# Check finalization progress
Expand All @@ -154,9 +154,11 @@ grep "finalized_slot=" ethlambda_0.log | tail -20
# If finalized_slot stays same for 50+ slots → finalization stalled
```

**Finalization requires >2/3 supermajority:**
- 6 validators → need 5 votes minimum
- 9 validators → need 7 votes minimum
**Justification requires a ≥2/3 supermajority** (`3 * votes >= 2 * validator_count`,
i.e. `ceil(2N/3)` — see `crates/blockchain/state_transition/src/lib.rs`):
- 6 validators → need 4 votes minimum
- 9 validators → need 6 votes minimum
- 16 validators → need 11 votes minimum

**See [references/FINALIZATION_DEBUG.md](references/FINALIZATION_DEBUG.md) for:**
- Common causes of finalization stalls
Expand Down Expand Up @@ -184,15 +186,15 @@ Different clients have different log formats and key patterns.

## Block Proposal Flow (ethlambda)

A healthy block proposal follows this sequence:
Since the pre-build change (#445), the proposer builds at the *previous* slot's
interval 4 and publishes aligned to the slot boundary. A healthy block proposal
follows this sequence:

1. `We are the proposer for this slot` - Node detects it's the proposer
2. `TODO precompute poseidons in parallel + SIMD` - XMSS aggregate proof starts
3. `packed_pcs_commit` - Proof commitment
4. `Logup data` - Logup protocol data
5. `AIR proof{table=poseidon16}` / `AIR proof{table=poseidon24}` - AIR proofs
6. `Published block` - Block successfully built and published
7. `Published block to gossipsub` - Block broadcast to network
1. `We are the proposer for this slot` - Node detects it's the proposer (fires one slot early)
2. leanVM proving output (`packed_pcs_commit`, `Logup data`, `AIR proof{table=poseidon16|poseidon24}`) - XMSS aggregate proof
3. `Finished building block` - Build complete (#474; this is where the build-time metric stops)
4. `Published block` - Published at the slot boundary (the gap to step 3 is idle wait, not build cost)
5. `Published block to gossipsub` - Block broadcast to network

## Summary Report Format

Expand Down
26 changes: 7 additions & 19 deletions .claude/skills/test-pr-devnet/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ Test ethlambda branch changes in a multi-client local devnet with zeam (Zig), re
**Success criteria:**
- ✅ No errors in ethlambda logs
- ✅ All 4 nodes at same head slot
- ✅ Finalization advancing (every 6-12 slots)
- ✅ Finalization advancing (every few slots on a healthy devnet)
- ✅ Each validator produces blocks for their slots

### Sync Recovery (~90-120s)
Expand Down Expand Up @@ -139,25 +139,13 @@ sleep 10 # Wait for sync
# Quick status
.claude/skills/test-pr-devnet/scripts/check-status.sh

# Detailed analysis (use devnet-log-review skill in lean-quickstart)
cd $LEAN_QUICKSTART
# Detailed analysis (use this repo's devnet-log-review skill; dump logs first)
for node in zeam_0 ream_0 qlean_0 ethlambda_0; do
docker logs "$node" > "${node}.log" 2>&1
done
.claude/skills/devnet-log-review/scripts/analyze-logs.sh
```

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Local Analyzer Script Is Missing

The new detailed-analysis step invokes .claude/skills/devnet-log-review/scripts/analyze-logs.sh, but that script is not present in the repository. Following this workflow collects the container logs and then fails with No such file or directory instead of analyzing them.

Prompt To Fix With AI
This is a comment left during a code review.
Path: .claude/skills/test-pr-devnet/SKILL.md
Line: 147

Comment:
**Local Analyzer Script Is Missing**

The new detailed-analysis step invokes `.claude/skills/devnet-log-review/scripts/analyze-logs.sh`, but that script is not present in the repository. Following this workflow collects the container logs and then fails with `No such file or directory` instead of analyzing them.

How can I resolve this? If you propose a fix, please make it concise.


## Protocol Compatibility

| Client | Status | Gossipsub | BlocksByRoot |
|--------|--------|-----------|--------------|
| ream | ✅ Full | ✅ Full | ✅ Full |
| zeam | ✅ Full | ✅ Full | ⚠️ Limited |
| qlean | ✅ Full | ✅ Full | ⚠️ Limited |
| ethlambda | ✅ Full | ✅ Full | ✅ Full |

**Notes:**
- zeam/qlean BlocksByRoot errors are expected (not a blocker)
- ream ↔ ethlambda BlocksByRoot should work perfectly
- All clients use Gossipsub for block propagation

## Verification Checklist

| Check | Command | Expected |
Expand Down Expand Up @@ -256,5 +244,5 @@ docker logs ethlambda_0 2>&1 | grep -i "peer\|connection" | head -20

## References

- **[ethlambda CLAUDE.md](../../CLAUDE.md)** - Development workflow, detailed debugging commands
- **[lean-quickstart devnet-log-review](../../../lean-quickstart/.claude/skills/devnet-log-review/SKILL.md)** - Comprehensive log analysis
- **[ethlambda CLAUDE.md](../../../CLAUDE.md)** - Development workflow, detailed debugging commands
- **[devnet-log-review](../devnet-log-review/SKILL.md)** - Comprehensive log analysis (in this repo)
50 changes: 30 additions & 20 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Not to be confused with Ethereum consensus clients AKA Beacon Chain clients AKA
**Rust version:** 1.92.0 (edition 2024)
**Test fixtures release:** Download latest production fixtures from leanSpec releases

## Codebase Structure (10 crates)
## Codebase Structure (12 workspace crates)

```
bin/ethlambda/ # Entry point, CLI, orchestration
Expand All @@ -18,17 +18,24 @@ crates/
blockchain/ # State machine actor (GenServer pattern)
├─ src/lib.rs # BlockChain actor, tick events, validator duties
├─ src/store.rs # Fork choice store, block/attestation processing
├─ src/block_builder.rs # Block assembly (pre-built at previous slot's interval 4)
├─ src/aggregation.rs # Interval-2 signature aggregation worker
├─ src/reaggregate.rs # Re-aggregation of block-borne votes on import
├─ src/sync_status.rs # Sync-gate tracker (suppresses duties while syncing)
├─ src/key_manager.rs # Validator key management and signing
├─ src/metrics.rs # Blockchain-level Prometheus metrics
├─ fork_choice/ # LMD GHOST implementation (3SF-mini)
└─ state_transition/ # STF: process_slots, process_block, attestations
├─ fork_choice/ # [crate] LMD GHOST implementation (3SF-mini)
└─ state_transition/ # [crate] STF: process_slots, process_block, attestations
├─ src/justified_slots_ops.rs # Relative-index helpers for justified_slots
└─ src/metrics.rs # State transition timing + counters
common/
├─ types/ # Core types (State, Block, Attestation, Checkpoint)
├─ crypto/ # XMSS aggregation (leansig wrapper)
└─ metrics/ # Prometheus re-exports, TimingGuard, gather utilities
├─ metrics/ # Prometheus re-exports, TimingGuard, gather utilities
└─ test-fixtures/ # Spec-fixture loading (prod dep of rpc's Hive test driver)
net/
├─ p2p/ # libp2p: gossipsub + req-resp (Status, BlocksByRoot)
├─ api/ # Actor protocol traits wiring BlockChain ↔ P2P
├─ p2p/ # libp2p: gossipsub + req-resp (Status, BlocksByRoot, BlocksByRange)
│ ├─ src/gossipsub/ # Topic encoding, message handling
│ ├─ src/req_resp/ # Request/response codec and handlers
│ └─ src/metrics.rs # Peer connection/disconnection tracking
Expand Down Expand Up @@ -56,12 +63,14 @@ Interval 4: Accept accumulated attestations; build the NEXT slot's block and pub

### Attestation Pipeline
```
Gossip → Signature verification → new_attestations (pending)
↓ (interval 4)
promote → known_attestations (fork choice active)
Gossip → Signature verification → new_payloads (pending)
↓ (intervals 0/4)
promote → known_payloads (fork choice active)
Fork choice head update
```
(Store buffer fields are `new_payloads`/`known_payloads`; the accessors are named
`extract_latest_new_attestations`/`extract_latest_known_attestations`.)

### State Transition Phases
1. **process_slots()**: Advance through empty slots, update historical roots
Expand Down Expand Up @@ -105,7 +114,7 @@ let byte: u8 = code.into();

### Ownership for Large Structures
```rust
// Prefer taking ownership to avoid cloning large data (signatures ~3KB)
// Prefer taking ownership to avoid cloning large data (signatures ~2.5KB)
pub fn insert_signed_block(&mut self, root: H256, signed_block: SignedBlock) { ... }

// Add .clone() at call site if needed - makes cost explicit
Expand Down Expand Up @@ -252,7 +261,7 @@ actual_slot = finalized_slot + 1 + relative_index

**XMSS (eXtended Merkle Signature Scheme):**
- Post-quantum signature scheme
- 52-byte public keys, 3112-byte signatures
- 52-byte public keys, 2536-byte signatures (`SIGNATURE_SIZE` in `common/types/src/signature.rs`)
- Epoch-based to prevent reuse
- Aggregation via leanVM (previously leanMultisig) for efficiency

Expand All @@ -268,11 +277,11 @@ actual_slot = finalized_slot + 1 + relative_index
- Topic: `/leanconsensus/{fork_digest}/{block|aggregation|attestation_N}/ssz_snappy`
- `fork_digest` is a 4-byte hex string (no `0x` prefix); currently the dummy `12345678` agreed across clients
- Mesh size: 8 (6-12 bounds), heartbeat: 700ms
- **Req/Resp**: Status, BlocksByRoot (snappy frame compression + varint length)
- **Req/Resp**: Status, BlocksByRoot, BlocksByRange (snappy frame compression + varint length)

### Retry Strategy on Block Requests
- Exponential backoff: 10ms, 40ms, 160ms, 640ms, 2560ms
- Max 5 attempts, random peer selection on retry
- Exponential backoff: doubling from `INITIAL_BACKOFF_MS` (5ms → 2560ms)
- Max `MAX_FETCH_RETRIES` (10) attempts, random peer selection on retry
Comment on lines 282 to +284

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Backoff Endpoint Is Unreachable

The handler stops when pending.attempts >= MAX_FETCH_RETRIES, before scheduling the next delay. With attempts starting at 1, it schedules nine retries from 5 ms through 1280 ms; the documented 2560 ms delay is never reached, so developers calculating fetch timing from this section get the wrong retry window.

Suggested change
### Retry Strategy on Block Requests
- Exponential backoff: 10ms, 40ms, 160ms, 640ms, 2560ms
- Max 5 attempts, random peer selection on retry
- Exponential backoff: doubling from `INITIAL_BACKOFF_MS` (5ms → 2560ms)
- Max `MAX_FETCH_RETRIES` (10) attempts, random peer selection on retry
### Retry Strategy on Block Requests
- Exponential backoff: doubling from `INITIAL_BACKOFF_MS` (5ms → 1280ms)
- Max `MAX_FETCH_RETRIES` (10) total attempts: the initial request plus 9 retries, with random peer selection on retry
Prompt To Fix With AI
This is a comment left during a code review.
Path: CLAUDE.md
Line: 282-284

Comment:
**Backoff Endpoint Is Unreachable**

The handler stops when `pending.attempts >= MAX_FETCH_RETRIES`, before scheduling the next delay. With attempts starting at 1, it schedules nine retries from 5 ms through 1280 ms; the documented 2560 ms delay is never reached, so developers calculating fetch timing from this section get the wrong retry window.

```suggestion
### Retry Strategy on Block Requests
- Exponential backoff: doubling from `INITIAL_BACKOFF_MS` (5ms → 1280ms)
- Max `MAX_FETCH_RETRIES` (10) total attempts: the initial request plus 9 retries, with random peer selection on retry
```

How can I resolve this? If you propose a fix, please make it concise.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!


### Message IDs
- 20-byte truncated SHA256 of: domain (valid/invalid snappy) + topic + data
Expand Down Expand Up @@ -301,9 +310,9 @@ GENESIS_VALIDATORS:
### Test Categories
1. **Unit tests**: Embedded in source files
2. **Spec tests**: From `leanSpec/fixtures/consensus/`
- `forkchoice_spectests.rs` (uses `on_block_without_verification`)
- `signature_spectests.rs`
- `stf_spectests.rs` (state transition)
- `crates/blockchain/tests/forkchoice_spectests.rs` (uses `on_block_without_verification` via `spec_test_runner`)
- `crates/blockchain/tests/signature_spectests.rs`
- `crates/blockchain/state_transition/tests/stf_spectests.rs` (state transition)

### Running Tests
```bash
Expand All @@ -316,7 +325,7 @@ cargo test -p ethlambda-blockchain --test forkchoice_spectests -- --test-threads

### Aggregator Flag Required for Finalization
- At least one node **must** be started with `--is-aggregator` to finalize blocks
- Without this flag, attestations pass signature verification and are logged as "Attestation processed", but the signature is never stored for aggregation (`store.rs:368`), so blocks are always built with `attestation_count=0`
- Without this flag, attestations pass signature verification and are logged as "Attestation processed", but the signature is never stored for aggregation (the `is_aggregator` gate in `on_gossip_attestation`, `store.rs`), so blocks are always built with `attestation_count=0`
- The attestation pipeline: gossip → verify signature → store gossip signature (only if `is_aggregator`) → aggregate at interval 2 → promote to known → pack into blocks
- **Symptom**: `justified_slot=0` and `finalized_slot=0` indefinitely despite healthy block production and attestation gossip

Expand Down Expand Up @@ -368,7 +377,7 @@ and are consumed during the tick pipeline (promotion at intervals 0/4,
aggregation at interval 2).

### State Root Computation
- Always computed via `tree_hash_root()` after full state transition
- Always computed via `hash_tree_root()` after full state transition
- Must match proposer's pre-computed `block.state_root`

### Finalization Checks
Expand All @@ -383,8 +392,8 @@ aggregation at interval 2).

**Critical:**
- `leansig`: XMSS signatures (leanEthereum project)
- `ethereum_ssz`: SSZ serialization
- `tree_hash`: Merkle tree hashing
- `libssz` / `libssz-derive` / `libssz-types`: SSZ serialization
- `libssz-merkle`: Merkle tree hashing (`hash_tree_root()`)
- `spawned-concurrency`: Actor model
- `libp2p`: P2P networking (custom LambdaClass fork)
- `vergen-git2`: Build-time git commit/branch info embedded in binary
Expand All @@ -397,6 +406,7 @@ aggregation at interval 2).

**Specs:** `leanSpec/src/lean_spec/` (Python reference implementation)
**Devnet:** `lean-quickstart` (github.com/blockblaz/lean-quickstart)
**Docs:** `docs/` — `rpc.md`, `metrics.md`, `checkpoint_sync.md`, `3sf_mini.md`, `lmd_ghost.md` (mdbook via `make docs`)
**Releases:** See `RELEASE.md` for release process documentation

## Other implementations
Expand Down
11 changes: 7 additions & 4 deletions crates/storage/src/api/tables.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,18 @@ pub enum Table {
BlockHeaders,
/// Block body storage: H256 -> BlockBody
BlockBodies,
/// Block signatures storage: H256 -> BlockSignatures
/// Block signatures storage: (slot || root) -> BlockSignatures
///
/// Stored separately from blocks because the genesis block has no signatures.
/// All other blocks must have an entry in this table.
/// Keyed by slot || root so pruning can scan in slot order and stop early.
/// Non-genesis blocks have an entry until finalized: signatures below the
/// finalized boundary are pruned (`prune_old_block_signatures`), while
/// headers and bodies are kept forever.
Comment on lines +11 to +14

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Pruning Boundary Is Misstated

Signatures are not pruned merely because they are below the finalized slot. The implementation first computes tip_slot - SIGNATURE_PRUNING_RANGE and only removes entries below that cutoff when the cutoff is finalized, so this comment can make maintainers expect recently finalized signatures to be absent when they are deliberately retained.

Suggested change
/// Keyed by slot || root so pruning can scan in slot order and stop early.
/// Non-genesis blocks have an entry until finalized: signatures below the
/// finalized boundary are pruned (`prune_old_block_signatures`), while
/// headers and bodies are kept forever.
/// Keyed by slot || root so pruning can scan in slot order and stop early.
/// Non-genesis blocks have an entry until they fall below the retention cutoff
/// (`tip_slot - SIGNATURE_PRUNING_RANGE`) and that cutoff is finalized. Headers
/// and bodies are kept forever.
Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/storage/src/api/tables.rs
Line: 11-14

Comment:
**Pruning Boundary Is Misstated**

Signatures are not pruned merely because they are below the finalized slot. The implementation first computes `tip_slot - SIGNATURE_PRUNING_RANGE` and only removes entries below that cutoff when the cutoff is finalized, so this comment can make maintainers expect recently finalized signatures to be absent when they are deliberately retained.

```suggestion
    /// Keyed by slot || root so pruning can scan in slot order and stop early.
    /// Non-genesis blocks have an entry until they fall below the retention cutoff
    /// (`tip_slot - SIGNATURE_PRUNING_RANGE`) and that cutoff is finalized. Headers
    /// and bodies are kept forever.
```

How can I resolve this? If you propose a fix, please make it concise.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

BlockSignatures,
/// State storage: H256 -> State
///
/// Holds full-state snapshots only: the bootstrap anchor plus one anchor per
/// 1024-slot window. Never pruned. Non-anchor states live in `StateDiffs` and
/// Holds full-state snapshots only: the bootstrap anchor plus one anchor
/// every `SNAPSHOT_ANCHOR_INTERVAL` slots. Never pruned. Non-anchor states live in `StateDiffs` and
/// are reconstructed on demand (memoized by an in-memory cache).
States,
/// State diffs: H256 -> StateDiff
Expand Down