mempool update stuck retrieving missing tx

[philippem]

We ran into a situation where electrs was stuck in its mempool update loop, trying to retrieve a tx (or txes) from bitcoin core.
This occurred on signet chain. The bitcoin core node serving electrs is not provisioned differently than mainnet or other nodes.

The only logs from electrs were from https://github.com/Blockstream/electrs/blob/new-index/src/new_index/mempool.rs#L590, example:

2025-09-22 18:29:50.739warn2025-09-22T18:29:50.738+00:00 - WARN failed to fetch 2 mempool txs, retrying...
2025-09-22 18:29:50.419warn2025-09-22T18:29:50.418+00:00 - WARN failed to fetch 2 mempool txs, retrying...
2025-09-22 18:29:50.099warn2025-09-22T18:29:50.098+00:00 - WARN failed to fetch 2 mempool txs, retrying...
2025-09-22 18:29:49.778warn2025-09-22T18:29:49.778+00:00 - WARN failed to fetch 2 mempool txs, retrying...
2025-09-22 18:29:49.458warn2025-09-22T18:29:49.458+00:00 - WARN failed to fetch 2 mempool txs, retrying...
2025-09-22 18:29:49.138warn2025-09-22T18:29:49.138+00:00 - WARN failed to fetch 2 mempool txs, retrying...
2025-09-22 18:29:48.818warn2025-09-22T18:29:48.817+00:00 - WARN failed to fetch 2 mempool txs, retrying...
2025-09-22 18:29:48.498warn2025-09-22T18:29:48.497+00:00 - WARN failed to fetch 2 mempool txs, retrying...
2025-09-22 18:29:48.178warn2025-09-22T18:29:48.178+00:00 - WARN failed to fetch 3 mempool txs, retrying...
2025-09-22 18:29:47.858warn2025-09-22T18:29:47.857+00:00 - WARN failed to fetch 2 mempool txs, retrying...
2025-09-22 18:29:47.538warn2025-09-22T18:29:47.538+00:00 - WARN failed to fetch 2 mempool txs, retrying...
2025-09-22 18:29:47.218warn2025-09-22T18:29:47.218+00:00 - WARN failed to fetch 2 mempool txs, retrying...
2025-09-22 18:29:46.897warn2025-09-22T18:29:46.897+00:00 - WARN failed to fetch 1 mempool txs, retrying...
2025-09-22 18:29:46.577warn2025-09-22T18:29:46.577+00:00 - WARN failed to fetch 1 mempool txs, retrying...
2025-09-22 18:29:46.258warn2025-09-22T18:29:46.258+00:00 - WARN failed to fetch 1 mempool txs, retrying...
2025-09-22 18:29:45.939warn2025-09-22T18:29:45.939+00:00 - WARN failed to fetch 1 mempool txs, retrying...
2025-09-22 18:29:45.620warn2025-09-22T18:29:45.619+00:00 - WARN failed to fetch 1 mempool txs, retrying...
2025-09-22 18:29:45.300warn2025-09-22T18:29:45.300+00:00 - WARN failed to fetch 1 mempool txs, retrying...
2025-09-22 18:29:44.981warn2025-09-22T18:29:44.981+00:00 - WARN failed to fetch 1 mempool txs, retrying...

Graph showing stats from the metrics. Note that "tx" line remained constant:

[philippem]

hit this one today while testing locally

2025-09-29T13:17:38.902-07:00 - DEBUG RPC HTTP 500 error: {"result":null,"error":{"code":-5,"message":"No such mempool transaction. Use -txindex or provide a block hash to enable blockchain transaction queries. Use gettransaction for wallet transactions."},"id":3938}
2025-09-29T17:15:15.688-07:00 - WARN failed to fetch 1 mempool txs, retrying...
2025-09-29T17:15:19.527-07:00 - WARN missing mempool txids: [73f8e6aedad871e23601fa563bd40b66bf4c929eb88f505742d8a4ffad71e5fd] (capped at 10)
2025-09-29T17:15:19.527-07:00 - WARN reconnecting to bitcoind: Error: Connection error: disconnected from daemon while receiving