Implement graceful shutdown procedure

[whitslack]

BOLT 2 says:

A node, […] upon reconnection, if a channel is [not] in an error state, […] MUST wait to receive the other node's channel_reestablish message before sending any other messages for that channel.

We can abuse this requirement to implement a graceful shutdown procedure:

1. Set a flag that precludes lightningd from sending channel_reestablish messages for any channels that have exactly zero outstanding HTLCs.
2. Disconnect all peers that have exactly zero outstanding HTLCs in all of their channels with this node.
3. If no channels are "reestablished" and no channels have any outstanding HTLCs, then it is now impossible for any peers to add any new HTLCs to our channels, so we can safely shut down.
4. Otherwise, wait for some outstanding HTLC to settle, and then return to step 2.
5. If graceful shutdown is taking too long, then report to the user the approximate time until an outstanding HTLC next expires, and abort.

This PR has two objectives:

• Add a snub-idle-channels dynamic config variable that, when set to true, makes lightningd:
  • no longer spawn channeld subdaemons for channels that have no outstanding HTLCs;
  • as an optimization, no longer attempt to auto-reconnect to peers with whom we have no outstanding HTLCs;
  • ignore all received channel_reestablish messages for channels that have no outstanding HTLCs,
    • …but send a warning to the peer informing them that their channel reestablishment is being ignored because this peer imminently will be shutting down.
• Add a contrib/lightning-graceful-stop.sh script that utilizes snub-idle-channels to implement the graceful shutdown procedure outlined above.

I have tested this graceful shutdown procedure on my own production node with great success. In under a minute my node dropped from over 30 outstanding HTLCs to 14, all of which were "stuck." The shutdown script reported that the next expiration was 140 blocks away, giving me plenty of time to power off my node and perform a hardware upgrade. If I had been willing to wait for all of my outstanding HTLCs to be resolved, then I could have stopped my node indefinitely with no danger of any forced unilateral closures. (Of course, my peers could still voluntarily choose to unilaterally close my channels with them if they grew tired of waiting for my node to reappear in the network, but that's not the concern that graceful shutdown is attempting to address.)

Note that there is still one edge case that this graceful shutdown strategy doesn't solve. If a peer has transmitted a new commitment containing a new HTLC, but we never transmitted our own new commitment containing that same new HTLC (either because we never received the peer's new commitment or because we restarted before we could send our own new commitment), then we will not know about (or will have forgotten) the new HTLC, and we will believe that the channel is safe to snub even though the peer would retransmit their new commitment containing the new HTLC if we allowed them to reestablish the channel. I am not certain, but it may be possible to use the fields in the channel_reestablish message received from the peer to ascertain whether the peer has new HTLCs that they need to retransmit to us, and if they do, then we shouldn't snub the channel even if we are currently aware of no outstanding HTLCs in it.

Checklist

Before submitting the PR, ensure the following tasks are completed. If an item is not applicable to your PR, please mark it as checked:

• The changelog has been updated in the relevant commit(s) according to the guidelines.
• Tests have been added or modified to reflect the changes.
• Documentation has been reviewed and updated as needed.
• Related issues have been listed and linked, including any that this PR closes.
• Important All PRs must consider how to reverse any persistent changes for tools/lightning-downgrade

[whitslack]

I just used this again to gracefully shut down my node, and this time my established channels count (but not my outstanding HTLCs count) reached zero, but then the node apparently restarted itself! I tried again, and the same thing happened again: the script works down the number of connected peers to zero, and then the daemon spontaneously restarts itself. I don't have any kind of service supervisor managing it, so it has to be something lightningd is doing by itself. Any ideas?

[madelinevibes]

@whitslack are you able to try and resolve the CI errors so we can review and get this into 26.06? Planning for a release candidate on 11 May

[whitslack]

are you able to try and resolve the CI errors so we can review and get this into 26.06?

@madelinevibes: Honestly, probably not. No PR that I have ever opened on this project has passed CI checks, so I can't imagine they were ever working to begin with. It has been a continual source of frustration for me, as obviously I would like to submit PRs that pass all the checks, but the failures don't ever seem related to anything I've touched. In the present case I am no longer able to view the CI logs for this PR, so I wouldn't know where to begin.

[madelinevibes]

Very fair.
@daywalker90 can you have a look at CI please? See if we can get this into 26.06

[rustyrussell]

Heh, we will restart if we spot the versions of subdaemons have changed underneath us. This usually is because you are running in place and do a make, OR, make install while lightningd is running.

[rustyrussell]

OK, no. This is very clever, but instead I have based a different approach on this: implementing a graceful command which does a similar thing to your script (only it's up to you to call stop when it returns).

Not because this is a bad idea, but because using a transient setconfig is dangerous: if you don't make it transient you'll wonder why your node never works well!