Fix SSH tunnel not being closed after remote server reboot (#406)

[debba]

Closes #406

What was happening

When a MySQL connection went through an SSH tunnel and the remote server rebooted, Tabularis correctly detected the drop and cleared the connection indicator. But trying to reconnect failed with a "port already in use" error, and the only way out was to restart the app.

The reason is that SSH tunnels are cached in a global map (keyed by user@host:port:remote->port) and reused across connects, but nothing ever tore them down. A tunnel could outlive the connection it belonged to. Once the remote host rebooted, the tunnel died while its local forward port was still held by the (now defunct) ssh process — or, in the russh case, by a forwarding loop that never noticed the session was gone — so the next reconnect happily picked up the stale map entry, pointed at that dead port, and blew up. Restarting the app "fixed" it only because that wiped the map.

What I changed

• ssh_tunnel.rs
  • The russh forwarding loop now checks handle.is_closed() on every poll iteration: when the SSH session dies it shuts itself down and releases the local port. Without this a russh tunnel stayed "alive" forever and the eviction below would never trigger for it.
  • The system-ssh command now passes ServerAliveInterval=15, ServerAliveCountMax=3 and ExitOnForwardFailure=yes, so the ssh process exits when the server goes away instead of holding the local port indefinitely.
  • stop() now also reaps the killed system-ssh child (wait()), so it doesn't linger as a zombie still holding the forwarded port.
  • Added is_alive() to tell whether a cached tunnel is still usable (system-ssh child exited, or russh loop shut down after session death / stop()).
  • Added remove_tunnel(key) to stop a tunnel and drop it from the map (no-op if it isn't there).
• commands.rs
  • Before reusing a cached tunnel, resolve_connection_params evicts a dead one via the new evict_dead_ssh_tunnel() and creates a fresh tunnel instead of reusing the broken port.
  • Tunnel-key construction from unresolved params is shared in ssh_tunnel_key_for(), used by both teardown_ssh_tunnel() and evict_dead_ssh_tunnel().
  • disconnect_connection tears the tunnel down so it no longer survives the connection.
• health_check.rs
  • Pings fail fast when the tunnel is dead (evict + immediate error) instead of letting param resolution block on rebuilding a tunnel toward a server that may still be down.
  • When a connection crosses the failure threshold (exactly the reboot case), the tunnel is torn down and the pool is closed from the expanded params, skipping tunnel resolution entirely — with a connection_id the pool key only depends on driver/id/database, and resolving would have recreated a tunnel toward the dead server.
  • The saved-connection lookup + SSH/K8s param expansion chain is deduplicated into expand_params().

Tests

Added unit tests in ssh_tunnel.rs for is_alive() on both backends (russh flag + system-ssh exited child), for stop() reaping the child, and for remove_tunnel (both the missing-key no-op and real removal), plus tests in commands.rs for the evict_dead_ssh_tunnel() no-op paths. Full backend suite passes (the only failures are the pre-existing, environment-specific askpass socket-path ones).

Notes

Tunnels are shared by key, so two connections to the same DB over the same SSH host share one tunnel — tearing it down on disconnect affects both, which is the intended behaviour here since a reboot takes them all down anyway.