[bensheldon]

When Chrome is reached through a proxy or router with an idle timeout, any Ferrum wait longer than that timeout drops the connection. Heroku's router, for example, closes websocket connections that are idle for 55 seconds unless "either client or server [sends] an occasional ping packet". Many load balancers default to 60 seconds.

Two waits send nothing over the websocket for their whole duration:

• Client#send_message waiting for a slow command response, e.g. Page.navigate to a page that takes a long time to respond.
• Page#command waiting for a navigation started by the command, e.g. a click that submits a slow form.

With timeout: 120, either wait can outlast the proxy's limit. The only workaround today is to keep :timeout below that limit and retry in application code. Retrying a click can submit a form twice.

This adds an opt-in :keepalive_interval option (default nil, no behavior change):

Ferrum::Browser.new(url: "ws://chrome:3333", timeout: 120, keepalive_interval: 25)

When set, both waits are split into slices of at most keepalive_interval seconds, and a Browser.getVersion command is sent between slices. The total wait is still bounded by :timeout, and TimeoutError is raised at the same point as before.

Browser.getVersion is used because it's read-only and works at the browser level, so it still works while the page is between documents. As a real CDP round-trip, it creates traffic on every hop between Ruby and Chrome, including intermediate proxies that might answer websocket ping frames themselves.

[route]

Thanks for the detailed write-up, Ben. The problem is real, but I don't think it belongs in Ferrum.

Ferrum talks CDP to whatever sits at url:. A proxy that drops connections idle for 55 seconds is a property of that infrastructure, not of the protocol or the library. Most Ferrum users connect to a local Chrome or one on the same network (compose, Kubernetes, ghcr.io/rubycdp/chrome, browserless) and never hit this. Adding an option, plus extra CDP traffic built into our wait logic, to work around one kind of deployment means we'd carry it forever for a small group of users. That group can already solve it on their side.

Some concerns with the approach itself, apart from scope:

• It covers only two waits. The connection is still idle in plenty of other places: time between commands, a slow Ruby step in a test, wait_for_idle, network waits, user code that sleeps. If we did solve this in Ferrum, it would be one idle-based keepalive at the connection level, not slicing chosen waits.
• It makes Client#wait_with_keepalive public API just so Page can call it.
• It sends fake CDP commands. Browser.getVersion fills the logs and goes out on every wait. A WebSocket ping frame is the standard way to keep a connection alive, and Heroku's docs say a ping is enough.
• It overlaps with timeout handling we're changing right now (infinite timeouts in send_message), which makes the slicing loop harder to reason about.

You can do this today without changes to Ferrum, e.g. a background thread in your setup:

Thread.new do
  loop do
    sleep 25
    browser.client.command("Browser.getVersion")
  end
end

Or fix it where the problem is: raise the proxy's idle timeout, run Chrome on the same network as the tests, or warm up whatever makes the wait so long.

Some questions so we understand your setup:

1. What sits between Ruby and Chrome? Heroku's router, a load balancer, a dev-environment port forwarder?
2. Which Chrome service is it: browserless, ghcr.io/rubycdp/chrome, plain Chrome behind a proxy?
3. Is this a test suite (the Cuprite PR suggests so) or production use?
4. Which wait actually goes past 55s, and why? Is it on-demand asset compilation on the first visit, a slow endpoint, something else?
5. Does a WebSocket ping frame from the client keep your proxy happy, or does it really need application-level traffic?

I'll leave this and rubycdp/cuprite#349 open for now. If your answers show this is common across setups and not specific to one deployment, I'm happy to revisit it as a connection-level keepalive.

[bensheldon]

Thanks for the review. And sorry, I should have checked that browser.client.command was threadsafe because I assumed it wasn't. Orchestrating a background thread works for me.

Or fix it where the problem is: raise the proxy's idle timeout, run Chrome on the same network as the tests, or warm up whatever makes the wait so long.

For context, none of those are possible.

• When running Browserless on Heroku, I'm behind Heroku's loadbalancer, whose connectio timeouts are outside of my control.
• This is for production operation of browser driving external services
• I don't have control over external services being ungodly slow

I appreciate the review and feedback. Regardless, you have resolved my problem ❤️