socket:send/2,3: distinguish eagain from closed, and retry transparently via write-select

[harmon25]

This pull request improves the accuracy of error reporting in the socket send implementation, and adds a proper write-select mechanism so that transient send backpressure is retried transparently instead of being leaked to callers.

Socket send error handling improvements:

• src/libAtomVM/otp_socket.c: do_socket_send now distinguishes transient backpressure (such as lwIP ERR_MEM or BSD EAGAIN/EWOULDBLOCK) from a closed connection at the NIF level.
• src/libAtomVM/otp_socket.c: When the peer has closed the connection, returns {error, closed} instead of {ok, Data} to clarify the connection state.

Write-select mechanism (enif_select_write):

• enif_select_write was declared in erl_nif.h but never implemented or called anywhere. It's now implemented, and enif_select/enif_select_read/enif_select_write were reworked so a resource can hold independent pending read and write selects (each with its own ref/message/pid) on the same event at the same time, instead of one direction silently overwriting the other's state. This is needed since a socket can have a permanent recv select active (e.g. socket_dist_controller) at the same time as a transient send-backpressure select. Along the way, fixed a latent ref-ticks computation bug (term_is_local_reference(ref) vs ref != UNDEFINED_ATOM). Added C-level unit test coverage in tests/test-enif.c (there was none for enif_select at all before).
• otp_socket.c now implements nif_select_write/2 for both BSD (enif_select_write) and lwIP (new tcp_sent/tcp_poll callbacks + a SocketStateSelectingWrite state, using tcp_sndbuf/tcp_sndqueuelen to detect when the send buffer has room again).
• socket:send/2,3 now retries partial sends and {error, eagain} internally by waiting on the new nif_select_write, so callers get ok | {error, Reason} — no more leaking {ok, Rest}/{error, eagain} and no need for every caller to implement its own retry/backoff loop.
• ssl.erl's handshake/close-notify/send/recv loops now wait for real write-readiness instead of busy-looping on want_write.
• socket_dist_controller.erl now terminates the distribution connection on socket:send/2 errors (tick + outgoing data loop) instead of silently ignoring them.
• Added a backpressure test to test_tcp_socket.erl: fills the send buffer via raw nif_send/2 until genuine EAGAIN, then verifies nif_select_write/2 correctly wakes up once the peer drains, and that socket:send/2 completes transparently afterward.

Documentation update:

• CHANGELOG.md: Updated to describe both the eagain/closed distinction and the new internal retry behavior.

Testing

Validated on real ESP32-S3 hardware (not host/qemu) using atomvm_httpd, a consumer library whose tcp_server:do_send/4 retries sends specifically keyed on the eagain
vs closed reason (src/tcp_server.erl#L407) — exactly the distinction this PR introduces at the NIF level.

Setup: flashed this branch's VM firmware, then ran the library's httpd_debug example with its send-retry logging temporarily enabled (?TRACE on do_send/retry_send), so every retry logs the literal error reason returned by socket:send/2.

Reproducing genuine backpressure: a naive high-concurrency/full-speed curl burst against a large generated response raced the client's own timeout (the connection was genuinely closed before the server got scheduled to send) rather than exercising real send backpressure. Reliable backpressure instead came from a slow-reading client (curl --limit-rate 1k) against a 64KB response, which fills the small lwIP TCP send buffer because the peer isn't draining it, without racing timeouts:

curl -m 180 --limit-rate 1k http://<device-ip>/api/generate?size=65536   # x3 concurrent

Result: 3 concurrent slow connections produced 12 retry events, all logged as Send error eagain; retry 0 — zero Send error closed — and every connection completed successfully (200, full 65646-byte body) after a single retry each time. This confirms:

• Transient lwIP ERR_MEM backpressure now surfaces distinctly as {error, eagain} (previously indistinguishable from {error, closed}).
• The condition really is transient/recoverable — the very next send attempt succeeded every time.

The write-select mechanism added on top of this was additionally verified with test-enif (C-level enif_select unit tests), the full test_estdlib suite (including the new backpressure test) on generic_unix, and a full ESP32 (esp32s3) build via ESP-IDF 5.5.4.

These changes are made under both the "Apache 2.0" and the "GNU Lesser General
Public License 2.1 or later" license terms (dual license).

SPDX-License-Identifier: Apache-2.0 OR LGPL-2.1-or-later

[petermm]

thank you for pushing the httpd side!

this is obviously correct - however it does increase the leakage of underlying implementation, and now any caller will also need to implement their own unique retry mechanism (gen_tcp_socket.erl, socket_dist_controller.erl, ssl.erl - and atomvm_lib httpd) - and is it 5ms with exponential backoff or xyz - and we end up with complexity, duplicated and slow (sleeping) code..

I believe the solution is to implement nif_select_write/2, with proper write-select and no sleeping/guessing, eg. handle the happy path out of the box, similar to otp:

see https://github.com/petermm/AtomVM/tree/nif_select_write/2 (based on this PR)

[harmon25]

Thanks @petermm! I agree, that sounds like a better approach.

Should we continue from your branch, or should i incorporate those changes here?

[harmon25]

@petermm Thanks for the review and for prototyping this on nif_select_write/2 — agreed this is the right fix, and I've implemented it on top of this branch (6 new commits, keeping the existing eagain/closed distinction as the low-level primitive underneath):

1. enif_select_write — was declared in erl_nif.h but never implemented or called anywhere. Implemented it, and reworked enif_select/enif_select_read/enif_select_write so a resource can hold independent pending read and write selects (each with its own ref/message/pid) on the same event at the same time, instead of one direction silently overwriting the other's state (needed since a socket can have a permanent recv select active at the same time as a transient send-backpressure select, e.g. socket_dist_controller). Also fixed a latent ref-ticks bug found along the way (using term_is_local_reference(ref) instead of ref != UNDEFINED_ATOM). Added C-level unit test coverage for this in tests/test-enif.c (there was none before for enif_select at all).
2. otp_socket.c — wired up nif_select_write/2 for both BSD (enif_select_write) and lwIP (new tcp_sent/tcp_poll callbacks + a SocketStateSelectingWrite state, using tcp_sndbuf/tcp_sndqueuelen to detect room again).
3. socket:send/2,3 — now retries partial sends and {error, eagain} internally by waiting on nif_select_write, so callers just see ok | {error, Reason} — no more leaking {ok, Rest}/{error, eagain}.
4. ssl.erl — replaced the 4 want_write busy-loops with a real wait-for-writable.
5. socket_dist_controller.erl — now actually terminates the connection on socket:send/2 errors instead of silently ignoring them (tick + outgoing data loop).
6. Added a real backpressure test to test_tcp_socket.erl (fills the send buffer via raw nif_send/2 until genuine EAGAIN, then verifies nif_select_write/2 correctly wakes up once the peer drains, and that socket:send/2 completes transparently afterward).

Verified: test-enif, the full test_estdlib suite (including the new backpressure test) on generic_unix, and a full ESP32 (esp32s3) build via ESP-IDF 5.5.4.

One thing worth calling out from writing the test: I initially had it drain only a small chunk (64KB) of the peer's backlog and expected the socket to become writable again, but that hung — traced it down (gdb backtraces + an independent raw C poll(2) repro, unrelated to any AtomVM code) to the sandbox's kernel/buffer accounting not reliably crossing the write low-water mark on a small partial drain, only on a full drain. Not an AtomVM bug, just a test design fix, but flagging in case it's relevant to anyone else validating this on constrained targets (rp2/esp32) with small buffers — might be worth a similar "drain fully, not partially" pattern if there's ever a device-side test for this.