tcp: strict RFC 5961 RST sequence validation (contributes to #1132)

[ibondarenko1]

Contribution to issue #1132 (open since 2019-08, "Validate segment handling in Netstack as per RFC 793 page 69"). That issue documents the gap explicitly: "We also need to verify which parts have been updated by RFC5961 to make TCP robust against attacks." This PR adds RFC 5961 section 3.2 strict-match RST acceptance.

Problem

pkg/tcpip/transport/tcp/connect.go handleReset accepts any RST whose sequence number is within the advertised receive window (pkg/tcpip/transport/tcp/rcv.go:70-81 acceptable). The source comment cites RFC 793 page 37.

With a 4 MB advertised window the per-packet probability of a blind off-path RST hitting an in-window sequence is roughly window / 2^32 (~0.1 percent). An off-path attacker that knows the 4-tuple can terminate an established TCP connection in a small number of packets.

CVE-2024-10603 (commit cbdb2c61b1 "Randomize TCP source port selection", 2023-11-10) already added source-port randomization, which raised the practical cost by a factor of ~2^16. With both source-port randomization AND the existing window-based RST check, the joint per-packet blind probability is ~window / 2^48. The remaining gap is the RST acceptance rule itself.

RFC 5961 section 3.2

RFC 5961 section 3.2 tightens RFC 793 page 37:

1. If the RST bit is set and the sequence number exactly matches the next expected sequence number (RCV.NXT), then TCP MUST reset the connection.

2. If the RST bit is set and the sequence number does not exactly match the next expected sequence value, yet is within the current receive window, TCP MUST send an acknowledgment (challenge ACK)

Linux net/ipv4/tcp_input.c tcp_validate_incoming has implemented this since version 3.6 (2012). gVisor partially implements RFC 5961: section 4.1 (SYN-in-window challenge ACK) is present at connect.go:1290-1312, with an inline RFC 5961 link in the comment. Section 3.2 (RST strict-match) was the remaining gap that #1132 tracks.

Change

handleReset is rewritten to RFC 5961 section 3.2:

• Out of window: silent drop. (Unchanged.)
• In window but not exactly RCV.NXT: send a challenge ACK via the existing rate-limited helper e.snd.maybeSendOutOfWindowAck(s), then drop. (NEW.)
• Exactly RCV.NXT: accept and reset the connection. (Unchanged for legitimate peers.)

The challenge ACK path reuses the same helper used by the existing RFC 5961 section 4.1 implementation in this same file. The rate-limit inside maybeSendOutOfWindowAck also defends against the CVE-2016-5696 class of challenge-ACK side channels.

Tests

New file pkg/tcpip/transport/tcp/test/e2e/handle_reset_rfc5961_test.go adds three focused regression tests:

• TestRFC5961_RSTOutOfWindowIsDropped (silent drop, unchanged path)
• TestRFC5961_RSTInWindowNotExactSendsChallengeAck (NEW behavior: no abort, challenge ACK observed via the test sniffer)
• TestRFC5961_RSTExactMatchAbortsConnection (legitimate RST still aborts)

Local run:

$ bazel test //pkg/tcpip/transport/tcp/test/e2e:handle_reset_rfc5961_test \
    --test_output=all --test_arg=-test.v --cache_test_results=no

--- PASS: TestRFC5961_RSTOutOfWindowIsDropped (0.05s)
--- PASS: TestRFC5961_RSTInWindowNotExactSendsChallengeAck (0.00s)
--- PASS: TestRFC5961_RSTExactMatchAbortsConnection (0.00s)
PASS

//pkg/tcpip/transport/tcp/test/e2e:handle_reset_rfc5961_test  PASSED in 0.1s
Executed 1 out of 1 test: 1 test passes.

Challenge-ACK behavior verified via the test sniffer:

recv tcp 10.0.0.2:4096 -> 10.0.0.1:65106 flags:  R    seqnum:1814 ack:0 win:30000
send tcp 10.0.0.1:65106 -> 10.0.0.2:4096 flags:    A  seqnum:854776498 ack:790 win:65535

RST at SEQ=1814 (in window, not equal to RCV.NXT=790) is answered with a challenge ACK at SND.NXT=854776498 / ACK=790, instead of a connection reset.

//pkg/tcpip/transport/tcp/test/e2e:rcv_test also passes on this branch (independent verification that the receiver path is unchanged for non-RST segments).

bazel build //pkg/tcpip/transport/tcp:tcp builds clean against master.

Framing

Hardening contribution. Class is denial of service (off-path blind RST termination of an established connection). Google VRP does not bounty DoS, and gVisor's published threat model does not claim RFC 5961 compliance. This PR is a Linux-parity uplift contributing to the existing open issue, not a security advisory.

References

• Issue Validate segment handling in Netstack as per https://tools.ietf.org/html/rfc793 page 69  #1132 (open) "Validate segment handling in Netstack as per RFC 793 page 69"
• RFC 5961 section 3.2 (RST acceptance) and section 4.1 (SYN-in-window challenge ACK)
• Linux net/ipv4/tcp_input.c tcp_validate_incoming (RFC 5961 section 3 since Linux 3.6, 2012)
• CVE-2016-5696 (Off-Path TCP Exploits, Cao et al., USENIX Security 2016)
• CVE-2024-10603 (gVisor TCP source-port randomization, commit cbdb2c61b1, 2023-11)