Repository navigation
Socket closed without shutdown() causes TIME_WAIT accumulation and data loss risk #488
Description
Socket closed without shutdown() causes TIME_WAIT accumulation and data loss risk
Summary
The clickhouse-cpp client calls close() / closesocket() directly on the TCP socket without first calling shutdown(). This is an improper socket teardown that causes two problems:
- TIME_WAIT socket accumulation on the client side when the client initiates the close, which can exhaust ephemeral ports under high connection churn.
- Potential data loss —
close()on a socket with unread data in the receive buffer causes the OS to send a RST instead of a graceful FIN, which can discard in-flight data and confuse the peer.
The issue affects both plain TCP and SSL connections (no SSL_shutdown() either).
Root Cause
Socket::Close() in socket.cpp:
void Socket::Close() {
CloseSocket(handle_);
handle_ = INVALID_SOCKET;
}Where CloseSocket is:
void CloseSocket(SOCKET socket) {
if (socket == INVALID_SOCKET)
return;
#if defined(_win_)
closesocket(socket);
#else
close(socket);
#endif
}There is no shutdown(fd, SHUT_RDWR) or shutdown(fd, SHUT_WR) call before close() anywhere in the library. A search of the entire clickhouse/ source tree confirms zero uses of shutdown().
For SSL sockets, SSLSocket inherits from Socket and its destructor chain also never calls SSL_shutdown() before closing the underlying file descriptor.
Affected Code Paths
Every socket close goes through Socket::Close() → CloseSocket():
Client::~Client()→~Impl()→~socket_→Socket::~Socket()→Close()Client::Impl::ResetConnection()→InitializeStreams()→ old socket swapped out →~Socket()SocketRAIIWrapper::~SocketRAIIWrapper()inSocketConnect()(on connection failure)
Consequences
1. TIME_WAIT accumulation
When the client closes first without shutdown(), the TCP stack enters TIME_WAIT on the client side. Under high connection churn (many short-lived client instances), this can exhaust the available ephemeral port range (~28,000 ports on Linux default), causing connect() failures with EADDRNOTAVAIL.
2. RST instead of graceful FIN
Per POSIX/RFC 793, calling close() on a socket that has unread data in the kernel receive buffer causes the OS to send a TCP RST to the peer instead of a graceful FIN. This can happen when:
- The server sent data the client never read (e.g., client destroys
Clientobject mid-query) - The server sent an exception or log packet the client didn't consume
The RST causes the peer to see ECONNRESET, and any data the client sent that is still in-flight may be discarded by the peer's TCP stack.
3. SSL session not terminated properly
For SSL connections, SSL_shutdown() is never called. This means the TLS close_notify alert is never sent, which:
- Prevents SSL session resumption/caching
- May cause the server to log TLS errors
- Violates the TLS specification (RFC 5246 §7.2.1)
Reproduction
void tcp_server(std::atomic<bool>& running, int port) {
int server_fd = socket(AF_INET, SOCK_STREAM, 0);
// ... bind, listen ...
while (running) {
int client_fd = accept(server_fd, nullptr, nullptr);
if (client_fd >= 0) {
// Read Hello, then close properly from server side
recv(client_fd, buffer, sizeof(buffer), 0);
shutdown(client_fd, SHUT_RDWR);
close(client_fd);
}
}
}
// Client side — creates and destroys many connections
for (int i = 0; i < 15000; ++i) {
try {
clickhouse::Client client(opts);
} catch (...) {}
}
// Check TIME_WAIT count — will be very high
// Linux: grep "06 " /proc/net/tcp | wc -lSuggested Fix
Add shutdown() before close() in Socket::Close():
void Socket::Close() {
if (handle_ == INVALID_SOCKET)
return;
#if defined(_win_)
shutdown(handle_, SD_BOTH);
closesocket(handle_);
#else
shutdown(handle_, SHUT_RDWR);
close(handle_);
#endif
handle_ = INVALID_SOCKET;
}For SSL sockets, add SSL_shutdown() in the SSLSocket destructor before the base Socket::Close() runs:
SSLSocket::~SSLSocket() {
if (ssl_) {
SSL_shutdown(ssl_.get());
}
}Note: shutdown() may fail if the socket is already in an error state — the return value can be safely ignored here since we are closing regardless.
Environment
- Library: clickhouse-cpp (all current versions)
- Affected platforms: All (Linux, macOS, Windows)
- Affected files:
clickhouse/base/socket.cpp,clickhouse/base/sslsocket.cpp
Activity
ansh0014 commentedon May 4, 2026
@jyberg
I understand the issue: Socket::Close() calls close()/closesocket() directly, and SSL sockets never call SSL_shutdown(). This can cause TIME_WAIT buildup and ungraceful teardown (RST) if unread data exists.
Suggested fix: call shutdown(SHUT_RDWR/SD_BOTH) before close in Socket::Close(), and call SSL_shutdown() for TLS sockets before the FD is closed.
jyberg commentedon Aug 25, 2026
Hi, It looks like there was already correction done, but why it was not merged to the main? Any information when this small change can be done?
slabko commentedon Aug 26, 2026
Hi @jyberg
Sorry I didn't reply to your issue report in time.
Some aspects of the network stack are being reworked in the library, and I am still thinking about a way to solve this problem, if it really needs solving at all. Let me elaborate. And please, if I miss anything, let me know. This is a complicated topic with many parts, and I can absolutely miss some of them.
It is indeed required that the closing peer:
FINsegmentACKand FIN from the other sideACKTIME_WAITperiod to allow any inflight segments to reach the peer. (I am simplifying the reason forTIME_WAITa bit, as it is not important for the problem at hand)From an OS API perspective, you can either let the system handle this flow and just close the socket (unless specific options are applied to prevent that, which is not the case here), or the OS can let you control the flow yourself. If you want to control it yourself, you call
shutdown(fd, SHUT_WR)orshutdown(fd, SHUT_RDWR)and then callrecv(...)to get the final empty response of size 0. This allows the application to send theFINsegment and wait for theFINsegment from the peer. Doing so, however, does not prevent the OS from sending anRSTif the peer sends any data after your call toshutdown(...). It doesn't change anything aboutTIME_WAITeither -TIME_WAITis always respected on the side that initiated the connection closure, regardless of whether you callshutdown(...)or not.Now let me address your concerns one by one:
TIME_WAITis going to happen regardless of whether you callshutdown(...)beforeclose(...). This is part of the closing handshake, the OS must do it to follow the protocol.As I said, this can happen even if you call
shutdown(...), and even if you follow the more advanced flow described earlier (i.e.shutdown(...),recv(...), and only thenclose(...)).However, if you are concerned about ephemeral port exhaustion, you are probably dealing with a much larger problem in general. In order to reach this point, you need to open and close 28K ports per
TIME_WAITperiod, whereTIME_WAITis usually 60 seconds. In other words, to reach that point, you need to open and close more than two sockets per millisecond.The exact same thing happens when calling
shutdown(fd, SHUT_RDWR).That happens when the user has already deleted their socket, so they are not interested in that data in the first place. And again, the exact same thing happens if you call
shutdown(fd, SHUT_RDWR).This can also happen when sending data. For example, the data has been sent and the peer closes the connection. If there is any data in both the read and write buffers, the OS will reset that socket, discarding data in the write buffer.
However, the ClickHouse protocol is a confirmation- and turn-based protocol, meaning that:
Insert(...),EndInsert(...),Execute(...), etc.), the call waits for server confirmation before returning control, so the client and its socket cannot be deleted at that point.close(), because the server only replies after it receives data.From all of this, however, I think callingJust realized thatSSL_shutdown(...)followed byclose(...)is a good idea, but the lack of it should not cause any problems.SSL_shutdowncan also block under some circumstances, which is a no-no in the destructor.@jyberg, all that said, I understand that you created the issue for a reason, and I believe you are facing a specific problem in your environment. I was hoping you could share it here so that we can address that problem instead of trying to solve the abstract issues initially listed in the ticket.
slabko commentedon Sep 16, 2026
Closing stale ticket