Apache 5 causing S3 cross-region errors

[kschwarz1116]

Describe the bug

With an S3 client with .crossRegionAccessEnabled(true), the recent update has started showing broken pipes on write requests.
Re-enabling expectContinue does not fix the behavior.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Cross region access to continue working out-of-the-box or the functionality to be removed if it is no longer supported.

Current Behavior

Here is a subset of the stack trace:

Caused by: java.net.SocketException: Broken pipe
	at java.base/sun.nio.ch.SocketDispatcher.write0(Native Method)
	at java.base/sun.nio.ch.SocketDispatcher.write(Unknown Source)
	at java.base/sun.nio.ch.NioSocketImpl.tryWrite(Unknown Source)
	at java.base/sun.nio.ch.NioSocketImpl.implWrite(Unknown Source)
	at java.base/sun.nio.ch.NioSocketImpl.write(Unknown Source)
	at java.base/sun.nio.ch.NioSocketImpl$2.write(Unknown Source)
	at java.base/java.net.Socket$SocketOutputStream.implWrite(Unknown Source)
	at java.base/java.net.Socket$SocketOutputStream.write(Unknown Source)
	at java.base/sun.security.ssl.SSLSocketOutputRecord.deliver(Unknown Source)
	at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(Unknown Source)
	at org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection$1.write(DefaultBHttpClientConnection.java:259)
	at org.apache.hc.core5.http.impl.io.SessionOutputBufferImpl.flushBuffer(SessionOutputBufferImpl.java:117)
	at org.apache.hc.core5.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:150)
	at org.apache.hc.core5.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:112)
	at org.apache.hc.core5.http.io.entity.InputStreamEntity.writeTo(InputStreamEntity.java:156)
	at org.apache.hc.core5.http.io.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:100)
	at software.amazon.awssdk.http.apache5.internal.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:178)

Reproduction Steps

Attempt writing to S3 with a cross-region client across regions.

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.46

JDK version used

25

Operating System and version

Amazon Linux 2023 (Lambda)

[dagnir]

hi @kschwarz1116 can you provide more details on your setup? I'm unable to repro with the following using 2.46.0

edit: description said write, missed that! Writing also seems to work with 2.46.0 (locally, and in Lambda with JDK 25):

public class Repro {
    private static final String BUCKET = "dongie-us-east-1";
    private static final String KEY = "test.txt";

    public static void main(String[] args) {
        S3Client s3 = S3Client.builder()
                .region(Region.US_WEST_2)
                .crossRegionAccessEnabled(true)
                .build();

        s3.putObject(r -> r.bucket(BUCKET).key(KEY), RequestBody.fromString("test"));

        ResponseBytes<GetObjectResponse> object = s3.getObject(r -> r.bucket(BUCKET).key(KEY), ResponseTransformer.toBytes());

        System.out.println(object.asUtf8String());

    }
}

2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-0 >> "PUT /test.txt HTTP/1.1[\r][\n]"
2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-0 >> "Host: dongie-us-east-1.s3.us-west-2.amazonaws.com[\r][\n]"
...
2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-0 << "HTTP/1.1 301 Moved Permanently[\r][\n]"
2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-0 << "x-amz-bucket-region: us-east-1[\r][\n]"
...
2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-1 >> "PUT /test.txt HTTP/1.1[\r][\n]"
2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-1 >> "Host: dongie-us-east-1.s3.us-east-1.amazonaws.com[\r][\n]"
2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-1 >> "Content-Length: 45[\r][\n]"
2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-1 >> "Connection: keep-alive[\r][\n]"
2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-1 >> "[\r][\n]"
2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-1 >> "4[\r][\n]"
2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-1 >> "test[\r][\n]"
2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-1 >> "0[\r][\n]"
2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-1 >> "x-amz-checksum-crc32:2H9+DA==[\r][\n]"
2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-1 >> "[\r][\n]"
...
2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-1 >> "GET /test.txt HTTP/1.1[\r][\n]"
2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-1 >> "Host: dongie-us-east-1.s3.us-east-1.amazonaws.com[\r][\n]"
...
2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-1 << "HTTP/1.1 200 OK[\r][\n]"
2026-06-17 16:04:32 [main] DEBUG org.apache.hc.client5.http.wire:92 - http-outgoing-1 << "Content-Length: 20[\r][\n]"
...
test

[dagnir]

Okay, looks like a larger request body is needed to trigger this:

public class Repro {
    private static final String BUCKET = "dongie-us-east-1";
    private static final String KEY = "test.dat";
    private static final Path FILE = Paths.get("128mb.dat");

    public static void main(String[] args) throws IOException {
        S3Client s3 = S3Client.builder()
                .region(Region.US_WEST_2)
                .crossRegionAccessEnabled(true)
                .build();

        InputStream is = new FileInputStream(FILE.toFile());
        RequestBody requestBody = RequestBody.fromInputStream(is, Files.size(FILE));

        s3.putObject(r -> r.bucket(BUCKET).key(KEY), requestBody);
    }
}

Will continue to investigate this. As a workaround, you can switch back to the 4.x client:

        S3Client s3 = S3Client.builder()
                .region(Region.US_WEST_2)
                .httpClientBuilder(ApacheHttpClient.builder())
                .crossRegionAccessEnabled(true)
                .build();

[kschwarz1116]

Thanks, yep we're writing large payloads! Currently getting around it by just instantiating separate clients per-region, but going back to 4.x works too.

[dagnir]

Looks like this is due to a difference in how Apache 5.x treats 3xx responses to requests with Expect: 100-continue, i.e. that they're not treated as final status codes. The client continues to send the entity even when a 3xx is received https://github.com/apache/httpcomponents-core/blob/d96a00fec9b2e19f8005e35681df5f6cd6e21a9e/httpcore5/src/main/java/org/apache/hc/core5/http/impl/io/HttpRequestExecutor.java#L186-L190

There is an open issue on the Apache tracker for this, and I'm requesting for it to be fixed https://issues.apache.org/jira/browse/HTTPCORE-796