Skip to content

fix(io): abort the multipart upload when an ObjectStoreWrite fails - #9783

Open
jackylee-ch wants to merge 1 commit into
vortex-data:developfrom
jackylee-ch:fix/objectstore-write-abort
Open

fix(io): abort the multipart upload when an ObjectStoreWrite fails#9783
jackylee-ch wants to merge 1 commit into
vortex-data:developfrom
jackylee-ch:fix/objectstore-write-abort

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

object_store's MultipartUpload::abort documents that "some stores, such as S3 and GCS,
cannot perform cleanup on drop", so a failed write left the already-uploaded parts staged —
billable, and invisible to a plain list. write_all, flush and shutdown all returned the
error without aborting.

The three failure paths now go through abort_with. It aborts at most once, since a second
abort is implementation-defined, and logs a failing abort rather than returning it in place of
the error the caller needs.

Dropping the writer without calling shutdown still leaks: doing that from Drop needs a
runtime handle, which is a design call rather than a fix. The struct doc already puts the
shutdown obligation on the caller.

Tests

cargo test --release -p vortex-io --features object_store,tokio: 184 passed, 181 on develop;
also --no-default-features --features tokio. A MultipartUpload that refuses every part
counts its aborts, since InMemory and LocalFileSystem do not expose staged parts. Removing
the abort fails all three cases; removing the once-only guard fails the third alone.

Not verified: no S3 or GCS run — no credentials here, and S3Mock needs a Docker daemon.

AI assistance

Written with agentic AI assistance; I read abort's contract in the vendored object_store
0.13.2 source before relying on it.

`write_all`, `flush` and `shutdown` returned the part or completion error
without aborting, and `object_store` documents that S3 and GCS keep the parts
that already landed when a `MultipartUpload` is dropped without an abort.

Route the three failure paths through `abort_with`, which aborts once — a
second abort is implementation-defined — logs a failing abort rather than
replacing the caller's error, and hands the original error back.

Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
@AdamGS

AdamGS commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

IDK what I think about these semantics, the comparable behavior for files would be to delete or truncate the file, which seems weird.

@robert3005

Copy link
Copy Markdown
Contributor

If you have incomplete multipart upload it will generally not be recoverable unless you stashed the upload id somewhere permanent. I never seen anything issue explicit aborts since most of the time you configure s3 buckets to automatically expire unfinished multipart uploads

@jackylee-ch

Copy link
Copy Markdown
Contributor Author

Fair — the parts aren't resumable either way, so it's only about who deletes them; lifecycle rules stay the backstop for what we can't abort at all. We did have this: WriteMultipart::finish aborts when complete() fails (upload.rs:234-241), and we called it until #2155 replaced it with a hand-rolled upload. Upstream just extended that abort to failed parts too (apache/arrow-rs-object-store#819, merged but unreleased; we're on 0.13.2). @AdamGS for files there's nothing to truncate — LocalUpload renames a {dest}#N staging file on complete, so the target path never holds a partial. Unless you'd rather close it, I'll cut this to shutdown only, the finish analogue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants