fix(io): abort the multipart upload when an ObjectStoreWrite fails - #9783
fix(io): abort the multipart upload when an ObjectStoreWrite fails#9783jackylee-ch wants to merge 1 commit into
Conversation
`write_all`, `flush` and `shutdown` returned the part or completion error without aborting, and `object_store` documents that S3 and GCS keep the parts that already landed when a `MultipartUpload` is dropped without an abort. Route the three failure paths through `abort_with`, which aborts once — a second abort is implementation-defined — logs a failing abort rather than replacing the caller's error, and hands the original error back. Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
|
IDK what I think about these semantics, the comparable behavior for files would be to delete or truncate the file, which seems weird. |
|
If you have incomplete multipart upload it will generally not be recoverable unless you stashed the upload id somewhere permanent. I never seen anything issue explicit aborts since most of the time you configure s3 buckets to automatically expire unfinished multipart uploads |
|
Fair — the parts aren't resumable either way, so it's only about who deletes them; lifecycle rules stay the backstop for what we can't abort at all. We did have this: |
object_store'sMultipartUpload::abortdocuments that "some stores, such as S3 and GCS,cannot perform cleanup on drop", so a failed write left the already-uploaded parts staged —
billable, and invisible to a plain
list.write_all,flushandshutdownall returned theerror without aborting.
The three failure paths now go through
abort_with. It aborts at most once, since a secondabort is implementation-defined, and logs a failing abort rather than returning it in place of
the error the caller needs.
Dropping the writer without calling
shutdownstill leaks: doing that fromDropneeds aruntime handle, which is a design call rather than a fix. The struct doc already puts the
shutdownobligation on the caller.Tests
cargo test --release -p vortex-io --features object_store,tokio: 184 passed, 181 ondevelop;also
--no-default-features --features tokio. AMultipartUploadthat refuses every partcounts its aborts, since
InMemoryandLocalFileSystemdo not expose staged parts. Removingthe abort fails all three cases; removing the once-only guard fails the third alone.
Not verified: no S3 or GCS run — no credentials here, and S3Mock needs a Docker daemon.
AI assistance
Written with agentic AI assistance; I read
abort's contract in the vendoredobject_store0.13.2 source before relying on it.