Skip to content

feat: add S3 support to spark connect #652

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
razvan merged 30 commits into from
Feb 23, 2026
Merged

feat: add S3 support to spark connect #652

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
c58acb4
WIP
razvan Feb 4, 2026
a16d2ab
Merge branch 'main' into feat/connect-s3
razvan Feb 5, 2026
72ef48b
add s3 props, secret volumes and mounts
razvan Feb 5, 2026
2d550b1
use higher level functions from op-rs
razvan Feb 5, 2026
2a49d9b
regenrate certs for "minio" host update test
razvan Feb 5, 2026
8c034a8
fix boolean path style property
razvan Feb 5, 2026
b549812
add connectors field and successfully ran non tls tests
razvan Feb 6, 2026
6682beb
add truststore init container and re-enable tls tests
razvan Feb 6, 2026
0daee5e
cleanup
razvan Feb 6, 2026
4253fe9
update changelog
razvan Feb 6, 2026
5f144cf
Update time to fix RUSTSEC-2026-0009
sbernauer Feb 6, 2026
c4386c0
update nix
sbernauer Feb 6, 2026
82c3d82
Merge branch 'main' into feat/connect-s3
razvan Feb 17, 2026
0725ab9
add spec.connectors.s3connection and rename buckets property.
razvan Feb 17, 2026
db9c514
merge main
razvan Feb 18, 2026
f98764b
update test and cleanup
razvan Feb 18, 2026
6fa8a75
added unit tests
razvan Feb 18, 2026
85ab9f6
added docs
razvan Feb 18, 2026
9068ff3
Merge branch 'main' into feat/connect-s3
razvan Feb 19, 2026
63c6ef9
docs : fix typo
razvan Feb 19, 2026
0a8368c
tests: expand test with s3connection
razvan Feb 19, 2026
809739f
Merge branch 'main' into feat/connect-s3
razvan Feb 20, 2026
bc9589b
cargo update after main merge
razvan Feb 20, 2026
15c94dd
attempt to silence clippy warning
razvan Feb 20, 2026
607cde6
Apply suggestions from code review
razvan Feb 20, 2026
8332bb9
Update rust/operator-binary/src/connect/s3.rs
razvan Feb 20, 2026
ebd8f51
extract volumes and mounts to local vars
razvan Feb 20, 2026
ada51e8
add is_truststore_needed
razvan Feb 20, 2026
af799e6
reduce unit test verbosity
razvan Feb 20, 2026
5bc1293
refactor ca secret generation to include metadata annotations
razvan Feb 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ All notable changes to this project will be documented in this file.
This has the effect that applications where the `spark-submit` Pod fails are not resubmitted.
Previously, Jobs were retried at most 6 times by default ([#647]).
- Support for Spark `3.5.8` ([#650]).
- First class support for S3 on Spark connect clusters ([#652]).

### Fixed

Expand Down Expand Up @@ -41,6 +42,7 @@ All notable changes to this project will be documented in this file.
[#649]: https://github.com/stackabletech/spark-k8s-operator/pull/649
[#650]: https://github.com/stackabletech/spark-k8s-operator/pull/650
[#651]: https://github.com/stackabletech/spark-k8s-operator/pull/651
[#652]: https://github.com/stackabletech/spark-k8s-operator/pull/652
[#655]: https://github.com/stackabletech/spark-k8s-operator/pull/655
[#656]: https://github.com/stackabletech/spark-k8s-operator/pull/656

Expand Down
Loading