Spark Connect Rust Client

A fast, native Rust client for Apache Spark Connect - and a drop-in pyspark replacement. It builds spark.connect protobuf plans, manages the gRPC channel, and decodes Arrow results in Rust, speaking the same protocol and returning the same results as the reference client.

📖 Documentation

Full documentation lives at apache.github.io/spark-connect-rust

• installation, quickstart, the DataFrame / Columns / SQL / Reading & Writing / Streaming / Catalog / Types API, Rust UDFs via WebAssembly, and the architecture.

Install

Python - a faster, drop-in replacement for the pyspark-client PyPI package (uninstall any existing pyspark / pyspark-client first):

pip install pyspark-client-rust

Your Spark Connect code then runs unchanged; use it exactly like PySpark.

Rust - the native crate:

[dependencies]
apache-spark-connect = "4.2"

Quickstart (Rust)

use spark_connect::{SparkSession, functions as f, lit};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let spark = SparkSession::builder()
        .remote("sc://localhost:15002")
        .get_or_create()?;

    let df = spark
        .range(1_000_000)?
        .select(vec![(f::col("id") * lit(2)).alias("x")])
        .filter((f::col("x") % lit(3)).eq(lit(0)));

    println!("count = {}", df.count()?);
    df.show(20)?;
    Ok(())
}

See the documentation for the full API, running a Spark Connect server, and more.

API Parity & Drop-in Replacement

The Python drop-in (pyspark-client-rust) achieves 100% public-API parity with PySpark 4.2.0. The Rust client (crate apache-spark-connect) has the same parity and supports the complete DataFrame, SQL, Streaming, Catalog, and Type system. The two internal APIs intentionally absent (Column.to_plan and SparkSession.client) are implementation details and not part of the public interface.

User-Defined Functions (UDFs)

Rust UDFs are compiled to WebAssembly, shipped to the executors, and can be used via the DataFrame API or registered by name for SQL. The runner is cloudpickled by value, so executors need only the wasmtime Python package:

use spark_connect::functions::col;
use spark_connect::wasm_udf::{udf, AbiType};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let spark = SparkSession::builder().remote("sc://localhost:15002").get_or_create()?;

    // Load a prebuilt module: fn shout(s: String) -> String
    let wasm = std::fs::read("shout.wasm")?;
    let shout = udf("shout", wasm, "shout", vec![AbiType::Str], AbiType::Str);

    // DataFrame API:
    let df = spark.range(0, 3)?.select(vec![shout.call(vec![col("id").cast_str("string")])?])?;
    df.show(20)?;

    // ...or register by name and call from SQL (mirrors spark.udf.register):
    spark.udf().register("shout", &shout)?;
    spark.sql("SELECT shout(name) FROM people")?.show(20)?;
    Ok(())
}

See the WASM UDF guide for building modules with the spark_wasm_udf macro and more examples.

Contributing

Issues are tracked in ASF JIRA under SPARK (GitHub Issues are disabled). See the contributing guide.

License

Apache License 2.0.