Skip to content

Spark Connect Rust Client

A fast, native Rust client for Apache Spark Connect - and a drop-in pyspark replacement.

It builds spark.connect protobuf plans, manages the gRPC channel, and decodes Arrow results in Rust - speaking the same protocol and returning the same results as the reference client.

PyPI Spark License

Get started Quickstart View on GitHub

Coming from Python?

pyspark-client-rust is a faster, drop-in replacement for the pyspark-client package on PyPI. Uninstall any existing pyspark / pyspark-client, pip install pyspark-client-rust, and your Spark Connect code runs unchanged - same imports, same API, same server (see Installation). Use it exactly like PySpark; the rest of these docs cover the native Rust API.

Why

The reference Spark Connect client is pure Python - it builds protobuf plans, manages the gRPC channel, and decodes Arrow results in Python. This project moves that work into Rust: a synchronous, PySpark-shaped DataFrame API you can use directly from Rust, and - through PyO3 - a byte-for-byte compatible pyspark package for the Python world.

Native Rust API

A synchronous spark_connect crate that mirrors PySpark's DataFrame, Column, functions, SQL, streaming, and catalog surface.

Same protocol & results

Speaks the same spark.connect gRPC/protobuf protocol against the same server, validated against the official Apache Spark test suite.

Arrow-native results

Results decode through Apache Arrow; optionally convert to DataFusion or Polars.

Rust UDFs via WebAssembly

Compile a Rust function to WASM and run it as a Spark UDF on the executors - no server-side plugin. See Rust UDFs.

Drop-in pyspark

A faster replacement for the pyspark-client PyPI package - existing Python code runs unchanged.

Spark 4.2.0+

The crate and wheel version tracks the Spark release it targets, so the version number tells you which Spark it speaks.

Hello, Spark Connect (in Rust)

use spark_connect::{SparkSession, functions as f, lit};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let spark = SparkSession::builder()
        .remote("sc://localhost:15002")
        .get_or_create()?;

    let df = spark
        .range(1_000_000)?
        .select(vec![(f::col("id") * lit(2)).alias("x")])
        .filter((f::col("x") % lit(3)).eq(lit(0)));

    println!("count = {}", df.count()?);
    df.show(20)?;
    Ok(())
}

Where to next