Apache Spark

What is Apache Spark?

Apache Spark is an open-source engine for large-scale data processing. You write a query or transformation in SQL, Python, Scala, Java, or R, and Spark distributes the work across a cluster of machines.

Spark is used when one machine is not enough, or when the same platform needs to handle several kinds of analytical work: batch transformations, SQL, streaming, machine learning, and graph processing. It is the engine underneath many modern data platforms, including Databricks, Microsoft Fabric notebooks, Amazon EMR, Google Dataproc, and many self-hosted lakehouse setups.

Spark started as a research project at UC Berkeley's AMPLab. It became popular because it made many workloads faster and more flexible than older MapReduce-style jobs, especially where data had to be reused across several processing steps.

The main parts of Spark

Spark Core
The foundation: task scheduling, memory management, fault tolerance, and distributed execution. This is also where the original low-level RDD API comes from.

Spark SQL and DataFrames
The most common way to use Spark today. A DataFrame is a distributed table with named columns and data types. Spark can optimise DataFrame and SQL queries because it understands the structure of the data.

Structured Streaming
Spark's streaming layer. You describe a query against incoming data and Spark keeps updating the result as new data arrives. It is often used with Kafka, Event Hubs, files, or cloud object storage.

MLlib
Spark's machine-learning library for feature engineering and algorithms such as classification, regression, clustering, and recommendation. It is useful when the training data is too large for a single machine workflow.

GraphX
A graph-processing module for workloads such as PageRank, connected components, and network analysis. It is more specialised than Spark SQL, but it is part of the broader Spark toolkit.

RDD, DataFrame, and Dataset

Spark has several APIs because the project evolved over time.

RDD
A Resilient Distributed Dataset is the original Spark abstraction: a distributed collection of records with operations such as map, filter, and reduce. It gives low-level control, but Spark has less structure to optimise.

DataFrame
A DataFrame is a distributed table. Spark knows the columns and types, so the Catalyst optimiser can rearrange operations, push filters down, and choose better join strategies. For most teams, DataFrames are the default.

Dataset
A Dataset adds compile-time type safety for Scala and Java users while keeping many DataFrame optimisations. In Python, people generally work with DataFrames rather than Datasets.

The usual advice is simple: use SQL or DataFrames unless you have a very specific reason to drop down to RDDs.

Where Spark fits in a lakehouse

In a lakehouse, Spark is often the general-purpose processing engine. It reads raw files, transforms them into cleaned tables, runs notebook analysis, trains features for machine learning, and writes results back as Delta, Iceberg, Hudi, or Parquet tables.

For example, a nightly Spark job might read raw sales events from object storage, join them to product and customer dimensions, write a clean fact table, and prepare the gold layer used by Power BI. A streaming Spark job might read events from Kafka and update near-real-time tables every few seconds or minutes.

Spark is not the only engine in a lakehouse. SQL warehouses, DuckDB, Trino, dbt, Fabric Data Pipelines, and pandas or Polars may all have a role. Spark earns its place when the workload is too large, too distributed, or too mixed for a single-node tool.

Spark versus pandas, DuckDB, and Flink

pandas and Polars
Great for work that fits on one machine. They start quickly and are often faster for small or medium data. Do not use a Spark cluster just to process a few CSV files that fit comfortably on a laptop.

DuckDB
Excellent for local SQL analytics on Parquet, CSV, and other files. It can be simpler than Spark for exploratory or embedded analytical work.

Apache Flink
Flink is streaming-first and often better for very low-latency event processing. Spark Structured Streaming is easier to adopt for many data engineering teams, but it is usually micro-batch oriented unless configured for specific continuous patterns.

dbt
dbt is a transformation framework, not a distributed execution engine. It can run SQL on warehouses or lakehouse engines, including Spark-backed platforms, but it is not a Spark replacement.

What to watch out for with Spark

It is overkill for small data
Cluster startup, scheduling, and JVM overhead can make Spark slower than pandas, Polars, or DuckDB for small datasets.

Lazy evaluation can surprise people
Transformations do not run immediately. Spark builds a plan and only executes it when an action such as count, show, collect, or write is called.

Shuffles are expensive
Wide joins and group-bys move data between machines. Poor partitioning, skewed keys, or missing broadcast joins can turn a simple query into a long-running job.

Memory issues are real
Distributed does not mean unlimited. Executors can run out of memory, spill heavily to disk, or retry stages repeatedly when data is skewed.

Version and connector compatibility matters
Spark, Python, Scala, Delta, Iceberg, Kafka, and cloud connectors all have compatibility boundaries. Pin versions in production and test upgrades per workload.