DAG (directed acyclic graph)

What is a DAG?

A DAG is a directed acyclic graph. In data work, it is the shape used to describe a pipeline: boxes for steps, arrows for dependencies.

The name has three parts.

Graph
A graph is a set of points with connections between them. In a pipeline, the points are tasks, models, or assets.

Directed
The connections have a direction. An arrow from A to B means A must finish before B can start.

Acyclic
There are no loops. If you follow the arrows, you can never end up back at the same step. That matters because a loop could make a task wait, directly or indirectly, for itself.

In plain English: a DAG tells a system what needs to happen, what depends on what, and what can run at the same time.

Why data pipelines use DAGs

A data pipeline is rarely just one script. It has steps: extract orders, extract invoices, load raw files, transform tables, run tests, publish a model, refresh a report. Some steps must wait. Others can run in parallel.

A DAG solves three problems.

Order
The arrows define the order. The orchestrator does not have to guess or rely on fixed times.

Parallelism
If two tasks have no dependency between them, they can run at the same time. That shortens the pipeline without making it less correct.

Deadlock prevention
Because loops are not allowed, no chain of tasks can end up waiting for itself.

A four-step example

Take a nightly finance pipeline.

1. Fetch invoices from the accounting system.

2. Fetch payments from the bank.

3. Match invoices and payments into an open-balance table.

4. Refresh the finance report.

Step 1 and step 2 can run together. Step 3 waits for both. Step 4 waits for step 3. Draw those dependencies as arrows and you have a DAG.

If the bank extract fails, the matching step should not run. If the invoice extract is slow, the report refresh should wait. The DAG gives the workflow engine enough structure to make those decisions.

DAGs in common data tools

Apache Airflow
Airflow uses the term DAG directly. A DAG contains tasks, dependencies, schedule, retries, and runtime settings. Airflow's scheduler decides which task instances are ready to run based on that graph.

dbt
dbt builds a dependency graph from functions such as ref(). If a model references another model, dbt knows the referenced model must be built first. That graph drives run order, documentation, and lineage.

Apache Spark
Spark builds execution plans as graphs of transformations. Its scheduler breaks work into stages and runs the stages in an order that respects dependencies.

Dagster
Dagster usually talks about assets and asset graphs. The idea is similar: define what data assets exist, how they are produced, and which upstream assets they depend on.

The words differ by tool, but the shape is familiar: nodes, arrows, and no circular dependency.

DAG versus schedule

A schedule says run at 02:00. A DAG says run this after those two things have succeeded.

Both can exist together. A nightly DAG may start at 02:00, but once it starts, the dependency graph controls the order. If one extract takes longer than usual, downstream tasks wait. If two independent extracts are ready, they can run together.

This is why DAGs are a step up from a pile of cron jobs. Cron can start scripts. A DAG-aware orchestrator can understand the state of the whole workflow.

What to watch out for

Hidden dependencies
A DAG is only useful if the arrows are honest. If a task quietly relies on a file written by another system, but that dependency is not in the graph, the pipeline can turn green while the data is still wrong.

Tasks that are too large
If one task does everything, the graph is easy to read but hard to retry. A small failure reruns too much work.

Tasks that are too small
If every tiny operation becomes a task, the graph becomes unreadable. Split work where a real dependency, retry boundary, or ownership boundary exists.

Cycles in business logic
Sometimes the data model itself has a circular dependency: table A needs table B, while table B needs table A. The DAG forces that issue into the open. You usually need to break the loop with a snapshot, staging step, seed value, or redesigned model.

Idempotency
DAG tools often retry failed tasks. A retried task should be safe to run again with the same inputs. If it appends duplicates or sends the same invoice twice, the graph is not the real problem: the task design is.