Dagster

What is Dagster?

Dagster is an open-source data orchestrator built around data assets. An asset is something your data platform should produce and keep reliable: a table in a warehouse, a file in a lakehouse, a machine learning feature set, a dashboard input, or a trained model.

Instead of only describing a list of tasks, you describe the assets that should exist and the Python code that produces them. Dagster then understands the dependency graph: which assets depend on which other assets, what needs to run first, and what downstream data is affected when something changes.

Dagster calls this pattern software-defined assets. The useful shift is subtle but important. You stop asking only which script runs at 02:00? and start asking which data product should be up to date, tested, and trustworthy?

Assets instead of loose tasks

In Dagster, an asset definition says how an asset is computed. If a table depends on three upstream tables, those dependencies are part of the definition. Dagster uses that information to build lineage, run the right assets in the right order, and show the state of the data in its UI.

Materialization
Materializing an asset means actually creating or updating it. Dagster runs the asset code and writes the result to the warehouse, lakehouse, file store, or other target.

Asset checks
Asset checks are data quality checks attached to an asset. For example: this column must not contain nulls, row count must stay within an expected range, or the table must be refreshed recently. The check runs alongside the asset workflow, so a problem is visible near the place where the data is produced.

Lineage
Because assets and dependencies are defined in code, Dagster can show where a data asset came from and which downstream assets depend on it. That helps when a source is late, a test fails, or a team wants to know what will break if a table changes.

Dagster versus Apache Airflow

Apache Airflow is the older and better-known open-source orchestrator. Airflow is task-oriented: you define tasks and connect them in a DAG. It is strong for scheduled workflows across many systems, and it has a large ecosystem of operators and managed services.

Dagster is more asset-oriented. It asks what data assets should exist, how they are produced, and whether they are healthy. Lineage, freshness, and checks are closer to the centre of the model.

That does not make one tool universally better. Airflow can be a good fit for mature task workflows across many platforms. Dagster often feels natural in an analytics engineering stack where the main objects are warehouse tables, dbt models, feature tables, and downstream reports.

Dagster and dbt

Dagster and dbt work well together because both think in dependencies between data objects.

dbt defines SQL models and uses ref() calls to understand which model depends on which other model. Dagster can read a dbt project and represent dbt models as Dagster assets. That gives you one graph that can include upstream extracts, Python assets, dbt models, checks, and downstream jobs.

In practice, dbt remains the transformation framework and Dagster becomes the orchestrator around it. Dagster can run dbt, show model-level lineage, combine dbt with Python or Spark work, and trigger only the parts of the graph that need to be rebuilt.

When does Dagster fit?

• Your team thinks in tables, models, and data products rather than scripts and schedules alone.

• You want lineage and data quality checks built into orchestration, not added as an afterthought.

• You use dbt and want an orchestrator that understands dbt models as individual assets.

• You want data engineers to build and test pipelines locally before deploying them.

• You have enough Python capability to own code-first orchestration.

If your whole analytics setup lives inside one low-code platform and only has a few simple refreshes, Dagster may be more machinery than you need. If your stack spans a warehouse, lakehouse, dbt, notebooks, ML jobs, and custom Python code, the asset model starts to pay off.

What to watch out for

Dagster still needs ownership
Open-source does not mean maintenance-free. You run it yourself or use Dagster's hosted option, and someone must own deployment, secrets, upgrades, alerting, and conventions.

The asset mindset takes practice
Teams used to task chains may initially model everything as one more script. The benefit comes when you name the assets clearly and attach useful checks to them.

Checks need business meaning
A green materialization only says the code ran. It does not prove the table is useful. Add checks for freshness, row counts, nulls, uniqueness, accepted values, and domain rules that matter to the report or product.

Do not over-split the graph
Fine-grained lineage is helpful, but thousands of tiny assets can become hard to read. Split where there is a real dependency, ownership boundary, retry boundary, or reusable data product.