Apache Airflow

What is Apache Airflow?

Apache Airflow is an open-source platform for developing, scheduling, and monitoring batch-oriented workflows. In data teams, that usually means data pipelines: extract data, load it, transform it, run checks, refresh a report, and alert someone if a step fails.

The important difference with a pile of scripts is that Airflow makes the workflow explicit. You write the pipeline in Python, describe the tasks and their dependencies, and Airflow decides what can run, what must wait, what should retry, and what should stop because an earlier step failed.

The pipeline is represented as a DAG: a directed acyclic graph. Directed means the arrows go one way. Acyclic means the workflow cannot loop back on itself. In plain English: Airflow needs to know that task B waits for task A, and that no task is waiting in a circle.

Airflow is often the point where scattered cron jobs turn into an observable workflow engine. Cron can say 'run this script at 03:00'. Airflow can say 'run this transformation after both source extracts have finished, retry the webshop extract twice if the API is down, and do not refresh the dashboard unless the quality checks passed'.

The main parts of Airflow

DAG
A DAG is one workflow. It contains the schedule, tasks, dependencies, callbacks, and other settings that tell Airflow how the workflow should run.

Task
A task is one unit of work inside the DAG: run a Python function, execute a SQL statement, call an API, launch a dbt job, copy a file, or refresh a downstream system.

Operator
An operator is a reusable task template. Airflow has operators for common actions, and provider packages add integrations for cloud platforms, databases, warehouses, message queues, and other tools. You can also write your own.

Scheduler and workers
The scheduler reads DAGs and decides which task instances are ready to run. Workers do the actual work. On a small setup, that may happen on one machine. On a larger setup, work is distributed across several workers.

Web UI and logs
The web interface shows which DAGs ran, which tasks failed, how long each step took, and where to find the logs. That visibility is often the reason teams adopt Airflow: when a nightly pipeline fails, you look in one place instead of hunting through separate log files.

A nightly pipeline in practice

Imagine a wholesaler with a webshop, an accounting system, a data warehouse, and a Power BI report. The nightly Airflow DAG could look like this:

1. Extract orders from the webshop and invoices from accounting in parallel.

2. Load both raw extracts into the warehouse once both source tasks succeeded.

3. Run transformations that clean the data, join orders to invoices, and calculate margin by product group.

4. Run quality checks: row counts, dates, duplicate keys, and missing required fields.

5. Refresh the Power BI report only when the checks pass.

If the webshop API is temporarily unavailable, Airflow can retry that one extract after a delay. If it keeps failing, the downstream tasks stay blocked. The report keeps yesterday's numbers instead of showing a half-loaded version of last night's data. That is the practical value: the workflow fails visibly and in the right place.

Where Airflow fits

Airflow is a good fit for ETL / ELT pipelines with several dependent steps, especially when those steps cross tools or platforms. It shines when a workflow touches an API, object storage, a data warehouse, dbt, Spark, a reporting tool, and a notification channel in one chain.

It is less useful for three isolated tasks that only need a simple schedule. It is also not a stream processing engine. Airflow can trigger or monitor streaming jobs, but tools such as Kafka, Flink, Spark Structured Streaming, or cloud streaming services do the continuous event processing itself.

Airflow is code-first. That gives you version control, review, tests, reusable code, and dynamic workflow generation. It also means someone on the team needs to be comfortable with Python and with the operational model behind Airflow.

Managed Airflow options

You can run Airflow yourself, but then you own the scheduler, workers, metadata database, upgrades, scaling, security, and runtime images. Many teams choose a managed service so they can focus on DAG code and pipeline behaviour.

Common options include Amazon Managed Workflows for Apache Airflow (MWAA), Google's Managed Service for Apache Airflow (formerly Cloud Composer), Astro by Astronomer, and Apache Airflow jobs in Microsoft Fabric. The promise is similar: keep the Airflow programming model, but hand more of the infrastructure work to the platform.

In the Microsoft ecosystem, this creates an interesting choice. Fabric Data Pipelines are visual and work well when you want low-code orchestration inside Fabric. Apache Airflow jobs fit better when you prefer Python, want code review, or need to coordinate Fabric with systems outside Fabric.

Airflow versus cron and visual pipelines

Airflow versus cron
Cron starts things at a time. Airflow starts things based on a graph of dependencies and records the state of every task. If step one runs late, step two waits. If step three fails, step four does not pretend everything is fine.

Airflow versus visual pipelines
Visual tools such as Fabric Data Pipelines or Azure Data Factory are easier for non-developers to read and edit. Airflow is stronger when the orchestration logic belongs in code: reusable functions, branching, tests, pull requests, and many similar pipelines generated from configuration.

Airflow versus newer orchestrators
Tools such as Dagster and Prefect often put more emphasis on developer experience, assets, and local testing. Airflow's strength is maturity: a large ecosystem, many operators, many managed services, and a big labour market of people who already know it.

What to watch out for with Apache Airflow

Retries need idempotent tasks
Airflow can retry failed tasks. That is only safe when rerunning a task with the same inputs produces the same result. A task that blindly inserts rows on every run can turn one failure into duplicate data.

Do not put heavy processing in the orchestrator
Airflow should coordinate work, not become the compute engine for every transformation. Let the warehouse, Spark cluster, dbt job, or API do the heavy work, and let Airflow manage the sequence.

Keep DAGs readable
Because DAGs are Python, it is tempting to hide too much logic in clever code. Future you still needs to open the graph and understand what happens at 02:00.

Alert fatigue
If every tiny retry sends a message, people stop looking. Alert on failures that need action, and keep the rest in logs and dashboards.

Operational ownership
Even managed Airflow needs ownership. Someone has to review DAG changes, tune retries, manage secrets, update dependencies, and check failed runs.