Data Pipeline Orchestration

Data pipeline orchestration is the automated management of a data workflow — deciding when each task runs, in what order, and what happens when one fails — sitting one level above the ETL process itself, covered in the lesson on data warehouses and OLAP technology. This lesson works through the one concept that makes orchestration a genuinely distinct discipline rather than just "running scripts in sequence": the DAG (Directed Acyclic Graph), and specifically why the "acyclic" part is a hard requirement, not a stylistic preference.

A Worked DAG Example

A typical pipeline has some tasks that must happen strictly in order and others that can run independently at the same time. Consider four tasks: Extract pulls raw data from a source system; Validate checks that extract for schema and completeness issues; Clean fixes missing values and duplicates; and Load writes the final result into a warehouse. The dependencies among them:

Extract -> Validate -> Load
Extract -> Clean     -> Load

Both Validate and Clean depend only on Extract finishing — they don't depend on each other — so an orchestrator can run them in parallel once Extract completes, and only start Load once both of them have finished. This is exactly what a DAG diagram makes explicit: arrows show what has to finish before what starts, and any two tasks with no arrow between them (directly or indirectly) are free to run at the same time, which is precisely the parallel-execution benefit orchestration tools like Apache Airflow are built around.

Why "Acyclic" Is a Hard Requirement, Not a Preference

Now suppose a pipeline was accidentally configured so that Clean depends on Transform finishing first, and Transform depends on Clean finishing first:

Clean -> Transform -> Clean  (a cycle)

There is no valid order to run these two tasks in: starting either one requires the other to have already finished, which requires the first one to have already finished — an infinite regress with no valid starting point. This is precisely why a workflow graph must be acyclic: an orchestrator determines execution order by finding a valid sequence through the dependency graph (a "topological sort"), and that sequence simply does not exist if the graph contains a cycle anywhere in it. A cyclic dependency isn't a slow or inefficient pipeline — it's one that cannot be scheduled at all, which is why every major orchestration tool rejects a workflow definition containing a cycle before ever attempting to run it, rather than discovering the problem partway through execution.

The Components a DAG Alone Doesn't Handle

The DAG determines order; the rest of what an orchestrator does is what happens around that order:

• Task scheduling decides when the DAG runs at all — on a fixed schedule (daily, hourly), triggered by an event (new data has arrived), or run manually.
• Error handling and recovery determines what happens when one node in the DAG fails partway through — commonly retrying that specific task a set number of times, or falling back to an alternative path, rather than restarting the entire pipeline from Extract.
• Resource management and monitoring track what's actually happening while the DAG executes — which tasks are running, how much compute they're using, and whether the pipeline as a whole is keeping pace with the data arriving.

  Common Orchestration Tools

  Apache Airflow defines workflows as Python-coded DAGs and remains the most widely adopted tool for complex, heterogeneous pipelines, with a strong web UI for monitoring runs. Prefect offers a comparable DAG-based model with a lighter-weight setup and stronger built-in error handling, aimed at teams that find Airflow's operational overhead heavier than they need. Dagster puts particular emphasis on data quality and type-checking between pipeline steps, which suits analytics and machine-learning pipelines where a silent schema mismatch between two steps is a common source of hard-to-diagnose failures. Each tool implements the same underlying DAG concept described above — the differences are in operational overhead, UI, and how much structure they enforce around data quality between steps, not in the fundamental scheduling problem they're all solving.