Data Pipelines & Orchestration

ML data pipelines have many sequential, dependent steps: raw data → validation → cleaning → feature engineering → training → evaluation → serving. Orchestrators: • Schedule pipelines on a cron or event trigger • Manage dependencies between tasks • Retry failed steps automatically • Provide visibility into what ran, when, and with what data

Apache Airflow — Most popular. DAG-based. Strong ecosystem. Heavy to self-host. Prefect — Python-native. Better DX than Airflow. Cloud-managed option. Metaflow (Netflix) — Data science focused. Excellent versioning. Kubeflow Pipelines — Kubernetes-native. Best for large-scale distributed training. ZenML — MLOps framework that can use any orchestrator as backend.