How we think about Data Pipelines is changing | Towards Data Science

Data Pipelines are series of tasks organised in a directed acyclic graph or "DAG". Historically, these are run on open-source workflow orchestration packages like Airflow or Prefect, and require infrastructure managed by data engineers or platform teams. These data pipelines typically run on a schedule, and allow data engineers to update data in locations such […]