PyConf Hyderabad 2017

PyConf Hyderabad

Airflow: To Manage Data Pipelines

Submitted by Mahendra Yadav (@userimack) on Thursday, 17 August 2017

videocam_off

Technical level

Beginner

Status

Confirmed

Vote on this proposal

Login to vote

Total votes:  +12

Abstract

A significant part of the IT/Data Engineering team is spent on writing and scheduling jobs, monitoring and troubleshooting the issues. Enterprise data originates from various sources and there are various business rules and processes that govern how that data can be consumed.

Airflow is a platform to programmatically author, schedule and monitor workflows. (https://airflow.incubator.apache.org/)

The various tasks in the workflow(s) are configured as a Directed Acyclic Graph. This talk covers how Airflow is used to establish better workflows for data engineering.

P.S: This talk is inspired from Bargava Subramanian (@barsubra) proposal.

Outline

  1. Existing challenges in data engineering - creating/monitoring/troubleshooting workflows
  2. Introduction to Airflow
  3. Main advantages of Airflow
  4. Tasks as DAG
  5. Airflow in practice - case study
  6. Dynamic pipeline generation
  7. Demo UI dashboards
  8. Data Engineering at Scale
  9. Brief overview of what other options exists

Speaker bio

Mahendra Yadav is a Data Engineer at Azri Solutions, Hyderabad. In his day to day work he processes a lot of data from different sources.

Links

Comments

  • 0
    Sarith Nippu a year ago

    mahendra

Login with Twitter or Google to leave a comment