Software Engineering Daily

Software Engineering Daily


Prefect Dataflow Scheduler with Jeremiah Lowin

April 29, 2020

A data workflow scheduler is a tool used for connecting multiple systems together in order to build pipelines for processing data. A data pipeline might include a Hadoop task for ETL, a Spark task for stream processing, and a TensorFlow task to train a machine learning model.  The workflow scheduler manages the tasks in that