Second generation "workflow managers" for big data by Alex Van Boxel
3,102 views · Published 11 November 2015 · 29:44 · Indexed 14 November 2026
Channel: Devoxx · 2015 · People & Blogs
The big data landscape is a fast moving. You quickly outgrow cron as scheduling you jobs and need a workflow manager. But the first generation doesn't cut it anymore. Now you need an agile and extendable tool. The new open source contenters in this space are all Python based, moving away from describing workflows in XML or special DSL's. Having your workflows described in code makes it more versatile. We'll have a look at one of the new open source workflow managers Luigi from Spotify. Then have a quick look at the feature set of the others like Pinball from Pinterest and Airflow from AirBnB.
More from this channel
-
32:13
Writing Groovy AST Transformations: Getting Practical in an hour! by Baruch Sadogursky
-
25:58
TweetWallFX - Behind the Scenes by Sven Reimers
-
29:13
Secrets are secrets. Please maintain them as secrets by Alex Soto
-
2:13:40
JavaScript build pipelines with Gulp.js by Stefan Baumgartner
-
2:04:09
Powerful Metaprogramming Techniques With Groovy by Jeff Brown
-
2:38:32
Distributed Machine Learning 101 using Apache Spark from a Browser by Xavier Tordoir/Andy Petrella
-
2:36:21
Caching 101: Caching on the JVM (and beyond) by Alex Snaps/Louis Jacomet
-
59:23
Modern Java Component Design with Spring 4.2 by Juergen Hoeller