Astronomer's Journey from Cloud Lock-in to OSS Independence
180 views · Published 21 September 2017 · 23:36 · Indexed 24 October 2026
Channel: The Linux Foundation · 2017 · Science & Technology
Astronomer's Journey from Cloud Lock-in to OSS Independence - Aaron Brongersma, Astronomer Astronomer is a data engineering platform that collects, processes and unifies users' data so that they can get straight to analytics and data science. Initially, Astronomer built their pipeline out of AWS-managed services. But, when they ran into the limitations of these services, they realized that they had committed to proprietary components, which prevented them from changing cloud providers. In this talk, Astronomer outlines their transition to open source services running on Apache Mesos and DC/OS. They started by using Apache Airflow instead of AWS Simple Workflow Service and went on to replace their Kinesis streams with Apache Kafka, and deploy their stream processing applications with Marathon. They will review their remaining challenges and the solutions they are considering. It took some time and effort to for Astronomer decouple from AWS, but they are glad they did. About Aaron Brongersma VP of Engineering, Astronomer
More from this channel
-
0:37
The Linux Foundation Video Site:: Chicks dig it
-
1:01
The Origin (Japanese Subtitles)
-
5:19
The Linux Foundation Video Site:: Flock
-
0:35
The Linux Foundation Video Site:: Gnu/Linux-Desktop
-
5:51
The Linux Foundation Video Site:: Debian & Cooliris
-
6:09
The Linux Foundation Video Site:: Slackware Linux
-
1:57
The Linux Foundation Video Site:: Bdale Garbee (Hewlett-Pack
-
1:12
The Linux Foundation Video Site:: James Morris (Red Hat) at