Look Who it Was Random video

Resilient Predictive Data Pipelines • Siddharth "Sid" Anand • GOTO 2016

1,033 views · Published 20 October 2016 · 45:29 · Indexed 12 November 2026

Channel: GOTO Conferences · 2016 · Science & Technology

Watch on YouTube

This presentation was recorded at GOTO Chicago 2016
http://gotochgo.com

Siddharth "Sid" Anand - Data Architect at Agari Inc

ABSTRACT
Big Data companies (e.g. LinkedIn, Facebook, Google, and Twitter) have historically built custom data pipelines over bare metal in custom-designed data centers. In order to meet strict requirements on data security, fault-tolerance, cost control, job scalability, and uptime, they need [...]

TIMECODES
0:00 Introduction
0:33 About Me
3:21 Motivation
3:50 Data Products
6:10 Serving + Data Pipelines
8:05 The Blast Radius Problem
12:11 Timeliness
20:52 SOS - Dead Letter Queue
22:52 SNS + SOS Design Pattern
25:02 What Does Agari Do?
33:22 Apache Airflow - Authoring DAGS
34:25 Apache Airflow - Perf. Insights
34:50 Apache Airflow - Alerting
35:53 NRT Architecture
39:10 Schema Registry
40:53 What is AWS Lambda?
41:54 Elastic Stream Processing
42:30 Open Source Plans
42:38 Questions? (@r39132)
42:44 SOFTWARE DEVELOPMENT CONFERENCE

Download slides and read the full abstract here:
https://gotocon.com/chicago-2016/presentation/Resilient%20Predictive%20Data%20Pipelines

https://twitter.com/gotochgo
https://www.facebook.com/GOTOConference
http://gotocon.com

Looking for a unique learning experience?
Attend the next GOTO conference near you! Get your ticket at https://gotopia.tech
Sign up for updates and specials at https://gotopia.tech/newsletter

SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily.
https://www.youtube.com/user/GotoConferences/?sub_confirmation=1

More from this channel