SOS - Optimizing Shuffle (Brian Cho and Ergin Seyfe)
2,787 views · Published 24 September 2018 · 31:06 · Indexed 22 October 2026
Channel: Databricks · 2018 · Science & Technology
Brian Cho has been working on Spark at Facebook for over a year. and Ergin Seyfe is also a software engineer on BigCompute team at Facebook. In this session, we present SOS’s multi-stage shuffle architecture and implementation. We will also share our production results and future optimizations. To learn more: https://databricks.com/product/getting-started-guide About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business. Read more here: https://databricks.com/product/unified-data-analytics-platform Connect with us: Website: https://databricks.com Facebook: https://www.facebook.com/databricksinc Twitter: https://twitter.com/databricks LinkedIn: https://www.linkedin.com/company/databricks Instagram: https://www.instagram.com/databricksinc/ Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
More from this channel
-
28:25
Riot Games: Improving the Gaming Experience With Apache Spark
-
10:06
Big Data Meets Learning Science
-
29:07
Dependency injection in Apache Spark Applications by Shu Das and Sandeep Prabhakar
-
10:55
#SparkSummit 2017 Structured Streaming Demo - Michael Armbrust
-
15:27
Combating Abusive Language in Chat with Apache Spark - Wes Kerr
-
30:18
Yelp Ad Targeting at Scale with Apache Spark - Inaz Alaei-Novin and Joe Malicki
-
37:42
Scaling Data Science Capabilities with Apache Spark at Stitch Fix - Derek Bennett
-
30:08
Rental Cars and Industrialized Learning to Rank - Sean Downes