Look Who it Was Random video

Taking Advantage of a Disaggregated Storage and Compute Architecture - Brian Cho and Ergin Seyfe

865 views · Published 20 September 2018 · 30:09 · Indexed 4 November 2026

Channel: Databricks · 2018 · Science & Technology

Watch on YouTube

Brian Cho, a software engineer at Facebook, and Ergin Seyfe, a software engineer on BigCompute team at Facebook, talk about how the hardware trend is moving to disaggregated storage and compute architecture in order to improve cost efficiency and scale at finer granular. For Spark, this means that compute machines store and access temporary data (shuffle, spill, persisted RDD/dataframes) on storage machines over the network. Disaggregated storage has many advantages over collocated storage where each node is constrained to the space of its local disks. Over half of our Spark workload is now running on disaggregated infrastructure with individual jobs shuffling more than 300 TB compressed.

Learn more here: https://databricks.com/session/taking-advantage-of-a-disaggregated-storage-and-compute-architecture

Article you might like: https://databricks.com/session/transparent-gpu-exploitation-on-apache-spark
About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business.
Read more here: https://databricks.com/product/unified-data-analytics-platform

Connect with us:
Website: https://databricks.com
Facebook: https://www.facebook.com/databricksinc
Twitter: https://twitter.com/databricks
LinkedIn: https://www.linkedin.com/company/databricks
Instagram: https://www.instagram.com/databricksinc/ Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner

More from this channel