Autoplay Next video from Max length

Using AI to Build a Self-Driving Query Optimizer (Shivnath Babu and Adrian Popescu)

487 views · Published 25 September 2018 · 30:53 · Indexed 30 October 2026

Channel: Databricks · 2018 · Science & Technology

Shivnath Babu is a CTO at Unravel Data Systems, and Adrian Popescu is a data engineer at Unravel Data Systems.

We discuss Spark’s Catalyst Optimizer uses cost-based optimization (CBO) to pick the best execution plan for a SparkSQL query. The CBO can choose which join strategy to use (e.g., a broadcast join versus repartitioned join), which table to use as the build side for the hash-join, which join order to use in a multi-way join query, which filter to push down, and others. To get its decisions right, the CBO makes a number of assumptions including availability of up-to-date statistics about the data, accurate estimation of result sizes, and availability of accurate models to estimate query costs.

Learn more here: https://databricks.com/session/using-ai-to-build-a-self-driving-query-optimizer

To learn more: https://databricks.com/session/merchant-churn-prediction-using-sparkml-at-paypal
About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business.
Read more here: https://databricks.com/product/unified-data-analytics-platform

Connect with us:
Website: https://databricks.com
Facebook: https://www.facebook.com/databricksinc
Twitter: https://twitter.com/databricks
LinkedIn: https://www.linkedin.com/company/databricks
Instagram: https://www.instagram.com/databricksinc/ Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner

Using AI to Build a Self-Driving Query Optimizer (Shivnath Babu and Adrian Popescu)

More from this channel