Look Who it Was Random video

Demystifying DataFrame and Dataset - Dr. Kazuaki Ishizaki

4,972 views · Published 12 June 2017 · 25:42 · Indexed 30 October 2026

Channel: Databricks · 2017 · Science & Technology

Watch on YouTube

"Apache Spark achieves high performance with ease of programming due to a well-balanced design between ease of usage of APIs and the state-of-the-art runtime optimization. In Spark 1.3, DataFrame API was introduced to write a SQL-like program in a declarative manner. It can achieve superior performance by leveraging advantages in Project Tungsten. In Spark 1.6, Dataset API was introduced to write a generic program, such as machine learning in a functional manner. It was also designed to achieve superior performance by reusing the advantages in Project Tungsten. The differences between DataFrame and Dataset are not fully understood in the community, and it is worth understanding these differences because it is becoming popular to write programs in Dataset and for a transition of programs from RDD to Dataset.
 
 This session will explore the differences between DataFrame and Dataset using programs that performs the same operations (e.g. filter()). Dr. Ishizaki will give several comparisons from levels of source code, SQL execution plans, SQL optimizations, generated Java code, data representations and runtime performance. He will show performance difference of the programs between DataFrame and Dataset, and will identify the cause of the difference. He will also explain opportunities and approaches to improve performance of Dataset programs by alleviating some of issues.
 
 Learn to understand the differences between DataFrame and Dataset from several views; get to know performance differences of programs, which perform the same computation, by using the DataFrame API and the Dataset API; and understand opportunities to improve performance of programs in the Dataset API.

Session hashtag: #SFdev20"

About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business.
Read more here: https://databricks.com/product/unified-data-analytics-platform

Connect with us:
Website: https://databricks.com
Facebook: https://www.facebook.com/databricksinc
Twitter: https://twitter.com/databricks
LinkedIn: https://www.linkedin.com/company/databricks
Instagram: https://www.instagram.com/databricksinc/ Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner

More from this channel