Autoplay Next video from Max length

Making PySpark Amazing—From Faster UDFs to Graphing! (Holden Karau and Bryan Cutler)

952 views · Published 24 September 2018 · 30:50 · Indexed 12 November 2026

Channel: Databricks · 2018 · Science & Technology

Holden is a transgender Canadian open source developer advocate @ Google with a focus on Apache Spark, BEAM, and related "big data" tools. Bryan Cutler is a software engineer at IBM's Spark Technology Center, where he works on big data analytics and machine learning systems.

PySpark is getting awesomer in Spark 2.3 with vectorized UDFs, and there is even more wonderful things on the horizon (and currently available as WIP packages). This talk will start by illustrating how to use PySpark’s new vectorized UDFs to make ML pipeline stages. Since most of us use Python in part because of its wonderful libraries, like pandas, numpy, and antigravity*, it’s important to be able to make sure that our dependencies are available on our cluster. Historically there’s been a few If there is time near the end we will talk about how to expose your Python code to Scala so everyone can use your fancy deep learning code (if you want them to). *Ok maybe not a real thing, but insert super specialized domain specific library you use instead 🙂

To learn more: https://docs.databricks.com/spark/latest/dataframes-datasets/introduction-to-dataframes-python.html
About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business.
Read more here: https://databricks.com/product/unified-data-analytics-platform

Connect with us:
Website: https://databricks.com
Facebook: https://www.facebook.com/databricksinc
Twitter: https://twitter.com/databricks
LinkedIn: https://www.linkedin.com/company/databricks
Instagram: https://www.instagram.com/databricksinc/ Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner

Making PySpark Amazing—From Faster UDFs to Graphing! (Holden Karau and Bryan Cutler)

More from this channel