Li Jin - Improving Pandas and PySpark performance and interoperability with Apache Arrow

3,301 views · Published 21 December 2017 · 24:16 · Indexed 12 November 2026

Channel: PyData · 2017 · Science & Technology

Watch on YouTube

PyData New York City 2017

Slides: https://www.slideshare.net/PyData/improving-pandas-and-pyspark-performance-and-interoperability-with-apache-arrow

Apache Spark has become a popular and successful way for Python programming to parallelize and scale up data processing. However, it's not well integrated with popular Python tools such as Pandas, and often result in poor performance when using Pandas with PySpark. In this talk, we will demonstrate how we improve PySpark performance with Apache Arrow. 00:00 Welcome!
00:10 Help us add time stamps or captions to this video! See the description for details.

Want to help add timestamps to our YouTube videos to help with discoverability? Find out more here: https://github.com/numfocus/YouTubeVideoTimestamps

More from this channel