Li Jin - Improving Pandas and PySpark performance and interoperability with Apache Arrow
3,301 views · Published 21 December 2017 · 24:16 · Indexed 12 November 2026
Channel: PyData · 2017 · Science & Technology
PyData New York City 2017 Slides: https://www.slideshare.net/PyData/improving-pandas-and-pyspark-performance-and-interoperability-with-apache-arrow Apache Spark has become a popular and successful way for Python programming to parallelize and scale up data processing. However, it's not well integrated with popular Python tools such as Pandas, and often result in poor performance when using Pandas with PySpark. In this talk, we will demonstrate how we improve PySpark performance with Apache Arrow. 00:00 Welcome! 00:10 Help us add time stamps or captions to this video! See the description for details. Want to help add timestamps to our YouTube videos to help with discoverability? Find out more here: https://github.com/numfocus/YouTubeVideoTimestamps
More from this channel
-
31:23
Kostas Perifanos - Authorship Attribution Using Python
-
31:08
Prash Majmudar - Measuring the Digital Economy Using Big Data
-
36:26
Ian Ozsvald - The High Performance Python Landscape
-
37:48
Mike Muller - Shared Memory Parallelism with Python
-
35:01
Martin Goodson - Most Winning A/B Test Results are Illusory
-
30:19
Mark Basham - Python for High Throughput Science
-
1:13:42
David Cournapeau & Simon Jagoe - Narrowing the Gap Between Array Computing and Databases
-
31:38
Emlyn Clay - Phython, Pharmaceuticals & Discovery