Simrat Hanspal – Looking under the hood - demystifying data tools

332 views · Published 31 July 2016 · 18:14 · Indexed 19 October 2026

Channel: Hasgeek TV · 2016 · Science & Technology

Watch on YouTube

Slides: https://docs.google.com/presentation/d/1azppoHqOWqxwAqA6yl337XlhAHGOL0ojNhanlCPPoKg/edit#slide=id.g15f04aaed7_0_46

The goal of this talk is to help build an understanding of the performances of the following packages -
R Dataframe
R data.table
Pandas
Numpy
PySpark RDDs
PySpark Dataframes
RedShift
While these packages are operating in different but intersecting realms of use cases, depending on the cardinality of the data and the operations that will be performed on it, some are more suited than others for the task at hand. Before making the plunge into ‘Big Data’ it is important to understand the point at which one is trying to kill an ant with a sledgehammer. This talk outlines our attempts at grasping this. We will not evaluate a plethora of tools, just the ones that we considered for our requirements.

More from this channel