Look Who it Was Random video

DEF CON 17 - Joey Calca and Ryan Anguiano - Hadoop Apaches Implementation of Googles MapReduce

147 views · Published 20 November 2013 · 37:26 · Indexed 19 October 2026

Channel: DEFCONConference · 2013 · Science & Technology

Watch on YouTube

Hadoop: Apache's Open Source Implementation of Google's MapReduce Framework
Joey Calca Hacked Existence Team
Ryan Anguiano Hacked Existence Team

This presentation will begin with a brief overview of Google's Map/Reduce Framework. Map/Reduce is built to analyze extremely large datasets. We will first look at what a Mapper and Reducer are, the inputs they take and the outputs they generate. From there, we will look at the open source java based implementation of the Map/Reduce Framework by the Apache Team's Hadoop Project. Since Hadoop is Java based, we will then look at using the Hadoop framework in order to build Mappers and Reducers in Python, as well as running Mappers written in the AWK scripting language. A brief comparison of compile times and efficiencies between the three will be shown, as well as the results from running our code on ASU's Saguaro Cluster. After that, we will brush over HBase, the Hadoop equivalent to Google's BigTable, a non-relational database for Map/Reduce. Finally, we will look at some demo code, including a machine learning algorithm based on the Netflix Prize Dataset, a 2 gigabyte dataset of movie ratings from the Netflix Database.

Also, we will not present on, but will include source code from different team's projects, including Map/Reduce programs for image analysis and recognition, analyzing air traffic data, analyzing package delivery systems for use with swarm theory and a Map/Reduce program that analyzes patterns in large literature as a response to "The Bible Code", most of which use public datasets as inputs.

Joey Calca is a member of the Hacked Existence Team. He is a recent graduate from Arizona State University with a Business Degree in Computer Information Systems and a background in Computer Systems Engineering. He has done various consulting projects including the design and implementation of large scale multimedia delivery systems, and is currently pursuing Cloud Computing as the next big technology boom.

Ryan Anguiano is a member of the Hacked Existence team. He is also a web application developer employed by The Forum Agency (theforumagency.com), having over 7 years of experience in the industry. Ryan is currently an undergraduate student at ASU, and is leading the web development team for MH+L Magazine (mhlmag.com).

For copies of the slides and additional materials please see the DEF CON 17 Archive here: https://defcon.org/html/links/dc-archives/dc-17-archive.html

More from this channel