Autoplay Next video from Max length

Large Scale Feature Aggregation Using Apache Spark - Pulkit Bhanot

1,227 views · Published 19 September 2018 · 29:27 · Indexed 6 November 2026

Channel: Databricks · 2018 · Science & Technology

Pulkit Bhanot, a tech lead at Uber, and Amit Nene, a manager and founding member of Uber’s Risk Data Team share about large scale feature aggregation. Aggregation based features account for a quarter of the several thousands of features used by the ML-based decisioning system by the Risk team at Uber. They observed several repetitive, cumbersome steps needed for onboarding a feature, every single time. To accelerate developer velocity, and to enable Feature Engineering at scale, they decided to develop a generic spark based infrastructure to simplify the process to no more than a simple spec file, containing a parameterized query, along with some metadata on where the feature should be aggregated and stored.

Learn more here: https://databricks.com/blog/2017/01/19/real-time-streaming-etl-structured-streaming-apache-spark-2-1.html

About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business.
Read more here: https://databricks.com/product/unified-data-analytics-platform

Connect with us:
Website: https://databricks.com
Facebook: https://www.facebook.com/databricksinc
Twitter: https://twitter.com/databricks
LinkedIn: https://www.linkedin.com/company/databricks
Instagram: https://www.instagram.com/databricksinc/ Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner

Large Scale Feature Aggregation Using Apache Spark - Pulkit Bhanot

More from this channel