Autoplay Next video from Max length

Building Competing Models Using Apache Spark DataFrames - Abdulla Al-Qawasmeh

474 views · Published 12 June 2017 · 29:24 · Indexed 15 October 2026

Channel: Databricks · 2017 · Science & Technology

"Credit Karma serves over 60 million members wth vastly different credit profiles. Therefore, it is critically important that they recommend financial products that are tailored to each unique member, and that they do so efficiently at scale. To solve this recommendation problem, they build predictive models that compete against each other both vertically (in different stages of the conversion pipeline) and horizontally (for showing different offers) to provide the final recommendation.

In this session, Al-QawasmehI will demonstrate how Credit Karma builds predictive recommendation models using Spark DataFrames, and how they provide type safety while using DataFrames. See how Credit Karma's choice of metric evaluation helps them calibrate the models to obtain the best global result, and hear about lessons learned when they scaled their model development environment to handle Terabyte-scale data with thousands of features.

You'll leave with two key takeaways: an understanding of how model calibration plays a crucial role in the performance of a recommendation system redit Karma builds predictive recommendation models using Spark DataFrames and how we provide type safety while using DataFrames. I will show how our choice of metric evaluation helps us calibrate the models to obtain the best global result. I will also touch on lessons learned when we scaled our model development environment to handle Terabyte-scale data with thousands of features.
Participants will walk away with two takeaways: an understanding of how model calibration plays a crucial role in the performance of a recommendation system when competing models are involved, and an example of how to scale Spark.ml to work with Terabyte-scale data.

Session hashtag: #SFml11"

About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business.
Read more here: https://databricks.com/product/unified-data-analytics-platform

Connect with us:
Website: https://databricks.com
Facebook: https://www.facebook.com/databricksinc
Twitter: https://twitter.com/databricks
LinkedIn: https://www.linkedin.com/company/databricks
Instagram: https://www.instagram.com/databricksinc/ Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner

Building Competing Models Using Apache Spark DataFrames - Abdulla Al-Qawasmeh

More from this channel