Look Who it Was Random video

Classifying Text in Money Transfers with Apache Spark - Jose A. Rodriguez-Serrano

2,783 views · Published 3 November 2017 · 32:02 · Indexed 30 October 2026

Channel: Databricks · 2017 · Science & Technology

Watch on YouTube

"At BBVA (second biggest bank in Spain), every money transfer a customer makes goes through an engine that infers a category from its textual description. This engine has been developed in Spark, mixes MLLib and own implementations, and is currently into production serving more than 5M customers daily. In our proposed presentation, we plan to describe the process undergone by the Data Science team. This includes the problem (classify 700K daily transfers by its text), the data science challenges, the algorithmic and engineering solution, and the achievements and learnings. To make the talk practical and focused, we will walk through our best performing pipeline, which uses (i) word2vec embeddings to represent words, (ii) a pooling algorithm to aggregate them into sentence embeddings, (iii) a supervised classifier. We believe this is relevant because it mixes (i) off-the-shelf MLLib funcions, such as the word2vec embeddings, (ii) components that build on Mllib and were adapted (e.g. a calibrated multic-class logistic regression which outputs probabilities) and (iii) an own implementations of the pooling algorithm, which is known as the Vector of Locally Aggregated Descriptors (VLAD). We highlight that we are not aware of the previous application of word2vec with VLAD in NLP, so this would introduce a novelty. The relevant audience are data scientists, engineers, team leaders and executives interested in Spark and seeking examples of machine learning deployments in a real-world productive environment. The main takeaways will be: (i) it is relatively simple to build a text classification system in Spark, (ii) with extra effort one it is feasible to build state-of-the-art, productive solutions. We will mention the problems we found in practice, such as how to design a training corpus to maximize precision, not recall, or how we designed the system against ""catastrophic"" classification mistakes.


Session hashtag: #EUds7"

About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business.
Read more here: https://databricks.com/product/unified-data-analytics-platform

Connect with us:
Website: https://databricks.com
Facebook: https://www.facebook.com/databricksinc
Twitter: https://twitter.com/databricks
LinkedIn: https://www.linkedin.com/company/databricks
Instagram: https://www.instagram.com/databricksinc/ Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner

More from this channel