Debugging machine learning - Michał Łopuszyński

1,297 views · Published 13 November 2017 · 30:39 · Indexed 24 October 2026

Channel: PyData · 2017 · Science & Technology

Watch on YouTube

Description
Debugging machine learning apps is hard. I feel that this topic is important, however, relatively rarely touched compared to, e.g., latest models or new interesting ML applications. In this talk, I will try try to fill that gap by discussing best practices, recommendations and my own experience on the subject.

Abstract
Recently, OpenAI examined ten popular reinforcement learning algorithms reimplementation and found that six (!) of them contained significant non-breaking bugs [1]. This illustrates the fact that debugging and testing machine learning software is difficult and requires extra attention.

In particular, machine learning (ML) driven apps exhibit rich spectrum of failure types, specific to the domain. ML systems can be diagnosed:

not to work at all (e.g., providing nearly random outcomes);

to work with (say) 70% accuracy, but with user expectation of much better performance;

to work on the first batch of data, but miserably failing in production, when new data arrive;

to initially work very well in production, but deteriorating significantly with time.

In this talk, I will touch on each of the above scenario, describing possible pitfalls and recommended engineering practices, as well as providing real-life examples.

[1] https://blog.openai.com/openai-baselines-dqn/

www.pydata.org

PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R. 

PyData conferences aim to be accessible and community-driven, with novice to advanced level presentations. PyData tutorials and talks bring attendees the latest project features along with cutting-edge use cases.

Debugging machine learning. Mostly for profit, but with a bit of fun too.
Michał Łopuszyński 00:00 Welcome!
00:10 Help us add time stamps or captions to this video! See the description for details.

Want to help add timestamps to our YouTube videos to help with discoverability? Find out more here: https://github.com/numfocus/YouTubeVideoTimestamps

More from this channel