Look Who it Was Random video

Masking personal data in medical documents - Kornel Lewandowski

408 views · Published 13 November 2017 · 31:38 · Indexed 21 October 2026

Channel: PyData · 2017 · Science & Technology

Watch on YouTube

Description
The medical and pharmaceutical companies create a lot of documents. Analyzing them all could provide meaningful insights. Unfortunately, these texts usually contain personal data and that is a real blocker for passing documents to any external analytics tools. I will present our scalable solution that achieves excellent results and handles a few European languages.

Abstract
Masking personal data in medical documents
Introduction
The medical and pharmaceutical companies create a lot of documents. Analyzing them all could provide meaningful insights. Unfortunately, these texts usually contain personal data and that is a real blocker for passing documents to any external analytics tools. I will present our scalable solution achieving excellent results, which handles this problem and is able to handle few European languages.

Solution overview
Masking the data should be supported by high quality algorithms to maximize the number of true positives and minimize the error rate. Our solution combines a few different techniques from the fields of natural language processing, text mining and advanced data analysis to find as many pieces of personal data as possible and custom filters and taxonomy-based solutions decreasing the number of false positives. Our results are really promising and the process performed on the documents prepares them to be passed to all Roche's units and even to external companies with respect to the most of data privacy rules.

Technical details
Our solution is highly scalable because we use Hadoop Streaming for parallel and distributed processing. Most of the code is written in Python and uses popular natural language processing tools, models and libraries such as NTLK and CoreNLP.

www.pydata.org

PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R. 

PyData conferences aim to be accessible and community-driven, with novice to advanced level presentations. PyData tutorials and talks bring attendees the latest project features along with cutting-edge use cases. 00:00 Welcome!
00:10 Help us add time stamps or captions to this video! See the description for details.

Want to help add timestamps to our YouTube videos to help with discoverability? Find out more here: https://github.com/numfocus/YouTubeVideoTimestamps

More from this channel