Look Who it Was Random video

Juha Suomalainen - Visualizing research data: Challenges of combining different datasources

420 views · Published 1 June 2016 · 34:38 · Indexed 20 October 2026

Channel: PyData · 2016 · Science & Technology

Watch on YouTube

PyData Berlin 2016

Your source data has multiple formats? You have multiple API’s to pull data from? This talk will go through some common problems with solutions that you will face when trying to combine multiple different research data sources in programmatic way. We go through a real world web project on that visualizes poverty data with JSON API's, Shapefiles and Excel spreadsheets as data sources.

Introduction

- Give talk goals: This talk aims to give the tools to solve the engineering challenges related to combining different datasources
- Set the context: We use geodata from humanitarian projects as an example, but solutions will apply to other areas as well.
- Go through talk outline

Part 2: Quickly introduce the project

Give the audience idea of real world project in preparation for the part 3
- Show screenshots of the final project
- Go through the used technologies (ESRI shapefiles, geo/topo json, xls, API’s, python libraries)
- Introduce the data pipeline

Part 3: Explain common problems and our solutions for them

This is the meat of the talk, each point introduces problem and suggests at least one solution. Solutions are based on Python technologies - Handling different data formats - How to manage the data sources (validation, automation, etc) - Normalizing units - Mapping problems (different projects may follow different standards for the id’s) - Normalizing data and metadata

Part 4: Wrap up

- Quickly explain how we applied these problems in the project
- Sum up the things you should consider (check-list) 00:00 Welcome!
00:10 Help us add time stamps or captions to this video! See the description for details.

Want to help add timestamps to our YouTube videos to help with discoverability? Find out more here: https://github.com/numfocus/YouTubeVideoTimestamps

More from this channel