Mnemosyne: a distributed bitmapped indexing layer for big data with Phil Messenger
1,203 views · Published 17 May 2018 · 52:21 · Indexed 10 November 2026
Channel: Devoxx · 2018 · Science & Technology
The Brandwatch Audiences product allows adhoc queries joining hundreds of millions of social network profiles with billions of post and tens of billions of follower graph edges, all updated in realtime at thousands of transactions per second. Face with scalability limitations in our original data backend, we opted to build Mnemosyne, our own distributed indexing layer. Fusing succinct data structures, free text search, in memory computing with the JVM, CUDA and Kafka, the final system is able to ingest millions of entities a second whilst still answering complex queries. This talk is the story of this build, diving into how Mnemosyne works and revealing some surprising things we learned along the way. We'll cover CAP theorem trade-offs, how brute force approaches are sometimes better than indexes, the data structures and techniques required to sort billions of records in milliseconds, how GPU's can solve unexpected problems and how to do all this on the JVM.
More from this channel
-
32:13
Writing Groovy AST Transformations: Getting Practical in an hour! by Baruch Sadogursky
-
25:58
TweetWallFX - Behind the Scenes by Sven Reimers
-
29:44
Second generation "workflow managers" for big data by Alex Van Boxel
-
29:13
Secrets are secrets. Please maintain them as secrets by Alex Soto
-
2:13:40
JavaScript build pipelines with Gulp.js by Stefan Baumgartner
-
2:04:09
Powerful Metaprogramming Techniques With Groovy by Jeff Brown
-
2:38:32
Distributed Machine Learning 101 using Apache Spark from a Browser by Xavier Tordoir/Andy Petrella
-
2:36:21
Caching 101: Caching on the JVM (and beyond) by Alex Snaps/Louis Jacomet