Autoplay Next video from Max length

Analytics on Large Scale, Unstructured, Dynamic Data using Lambda Architecture

1,271 views · Published 13 September 2014 · 33:16 · Indexed 27 October 2026

Channel: Hasgeek TV · 2014 · Science & Technology

Lambda architecture, at its core, is a set of architecture principles that allows both batch and real-time or stream data processing to work together while building immutability, recomputation and human fault tolerance into the system.

It has three layers - batch, serving and speed:

The batch layer is responsible for computing arbitrary views on the master data. Our master data is an immutable store in HDFS and we compute views using a series of Map Reduce jobs using Scalding and Spark. Our batch system runs recomputation every day on our entire data set.

The serving layer indexes and exposes precomputed views to be queried ad-hoc with low latency. We use HBase, Solr and our own inhouse inmemory implementation for the serving layer.

The speed layer deals only with new data and compensates for the high latency updates of the serving layer by creating realtime views. Our real time latency requirements are in few hours and not in seconds, which allows us to use a micro-batch architecture that is a stripped down version of our batch layer and uses the same technologies.

To get the final result, the batch and realtime views must be queried and the results merged together.

In this talk, the speaker explains the following:
Why Lambda Architecture? What problems did it solve for us?
Technical Challenges encountered in building the lambda architecture
Schema Evolution
HDFS Small Files Issue
Code re-use between batch and real time systems
Modeling the data pipelines for each layer
Open problems

Analytics on Large Scale, Unstructured, Dynamic Data using Lambda Architecture

More from this channel