Concurrent Stream Processing - David McNeil
1,238 views · Published 7 January 2013 · 41:18 · Indexed 27 October 2026
Channel: ClojureTV · 2013 · Science & Technology
The foundation of our query processing engine is a concurrent data stream processor. This processor is characterized by the need to efficiently perform parallel, non-blocking processing of multiple data streams which are too large to fit in memory. Many such executions need to be executed simultaneously and fairly. The ideas in this talk are relevant to those who work with large scale, parallel data processing within the scope of a single process. A central theme of the talk is the creation of layers of abstractions to eventually create a language tailored to the problem. The talk discusses characteristics of the concurrent stream processor including: core data structures to represent processing nodes connected by data streams, processing plans represented as s-expressions, compiling s-expressions into processing nodes and streams, processing plan optimizations via s-expression manipulations, concurrent processing via a fork/join pool, facilities for debugging and cancelling executions and using the data stream processor as the core of a federated query processor.
More from this channel
-
18:41
Fertile Ground, the Roots of Clojure - Michael Fogus
-
39:41
Lisp, Functional Programming, and the State of Flow - Tom Faulhaber
-
39:17
Protocol XIII: Clojure Protocols Explained - Sean Devlin
-
5:23
Aleph: A Framework for AsynchronousCommunication - Zach Tellman
-
26:57
Clojure and Android - Daniel Solano Gomez
-
38:24
Performance in the Wild - Craig Andera
-
42:20
Ousterhout's Dichotomy Isn't (Part 2 of the Simplicity/Power/Focus Series) - Stuart Halloway
-
42:08
From Linear to Incremental - Christophe Grand