From pipelines to graphs Escape the tyranny of the shell’s linear pipelines with dgsh

160 views · Published 10 March 2018 · 16:02 · Indexed 16 November 2026

Channel: FOSDEM · 2018 · Science & Technology

Watch on YouTube

by Diomidis Spinellis

At: FOSDEM 2017

The Unix [dgsh](http://www.spinellis.gr/sw/dgsh/) shell provides an expressiveway to construct sophisticated and efficient data processing pipelines usingstandard Unix tools, as well as third-party and custom-built components. Dgshallows the specification of pipelines of non-uniform non-linear operations.For example _tee_ can feed three processes whose output can then be collectedby _paste_. The pipelines form a directed acyclic process graph, which istypically executed by multiple processor cores, thus increasing the task'sprocessing throughput. We will see how to use _dgsh_ in practice through anumber of general data processing and domain-specific examples, and how toadapt tools for use with _dgsh_.


Room: H.2215 (Ferrer)
Scheduled start: 2017-02-04 16:00:00

More from this channel