Nezha: A Kubernetes Native Big Data Accelerator For Machine Learning - Huamin Chen & Yuan Zhou
229 views · Published 16 December 2018 · 30:23 · Indexed 28 October 2026
Channel: CNCF [Cloud Native Computing Foundation] · 2018 · Film & Animation
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects Nezha: A Kubernetes Native Big Data Accelerator For Machine Learning - Huamin Chen, Red Hat & Yuan Zhou, Intel Large training datasets used by machine learning frameworks, such as Kubeflow, are usually stored in low cost and high capacity S3 or Google Cloud Storage. However, S3’s rating limiting and slow data downloading significantly challenges training performance and limits compute scalability. We introduce NeZha and explain how it improves Kubeflow’s training. Nezha is an open source, community driven, and highly collaborative project, contributed by storage and big data engineers. Nezha is based on Kubernetes Initializer: it rewrites Pod spec, adds a sidecar S3 cache, and redirects Pod to use local cache to accelerate. Nezha is self contained and easy to use. It does not require modification to existing applications or user visible Pod changes. Nezha improves big data application performance. Our initial Kubeflow benchmark results using MNIST dataset shows NeZha achieves ~2x speedup. To learn more: https://sched.co/GrWB
More from this channel
-
3:33
Closing Remarks - Dan Kohn, Executive Director, Cloud Native Computing Foundation
-
18:53
2nd Day Operations: Your Checklist for Cloud Native World - Ben Hindman, Mesosphere
-
35:01
Containerizing PostgreSQL and Making it Cloud Native Ready - Jeff McCormick, Crunchy Data Solutions
-
23:56
Scaling Containers from Sandbox to Production - Courtesy of CNCF - Angel Diaz, IBM
-
5:18
Whats Hot in Containers & OpenStack- Duane De Capite, Cisco
-
44:48
Learning How to Pronounce Kubernetes to Production in 3 Months! by Sheriff Mohamed, GolfNow
-
35:09
Imperative to Declarative Infrastructure by For Developers and Operators by Ashwin Raveendran, eBay
-
42:47
Self-hosted, Scale, and Federation with Kubernetes v1.4 and Beyond by Brandon Philips, CoreOS, Inc.