Autoplay Next video from Max length

Writing and Deploying Interactive Applications Based on Apache Spark

4,977 views · Published 11 October 2018 · 30:04 · Indexed 19 October 2026

Channel: Databricks · 2018 · Science & Technology

Talk by: Matteo Pelati and Chandra Sekhar Saripaka (DBS Bank)

Very often it is useful to create Spark applications which runs in interactive mode rather than batch mode. Think, for instance, in Spark notebook. This requires exposing a UI and Rest APIs which will interact with the core spark engine. On top of this, in an enterprise environment, it is always necessary to integrate with authentication and authorization services, in order to impersonate the correct user who is logging in and accessing the data interactively.

In this talk we will discuss how we have built an interactive Spark application which is fully integrated with our enterprise environment at DBS Bank. We will showcase the entire architecture of the framework we have built, showcasing how we embedded REST APIs and a web UI, how we can provision YARN containers dynamically and impersonating the proper user using Kerberos authentication, and how we perform service discovery across the various YARN instances to make the Spark engine accessible from the web.

At the end of the talk, the audience will have a clear understanding of how an interactive enterprise application can be built on top of Spark, and will be able to follow a similar design to implement and deploy interactive applications in their enterprise environment.

About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business.
Read more here: https://databricks.com/product/unified-data-analytics-platform

Connect with us:
Website: https://databricks.com
Facebook: https://www.facebook.com/databricksinc
Twitter: https://twitter.com/databricks
LinkedIn: https://www.linkedin.com/company/databricks
Instagram: https://www.instagram.com/databricksinc/ Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner

Writing and Deploying Interactive Applications Based on Apache Spark

More from this channel