Autoplay Next video from Max length

Analytical DBMS to Apache Spark Auto Migration Framework with Edward Zhang and Lipeng Zhu (eBay)

341 views · Published 11 October 2018 · 35:54 · Indexed 10 November 2026

Channel: Databricks · 2018 · Science & Technology

eBay has been using Analytical DBMS (ADBMS) data warehouse solution for over a decade, there are millions of batch queries running every day against 6000+ key DW tables, which contains over 22PB data (compressed) and still keeps booming every year. Based upon that, data services and products enables eBay business decisions and site features, so it has to be always available and accurate.

Apache Spark provides an open source and more scalable way of solution for such amount of data. Starting from beginning of this year, eBay has been working on migrating ADBMS batch workload to Spark, about 90% of them migrated in automatic way. Our team is leading the automation tools and pipeline to commit the accomplishment within this year.

In today’s session, we will introduce:

1. Tool sets which enables the auto migration engine: including metadata services, SQL convertor, Table/View generator, data mover, optimizer, pipeline generator, data validator, workflow controller many not only contributes in auto migration but also enables development work of individual engineers

2. End to end auto migration steps till cut over on production, starting from initializing on dev environment, unit test, data validation, integration test, release, parallel run, monitoring and cut over

About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business.
Read more here: https://databricks.com/product/unified-data-analytics-platform

Connect with us:
Website: https://databricks.com
Facebook: https://www.facebook.com/databricksinc
Twitter: https://twitter.com/databricks
LinkedIn: https://www.linkedin.com/company/databricks
Instagram: https://www.instagram.com/databricksinc/ Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner

Analytical DBMS to Apache Spark Auto Migration Framework with Edward Zhang and Lipeng Zhu (eBay)

More from this channel