Autopsy of an automation disaster
39 views · Published 12 March 2018 · 23:16 · Indexed 19 November 2026
Channel: FOSDEM · 2018 · Science & Technology
by Jean-François Gagné At: FOSDEM 2017 You’ve deployed automation, enabled automatic master failover and tested itmany times: great, you can now sleep at night without being paged by a failingserver. However, when you wake up in the morning, things might not have gonethe way you expect. This talk will be about such a surprise. Once upon a time, a failure brought down a master. Automation kicked in andfixed things. However, a fancy failure, combined with human errors, with anedge-case recovery, and a lack of oversight in automation, lead to a split-brain. This talk will go into details about the convoluted - but still realworld - sequence of events that lead to this disaster. I will cover what couldhave avoided the split-brain and what could have make things easier to fix it. Room: H.1309 (Van Rijn) Scheduled start: 2017-02-04 12:05:00