Friday, March 5, 2010

Google App Engine downtime postmortem, nearly a perfect model for others

Google posted one of the most detailed and well thought out postmortems I've seen to explain what happened around their 2/24/10 App Engine Downtime. Let's run it through the gauntlet:

Prerequisites:
  1. Admit failure - Yes
  2. Sound like a human - Yes, more in some sections then others
  3. Have a communication channel - Yes, the Google App Engine Downtime Notify group. Ideally it would have been linked to from the App Engine System Status Dashboard as well.
  4. Above all else, be authentic - Yes
Requirements:
  1. Start time and end time of the incident - Yes, including GMT time, and a highly detailed timeline of the entire event
  2. Who/what was impacted - Partly, and also partly covered during the actual incident
  3. What went wrong - Yes, yes, and yes! Incredible amount of detail here.
  4. Lessons learned - Yes! Not only are there five specific action items, they are also introducing new architectural changes and customer choice as a result.
Bonus:
  1. Details on the technologies involved - Somewhat
  2. Answers to the Five Why's - No
  3. Human elements - heroic efforts, unfortunate coincidences, effective teamwork, etc - Partial
  4. What others can learn from this experience - Partial
Takeaways and thoughts:
  1. A vast majority of the issues were training related. This is an important lesson: all of the technology and process in the world won't help you if your on-call team is unaware of what to do. This is especially true during the stress of a large incident. Follow the advice of Google and run regular on-call drills (including rare issues), keep documentation updated, and give on-call people the authority to make decisions on the spot.
  2. Extremely impressed with the decision to use this opportunity to improve their service by giving their customers the choice between datastore performance and reliability. This is a perfect example of turning downtime into a positive.
  3. Interesting insight into their process of detecting an incident to communicating externally. From the start of the incident at 7:48am to the first external communication at 8:01am is not too shabby. Not sure why it took so long to post the update to the downtime forum and the health status dashboard (8:36am).
  4. The amount of time and thought that went into this postmortem shows how much Google is concerned about their service, and impressions around its reliability.
What could be improved:
  1. External communication could be faster. No reason not to post something as soon as the investigation begins, not to mention posting to the forum dedicated to downtime notifications and the health status dashboard immediately. When the incident started the dashboard had very limited data, which should be automatic and real-time.
  2. A post to this postmortem from the health status dashboard would make it a lot easier to find. I didn't see this until someone sent it to me.
  3. Timelines and concrete deliverables on the changes (e.g. on-call training sessions, documentation updates, new datastore feature release) would give us more confidence that things will actually change.

4 comments:

  1. Google was unable to post in dashboard as it was down and I imagine they had issues with its admin interface. The live health data stopped appearing for majority of parameters, while the AppEngine Status Dashboard sometimes promptly appeared within downtime incident.

    The Status Dashboard have to be the most reliable part in issues like this, but it was not due to dependency upon AppEngine infrastructure.

    ReplyDelete
  2. 1. goof up with help from a power outage
    2. publish details of goof up, such that fandom hype up over this act itself, and everyone forgets about the goof up
    3. ...
    4. profit

    ReplyDelete
  3. Coach Outlet http://www.coachstoreoutlet.us.org

    Christian Louboutin Shoes http://www.christianlouboutinshoe.eu.com

    Valentino Shoes http://www.valentinoshoes.eu.com

    Michael Kors Outlet http://www.michaelkorsoutlet.us.org

    Coach Factory Outlet http://www.coachfactoryoutlet.us.org

    Coach Outlet Online http://www.coachoutletonlinestore.us.org

    Coach Purses http://www.coachpurse.us.com

    Kate Spade Outlet http://www.katespadefactoryoutlet.us.com

    Toms Shoes http://www.toms-shoes.name

    Hermes Belts http://www.hermesbelts.us.org

    Louis Vuitton http://www.louisvuitton-outletstores.us.com

    Fendi Handbags http://www.fendibelt.us.com

    Giuseppe Shoes http://www.giuseppezanotti.eu.com

    Michael Kors Outlet http://www.michael-kors-outlet.us.com

    Stephen Curry Shoes http://www.stephencurryshoes.us.com

    Salomon Shoes http://www.salomonshoes.eu.com

    North Face Outlet http://www.northfacesoutlet.in.net

    Coach Outlet http://www.outletcoach.in.net

    North Face Outlet http://www.northfacesoutlet.us.com

    Burberry Outlet http://www.burberryoutletstore.us.org

    North Face Outlet http://www.north-faceoutlets.in.net

    North Face Jackets http://www.outletnorthface.in.net

    Skechers Shoes http://www.skechersshoes.us.com

    Toms Outlethttp://www.tomsoutletstore.us.org

    North Face Outlet http://www.northfacestoreoutlet.us.org

    Nike Air Max http://www.nikeoutlet.us.org

    Nike Hoodies http://www.nikehoodies.us.com

    Marc Jacobs Handbags http://www.marcjacobshandbags.name

    Marc Jacobs Outlet http://www.marcjacobsoutletstore.us.com

    Jimmy Choo Shoes http://www.jimmy-chooshoes.us.com

    Jimmy Choos http://www.jimmychoos.in.net

    Burberry Belt http://www.burberrybelt.us.com

    Louis Vuitton Belt http://www.louisvuitton-belt.com

    Salvatore Ferragamo http://www.ferragamo-belt.us.com

    Marc Jacobs Handbags http://marcjacobshandbagsoutlet.blog.com

    Lululemon Outlet http://www.lululemonsoutlet.us.com

    True Religion Outlet http://www.truereligion-outlets.in.net

    Tommy Hilfiger http://www.tommyhilfigeroutlet.us.com

    Michael Kors Outlet http://www.outletmichaelkors.eu.com

    Coach Outlet http://www.outletcoach.eu.com

    Red Bottoms http://www.redbottom.in.net

    Kevin Durant Shoes http://www.kevindurantshoes.us.org

    New Balance Outlet http://www.newbalanceoutlet.in.net

    Adidas Outlet http://www.adidasoutlet.in.net

    Coach Outlet Online http://www.coachoutlet-online.eu.com

    Stephen Curry Jersey http://www.stephencurryjersey.us.com

    Vans Outlet http://www.vansoutlet.us.com

    Ralph Lauren Outlet http://www.ralphlauren-outletstore.in.net

    True Religion Outlet http://www.true-religionoutlets.in.net

    ED Hardy Outlet http://www.edhardyoutlet.in.net

    North Face Outlet http://www.northfaceoutlet.eu.com

    UGG Outlet http://www.uggoutlet.us.org

    UGG Outlet http://www.outletugg.in.net

    North Face Outlet http://www.northface--outlet.us.com

    Ugg Boots Sale http://www.uggbootssale.in.net

    UGGS For Women http://www.uggsforwomen.us.com

    Skechers Go Walk http://www.skechersgowalk.us.com

    Adidas Yeezy Boost http://www.adidasyeezyboost.in.net

    Adidas Yeezy http://www.adidasyeezy.us.com

    Adidas NMD http://www.adidas-nmd.us.org

    Coach Outlet http://www.coachoutletinc.us.com

    ReplyDelete