Prerequisites:
- Admit failure - Yes
- Sound like a human - Yes, more in some sections then others
- Have a communication channel - Yes, the Google App Engine Downtime Notify group. Ideally it would have been linked to from the App Engine System Status Dashboard as well.
- Above all else, be authentic - Yes
Requirements:
- Start time and end time of the incident - Yes, including GMT time, and a highly detailed timeline of the entire event
- Who/what was impacted - Partly, and also partly covered during the actual incident
- What went wrong - Yes, yes, and yes! Incredible amount of detail here.
- Lessons learned - Yes! Not only are there five specific action items, they are also introducing new architectural changes and customer choice as a result.
Bonus:
- Details on the technologies involved - Somewhat
- Answers to the Five Why's - No
- Human elements - heroic efforts, unfortunate coincidences, effective teamwork, etc - Partial
- What others can learn from this experience - Partial
Takeaways and thoughts:
- A vast majority of the issues were training related. This is an important lesson: all of the technology and process in the world won't help you if your on-call team is unaware of what to do. This is especially true during the stress of a large incident. Follow the advice of Google and run regular on-call drills (including rare issues), keep documentation updated, and give on-call people the authority to make decisions on the spot.
- Extremely impressed with the decision to use this opportunity to improve their service by giving their customers the choice between datastore performance and reliability. This is a perfect example of turning downtime into a positive.
- Interesting insight into their process of detecting an incident to communicating externally. From the start of the incident at 7:48am to the first external communication at 8:01am is not too shabby. Not sure why it took so long to post the update to the downtime forum and the health status dashboard (8:36am).
- The amount of time and thought that went into this postmortem shows how much Google is concerned about their service, and impressions around its reliability.
What could be improved:
- External communication could be faster. No reason not to post something as soon as the investigation begins, not to mention posting to the forum dedicated to downtime notifications and the health status dashboard immediately. When the incident started the dashboard had very limited data, which should be automatic and real-time.
- A post to this postmortem from the health status dashboard would make it a lot easier to find. I didn't see this until someone sent it to me.
- Timelines and concrete deliverables on the changes (e.g. on-call training sessions, documentation updates, new datastore feature release) would give us more confidence that things will actually change.
Google was unable to post in dashboard as it was down and I imagine they had issues with its admin interface. The live health data stopped appearing for majority of parameters, while the AppEngine Status Dashboard sometimes promptly appeared within downtime incident.
ReplyDeleteThe Status Dashboard have to be the most reliable part in issues like this, but it was not due to dependency upon AppEngine infrastructure.
1. goof up with help from a power outage
ReplyDelete2. publish details of goof up, such that fandom hype up over this act itself, and everyone forgets about the goof up
3. ...
4. profit
ray ban sunglasses outlet
Deleteair jordan shoes
gucci handbags
oakley sunglasses wholesale
michael kors outlet online
kate spade outlet
ugg canada
polo ralph lauren
oakley outlet
canada goose outlet
michael kors outlet
louboutin shoes
nike air max pas cher
michael kors outlet online
burberry sale
lebron 13 shoes
louis vuitton handbags
air jordan 8
ladies cartier watches
adidas outlet store
cheap jerseys
coach outlet
ugg italia
coach outlet store online
coach factory outlet
coach outlet store online
ugg outlet
longchamp outlet
cheap uggs
christian louboutin wedges
20168.17chenjinyan
Coach Outlet http://www.coachstoreoutlet.us.org
ReplyDeleteChristian Louboutin Shoes http://www.christianlouboutinshoe.eu.com
Valentino Shoes http://www.valentinoshoes.eu.com
Michael Kors Outlet http://www.michaelkorsoutlet.us.org
Coach Factory Outlet http://www.coachfactoryoutlet.us.org
Coach Outlet Online http://www.coachoutletonlinestore.us.org
Coach Purses http://www.coachpurse.us.com
Kate Spade Outlet http://www.katespadefactoryoutlet.us.com
Toms Shoes http://www.toms-shoes.name
Hermes Belts http://www.hermesbelts.us.org
Louis Vuitton http://www.louisvuitton-outletstores.us.com
Fendi Handbags http://www.fendibelt.us.com
Giuseppe Shoes http://www.giuseppezanotti.eu.com
Michael Kors Outlet http://www.michael-kors-outlet.us.com
Stephen Curry Shoes http://www.stephencurryshoes.us.com
Salomon Shoes http://www.salomonshoes.eu.com
North Face Outlet http://www.northfacesoutlet.in.net
Coach Outlet http://www.outletcoach.in.net
North Face Outlet http://www.northfacesoutlet.us.com
Burberry Outlet http://www.burberryoutletstore.us.org
North Face Outlet http://www.north-faceoutlets.in.net
North Face Jackets http://www.outletnorthface.in.net
Skechers Shoes http://www.skechersshoes.us.com
Toms Outlethttp://www.tomsoutletstore.us.org
North Face Outlet http://www.northfacestoreoutlet.us.org
Nike Air Max http://www.nikeoutlet.us.org
Nike Hoodies http://www.nikehoodies.us.com
Marc Jacobs Handbags http://www.marcjacobshandbags.name
Marc Jacobs Outlet http://www.marcjacobsoutletstore.us.com
Jimmy Choo Shoes http://www.jimmy-chooshoes.us.com
Jimmy Choos http://www.jimmychoos.in.net
Burberry Belt http://www.burberrybelt.us.com
Louis Vuitton Belt http://www.louisvuitton-belt.com
Salvatore Ferragamo http://www.ferragamo-belt.us.com
Marc Jacobs Handbags http://marcjacobshandbagsoutlet.blog.com
Lululemon Outlet http://www.lululemonsoutlet.us.com
True Religion Outlet http://www.truereligion-outlets.in.net
Tommy Hilfiger http://www.tommyhilfigeroutlet.us.com
Michael Kors Outlet http://www.outletmichaelkors.eu.com
Coach Outlet http://www.outletcoach.eu.com
Red Bottoms http://www.redbottom.in.net
Kevin Durant Shoes http://www.kevindurantshoes.us.org
New Balance Outlet http://www.newbalanceoutlet.in.net
Adidas Outlet http://www.adidasoutlet.in.net
Coach Outlet Online http://www.coachoutlet-online.eu.com
Stephen Curry Jersey http://www.stephencurryjersey.us.com
Vans Outlet http://www.vansoutlet.us.com
Ralph Lauren Outlet http://www.ralphlauren-outletstore.in.net
True Religion Outlet http://www.true-religionoutlets.in.net
ED Hardy Outlet http://www.edhardyoutlet.in.net
North Face Outlet http://www.northfaceoutlet.eu.com
UGG Outlet http://www.uggoutlet.us.org
UGG Outlet http://www.outletugg.in.net
North Face Outlet http://www.northface--outlet.us.com
Ugg Boots Sale http://www.uggbootssale.in.net
UGGS For Women http://www.uggsforwomen.us.com
Skechers Go Walk http://www.skechersgowalk.us.com
Adidas Yeezy Boost http://www.adidasyeezyboost.in.net
Adidas Yeezy http://www.adidasyeezy.us.com
Adidas NMD http://www.adidas-nmd.us.org
Coach Outlet http://www.coachoutletinc.us.com
Highly qualified information because it can provide useful info thanks.
ReplyDeleteobat herbal psoriasis vulgaris
kobe shoes
ReplyDeletetrue religion outlet
kate spade handbags
coach outlet
birkenstock sandals
true religion outlet
nike free run 5.0
jordan shoes
polo shirts
toms wedges
hzx20170412
شركة المتحدة
ReplyDeleteدليل شركات نقل العفش
شركة نقل اثاث بالمدينة المنورة
شركة نقل اثاث بالرياض
شركة نقل اثاث بجدة
شركة نقل اثاث بمكة
ReplyDeleteشركة نقل اثاث بالطائف
شركة نقل اثاث بينبع
شركة نقل اثاث بالدمام
شركة نقل اثاث بنجران
شركة نقل اثاث بخميس مشيط
ReplyDeleteشركة نقل اثاث بابها
شركة نقل اثاث بحائل
شركة نقل اثاث ببريدة
ReplyDeleteشركة نقل اثاث بالقصيم
شركة نقل اثاث بحفر الباطن
شركة نقل اثاث برابغ
very good article and useful once for my admin and pardon me permission to share articles herein may be useful and helpful Obat amandel bengkak
ReplyDeletemay be useful for all, helpful article once and pardon me permission to share also here :
ReplyDeleteCara menyembuhkan lambung perih
Obat lambung bengkak
Cara menyembuhkan thalasemia
By reading this article I get a lot of lessons and this is very useful . cara menggugurkan kandungan
ReplyDeleteThanks for the very helpful information. Because, I just know there is a very extraordinary article like this, thanks to
ReplyDeleteToko Obat Herbal
Obat Nyeri Sendi
Thanks for the very useful information, If you do not know then we will share the article. Please read because this is very important for us all.
ReplyDeleteObat herbal osteoporosis
Obat Herbal Campak
Obat Sakit Tenggorokan
Obat Luka Diabetes
Obat Herbal Prurigo
Packers and Movers Chandigarh to Delhi
ReplyDeleteMovers and Packers Chandigarh to Gurgaon
Movers and Packers Chandigarh to Kolkata
Movers and Packers Chandigarh to Mumbai
Movers and Packers Chandigarh to Pune
Packers and Movers Bangalore to Ahmedabad
Packers and Movers Bangalore to Chandigarh
Packers and Movers Bangalore to Gurgaon
Packers and Movers Bangalore to Hyderabad
Packers and Movers Bangalore to Lucknow