Wednesday, September 29, 2010

Case Study: Facebook outage

I'm a bit late to the story (something called a day job getting in the way!) but I can't pass up an opportunity to discuss how Facebook handled the "worst outage [they've] had in over four years".  I blogged about the intra-incident communication the day they had the outage, so let's review the postmortem that came out after they had recovered, and how they handled the downtime as a whole.






Using the "Upside of Downtime" framework (above) as a guide:
  1. Prepare: Much room for improvement. The health status feed is hard to find for the average user/developer, and the information was limited. On the plus side, it exists. Twitter was also used to communicate updates, but again the information was limited.
  2. Communicate: Without a strong foundation create by the Prepare step, you don't have much opportunity to excel at the Communicate step. There was an opportunity to use the basic communication channels they had in place (status feed, twitter) more effectively by communicating throughout the incident, with more actionable information, but alas this was not the case. Instead, there was mass speculation about the root cause and the severity. That is exactly what you want to strive to avoid.
  3. Explain: Let's find out by running the postmortem through our guideline for postmortem communication...


Prerequisites:
  1. Admit failure - Excellent, almost a textbook admittance without hedging or blaming.
  2. Sound like a human - Well done. Posted from Director of Engineering at Facebook Robert Johnson's personal account, the tone and style was personal and effective.
  3. Have a communication channel - Can be improved greatly. Making the existing health status page easier to find, more public, and more useful would help in all future incidents. I've covered how Facebook can improve this page in a previous post.
  4. Above all else, be authentic - No issues here.
Requirements:
  1. Start time and end time of the incident - Missing.
  2. Who/what was impacted - Partial. I can understand this being difficult in the case of Facebook, but I would have liked to see more specifics around how many many users were affected. On one hand this is a global consumer service that may not be critical to people's lives. On the other hand though, if you treat your users with respect, they'll reward you for it.
  3. What went wrong - Well done, maybe the best part of the postmortem.
  4. Lessons learned - Partial. It sounds like many lessons were certainly learned, but they weren't directly shared. I'd love to know what the "design patterns of other systems at Facebook that deal more gracefully with feedback loops and transient spikes" look like.
Bonus:
  1. Details on the technologies involved - No
  2. Answers to the Five Why's - No
  3. Human elements - heroic efforts, unfortunate coincidences, effective teamwork, etc - No
  4. What others can learn from this experience - Marginal


Biggest lesson for us to take away: Preparation is key to successfully managing outages, and using them to build trust with your users.

53 comments:

  1. All the best blogs that is very useful for keeping me share the ideas
    of the future as well this is really what I was looking for, and I am
    very happy to come here. Thank you very much
    earn to die
    earn to die 2
    earn to die 3
    Hi! I’ve been reading your blog for a while now and finally got the
    earn to die 4
    courage to go ahead and give youu a shout out from
    earn to die 6
    Austin Texas! Just wanted to tell
    earn to die 5
    you keep up the fantastic work!my weblog
    age of war
    Hi! I’ve been reading your blog for a while now and finally got the
    happy wheels
    strike force heroes
    slitherio
    good game empire

    ReplyDelete
  2. Welcomes to google terminal from Austin Texas! google snake
    Amazing insight you have on this, Happy wheels it's nice to find a website that details so much information about different artists... Age of war 2 This article always blew me... Earn to die For how many times I have read this.
    slither io Hi! I’ve been reading your blog for a while slitherio it's nice to find a website that details so much big farm

    ReplyDelete
  3. Coach Outlet http://www.coachstoreoutlet.us.org

    Christian Louboutin Shoes http://www.christianlouboutinshoe.eu.com

    Valentino Shoes http://www.valentinoshoes.eu.com

    Michael Kors Outlet http://www.michaelkorsoutlet.us.org

    Coach Factory Outlet http://www.coachfactoryoutlet.us.org

    Coach Outlet Online http://www.coachoutletonlinestore.us.org

    Coach Purses http://www.coachpurse.us.com

    Kate Spade Outlet http://www.katespadefactoryoutlet.us.com

    Toms Shoes http://www.toms-shoes.name

    Hermes Belts http://www.hermesbelts.us.org

    Louis Vuitton http://www.louisvuitton-outletstores.us.com

    Fendi Handbags http://www.fendibelt.us.com

    Giuseppe Shoes http://www.giuseppezanotti.eu.com

    Michael Kors Outlet http://www.michael-kors-outlet.us.com

    Stephen Curry Shoes http://www.stephencurryshoes.us.com

    Salomon Shoes http://www.salomonshoes.eu.com

    North Face Outlet http://www.northfacesoutlet.in.net

    Coach Outlet http://www.outletcoach.in.net

    North Face Outlet http://www.northfacesoutlet.us.com

    Burberry Outlet http://www.burberryoutletstore.us.org

    North Face Outlet http://www.north-faceoutlets.in.net

    North Face Jackets http://www.outletnorthface.in.net

    Skechers Shoes http://www.skechersshoes.us.com

    Toms Outlethttp://www.tomsoutletstore.us.org

    North Face Outlet http://www.northfacestoreoutlet.us.org

    Nike Air Max http://www.nikeoutlet.us.org

    Nike Hoodies http://www.nikehoodies.us.com

    Marc Jacobs Handbags http://www.marcjacobshandbags.name

    Marc Jacobs Outlet http://www.marcjacobsoutletstore.us.com

    Jimmy Choo Shoes http://www.jimmy-chooshoes.us.com

    Jimmy Choos http://www.jimmychoos.in.net

    Burberry Belt http://www.burberrybelt.us.com

    Louis Vuitton Belt http://www.louisvuitton-belt.com

    Salvatore Ferragamo http://www.ferragamo-belt.us.com

    Marc Jacobs Handbags http://marcjacobshandbagsoutlet.blog.com

    Lululemon Outlet http://www.lululemonsoutlet.us.com

    True Religion Outlet http://www.truereligion-outlets.in.net

    Tommy Hilfiger http://www.tommyhilfigeroutlet.us.com

    Michael Kors Outlet http://www.outletmichaelkors.eu.com

    Coach Outlet http://www.outletcoach.eu.com

    Red Bottoms http://www.redbottom.in.net

    Kevin Durant Shoes http://www.kevindurantshoes.us.org

    New Balance Outlet http://www.newbalanceoutlet.in.net

    Adidas Outlet http://www.adidasoutlet.in.net

    Coach Outlet Online http://www.coachoutlet-online.eu.com

    Stephen Curry Jersey http://www.stephencurryjersey.us.com

    Vans Outlet http://www.vansoutlet.us.com

    Ralph Lauren Outlet http://www.ralphlauren-outletstore.in.net

    True Religion Outlet http://www.true-religionoutlets.in.net

    ED Hardy Outlet http://www.edhardyoutlet.in.net

    North Face Outlet http://www.northfaceoutlet.eu.com

    UGG Outlet http://www.uggoutlet.us.org

    UGG Outlet http://www.outletugg.in.net

    North Face Outlet http://www.northface--outlet.us.com

    Ugg Boots Sale http://www.uggbootssale.in.net

    UGGS For Women http://www.uggsforwomen.us.com

    Skechers Go Walk http://www.skechersgowalk.us.com

    Adidas Yeezy Boost http://www.adidasyeezyboost.in.net

    Adidas Yeezy http://www.adidasyeezy.us.com

    Adidas NMD http://www.adidas-nmd.us.org

    Coach Outlet http://www.coachoutletinc.us.com

    ReplyDelete
  4. This reminds me when this one time I went to this new restaurant and I was amazed that so many people are willing to pay extra for what they could get at half the price somewhere else. Agen Poker

    ReplyDelete
  5. I find your opinion quite interesting, but the other day I stumbled upon a completely different advice from another blogger, I need to think that one through, thanks for posting. Poker Online Indonesia

    ReplyDelete
  6. Thanks, great post. I really like your point of view! Judi Poker

    ReplyDelete
  7. I really liked this part of the article, with a nice and interesting topics
    Agen Domino

    ReplyDelete
  8. Your post so Inspirative and good point.
    Thanks, great post. I really like your point of view!
    Bandar Ceme

    ReplyDelete
  9. That’s awesome! This post gives truly quality information. I love this post it very interesting and helpful for us, Thank for sharing all, Thank you so much. Poker Bonus Deposit Terbesar

    ReplyDelete
  10. This is great, you are good, i like your post and i still waiting our next post
    Bandar Ceme

    ReplyDelete
  11. Wow. good post. I’d like to write like this too , taking time and real hard work to make a great article. Thanks for sharing. Bonus Deposit Poker

    ReplyDelete
  12. Thanks for sharing nice information with us. I really liked this part of the article, This is truly awesome article. Bonus Member Baru Poker Online

    ReplyDelete
  13. I would like to thank you for the efforts you have made in writing this article , Your article very well and good, thanks for posting. Judi Capsa

    ReplyDelete
  14. It is a good and amazing article. Your information is very useful for me and for others. thank you so much. BoyaQQ

    ReplyDelete
  15. This article is very good, I like this article, thanks for sharing. Link Alternatif BoyaQQ

    ReplyDelete
  16. Good Job And continue to make useful articles.
    Link Alternatif BoyaQQ

    ReplyDelete
  17. Your articles were very nice and full of information.
    BoyaQQ

    ReplyDelete
  18. Very Fantastic article, I really like your point of view!
    Good Job And continue to make useful articles.
    DominoHP.net Agen Poker Online Terpercaya

    ReplyDelete
  19. This is great, you are good, i like your post and i still waiting our next post
    Bandar Ceme

    ReplyDelete
  20. This is so interesting, I have never seen this before , Nice Post. Bandar Domino
    a

    ReplyDelete
  21. Thanks for this wonderful post and hoping to post more of this. Bandar QQ

    ReplyDelete
  22. great post. I really like your point of view!
    Bandar Ceme

    ReplyDelete
  23. I need to think that one through, thanks for posting.
    Domino Qiu Qiu

    ReplyDelete
  24. Thanks, guys you always provide useful information through your articles
    Agen Domino

    ReplyDelete
  25. Really a nice content i am going to bookmark your site thanks
    Domino QQ

    ReplyDelete
  26. Your blog information is nice and friendly. This post is very helpful for me. Thank for post.
    Judi Ceme

    ReplyDelete
  27. This is impressive and also great information, This is excellent. Judi Domino

    ReplyDelete
  28. This is really very nice blog and so informative, I would like to thank you for all the information you give. Domino 99

    ReplyDelete

  29. Absolutely really Amazing Articel and good point in there.
    Thanks shared.
    Dominoqiu

    ReplyDelete
  30. Nice Post, and good Information for this site.
    Thanks for shared!!
    Dominoqq

    ReplyDelete
  31. Nice Post, and good Information for this site.
    Thanks for shared!!
    Dominoqq

    ReplyDelete
  32. Thanks for posting, this is really amazing.
    BandarQQ Online

    ReplyDelete
  33. Some of these information are really amazing. Thank you for giving me good information.
    Situs BandarQQ

    ReplyDelete
  34. I would like to thank you for all the information you give. information your is useful to me.
    BandarQQ Terpercaya

    ReplyDelete
  35. I was working and suddenly I visits your site frequently and recommended it to me to read also. The writing style is superior and the content is relevant. Thanks for the insight you provide the readers!
    run2online.net

    ReplyDelete
  36. Thanks for sharing this quality information with us. I really enjoyed reading.

    supersmashflash2s.com

    ReplyDelete
  37. I was very impressed by this post, this site has always been pleasant news. Thank you very much for such an interesting post. Keep working, great job! In my free time, I like play game: mutilateadoll2game.com. What about you?

    ReplyDelete
  38. I like your all post. You have done really good work. Thank you for the information you provide, it helped me a lot. I hope to have many more entries or so from you.
    Very interesting blog.
    www.baixarfacebookgratis.com.br

    ReplyDelete
  39. I suppose you will keep the quality work going on. It's for some other informative blog. Where else could I get that type of information obat kelenjar getah bening obat benjolan di tangan obat infeksi saluran kencing

    ReplyDelete
  40. good articles and solid content is very interesting
    http://www.sinidomino.com/

    ReplyDelete
  41. interesting articles and this is my first first reading a very interesting article thanks for this article that fits with the theme news.
    raja poker

    ReplyDelete
  42. REALLY GOOD! i like it so much<3 Thanks for the Good Artickle Sir.
    sanadomino

    ReplyDelete
  43. REALLY GOOD! i like it so much<3 Thanks for the Good Artickle Sir.
    agen judi poker online terpercaya di indonesia

    ReplyDelete
  44. This article is very interesting to read and very easy to understand I am very interested in your article thanks.
    raja poker

    ReplyDelete