Friday, October 8, 2010 opens the kimono and talks frankly about outages

You know when John Allspaw (VP of Ops at Etsy, Manager of Operations at Flickr, Infrastructure Architect at Friendster) is involved, you're going to get a unique perspective on things. A few weeks ago was down. John (and his operations) department decided it would be a good opportunity to take what I'll call an "outage bankruptcy" and basically reset expectations. In an extremely detailed and well thought out post (titled "Frank Talk about  Site Outages") he goes on to describe the entire end-to-end processes that go into managing uptime at Etsy. I would recommend reading the entire post, but I thought it would be useful to point out the things that we can all take away from the experience of one of the most well respected operations people in the industry:

"Today, we gather a little over 30,000 metrics, on everything from CPU usage, to network bandwidth, to the rate of listings and re-listings done by Etsy sellers. Some of those metrics are gathered every 20 seconds, 24 hours a day, 365 days a year. About 2,000 metrics will alert someone on our operations staff (we have an on-call rotation) to wake up in the middle of the night to fix a problem."
Takeaway: Capture data on every part of your infrastructure, and later decide which metrics are leading indicators of problems. He goes on to talk about the importance of external monitoring (outside of your firewall) to measure the actual end-user experience.
"When we have an outage or issue that affects a measurable portion of the site’s functionality, we quickly group together to coordinate our response. We follow the same basic approach as most incident response teams. We assign some people to address the problem and others to update the rest of the staff and post to to alert the community. Changes that are made to mitigate the outage are largely done in a one-at-a-time fashion, and we track both our time-to-detect as well as our time-to-resolve, for use in a follow-up meeting after the outage, called a “post-mortem” meeting. Thankfully, our average time-to-detect is on the order of 2 minutes for any outages or major site issues in the past year. This is mostly due to continually tuning our alerting system."
Takeaway: Two important points here. First, communication and collaboration are key to successfully managing issues. Second, and even more interesting, is the need for two to address the problem and one to communicate status updates both internally and externally. This is often a missing piece for companies, where no updates go out because everyone is busy fixing the problem.
"After any outage, we meet to gather information about the incident. We reconstruct the time-line of events; when we knew of the outage, what we did to fix it, when we declared the site to be stable again. We do a root cause analysis to characterize why the outage happened in the first place. We make a list of remediation tasks to be done shortly thereafter, focused on preventing the root cause from happening again. These tasks can be as simple as fixing a bug, or as complex as putting in new infrastructure to increase the fault-tolerance of the site. We document this process, for use as a reference point in measuring our progress."
Takeaway: Fixing the problem and getting back online is not enough. Make it a an automatic habit to schedule a postmortem to do a deep dive into the root cause(s) of the problem, and address not only the immediate bugs but also the deeper issues that led to the root cause. The Five Why's can help here, as can the Lean methodology of investing a proportional number of hours into the most problematic parts of the infrastructure.
Single Point of Failure Reduction
"As Etsy has grown from a tiny little start-up to the mission-critical service it is today, we’ve had to outgrow some of our infrastructure. One reason we have for this evolution is to avoid depending on single pieces of hardware to be up and running all of the time. Servers can fail at any time, and should be able to keep working if a single server dies. To do that, we have to put our data in multiple places, keep them in sync, and make sure our code can route around any individual failures.
So we’ve been working a lot this year to reduce those “single points of failure,” and to put in redundancy as fast as we safely can. Some of this means being very careful (paranoid) as we migrate data from the single instances to multiple or replicated instances. As you can imagine, it’s a bit of a feat to move that volume of data around while still seeing a peak of 15 new listings per second, all the while not interrupting the site’s functionality."
Takeaway: Reduce single points of failure incrementally. Do what you can in the time you have.
Change Management and Risk
"For every type of technical change, we have answers to questions like:
  • What problem does the change solve?
  • Has this kind of change happened before? Is there a successful history?
  • When is the change going to start? When is it expected to end?
  • What is the expected effect of this change on the Etsy community? Is a downtime required for the change?
  • What is the rollback plan, if something goes wrong?
  • What test is needed to make sure that the change succeeded?
As with all change, the risk involved and the answers to these questions are largely dependent on the judgment of the person at the helm. At Etsy, we believe that if we understand the likely failures, and if there’s a plan in place to fix any unexpected issues, we’ll make progress.
Just as important, we also track the results of changes. We have an excellent history with respect to the number of successful changes. This is a good record that we plan on keeping."
Takeway: Be prepared for failure by anticipating worst-case scenario's for every change. Be ready to roll back and respond. More importantly, make sure to track when things go right to have a realistic measure of risk.
Other takeaways:
  • Declaring "outage bankruptcy" is not the ideal approach. But it is better than simply going along without any authentic communication with your customers throughout a period of instability. Your customers will understand, if you act human.
  • Etsy has been doing a great job keeping customers up to date at
  • A glance at the comments on the page shows a few upset customers, but a generally positive response.


  1. Make generate fast and easy beta key generator and beta key giveaway for heroes of the storm and access all the resource for free for heroes of the storm and make ore idea over: heroes of the storm unlimited gold cheats

  2. Great post Digital tickets I would like to thank you for the efforts you have made in writing this interesting and knowledgeable article.

  3. Writing essays normally requires reading or research. But you can do it easier: just buy custom essays cheap. No regrets!

  4. Valuable information. Lucky me I found your web site by accident, and I am surprised why this coincidence did not took place earlier! I bookmarked it. amazon gift card

  5. Really impressive post. I read it whole and going to share it with my social circules. I enjoyed your article and planning to rewrite it on my own blog.
    mahjong | agario | minecraft | pacman| kizi | hulk | mickey mouse games | sniper games

  6. I just want to say thanks for your wonderful post, it is contain a lot of knowledge and information that i needed right now. You really help me out my friend, thanks!
    halloween| animal jam| happy wheels| cool math games| 8 ball pool| sudoku| yoob| friv| tetris| friv4school| monster high| agario

  7. This is such a great experience hope to see and read more essay here. I cant wait for your next post. By the way, the photo looks so stunning.
    happy wheels
    car games
    plants vs zombies



  9. Very useful post. This is my first time i visit here. I found so many interesting stuff in your blog especially its discussion. Really its great article. Keep it up

    - Mortal Kombat XL
    - Atari Breakout
    - Dragon Ball Z Games

  10. This is a really super post. Must admit that you are amid the best writer I have read. I appreciate your making the effort to discuss this class of article.
    usps tracking
    iphone 7 release date

  11. Thanks for the best was very useful for me.keep sharing such ideas in the future as well. Thanks for giving me the useful information. I think I need it!
    Plants vs Zombies , Solitaire,Tom And Jerry Games, Brain Games, Happy Wheels , FNAF World , FNAF

  12. Thanks for giving me the useful information. I think I need it! –
    minecraft games | fnaf |tetris - puzzle games
    happy wheels | abcdeya |agario hi | brain games

  13. Let’s keep out sites for your child! click:
    brain games | puzzle games | tetris | happy wheels | agario | abcya | fnaf 4 | super mario games
    To play for free!

  14. All the best blogs that is very useful for keeping me share the ideas
    of the future as well this is really what I was looking for, and I am
    very happy to come here. Thank you very much
    earn to die play
    earn to die
    earn to die 3
    Hi! I’ve been reading your blog for a while now and finally got the
    earn to die 4
    courage to go ahead and give youu a shout out from
    earn to die 6
    Austin Texas! Just wanted to tell
    earn to die 5
    you keep up the fantastic work!my weblog
    age of war
    Hi! I’ve been reading your blog for a while now and finally got the
    happy wheels
    tank trouble 3
    slither io

  15. Let’s keep out sites for your child! click:
    brain games | puzzle games | tetris | happy wheels | agario | abcya | fnaf 4
    To play for free!

  16. Thanks for sharing. I hope it will be helpful for too many people that are searching for this topic.
    shooting games | dog games | fighting games | mickey mouse games | Subway Surfers

  17. This blog is so nice to me. I will continue to come here again and again. Visit my link as well. Good luck
    obat aborsi
    cara menggugurkan kandungan
    obat telat datang bulan
    obat penggugur kandungan

  18. Let’s keep out sites for your child! click: | | happy wheels | abcya | fnaf 4 | games for kids | racing games to play for free!

  19. A woman who truly loves you will be angry at you for so many things but will always be there to stick around
    mutant fighting cup 2 | manny pacquiao | stardew valley wiki | | powerpuff yourself | zumbi tsunami | fishdom | quotes about love | chinese food near me | snake game

  20. Great! Thanks for sharing the information. I like this information. Keep posting
    happy wheels | friv | girlsgogames | games2girls | happy wheels | fireboy and watergirl

  21. Welcomes to google terminal from Austin Texas! google snake
    Amazing insight you have on this, Happy wheels it's nice to find a website that details so much information about different artists... Age of war 2 This article always blew me... Earn to die For how many times I have read this.
    slither io Hi! I’ve been reading your blog for a while slitherio it's nice to find a website that details so much big farm

  22. This app is great it gives you those moments of relaxation and incredibly wonderful
    age of war 2 | tank trouble | gold miner | tank trouble 3
    Just wanted to tell you keep up the fantastic work!my weblog:
    age of war 3 | earn to die 6 | earn to die 1

  23. The best space for your child to relax!: wingsio | slither io | ya

  24. With a Hotmail account, you can send and receive emails quickly and easily as well as login and use all Microsoft services.
    hotmail login | login to hotmail
    This is the game so players feel very fierce , adventurous . You try to join and play this game , you will feel great like.
    happy wheels | fireboy and watergirl | goodgame empire
    learn to fly | slither io | gun mayhem | age of war
    io games | strike force heroes | strike force heroes 4

  25. تعتبر العاب تلبيس بنات من اشهر الانواع في هذا المجال وهي بدورها تتضمن عدة اصناف جميلة ويعشقها الكتير وخاصة البنات منها العاب تلبيس ومكياج التي تمزج بين التلبيس وكذلك الميك اب في آن واحد هذا الامر الدي يزيد من جمالها وتجعل كل من يلعبها يستمتع بذلك زد على ذلك العاب تلبيس باربي التي تعرف شعبية كبيرة لانها شخصية مشهورة ويعرفها الصغير والكبير ولهم ذكريات جميلة معها لانها اشتهرت في عالم الكارتون والان اصبح الامر كذلك في مجال الالعاب وغير هذا هناك كذلك نوع آخر مميز ايضا وهو العاب تلبيس عرائس فالجميع يحلم ان يقوم بتلبيسهما لانها تذكرهم بهذه المناسبة الجميلة الا وهي الزواج التي تعتبر اهم مرحلة في حياة الانسان وهناك انواع مغايرة لها جمهور كبير في كل انحاء العالم وهي العاب قص الشعر ليس هي فقط بل توجد ايضا العاب طبخ التي يمكن للجميع لعبها سواء كانوا اولادا او بناتا وهي الاكتر طلبا في النت ويحبها الجميع ومعها ايضا العاب باربي التي تكلمنا عليها بكل انواعها
    لعبة تحضير السلطة
    لعبة طبخ الدونات

  26. Run 3 unblocked is the third install of the Run Serie. Skate, float, bounce, and/or jump your way through even more space tunnels! Run 2 unblocked game is next games in run series, now you can play run game with more and more level. Try to run as far as possible, good luck. Tanks Trouble is a game designed by danish Mads Purup. It is a maze party game with 1-3 players on one computer. The objective of the game is to destroy the other tanks to get a point.
    Run 2 - Run 2 unblocked - Tank Trouble - Tank Trouble 2

  27. I enjoyed over read your blog post. Your blog have nice information, I got good ideas from this amazing blog. I am always searching like this type blog post.
    wingsio | sesxebi | y8 | gunblood | Happy Wheels

  28. Great! Thanks for sharing the information. That is very helpful for increasing my knowledge in this field.
    Subway Surfers | Run 3 | Click Jogos

  29. I would like more information about this, because it is very nice. Thanks for sharing!
    Geometry Dash

  30. Coach Outlet

    Christian Louboutin Shoes

    Valentino Shoes

    Michael Kors Outlet

    Coach Factory Outlet

    Coach Outlet Online

    Coach Purses

    Kate Spade Outlet

    Toms Shoes

    Hermes Belts

    Louis Vuitton

    Fendi Handbags

    Giuseppe Shoes

    Michael Kors Outlet

    Stephen Curry Shoes

    Salomon Shoes

    North Face Outlet

    Coach Outlet

    North Face Outlet

    Burberry Outlet

    North Face Outlet

    North Face Jackets

    Skechers Shoes

    Toms Outlet

    North Face Outlet

    Nike Air Max

    Nike Hoodies

    Marc Jacobs Handbags

    Marc Jacobs Outlet

    Jimmy Choo Shoes

    Jimmy Choos

    Burberry Belt

    Louis Vuitton Belt

    Salvatore Ferragamo

    Marc Jacobs Handbags

    Lululemon Outlet

    True Religion Outlet

    Tommy Hilfiger

    Michael Kors Outlet

    Coach Outlet

    Red Bottoms

    Kevin Durant Shoes

    New Balance Outlet

    Adidas Outlet

    Coach Outlet Online

    Stephen Curry Jersey

    Vans Outlet

    Ralph Lauren Outlet

    True Religion Outlet

    ED Hardy Outlet

    North Face Outlet

    UGG Outlet

    UGG Outlet

    North Face Outlet

    Ugg Boots Sale

    UGGS For Women

    Skechers Go Walk

    Adidas Yeezy Boost

    Adidas Yeezy

    Adidas NMD

    Coach Outlet

  31. Great post I would like to thank you for the efforts you have made in writing this interesting post.
    Mehndi designs
    Best arabic mehndi desings
    Simple mehndi desings
    Best mehndi desings
    Latest mehndi desings

  32. Our Detroit Lions Banner Flag measures 28x40 inches, is constructed of 2-ply 100% polyester, and provides a top sleeve for hanging vertically from a wood flagpole. The Detroit Lions Banner Flag is high quality dye sublimated with the NFL Lions logo as shown. NFL Banners Flags are a perfect way to show neighbors your NFL team allegiance by flying from your front porch, patio, or deck.
    cowboys stars and stripes flags
    dallas cowboys house flagswholesale Indiana Colts american flag
    cheap Indianapolis Colts american flag

  33. Considerando che il nuovo casa Tottenham e corredi assenti presentano finiture in oro, il Tottenham Hotspur 2016-17 terzo camicia vede il colore prendere in consegna la totalità della camicia, la creazione di un look completamente nuovo per il club.maglie calcio 2017,
    maglie calcio poco prezzo, Maglia Barcelona 2017 vendita
    Maglia Deportivo a poco prezzo shop