Friday, October 8, 2010

Etsy.com opens the kimono and talks frankly about outages

You know when John Allspaw (VP of Ops at Etsy, Manager of Operations at Flickr, Infrastructure Architect at Friendster) is involved, you're going to get a unique perspective on things. A few weeks ago Etsy.com was down. John (and his operations) department decided it would be a good opportunity to take what I'll call an "outage bankruptcy" and basically reset expectations. In an extremely detailed and well thought out post (titled "Frank Talk about  Site Outages") he goes on to describe the entire end-to-end processes that go into managing uptime at Etsy. I would recommend reading the entire post, but I thought it would be useful to point out the things that we can all take away from the experience of one of the most well respected operations people in the industry:


Metrics
"Today, we gather a little over 30,000 metrics, on everything from CPU usage, to network bandwidth, to the rate of listings and re-listings done by Etsy sellers. Some of those metrics are gathered every 20 seconds, 24 hours a day, 365 days a year. About 2,000 metrics will alert someone on our operations staff (we have an on-call rotation) to wake up in the middle of the night to fix a problem."
Takeaway: Capture data on every part of your infrastructure, and later decide which metrics are leading indicators of problems. He goes on to talk about the importance of external monitoring (outside of your firewall) to measure the actual end-user experience.
Communication
"When we have an outage or issue that affects a measurable portion of the site’s functionality, we quickly group together to coordinate our response. We follow the same basic approach as most incident response teams. We assign some people to address the problem and others to update the rest of the staff and post to http://etsystatus.com to alert the community. Changes that are made to mitigate the outage are largely done in a one-at-a-time fashion, and we track both our time-to-detect as well as our time-to-resolve, for use in a follow-up meeting after the outage, called a “post-mortem” meeting. Thankfully, our average time-to-detect is on the order of 2 minutes for any outages or major site issues in the past year. This is mostly due to continually tuning our alerting system."
Takeaway: Two important points here. First, communication and collaboration are key to successfully managing issues. Second, and even more interesting, is the need for two teams...one to address the problem and one to communicate status updates both internally and externally. This is often a missing piece for companies, where no updates go out because everyone is busy fixing the problem.
Post-Mortems
"After any outage, we meet to gather information about the incident. We reconstruct the time-line of events; when we knew of the outage, what we did to fix it, when we declared the site to be stable again. We do a root cause analysis to characterize why the outage happened in the first place. We make a list of remediation tasks to be done shortly thereafter, focused on preventing the root cause from happening again. These tasks can be as simple as fixing a bug, or as complex as putting in new infrastructure to increase the fault-tolerance of the site. We document this process, for use as a reference point in measuring our progress."
Takeaway: Fixing the problem and getting back online is not enough. Make it a an automatic habit to schedule a postmortem to do a deep dive into the root cause(s) of the problem, and address not only the immediate bugs but also the deeper issues that led to the root cause. The Five Why's can help here, as can the Lean methodology of investing a proportional number of hours into the most problematic parts of the infrastructure.
Single Point of Failure Reduction
"As Etsy has grown from a tiny little start-up to the mission-critical service it is today, we’ve had to outgrow some of our infrastructure. One reason we have for this evolution is to avoid depending on single pieces of hardware to be up and running all of the time. Servers can fail at any time, and Etsy.com should be able to keep working if a single server dies. To do that, we have to put our data in multiple places, keep them in sync, and make sure our code can route around any individual failures.
So we’ve been working a lot this year to reduce those “single points of failure,” and to put in redundancy as fast as we safely can. Some of this means being very careful (paranoid) as we migrate data from the single instances to multiple or replicated instances. As you can imagine, it’s a bit of a feat to move that volume of data around while still seeing a peak of 15 new listings per second, all the while not interrupting the site’s functionality."
Takeaway: Reduce single points of failure incrementally. Do what you can in the time you have.
Change Management and Risk
"For every type of technical change, we have answers to questions like:
  • What problem does the change solve?
  • Has this kind of change happened before? Is there a successful history?
  • When is the change going to start? When is it expected to end?
  • What is the expected effect of this change on the Etsy community? Is a downtime required for the change?
  • What is the rollback plan, if something goes wrong?
  • What test is needed to make sure that the change succeeded?
As with all change, the risk involved and the answers to these questions are largely dependent on the judgment of the person at the helm. At Etsy, we believe that if we understand the likely failures, and if there’s a plan in place to fix any unexpected issues, we’ll make progress.
Just as important, we also track the results of changes. We have an excellent history with respect to the number of successful changes. This is a good record that we plan on keeping."
Takeway: Be prepared for failure by anticipating worst-case scenario's for every change. Be ready to roll back and respond. More importantly, make sure to track when things go right to have a realistic measure of risk.
Other takeaways:
  • Declaring "outage bankruptcy" is not the ideal approach. But it is better than simply going along without any authentic communication with your customers throughout a period of instability. Your customers will understand, if you act human.
  • Etsy has been doing a great job keeping customers up to date at http://etsystatus.com/.
  • A glance at the comments on the page shows a few upset customers, but a generally positive response.

33 comments:

  1. Make generate fast and easy beta key generator and beta key giveaway for heroes of the storm and access all the resource for free for heroes of the storm and make ore idea over: heroes of the storm unlimited gold cheats

    ReplyDelete
  2. Great post Digital tickets I would like to thank you for the efforts you have made in writing this interesting and knowledgeable article.

    ReplyDelete
  3. Writing essays normally requires reading or research. But you can do it easier: just buy custom essays cheap. No regrets!

    ReplyDelete
  4. Valuable information. Lucky me I found your web site by accident, and I am surprised why this coincidence did not took place earlier! I bookmarked it. amazon gift card

    ReplyDelete
  5. Really impressive post. I read it whole and going to share it with my social circules. I enjoyed your article and planning to rewrite it on my own blog.
    mahjong | agario | minecraft | pacman| kizi | hulk | mickey mouse games | sniper games

    ReplyDelete
  6. I just want to say thanks for your wonderful post, it is contain a lot of knowledge and information that i needed right now. You really help me out my friend, thanks!
    halloween| animal jam| happy wheels| cool math games| 8 ball pool| sudoku| yoob| friv| tetris| friv4school| monster high| agario

    ReplyDelete
  7. This is such a great experience hope to see and read more essay here. I cant wait for your next post. By the way, the photo looks so stunning.
    agario
    happy wheels
    car games
    pacman
    plants vs zombies
    solitaire

    ReplyDelete
  8. http://www.searchenginecaffe.com/2011/06/inside-acl-building-watson-deepqa.html?showComment=1458707706857#c6979309111936844419

    ReplyDelete

  9. Very useful post. This is my first time i visit here. I found so many interesting stuff in your blog especially its discussion. Really its great article. Keep it up

    - Mortal Kombat XL
    - Atari Breakout
    - Dragon Ball Z Games

    ReplyDelete
  10. This is a really super post. Must admit that you are amid the best writer I have read. I appreciate your making the effort to discuss this class of article.
    usps tracking
    iphone 7 release date
    Netflix

    ReplyDelete
  11. Thanks for the best blog.it was very useful for me.keep sharing such ideas in the future as well. Thanks for giving me the useful information. I think I need it!
    Plants vs Zombies , Solitaire,Tom And Jerry Games, Brain Games, Happy Wheels , FNAF World , FNAF

    ReplyDelete
  12. Thanks for giving me the useful information. I think I need it! –
    minecraft games | fnaf |tetris - puzzle games
    happy wheels | abcdeya |agario hi | brain games

    ReplyDelete
  13. Let’s keep out sites for your child! click:
    brain games | puzzle games | tetris | happy wheels | agario | abcya | fnaf 4 | super mario games
    To play for free!

    ReplyDelete
  14. All the best blogs that is very useful for keeping me share the ideas
    of the future as well this is really what I was looking for, and I am
    very happy to come here. Thank you very much
    earn to die play
    earn to die
    earn to die 3
    Hi! I’ve been reading your blog for a while now and finally got the
    earn to die 4
    courage to go ahead and give youu a shout out from
    earn to die 6
    Austin Texas! Just wanted to tell
    earn to die 5
    you keep up the fantastic work!my weblog
    age of war
    Hi! I’ve been reading your blog for a while now and finally got the
    happy wheels
    tank trouble 3
    slither io
    slitherio

    ReplyDelete
  15. Let’s keep out sites for your child! click:
    brain games | puzzle games | tetris | happy wheels | agario | abcya | fnaf 4
    To play for free!

    ReplyDelete
  16. Thanks for sharing. I hope it will be helpful for too many people that are searching for this topic.
    shooting games | dog games | fighting games | mickey mouse games | Subway Surfers

    ReplyDelete
  17. This blog is so nice to me. I will continue to come here again and again. Visit my link as well. Good luck
    obat aborsi
    cara menggugurkan kandungan
    obat telat datang bulan
    obat penggugur kandungan

    ReplyDelete
  18. Let’s keep out sites for your child! click:
    slither.io | wings.io | happy wheels | abcya | fnaf 4 | games for kids | racing games to play for free!

    ReplyDelete
  19. A woman who truly loves you will be angry at you for so many things but will always be there to stick around
    mutant fighting cup 2 | manny pacquiao | stardew valley wiki | slither.io | powerpuff yourself | zumbi tsunami | fishdom | quotes about love | chinese food near me | snake game

    ReplyDelete
  20. Great! Thanks for sharing the information. I like this information. Keep posting
    happy wheels | friv | girlsgogames | games2girls | happy wheels | fireboy and watergirl

    ReplyDelete
  21. Welcomes to google terminal from Austin Texas! google snake
    Amazing insight you have on this, Happy wheels it's nice to find a website that details so much information about different artists... Age of war 2 This article always blew me... Earn to die For how many times I have read this.
    slither io Hi! I’ve been reading your blog for a while slitherio it's nice to find a website that details so much big farm

    ReplyDelete
  22. This app is great it gives you those moments of relaxation and incredibly wonderful
    age of war 2 | tank trouble | gold miner | tank trouble 3
    Just wanted to tell you keep up the fantastic work!my weblog:
    age of war 3 | earn to die 6 | earn to die 1

    ReplyDelete
  23. The best space for your child to relax!: wingsio | slither io | ya

    ReplyDelete
  24. With a Hotmail account, you can send and receive emails quickly and easily as well as login and use all Microsoft services.
    hotmail login | login to hotmail
    This is the game so players feel very fierce , adventurous . You try to join and play this game , you will feel great like.
    happy wheels | fireboy and watergirl | goodgame empire
    learn to fly | slither io | gun mayhem | age of war
    io games | strike force heroes | strike force heroes 4

    ReplyDelete
  25. تعتبر العاب تلبيس بنات من اشهر الانواع في هذا المجال وهي بدورها تتضمن عدة اصناف جميلة ويعشقها الكتير وخاصة البنات منها العاب تلبيس ومكياج التي تمزج بين التلبيس وكذلك الميك اب في آن واحد هذا الامر الدي يزيد من جمالها وتجعل كل من يلعبها يستمتع بذلك زد على ذلك العاب تلبيس باربي التي تعرف شعبية كبيرة لانها شخصية مشهورة ويعرفها الصغير والكبير ولهم ذكريات جميلة معها لانها اشتهرت في عالم الكارتون والان اصبح الامر كذلك في مجال الالعاب وغير هذا هناك كذلك نوع آخر مميز ايضا وهو العاب تلبيس عرائس فالجميع يحلم ان يقوم بتلبيسهما لانها تذكرهم بهذه المناسبة الجميلة الا وهي الزواج التي تعتبر اهم مرحلة في حياة الانسان وهناك انواع مغايرة لها جمهور كبير في كل انحاء العالم وهي العاب قص الشعر ليس هي فقط بل توجد ايضا العاب طبخ التي يمكن للجميع لعبها سواء كانوا اولادا او بناتا وهي الاكتر طلبا في النت ويحبها الجميع ومعها ايضا العاب باربي التي تكلمنا عليها بكل انواعها
    لعبة تحضير السلطة
    لعبة طبخ الدونات

    ReplyDelete
  26. Run 3 unblocked is the third install of the Run Serie. Skate, float, bounce, and/or jump your way through even more space tunnels! Run 2 unblocked game is next games in run series, now you can play run game with more and more level. Try to run as far as possible, good luck. Tanks Trouble is a game designed by danish Mads Purup. It is a maze party game with 1-3 players on one computer. The objective of the game is to destroy the other tanks to get a point.
    Run 2 - Run 2 unblocked - Tank Trouble - Tank Trouble 2

    ReplyDelete
  27. I enjoyed over read your blog post. Your blog have nice information, I got good ideas from this amazing blog. I am always searching like this type blog post.
    wingsio | sesxebi | y8 | gunblood | Happy Wheels

    ReplyDelete