Friday, October 8, 2010

Etsy.com opens the kimono and talks frankly about outages

You know when John Allspaw (VP of Ops at Etsy, Manager of Operations at Flickr, Infrastructure Architect at Friendster) is involved, you're going to get a unique perspective on things. A few weeks ago Etsy.com was down. John (and his operations) department decided it would be a good opportunity to take what I'll call an "outage bankruptcy" and basically reset expectations. In an extremely detailed and well thought out post (titled "Frank Talk about  Site Outages") he goes on to describe the entire end-to-end processes that go into managing uptime at Etsy. I would recommend reading the entire post, but I thought it would be useful to point out the things that we can all take away from the experience of one of the most well respected operations people in the industry:


Metrics
"Today, we gather a little over 30,000 metrics, on everything from CPU usage, to network bandwidth, to the rate of listings and re-listings done by Etsy sellers. Some of those metrics are gathered every 20 seconds, 24 hours a day, 365 days a year. About 2,000 metrics will alert someone on our operations staff (we have an on-call rotation) to wake up in the middle of the night to fix a problem."
Takeaway: Capture data on every part of your infrastructure, and later decide which metrics are leading indicators of problems. He goes on to talk about the importance of external monitoring (outside of your firewall) to measure the actual end-user experience.
Communication
"When we have an outage or issue that affects a measurable portion of the site’s functionality, we quickly group together to coordinate our response. We follow the same basic approach as most incident response teams. We assign some people to address the problem and others to update the rest of the staff and post to http://etsystatus.com to alert the community. Changes that are made to mitigate the outage are largely done in a one-at-a-time fashion, and we track both our time-to-detect as well as our time-to-resolve, for use in a follow-up meeting after the outage, called a “post-mortem” meeting. Thankfully, our average time-to-detect is on the order of 2 minutes for any outages or major site issues in the past year. This is mostly due to continually tuning our alerting system."
Takeaway: Two important points here. First, communication and collaboration are key to successfully managing issues. Second, and even more interesting, is the need for two teams...one to address the problem and one to communicate status updates both internally and externally. This is often a missing piece for companies, where no updates go out because everyone is busy fixing the problem.
Post-Mortems
"After any outage, we meet to gather information about the incident. We reconstruct the time-line of events; when we knew of the outage, what we did to fix it, when we declared the site to be stable again. We do a root cause analysis to characterize why the outage happened in the first place. We make a list of remediation tasks to be done shortly thereafter, focused on preventing the root cause from happening again. These tasks can be as simple as fixing a bug, or as complex as putting in new infrastructure to increase the fault-tolerance of the site. We document this process, for use as a reference point in measuring our progress."
Takeaway: Fixing the problem and getting back online is not enough. Make it a an automatic habit to schedule a postmortem to do a deep dive into the root cause(s) of the problem, and address not only the immediate bugs but also the deeper issues that led to the root cause. The Five Why's can help here, as can the Lean methodology of investing a proportional number of hours into the most problematic parts of the infrastructure.
Single Point of Failure Reduction
"As Etsy has grown from a tiny little start-up to the mission-critical service it is today, we’ve had to outgrow some of our infrastructure. One reason we have for this evolution is to avoid depending on single pieces of hardware to be up and running all of the time. Servers can fail at any time, and Etsy.com should be able to keep working if a single server dies. To do that, we have to put our data in multiple places, keep them in sync, and make sure our code can route around any individual failures.
So we’ve been working a lot this year to reduce those “single points of failure,” and to put in redundancy as fast as we safely can. Some of this means being very careful (paranoid) as we migrate data from the single instances to multiple or replicated instances. As you can imagine, it’s a bit of a feat to move that volume of data around while still seeing a peak of 15 new listings per second, all the while not interrupting the site’s functionality."
Takeaway: Reduce single points of failure incrementally. Do what you can in the time you have.
Change Management and Risk
"For every type of technical change, we have answers to questions like:
  • What problem does the change solve?
  • Has this kind of change happened before? Is there a successful history?
  • When is the change going to start? When is it expected to end?
  • What is the expected effect of this change on the Etsy community? Is a downtime required for the change?
  • What is the rollback plan, if something goes wrong?
  • What test is needed to make sure that the change succeeded?
As with all change, the risk involved and the answers to these questions are largely dependent on the judgment of the person at the helm. At Etsy, we believe that if we understand the likely failures, and if there’s a plan in place to fix any unexpected issues, we’ll make progress.
Just as important, we also track the results of changes. We have an excellent history with respect to the number of successful changes. This is a good record that we plan on keeping."
Takeway: Be prepared for failure by anticipating worst-case scenario's for every change. Be ready to roll back and respond. More importantly, make sure to track when things go right to have a realistic measure of risk.
Other takeaways:
  • Declaring "outage bankruptcy" is not the ideal approach. But it is better than simply going along without any authentic communication with your customers throughout a period of instability. Your customers will understand, if you act human.
  • Etsy has been doing a great job keeping customers up to date at http://etsystatus.com/.
  • A glance at the comments on the page shows a few upset customers, but a generally positive response.

73 comments:

  1. Make generate fast and easy beta key generator and beta key giveaway for heroes of the storm and access all the resource for free for heroes of the storm and make ore idea over: heroes of the storm unlimited gold cheats

    ReplyDelete
  2. Writing essays normally requires reading or research. But you can do it easier: just buy custom essays cheap. No regrets!

    ReplyDelete
  3. Valuable information. Lucky me I found your web site by accident, and I am surprised why this coincidence did not took place earlier! I bookmarked it. amazon gift card

    ReplyDelete
  4. I just want to say thanks for your wonderful post, it is contain a lot of knowledge and information that i needed right now. You really help me out my friend, thanks!
    halloween| animal jam| happy wheels| cool math games| 8 ball pool| sudoku| yoob| friv| tetris| friv4school| monster high| agario

    ReplyDelete
  5. This is such a great experience hope to see and read more essay here. I cant wait for your next post. By the way, the photo looks so stunning.
    agario
    happy wheels
    car games
    pacman
    plants vs zombies
    solitaire

    ReplyDelete
  6. http://www.searchenginecaffe.com/2011/06/inside-acl-building-watson-deepqa.html?showComment=1458707706857#c6979309111936844419

    ReplyDelete

  7. Very useful post. This is my first time i visit here. I found so many interesting stuff in your blog especially its discussion. Really its great article. Keep it up

    - Mortal Kombat XL
    - Atari Breakout
    - Dragon Ball Z Games

    ReplyDelete
  8. A woman who truly loves you will be angry at you for so many things but will always be there to stick around
    mutant fighting cup 2 | manny pacquiao | stardew valley wiki | slither.io | powerpuff yourself | zumbi tsunami | fishdom | quotes about love | chinese food near me | snake game

    ReplyDelete
  9. This app is great it gives you those moments of relaxation and incredibly wonderful
    age of war 2 | tank trouble | gold miner | tank trouble 3
    Just wanted to tell you keep up the fantastic work!my weblog:
    age of war 3 | earn to die 6 | earn to die 1

    ReplyDelete
  10. تعتبر العاب تلبيس بنات من اشهر الانواع في هذا المجال وهي بدورها تتضمن عدة اصناف جميلة ويعشقها الكتير وخاصة البنات منها العاب تلبيس ومكياج التي تمزج بين التلبيس وكذلك الميك اب في آن واحد هذا الامر الدي يزيد من جمالها وتجعل كل من يلعبها يستمتع بذلك زد على ذلك العاب تلبيس باربي التي تعرف شعبية كبيرة لانها شخصية مشهورة ويعرفها الصغير والكبير ولهم ذكريات جميلة معها لانها اشتهرت في عالم الكارتون والان اصبح الامر كذلك في مجال الالعاب وغير هذا هناك كذلك نوع آخر مميز ايضا وهو العاب تلبيس عرائس فالجميع يحلم ان يقوم بتلبيسهما لانها تذكرهم بهذه المناسبة الجميلة الا وهي الزواج التي تعتبر اهم مرحلة في حياة الانسان وهناك انواع مغايرة لها جمهور كبير في كل انحاء العالم وهي العاب قص الشعر ليس هي فقط بل توجد ايضا العاب طبخ التي يمكن للجميع لعبها سواء كانوا اولادا او بناتا وهي الاكتر طلبا في النت ويحبها الجميع ومعها ايضا العاب باربي التي تكلمنا عليها بكل انواعها
    لعبة تحضير السلطة
    لعبة طبخ الدونات

    ReplyDelete
  11. Our Detroit Lions Banner Flag measures 28x40 inches, is constructed of 2-ply 100% polyester, and provides a top sleeve for hanging vertically from a wood flagpole. The Detroit Lions Banner Flag is high quality dye sublimated with the NFL Lions logo as shown. NFL Banners Flags are a perfect way to show neighbors your NFL team allegiance by flying from your front porch, patio, or deck.
    cowboys stars and stripes flags
    dallas cowboys house flagswholesale Indiana Colts american flag
    cheap Indianapolis Colts american flag

    ReplyDelete
  12. Your blog is really nice and unique.I like it so much. Looking forward to going over your web page again.
    ipage coupon codes

    ReplyDelete
  13. Great article, I was very impressed about it, wish you would have stayed next share
    atari breakout game | super mechs | wings.io | wingsio game | supermechs | wingsio play

    ReplyDelete
  14. Real love is the desire center nearby, is common to share a lifetime role. Whether thunderstorm made out there, just with him and have children, just double heart beats, every storm will calm goods.
    baixar musicas , snapchat , geometry dash

    ReplyDelete
  15. The blog you have written is precise. For some reason I found great interest in reading it and I would like to share it as well.
    Independent Escorts Services in Gurgaon

    ReplyDelete
  16. Your post is very interesting for see, I am personally very fond with your handwriting on this page.
    Independent Escort Services in Delhi

    ReplyDelete
  17. I am so delighted to be here and to read this perfect post. Thanks for share this with us.
    Independent Gurgaon Escorts Services

    ReplyDelete
  18. The information provide by you is very useful for me.thanks for sharing and hope you will share again.
    Pretty Escorts Service in Gugoan

    ReplyDelete
  19. Hello Sir my self Suhana Khan and i and an Independent Escorts in Bangalore and i love your blog because its such a great experience hope to see and read more essay here. I cant wait for your next post. By the way, the photo looks so stunning.

    ReplyDelete
  20. This comment has been removed by the author.

    ReplyDelete
  21. nice article very intersting and heer you can download filmywap apk

    ReplyDelete
  22. Amazing post here is more interesting things for us thanks for sharing with us.
    Escorts in 5 Star Hotel in Gurgaon

    ReplyDelete
  23. I find your opinion quite interesting, but the other day I stumbled upon a completely different advice from another blogger, I need to think that one through, thanks for posting.
    whatsapp baixar
    Square Quick
    Geometry Dash
    Baixar Musicas
    Snapchat

    ReplyDelete
  24. I would like to say thanks for share this amazing information.
    Model Call Girls Service Bangalore

    ReplyDelete
  25. I read deeply your article, the points you have mentioned in this article are useful. I am always pleased with what you bring to it very interesting and useful .


    Escorts in Chandigarh

    ReplyDelete

  26. It is a fantastic website i usually to come here to enhance the knowledge i spend a great time on your website i appreciate it.

    Russian escorts in Dwarka

    ReplyDelete
  27. Very effective post, it's give me new ideas on my life. obat benjolan di leher

    ReplyDelete
  28. Call Girls in Mahipalpur, contact us for the best Russian escort services in mahipalpur, Aerocity, Vasant Kunj in 5 star hotel facilitie http://www.russiongirlsinmahipalpur.in Call Girls in Mahipalpur

    ReplyDelete
  29. I still liked this article, good, good content, and unique design.Thank you for sharing the article.
    obat penyakit siliskosis tradisional

    ReplyDelete
  30. very helpful blog for me,please keep posting i gonna bookmark you. in my laptop.

    mcafee support number

    ReplyDelete
  31. contact us for the best Russian escort services in mahipalpur, Aerocity, Vasant Kunj in 5 star hotel facilitie http://www.russiongirlsinmahipalpur.in Call Girls in Mahipalpur

    ReplyDelete
  32. Thanks, guys you always provide useful information through your articles
    Agen Domino

    ReplyDelete
  33. Very interesting blog. Alot of blogs I see these days don't really provide anything that I'm interested in, but I'm most definately interested in this one. Just thought that I would post and let you know.
    facebook baixar l baixar facebook l baixar facebook gratis l facebook movel l facebook movel baixar

    ReplyDelete

  34. This is exactly what I was looking for. Thanks for sharing this great article! That is very interesting i love reading and I am always searching for informative information like this!

    http://www.gurgaonfairy.com/

    Escorts in Aerocity

    ReplyDelete
  35. Kini Telah Hadir Agen Poker, DominoQQ, Capsa Susun, & Bandar Ceme Online Terbaik dan Terpercaya.

    Hanya dengan Minimal Deposit 25 ribu Bonus 10 ribu dan Minimal Withdraw 25 ribu anda dapat bermain di Acehpoker;

    1. Acehpoker memberikan bonus Referral 15% (Seumur Hidup)
    2. Sistem keamanan terjamin dan Data pribadi akan aman bersama kami.
    3. Player Vs Player ( 100% No Robot)
    4. Dengan Sistem Confirm Deposit terbaru Max 2 menit (Jika semua Nama di Id dan di bank
    sama)
    5. CS yang ramah dan Melayani dengan Professional
    6. Online 365 Hari 24 Jam
    7. Anda Menang berapapun Kami Bayar tanpa Maksimal Withdraw dan Tidak ada batas maksimal
    withdraw dalam 1 hari.
    8. Dapat bermain 6 permainan hanya dengan 1 Web , yaitu Poker , DominoQQ , Bandar Ceme, Capsa Susun, Ceme Keliling, Live Poker
    9. Bisa bermain di Android dan IOS
    10. Sistem pembagian Kartu menggunakan teknologi yang mutakhir dengan sistem Random Permanent (acak) , Dan tersedia Ratusan meja untuk tempat anda bermain

    Pin BBM : 563AB8CA
    yahoo : mahadewi374@yahoo.com
    Line: Lilly080615

    Agen Bola
    Daftar Bola BCA
    Agen Bola Terpercaya
    Bandar Bola IBCBET
    Daftar ID Bola Sbobet
    Poker Online Terpercaya
    Poker Bonus New Member
    Judi Poker Online
    Capsa Susun Uang Asli
    Bandar Ceme Keliling
    Cara Bermain Poker
    Cara Menang Poker
    Cara Daftar Poker
    Qiu Qiu Uang Asli

    ReplyDelete
  36. Indian girls all very decent girls, stylish, verbally fluent and always be your good companion.
    Escort in Dubai

    ReplyDelete
  37. Thank your site! information your share is useful to me! it's great that I known this site!
    happywheels3game.com

    ReplyDelete
  38. Tomorrow is the last entry day. To all of the wonderful people whom I have had commented on their remarks and their comment's on mine. I thank you. I think the Blogs are great and what learning experiences we all get from each other's views. A true learning experience.Regards osappsbox

    ReplyDelete
  39. It proves you understand very well about this issue pocaauto

    ReplyDelete
  40. I have read your article, the information you give is very interesting.
    instagram viewer

    ReplyDelete
  41. Selamat Pagi bosku....!!
    Jangan bilang kalau kamu jago POKER sebelum main di www.gadispoker,com
    Buruan Gabung & Rasakan KESUKSESAN bersama kami.
    cukup hanya satu ID kamu uda bisa memainkan 6 permainan sekaligus loh..

    Ragu.. ROBOT ataupun ADMIN ikut bermain??
    tenang sajaa!! di gadispoker NO.1 terpercaya,
    Murni Member VS Member,Fairplay..
    Proses Depo/Tarik Dana Super cepat
    Berapapun kemenangan Anda akan Dibayar!

    kita mempunyai Promo mernarik Khusus untuK
    New MEMBER & MEMBER setia di gadispoker :
    *Minimal depo & With Draw Tarik Dana Rp.10.000,-
    *Bonus Deposit setiap hari untuk para MEMBER
    *bONUS Refferal terbesar 20% Seumur hidup
    *Bonus Mingguan up To 0,5%
    Silakan di klik link permainan dari kami : https://goo.gl/iXP5pY atau www.gadispoker.com/ref.php?ref=CSGADIS555
    Kalian juga akan di layani oleh CS yang Baik,Ramah, dan juga Cantik
    selama 24 jam, jadi tunggu apalagi..
    ayo segera gabung bersama kami.
    untuk info yang lebih lanjut silahkan hubungi Custumer Service kami di :
    -Live Chat : www.gadispoker,com
    Hubungi segera dan jangan lewatkan Kesempatan Menangmu..!

    ReplyDelete
  42. I think the Blogs are great and good learning experiences we all get from each other’s views. A true learning experience. I wish all of you best of luck, I will be envious if I don’t win but such is life. download free gmes.

    ReplyDelete
  43. Mari Boskuu...
    BermainLah Bersama Kami Di www.Gadispoker.com
    hanya Butuh 1 ID dan Anda Bisa Bermain 6 Permainan Di Sini.
    jadi Tunggu Apa lagi,, Ayo Bergabung Dan MenangKan Jutaan Rupiah.

    Jangan Ragu Dengan kami Di sini Kami MenjaminKan Pemainan Tanpa Robot.
    Ataupun Admin Kami Bermain Bersama anda.
    DI sini kami Fairplay, 100% Player VS player.

    Poker Aman Dan Terpercaya Hanya Bersama Gadispoker.
    Kami Juga Menyediakan Bermacam Promo untuk Para Member Setia Kami.
    -Minimal Depo / Wd Hanya : 10000,-
    -Bonus Deposit Setiap hari Untuk Para member.
    -Bonus Refferal 15% Seumur Hidup.
    -Bonus TO/Turn Over Up TO 0,3% - 0,5%

    Kalian Juga Akan Di layani dengan CS Kami Yang Ramah dan Sangat propesional.
    Selama 24jam Nonstop,
    JAdi Jangan Berpikir panjang lagi mari Bergabung bersama Gadispoker dan Di tunggu Kemenangannya,
    Dan jangan lupa kunjungi blogspot kami ( https://sumberberitasex.blogspot.co.id )

    Untuk Info Lebih lanjut Silakan hubungi CS (custumer Service).
    -Live chat ( https://goo.gl/iXP5pY )
    -Whatsapp : +855966624192
    -Skype : Gadispoker
    -Yahoo : gadispokercs
    -BBM : D87DB681
    Hubungi Segera Dan Jangan Lewatkan Kemenanganmu..!
    poker online

    dewa poker

    judi poker

    raja poker

    agen judi poker online terpercaya di indonesia

    bandar judi poker online yang paling aman

    bandar poker online indonesia

    judi poker online terbesar

    poker online betting

    ReplyDelete
  44. Anda Hobi Bermain POKER ??
    Yuk Gabung bersama kami di MGMDOMINO.COM
    dan dapatkan BONUS hingga JUTAAN Rupiah setiap harinya..

    Here You Can Find The Best and Reliable Authorized Body Only at
    http://MGMPOKER88.com you feel comfortable toplay https://goo.gl/UdJZKT

    Disini Anda Dapat Menemukan Agency Resmi terbaik dan terpercaya.
    Hanya di MGMPOKER88.com anda merasakan ke nyamanan untuk bermain.

    Join the trusted poker
    www.mgmpoker88.com
    Livechat : https://goo.gl/UdJZKT
    #MGMPOKER88

    Mari bergabung di poker terpercaya
    www.mgmpoker88.com
    agen judi poker online terpercaya di indonesia

    bandar judi poker online yang paling aman

    ReplyDelete
  45. With an Independent Call Girl like me in Mahipalpur, you can get the all erotic satisfaction in your life with all best recreations on demand and also the all best fondness delivered to you with all friendly lust and fun as you are looking forward the hot sex call girl on demand.
    http://www.escort-gurgaon.com/mahipalpur-escort.html Mahipalpur Escort

    ReplyDelete
  46. intext:"pembesar penis" "good post" "nice post"
    mgmcash88

    ReplyDelete
  47. "Bagi anda yang hobi bermain game online kini kami dompetmgm.com hadir untuk anda yang dimana kami adalah Agen poker yang paling terpercaya
    dan hanya dengan modal 20.000 saja anda sudah bisa bergabung dan bermain bersama kami dalam 7 jenis permainan seru
    menangkan ratusan sampai jutaan rupiah dipermainan ini.
    untuk info lebih lanjut silakan hubungi :
    BBM : D8BCE200
    LIVECHAT : DOMPETMGM.COM
    LINE : +855968725318
    "

    ReplyDelete
  48. I really like this article and it is very interesting to follow and the new times I see this article as good as this
    http://www.sinidomino.com/

    ReplyDelete
  49. very nice article http://www.mgmdomino.com/

    ReplyDelete
  50. Very informative post for us, thanks for sharing.
    Gmail Support.

    ReplyDelete
  51. really very good writing and this is an excelent article and very thankful for this article
    poker online

    ReplyDelete
  52. judi bola

    Wow. good post. I’d like to write like this too , taking time and real hard work to make a great article. Thanks for sharing.

    ReplyDelete
  53. I was very impressed by this post, this site has always been pleasant news. Thank you very much for such an interesting post. Keep working, great job! In my free time, I like play game
    togelpelangi


    ReplyDelete

  54. Your article is very interesting. I think this article has a lot of information needed, looking forward to your new posts. Get permission to share:
    Kizi games for kids

    ReplyDelete
  55. your article is really good, visit also my article in: http://www.mgmdomino.com/

    ReplyDelete
  56. This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work! apksfile.

    ReplyDelete
  57. nice article http://www.dompetmgm.com/

    ReplyDelete
  58. How To Get Periods Immediately How to Get Periods Immediately In One Day Natural Home Remedies 2018

    ReplyDelete

Note: Only a member of this blog may post a comment.