
So far, not too bad. Though note the broken rule in hosting your status page in the same location as your service. Lesson #1: Host your status page offsite. Let's keep moving with the timeline....At ~8:30AM Pacific Time we started experiencing networking issues at our El Segundo Data Center. We are working closely with them to determine the cause of these issues and will report any findings as they become available.
At this time we appear to be back fully. The tardiness of this update is a direct result of these networking issues.
About the same time the blog post went up, a Twitter message by @mt_monitor pointed to the official status update. Great to see that they actually use Twitter to communicate with their users, and judging by the 360 followers, I think this was a smart way to spread the news. On the other hand, this was the only Twitter update from Media Temple throughout the entire incident, which is strange. And it looks like some users were still in the dark for a bit too long. I was also surprised that the @mediatemple feed made no mention of this. Maybe they have a good reason to keep these separate? Looking at the conversation on Twitter, feels like most people by default use the @mediatemple label. Lesson #2: Don't confuse your users by splitting your Twitter identity.
From this point till about 9:40am PST, users were stuck wondering what was going on:

A few select tweets show us what users were thinking. The conversation on Twitter goes on for about 30 pages, or over 450 tweets from users wondering what the heck was going on.
Finally at 9:40am, Media Temple released their findings:
Now that everything is back up and users are "happy", what else can we learn from this experience?Our engineers have spoken with the engineers at our El Segundo Data Center (EL-IDC3). Here are their findings:
ASN number 47868 was broadcasting invalid BGP data that caused our routers, and a lot of other routers on the internet, to reboot. This invalid BGP data exploited a software bug in our routers. We have applied filters to prevent us from receiving this invalid data.
At this time they are in contact with their vendors to see if there is a firmware update that will address this. You can expect to see network delays and small outages across the internet as other providers try to address this same issue.
Lessons
- Host your status page offsite. (covered above)
- Don't confuse your users by splitting your Twitter identity. (covered above)
- Some transparency is better then no transparency. The basic status message helped calm people down and reduce support calls.
- There was a huge opportunity here for Media Temple to use the tools of social media (e.g. Twitter/Blogging) as a two-way communication channel. Instead, Media Temple used both their blog and Twitter as a broadcast mechanism. I guarantee that if there were just a few more updates throughout the downtime period the tone of the conversation on Twitter would have been much more positive. Moreover, the trust in the service would have been damaged less severely if users were not in the dark for so long.
- A health status dashboard would have been very effective in providing information to the public beyond the basic "we are looking into it" status update. Without any work on the part of Media Temple during the event, its users would have been able to better understand the scope of the event, and know instantly whether or not it was still a problem. It would have been extremely powerful when combined with lesson 4, if a representative on Twitter simply pointed users complaining of the downtime to the status page.
- The power of Twitter as a mechanism for determining whether a service is down (or whether it is just you), and in spreading news across the world in a matter of minutes, again proves itself.
ray ban sunglasses outlet
ReplyDeleteair jordan shoes
gucci handbags
oakley sunglasses wholesale
michael kors outlet online
kate spade outlet
ugg canada
polo ralph lauren
oakley outlet
canada goose outlet
michael kors outlet
louboutin shoes
nike air max pas cher
michael kors outlet online
burberry sale
lebron 13 shoes
louis vuitton handbags
air jordan 8
ladies cartier watches
adidas outlet store
cheap jerseys
coach outlet
ugg italia
coach outlet store online
coach factory outlet
coach outlet store online
ugg outlet
longchamp outlet
cheap uggs
christian louboutin wedges
20168.17chenjinyan
Coach Outlet http://www.coachstoreoutlet.us.org
ReplyDeleteChristian Louboutin Shoes http://www.christianlouboutinshoe.eu.com
Valentino Shoes http://www.valentinoshoes.eu.com
Michael Kors Outlet http://www.michaelkorsoutlet.us.org
Coach Factory Outlet http://www.coachfactoryoutlet.us.org
Coach Outlet Online http://www.coachoutletonlinestore.us.org
Coach Purses http://www.coachpurse.us.com
Kate Spade Outlet http://www.katespadefactoryoutlet.us.com
Toms Shoes http://www.toms-shoes.name
Hermes Belts http://www.hermesbelts.us.org
Louis Vuitton http://www.louisvuitton-outletstores.us.com
Fendi Handbags http://www.fendibelt.us.com
Giuseppe Shoes http://www.giuseppezanotti.eu.com
Michael Kors Outlet http://www.michael-kors-outlet.us.com
Stephen Curry Shoes http://www.stephencurryshoes.us.com
Salomon Shoes http://www.salomonshoes.eu.com
North Face Outlet http://www.northfacesoutlet.in.net
Coach Outlet http://www.outletcoach.in.net
North Face Outlet http://www.northfacesoutlet.us.com
Burberry Outlet http://www.burberryoutletstore.us.org
North Face Outlet http://www.north-faceoutlets.in.net
North Face Jackets http://www.outletnorthface.in.net
Skechers Shoes http://www.skechersshoes.us.com
Toms Outlethttp://www.tomsoutletstore.us.org
North Face Outlet http://www.northfacestoreoutlet.us.org
Nike Air Max http://www.nikeoutlet.us.org
Nike Hoodies http://www.nikehoodies.us.com
Marc Jacobs Handbags http://www.marcjacobshandbags.name
Marc Jacobs Outlet http://www.marcjacobsoutletstore.us.com
Jimmy Choo Shoes http://www.jimmy-chooshoes.us.com
Jimmy Choos http://www.jimmychoos.in.net
Burberry Belt http://www.burberrybelt.us.com
Louis Vuitton Belt http://www.louisvuitton-belt.com
Salvatore Ferragamo http://www.ferragamo-belt.us.com
Marc Jacobs Handbags http://marcjacobshandbagsoutlet.blog.com
Lululemon Outlet http://www.lululemonsoutlet.us.com
True Religion Outlet http://www.truereligion-outlets.in.net
Tommy Hilfiger http://www.tommyhilfigeroutlet.us.com
Michael Kors Outlet http://www.outletmichaelkors.eu.com
Coach Outlet http://www.outletcoach.eu.com
Red Bottoms http://www.redbottom.in.net
Kevin Durant Shoes http://www.kevindurantshoes.us.org
New Balance Outlet http://www.newbalanceoutlet.in.net
Adidas Outlet http://www.adidasoutlet.in.net
Coach Outlet Online http://www.coachoutlet-online.eu.com
Stephen Curry Jersey http://www.stephencurryjersey.us.com
Vans Outlet http://www.vansoutlet.us.com
Ralph Lauren Outlet http://www.ralphlauren-outletstore.in.net
True Religion Outlet http://www.true-religionoutlets.in.net
ED Hardy Outlet http://www.edhardyoutlet.in.net
North Face Outlet http://www.northfaceoutlet.eu.com
UGG Outlet http://www.uggoutlet.us.org
UGG Outlet http://www.outletugg.in.net
North Face Outlet http://www.northface--outlet.us.com
Ugg Boots Sale http://www.uggbootssale.in.net
UGGS For Women http://www.uggsforwomen.us.com
Skechers Go Walk http://www.skechersgowalk.us.com
Adidas Yeezy Boost http://www.adidasyeezyboost.in.net
Adidas Yeezy http://www.adidasyeezy.us.com
Adidas NMD http://www.adidas-nmd.us.org
Coach Outlet http://www.coachoutletinc.us.com
kobe shoes
ReplyDeletetrue religion outlet
kate spade handbags
coach outlet
birkenstock sandals
true religion outlet
nike free run 5.0
jordan shoes
polo shirts
toms wedges
hzx20170412
ReplyDeleteشركة المتحدة
دليل شركات نقل العفش
شركة نقل اثاث بالمدينة المنورة
شركة نقل اثاث بالرياض
شركة نقل اثاث بجدة
شركة نقل اثاث بمكة
ReplyDeleteشركة نقل اثاث بالطائف
شركة نقل اثاث بينبع
شركة نقل اثاث بالدمام
شركة نقل اثاث بنجران
شركة نقل اثاث بخميس مشيط
ReplyDeleteشركة نقل اثاث بابها
شركة نقل اثاث بحائل
ReplyDeleteشركة نقل اثاث ببريدة
شركة نقل اثاث بالقصيم
شركة نقل اثاث بحفر الباطن
شركة نقل اثاث برابغ