Zendesk - Transparency in action

A colleague pointed me to a simple postmortem written by the CEO of Zendesk, Mikkel Svane:

"Yesterday an unannounced DNS change apparently made our mail server go incognito to the rest of the world. The consequences of this came sneaking over night as the changes propagated through the DNS network. Whammy.

On top of this our upstream internet provider late last night PST (early morning CET) experienced a failure that prevented our servers from reaching external destinations. Web access was not affected but email, widget, targets, basically everything that relied on communication from our servers to the outside world were. Double whammy.

It took too long time to realize that we had two separate issues at hand. We kept focusing on the former as root cause for the latter. And it took unacceptably long to determine that we had a network outage."

How well does such an informal and simple postmortem stack up against the postmortem best practices? Let's find out:

Prerequisites:

1. Admit failure - Yes, no question.
   
2. Sound like a human - Yes, very much so.
   
3. Have a communication channel - Yes, both the blog and the Twitter account.
4. Above all else, be authentic - Yes, extremely authentic.
   

Requirements:

1. Start time and end time of the incident - No.
   
2. Who/what was impacted - Yes, though more detail would have been nice.
   
3. What went wrong - Yes, well done.
   
4. Lessons learned - Not much.
   

Bonus:

1. Details on the technologies involved - No
2. Answers to the Five Why's - No
3. Human elements - Some
   
4. What others can learn from this experience - Some

Conclusion:

The meat was definitely there. The biggest missing piece is insight into what lessons were learned and what is being done to improve for the future. Mikkel says that "We've learned an important lesson and will do our best to ensure that no 3rd parties can take us down like this again", but the specifics are lacking. The exact time of the start and end of the event would have been useful as well, for those companies wondering whether this explains their issues that day.

It's always impressive to see the CEO of a company put himself out there like this and admit failure. It is (naively) easier to pretend like everything is OK and hope the downtime blows over. In reality, getting out in front of the problem and being transparent, communicating during the downtime (in this case over Twitter), and after the event is over (in this postmortem), are the best things you can do to turn your disaster into an opportunity to increase customer trust.

As it happens, I will be speaking at the upcoming Velocity 2010 conference about this very topic!

Update: Zendesk has put out a more in-depth review of what happened, which includes everything that was missing from the original post (which as the CEO pointed out in the comments, was meant to be a quick update of what they knew at the time). This new post includes the time frame of the incident, details on what exactly went wrong with the technology, and most importantly lessons and takeaways to improve things for the future. Well done.

Top presentation of the day on Slideshare!

How to Trust the Cloud

Next week I head off to Beijing, China to speak at Cloud Computing Congress China. Below is a sneak peek at the talk I plan to give (let me know what you think!):

Google App Engine downtime postmortem, nearly a perfect model for others

Google posted one of the most detailed and well thought out postmortems I've seen to explain what happened around their 2/24/10 App Engine Downtime. Let's run it through the gauntlet:

Prerequisites:

1. Admit failure - Yes
2. Sound like a human - Yes, more in some sections then others
3. Have a communication channel - Yes, the Google App Engine Downtime Notify group. Ideally it would have been linked to from the App Engine System Status Dashboard as well.
4. Above all else, be authentic - Yes

Requirements:

1. Start time and end time of the incident - Yes, including GMT time, and a highly detailed timeline of the entire event
2. Who/what was impacted - Partly, and also partly covered during the actual incident
3. What went wrong - Yes, yes, and yes! Incredible amount of detail here.
4. Lessons learned - Yes! Not only are there five specific action items, they are also introducing new architectural changes and customer choice as a result.

Bonus:

1. Details on the technologies involved - Somewhat
2. Answers to the Five Why's - No
3. Human elements - heroic efforts, unfortunate coincidences, effective teamwork, etc - Partial
4. What others can learn from this experience - Partial

Takeaways and thoughts:

1. A vast majority of the issues were training related. This is an important lesson: all of the technology and process in the world won't help you if your on-call team is unaware of what to do. This is especially true during the stress of a large incident. Follow the advice of Google and run regular on-call drills (including rare issues), keep documentation updated, and give on-call people the authority to make decisions on the spot.
2. Extremely impressed with the decision to use this opportunity to improve their service by giving their customers the choice between datastore performance and reliability. This is a perfect example of turning downtime into a positive.
3. Interesting insight into their process of detecting an incident to communicating externally. From the start of the incident at 7:48am to the first external communication at 8:01am is not too shabby. Not sure why it took so long to post the update to the downtime forum and the health status dashboard (8:36am).
4. The amount of time and thought that went into this postmortem shows how much Google is concerned about their service, and impressions around its reliability.

What could be improved:

1. External communication could be faster. No reason not to post something as soon as the investigation begins, not to mention posting to the forum dedicated to downtime notifications and the health status dashboard immediately. When the incident started the dashboard had very limited data, which should be automatic and real-time.
2. A post to this postmortem from the health status dashboard would make it a lot easier to find. I didn't see this until someone sent it to me.
3. Timelines and concrete deliverables on the changes (e.g. on-call training sessions, documentation updates, new datastore feature release) would give us more confidence that things will actually change.

Cloud photos for presentations and documents

I've been working on a talk I'm giving at Cloud Computing Congress in China and (I'm not ashamed to admit) I spent a lot of time looking for just the right cloud photo. I thought I'd share the best photo's I came across, in case you ever need to give a talk about The Cloud (or just love photo's of clouds).

Note: These photos are all be licensed under Creative Commons, and came from a combination of Flickr Creative Commons search and Google Images (manually excluding non-Creative Commons images).

The photos are divided into four sections:

1. Simple clouds


2. Dark clouds


3. Powerful clouds


4. Unique clouds

Simple clouds

Dark clouds

Powerful clouds

Unique clouds

You can find all of these cloud photos here. If you have other favorites and are willing to share, please add a link to them in the comments.

A guideline for postmortem communication

Building on previous posts, the following is a proposed Guideline for Postmortem Communication:

Prerequisites:

1. Admit failure - Hiding downtime is no longer an option
2. Sound like a human - Do not use a standard template, do not apologize for "inconveniencing" us
3. Have a communication channel - Set up a process to handle incidents prior the event (e.g. public health dashboard, status blog, twitter account, etc.)
4. Above all else, be authentic - You must be believed to be heard

Requirements:

1. Start time and end time of the incident
2. Who/what was impacted - Should I be worried about this incident?
3. What went wrong - What broke and how you fixed it (with insight into the root cause analysis process)
4. Lessons learned - What's being done to improve the situation for the future, in technology, process, and communication

Bonus:

1. Details on the technologies involved
2. Answers to the Five Why's
3. Human elements - heroic efforts, unfortunate coincidences, effective teamwork, etc
4. What others can learn from this experience

The top 7 most overused cloud metaphors, sorted by weather pattern

Along with innovation, agility, and efficiency cloud computing has brought us a bevy of clever metaphors around the cloud concept. Unfortunately these play-on-words headlines are quickly becoming cliches. Below is a list of the most common metaphors I've come across, and headlines that use them. I hope this list will force us all to be a bit more original. (Note: Sorted by weather event chronological order)

Partly Cloudy

1. McKinsey's Cloud Computing Report Is Partly Cloudy
2. The 'partly cloudy' network: Amazon's new partial clouds via IPsec VPN
3. Cloud Computing's Partly Cloudy Future?
4. Partly Cloudy - Blue-Sky Thinking About Cloud Computing
5. Future of computing: Forecast calls for partly cloudy

Dark Lining

1. A look at cloud computing's dark lining
2. Multi-site bug exposes cloud computing's dark lining
3. Cloud Computing Has Dark Lining
4. Bill: Cloud Computing's Dark Lining
5. Amazon’s S3 Cloud Has a Dark Lining for Start-Ups

Dark Side

1. The Dark Side of the Cloud
2. Recent Foibles Expose Dark Side of Cloud Computing
3. The Dark Side of Cloud Computing
4. 5 lessons from the dark side of cloud computing
5. The Dark Side of Cloud Computing

Bursting

1. Bursting the Cloud
2. Cloud computing to burst in next year: Gartner
3. If The Clouds Burst
4. Microsoft predicts cloud computing will burst with growth, opportunity
5. Cloud Bursting Scenarios For Small Companies

Rain

1. Can Cloud Computing Rain on Your Business?
2. Security Issues Rain On Cloud Computing
3. Can Amazon's Cloud Computing Make Rain?
4. Will Cloud Computing Rain On IT's Parade?
5. How Microsoft could make cloud computing rain dollars

Clearing the Air

1. Clearing the Air on Cloud Computing
2. Ellen Rubin is clearing the air in cloud computing
3. Clearing the Air on Cloud Computing: Insights from Ten CIOs
4. Cloud Computing, Part 1: Clearing the Air
5. Clearing The Air On Cloud

Silver Lining

1. Not Every Cloud has a Silver Lining
2. Is Integration Cloud Computing's Silver Lining?
3. Not every cloud has a silver lining
4. Is Google Chrome OS cloud computing's silver lining?
5. Cloud Computing Is Silver Lining in Tough Economy

Bonus: aaS

1. Which “aaS” is Right for You?
2. Is Everyone an aaS?

Have you come across any others? Let me know and I'll update this post.