Atlassian has security breach, responds with transparency, sees benefits

This past Sunday, Atlassian (makers of Confluence, JIRA, and other popular collaboration tools), experienced a security breach:

Around 9pm U.S. PST Sunday evening, Atlassian detected a security breach on one of our internal systems. The breach potentially exposed passwords for customers who purchased Atlassian products before July 2008. During July 2008, we migrated our customer database into Atlassian Crowd, our identity management product, and all customer passwords were encrypted. However, the old database table was not taken offline or deleted, and it is this database table that we believe could have been exposed during the breach.

Instead of keeping the break-in private, and hoping for the incident to blow over quickly, they emailed their entire customer base the very next day with the gory details:


It turned out that this email alarmed a number of customers who had to no reason to worry (as their accounts were unaffected), which led to another email:


Along with this email, Atlassian went further and posted an extremely detailed postmortem of the entire event, detailing who was impacted, actions you need to take as a customer, lessons learned, and next steps that they are taking to improve for the future. Incidentally, this postmortem would fair very well if run through our postmortem best practices (even though the incident is completely different from a downtime event, for which the best practices were formed).

The Pay Off
Normally, an incident like this should create a large number of very unhappy customers. Instead, thanks to the quick, honest, and transparent response, we see the following reaction:










...and if you think I'm just picking out the positive reactions, compare a Twitter search for "Atlassian" with "positive" and "negative" sentiment. At the time of this writing, there are over 3 pages of positive results, and less than one page of negative (and many of the negative is unrelated to this incident). And this is after a major security breach.

Clearly a case of transparency turning a disaster into an opportunity, and how to take advantage of that opportunity by being open and honest with your users.

Zendesk - Transparency in action

A colleague pointed me to a simple postmortem written by the CEO of Zendesk, Mikkel Svane:

"Yesterday an unannounced DNS change apparently made our mail server go incognito to the rest of the world. The consequences of this came sneaking over night as the changes propagated through the DNS network. Whammy.

On top of this our upstream internet provider late last night PST (early morning CET) experienced a failure that prevented our servers from reaching external destinations. Web access was not affected but email, widget, targets, basically everything that relied on communication from our servers to the outside world were. Double whammy.

It took too long time to realize that we had two separate issues at hand. We kept focusing on the former as root cause for the latter. And it took unacceptably long to determine that we had a network outage."

How well does such an informal and simple postmortem stack up against the postmortem best practices? Let's find out:

Prerequisites:

1. Admit failure - Yes, no question.
   
2. Sound like a human - Yes, very much so.
   
3. Have a communication channel - Yes, both the blog and the Twitter account.
4. Above all else, be authentic - Yes, extremely authentic.
   

Requirements:

1. Start time and end time of the incident - No.
   
2. Who/what was impacted - Yes, though more detail would have been nice.
   
3. What went wrong - Yes, well done.
   
4. Lessons learned - Not much.
   

Bonus:

1. Details on the technologies involved - No
2. Answers to the Five Why's - No
3. Human elements - Some
   
4. What others can learn from this experience - Some

Conclusion:

The meat was definitely there. The biggest missing piece is insight into what lessons were learned and what is being done to improve for the future. Mikkel says that "We've learned an important lesson and will do our best to ensure that no 3rd parties can take us down like this again", but the specifics are lacking. The exact time of the start and end of the event would have been useful as well, for those companies wondering whether this explains their issues that day.

It's always impressive to see the CEO of a company put himself out there like this and admit failure. It is (naively) easier to pretend like everything is OK and hope the downtime blows over. In reality, getting out in front of the problem and being transparent, communicating during the downtime (in this case over Twitter), and after the event is over (in this postmortem), are the best things you can do to turn your disaster into an opportunity to increase customer trust.

As it happens, I will be speaking at the upcoming Velocity 2010 conference about this very topic!

Update: Zendesk has put out a more in-depth review of what happened, which includes everything that was missing from the original post (which as the CEO pointed out in the comments, was meant to be a quick update of what they knew at the time). This new post includes the time frame of the incident, details on what exactly went wrong with the technology, and most importantly lessons and takeaways to improve things for the future. Well done.

Google App Engine downtime postmortem, nearly a perfect model for others

Google posted one of the most detailed and well thought out postmortems I've seen to explain what happened around their 2/24/10 App Engine Downtime. Let's run it through the gauntlet:

Prerequisites:

1. Admit failure - Yes
2. Sound like a human - Yes, more in some sections then others
3. Have a communication channel - Yes, the Google App Engine Downtime Notify group. Ideally it would have been linked to from the App Engine System Status Dashboard as well.
4. Above all else, be authentic - Yes

Requirements:

1. Start time and end time of the incident - Yes, including GMT time, and a highly detailed timeline of the entire event
2. Who/what was impacted - Partly, and also partly covered during the actual incident
3. What went wrong - Yes, yes, and yes! Incredible amount of detail here.
4. Lessons learned - Yes! Not only are there five specific action items, they are also introducing new architectural changes and customer choice as a result.

Bonus:

1. Details on the technologies involved - Somewhat
2. Answers to the Five Why's - No
3. Human elements - heroic efforts, unfortunate coincidences, effective teamwork, etc - Partial
4. What others can learn from this experience - Partial

Takeaways and thoughts:

1. A vast majority of the issues were training related. This is an important lesson: all of the technology and process in the world won't help you if your on-call team is unaware of what to do. This is especially true during the stress of a large incident. Follow the advice of Google and run regular on-call drills (including rare issues), keep documentation updated, and give on-call people the authority to make decisions on the spot.
2. Extremely impressed with the decision to use this opportunity to improve their service by giving their customers the choice between datastore performance and reliability. This is a perfect example of turning downtime into a positive.
3. Interesting insight into their process of detecting an incident to communicating externally. From the start of the incident at 7:48am to the first external communication at 8:01am is not too shabby. Not sure why it took so long to post the update to the downtime forum and the health status dashboard (8:36am).
4. The amount of time and thought that went into this postmortem shows how much Google is concerned about their service, and impressions around its reliability.

What could be improved:

1. External communication could be faster. No reason not to post something as soon as the investigation begins, not to mention posting to the forum dedicated to downtime notifications and the health status dashboard immediately. When the incident started the dashboard had very limited data, which should be automatic and real-time.
2. A post to this postmortem from the health status dashboard would make it a lot easier to find. I didn't see this until someone sent it to me.
3. Timelines and concrete deliverables on the changes (e.g. on-call training sessions, documentation updates, new datastore feature release) would give us more confidence that things will actually change.

A guideline for postmortem communication

Building on previous posts, the following is a proposed Guideline for Postmortem Communication:

Prerequisites:

1. Admit failure - Hiding downtime is no longer an option
2. Sound like a human - Do not use a standard template, do not apologize for "inconveniencing" us
3. Have a communication channel - Set up a process to handle incidents prior the event (e.g. public health dashboard, status blog, twitter account, etc.)
4. Above all else, be authentic - You must be believed to be heard

Requirements:

1. Start time and end time of the incident
2. Who/what was impacted - Should I be worried about this incident?
3. What went wrong - What broke and how you fixed it (with insight into the root cause analysis process)
4. Lessons learned - What's being done to improve the situation for the future, in technology, process, and communication

Bonus:

1. Details on the technologies involved
2. Answers to the Five Why's
3. Human elements - heroic efforts, unfortunate coincidences, effective teamwork, etc
4. What others can learn from this experience

As Wordpress goes down, a chance to analyze another postmortem arrises

If you recall, we put together a proposed guideline for postmortem communication in a previous post:

Prerequisites

1. Admit failure - Hiding downtime is no longer an option (thanks to Twitter)
2. Sound like a human - Do not use a standard template, do not apologize for "inconveniencing" us.
3. Have a communication channel - Ideally you've set up a process to handle incidents before the event, and communicated publicly during the event. Customers will need to know where to find your updates.
4. Above all else, be authentic

Requirements:

1. Start time and end time of the incident.
2. Who/what was impacted.
3. What went wrong, with insight into the root cause analysis process.
4. What's being done to improve the situation, lessons learned.

Nice-to-have's:

1. Details on the technologies involved.
2. Answers to the Five Why's.
3. Human elements - heroic efforts, unfortunate coincidences, effective teamwork, etc.
4. What others can learn from this experience.

How did Wordpress do in their postmortem on 2/19/10?

Prerequisites:

1. Admit failure: Yes. The very first paragraph makes it clear they screwed up.
2. Sound like a human: Yes. Extremely personal post.
3. Have a communication channel: Yes, but not ideal. A combination of their general Twitter account, and the founders blog. Could be improved, but overall OK.
4. Be authentic: Yes. 110% authentic!

Requirements:

1. Start/end time: No. Only focused on duration.
2. Who/what was impacted: Yes. Describes that 10.2 million blogs were affected, for 110 minutes, taking away 5.5 million pageviews.
3. What went wrong: Yes. Router issues, though investigation is continuing.
4. Lessons learned: Partial. Mostly a promise to share the results of the investigation.

Nice-to-have's:

1. Technologies involved: No.
2. Answers to the Five Why's: No.
3. Human elements: Yes. "the entire team was on pins and needles trying to get your blogs back as soon as possible"
   
4. What others can learn: No.

Conclusion

The intent of the blog was to communicate quickly that they are aware of the severity of the issue and are taking it seriously. The details are lacking, mostly because it was posted so quickly. Still utility of this kind of post is extremely powerful, which makes me wonder if having a pre-postmortem with a simple admittance the issue, with an authentic voice and some detail, is a necessary step in the pre/during/post event communication process.

Downtime post-mortems, and a look at oneforty.com

Transparency happens one incident at a time, one process at a time. True transparency, the kind that benefits both the customer and the company, doesn't come easy. It requires pre-incident planning, intra-incident collaboration, and post-incident communication. I plan to blog about this holistic framework in the near future, but today I'd like to use the downtime postmortem posted by Mike Champion, describing the recent downtime of oneforty.com, to build a basic template for how to handle post-incident communication.

The Incident

On January 14th 2010, oneforty.com experienced a "hiccup":

A few weeks ago we rolled out an alpha version of our ecommerce platform and the news was covered on a few blogs, including TechCrunch. At roughly the same time (it seemed) there were alerts about the amount of swap space on one or more of our servers. The alerts would typically flap between a warning and then return to normal levels. I figured the two events were related and that the alerts were due to increased traffic, but not a serious issue.

The post goes on to describe (in detail) what went wrong, actions taken during the event, and lessons learned. An excellent post-mortem by any standard. Simply posting a post-mortem publicly is (sadly) a huge achievement. What can we learn from this post, and what should your post-mortem's include? Let me propose a rough guideline...

A guideline for post-mortem communication

Prerequisites

1. Admit failure - Hiding downtime is no longer an option (thanks to Twitter)
2. Sound like a human - Do not use a standard template, do not apologize for "inconveniencing" us.
3. Have a communication channel - Ideally you've set up a process to handle incidents before the event, and communicated publicly during the event. Customers will need to know where to find your updates.
4. Above all else, be authentic

Requirements:

1. Start time and end time of the incident.
2. Who/what was impacted.
3. What went wrong, with insight into the root cause analysis process.
4. What's being done to improve the situation, lessons learned.

Nice-to-have's:

1. Details on the technologies involved.
2. Answers to the Five Why's.
3. Human elements - heroic efforts, unfortunate coincidences, effective teamwork, etc.
4. What others can learn from this experience.

How did oneforty.com do?

Prerequisites:

1. Admit failure: Yes, using their Twitter account.
2. Sound like a human: Yes. Very non-generic and highly detailed post.
3. Have a communication channel: Partial. The Twitter account exists, and the blog by Mike Champion exists, but it would be hard pressed as a user to find these two venues when the service is down and I need to know what's going on.
4. Be authentic: Yes.

Requirements:

1. Start/end time: No. Can't find that anywhere, and have to extrapolate it from the first tweet to the last.
2. Who/what was impacted: No. I have to assume the entire site and all visitors were impacted, but there was no mention of this.
3. What went wrong: Yes. A lot of detail, takes us through the entire experience of root cause analysis.
4. Lessons learned: Yes. Extremely solid.

Nice-to-haves:

1. Technologies involved: Yes.
2. Answers to the Five Why's: No.
3. Human elements: Yes. An engaging story.
4. What others can learn from this experience: Yes, a lot to take away if you are an Engine Yard customer.

Conclusion

The basic aim of a post-mortem is to reassure your customers that you recognize there was a problem, that you have resolved it, and that you are going to learn from the experience. Holistically, oneforty.com accomplished this. When thinking about your own post-mortems postings, just imagine an angry customers that has lost faith in your service, and needs to be reassured that you know what you are doing. If you can accomplish that you've succeeded.