The Upside of Downtime (Velocity 2010)

Here is the full deck from my talk at Velocity, including two bonus sections at the end:

The Upside of Downtime (Velocity 2010)


Also, here is the "Upside of Downtime Framework" cheat-sheet (click through to download):

See you at Velocity 2010!

Tonight I leave for the (sold out) O'Reilly Velocity conference in Santa Clara, CA. I'll be presenting "The Upside of Downtime: How to Turn a Disaster into an Opportunity" on Wednesday at 4:35pm. If you're a reader of this blog and are at the conference, I'd love to meet up! Tweet me @lennysan or simply leave a comment here.

As soon as my talk ends, I will be posting the full slide-deck right here on this blog. Stay tuned!

P.S. If you're reading this post during my talk, here are some of the links I may or may not be referencing:

• 7 Keys To a Successful Public Health Dashboard
• A Guideline for Postmortem Communication
• Google App Engine Downtime Postmortem
• How Transparency Can Help your Business
• The Tao of Web Performance and Uptime

Quick update (and Velocity preview)

Alas this blog has been quite for too long. My pathetic excuse is that I'm channeling the efforts that would normally go to this blog into my upcoming talk at Velocity. To make up for my negligence, here is a sneak peek at the talk:

I will post the entire slide deck here on the blog immediately following the talk. Stay tuned!

A proposal for new community focused on web performance

I've been really impressed with the StackExchange platform (http://www.stackexchange.com, made by the same people that run stackoverflow.com), and I feel that it could be an extremely effective platform to host a web performance focused community. They built the platform from scratch in order to improve on the innate flaws with regular threaded discussion boards (e.g. Yahoo forums, Google Groups, phpBB, vBulletin, etc.). More importantly, the platform walks the line between incentivizing quick answers (for immediate feedback), and keeping answers from getting obsolete over time.

My hope is that this site becomes an evolving source of definitive answers on web performance best practices, tips, tool tricks, book recommendations, data exchange, etc.

The process to make this a reality is:

1. Submit a proposal for peer review

2. If there is enough support (votes), it moves on to the next stage.

3. People that would like to participate in the community (and help manage it) sign up

4. The details of the community get ironed out (moderators, name, tags, etc.)

5. It goes public

I've gone ahead and submitted the initial proposal (step 1):

http://meta.stackexchange.com/questions/5821/proposal-for-stackexchange-site-focused-on-web-site-performance

I'm just here to get the initial ball rolling, but from here on out it's going to be all about the greater community. This next stage, where everyone votes on the proposals, is going to make or break the concept. It's already received a good amount of votes, but it's going to take a lot more support to push it forward. If you think this has legs, and can see the value, vote it up!

Transparent censorship

Google has decided to fight censorship with transparency:

...it's no surprise that Google, like other technology and telecommunications companies, regularly receives demands from government agencies to remove content from our services. Of course many of these requests are entirely legitimate, such as requests for the removal of child pornography. We also regularly receive requests from law enforcement agencies to hand over private user data. Again, the vast majority of these requests are valid and the information needed is for legitimate criminal investigations. However, data about these activities historically has not been broadly available. We believe that greater transparency will lead to less censorship.

To this end they have launched a Government Requests tool:

We are today launching a new Government Requests tool to give people information about the requests for user data or content removal we receive from government agencies around the world. For this launch, we are using data from July-December, 2009, and we plan to update the data in 6-month increments. Read this post to learn more about our principles surrounding free expression and controversial content on the web.

Takeaway: If you are forced to do something bad, tell everyone about it.

A basic introduction to the Cloud

I've been getting requests recently to give a high-level overview of "The Cloud". What it means, why people are excited, and what they should care. I decided to put together a basic introduction of the Cloud (embeded below). Feel free to use parts of this in your own presentations, and I'd love to hear feedback on this:

The Cloud - An introduction

View more presentations from Lenny Rachitsky.

Atlassian has security breach, responds with transparency, sees benefits

This past Sunday, Atlassian (makers of Confluence, JIRA, and other popular collaboration tools), experienced a security breach:

Around 9pm U.S. PST Sunday evening, Atlassian detected a security breach on one of our internal systems. The breach potentially exposed passwords for customers who purchased Atlassian products before July 2008. During July 2008, we migrated our customer database into Atlassian Crowd, our identity management product, and all customer passwords were encrypted. However, the old database table was not taken offline or deleted, and it is this database table that we believe could have been exposed during the breach.

Instead of keeping the break-in private, and hoping for the incident to blow over quickly, they emailed their entire customer base the very next day with the gory details:


It turned out that this email alarmed a number of customers who had to no reason to worry (as their accounts were unaffected), which led to another email:


Along with this email, Atlassian went further and posted an extremely detailed postmortem of the entire event, detailing who was impacted, actions you need to take as a customer, lessons learned, and next steps that they are taking to improve for the future. Incidentally, this postmortem would fair very well if run through our postmortem best practices (even though the incident is completely different from a downtime event, for which the best practices were formed).

The Pay Off
Normally, an incident like this should create a large number of very unhappy customers. Instead, thanks to the quick, honest, and transparent response, we see the following reaction:










...and if you think I'm just picking out the positive reactions, compare a Twitter search for "Atlassian" with "positive" and "negative" sentiment. At the time of this writing, there are over 3 pages of positive results, and less than one page of negative (and many of the negative is unrelated to this incident). And this is after a major security breach.

Clearly a case of transparency turning a disaster into an opportunity, and how to take advantage of that opportunity by being open and honest with your users.