Twice I’ve been complimented on my ability to manage a crisis.
A couple of years ago a colleague called me “icy cool” whilst dealing with a room full of grumpy stakeholders during a failed rollout of a new CDN config, despite working flawlessly on production. It was drama-free and simple enough to activate a rollback plan and restore service. More recently, a client suggested I give classes to colleagues on crisis management.
Here’s a random list of possible commercial and technical disasters, do you recognise any of these?
- Unrealistic client-dictated deadlines
- CDN partner borked a configuration rollout
- DNS failure
- Two of your developers have the flu right before a launch
- Third party API doesn’t return data reliably
- Content is missing or appears in the wrong part of the product
I’ve worked on a number of high-profile web sites and mobile apps powered by complicated back-end services and inevitably something does go a bit wonky along the way.
Sometimes it’s an impossible deadline with CEO-level pressure to “just get it done”, a faulty 3rd party API, a failure of cloud-based services – or just a mistake. I’ve found that the old adage is true – the important thing in a crisis is how you deal with it.
Avoid the stench of fear and panic at all costs. I’ve had colleagues who simply couldn’t deal with chaos and, when the pressure was on, I spent as much time managing their flappy response as I did on the problem itself.
When a real emergency strikes, people often forget that communication – internally and externally – comes first. Someone with the ability to understand the nature of the problem and speak to multiple stakeholders at an appropriate level must be drafted into the crisis team and dispatch regular updates with clear, authoritative information. This person neutralises any and all drama and frees up the technical response team to methodically focus on solving the problem.
Whenever I’ve been on the client side of a crisis, nothing infuriates me more than a service rep asking me to “explain where on the site you think there’s a problem”. Similarly, opaque emails about “product teams working swiftly to resolve the problem” tell me nothing about who is working on what or that what they’re working on has any relevance to my complaint.
Even with a less-critical incident, one must communicate clearly – define the problem, be clear about the desired outcome and then bring stakeholders with you along the path toward a swift, successful resolution.