Incident Response in the time of Remote Work

Incident Communication

The cornerstone of any good incident response process is communication.

Document more: One thing to keep in mind to reduce the risk of misinformation or communication gaps, is to write more and write better. It’s always better to have a record of information and associated activity to go back to, if necessary. When in doubt, throw in a few more details.

Use a central Slack Channel: For those of you that love chatops or depend majorly on Slack for incident management, use a dedicated channel to bring in all your incidents. You might have to create separate channels for communicating with regard to specific issues and outages. But a central channel can act as an index, and prevent the chaos of looking for a specific incident and its status.

Virtual War Room: Goes without saying that collaboration is key to reducing your MTTR. You can mimic a traditional war room huddle by using a video conferencing tool or chat platform with the incident response team. With Squadcast, you can use our virtual war-room where you can chat, bring in other members from your team, SMEs, Stakeholders and business facing teams to ensure that all your goals are aligned.

Publish Meeting link: You can create a virtual meeting room for just the fire-fighting and keep that open throughout your on-call rotation. You can add the meeting ID along with the incident details or pin the details in the Slack channel or any other communication tool you use.

We typically use Zoom to keep an open meeting room that anyone with the meeting ID can join in. You should be able to do it with other tools as well.

Talking is faster than typing, so it can be tempting to just call up with every doubt; however, use it only if the situation calls for it. No one likes to be constantly interrupted.

Be transparent: Communication can take a major hit with work from home teams. This happens simply because you may think you’re communicating all the available information you know but may miss out on some prerequisites needed to comprehend the information better.

To avoid this, it’s great to just add in all of the relevant teams while dealing with an incident. Also, remember to update your status page with any new necessary information immediately post incident resolution.

This opens them up to all the activity taking place, the severity of the issue and acts as a single platform to discuss and share. With everyone always informed, you don’t have to struggle with context switching for just drafting the right message to send to external teams or customers.

Incident Response

Once you get the communication processes right, incident response gets simpler. You can focus on firefighting without having to worry about anything else.

Assign Roles: You are quickest when you know what you need to do. This kind of clarity can be achieved by simply assigning roles to your incident response team. This also helps distribute work that would otherwise fall on just the one person figuring out a fix.

For teams of just 1 or 2, just a checklist of items to do when an incident hits can go a long way. This clears the mind of any doubts about pending work to do.

Timeline of Incident Activity: Usually, a scribe is expected to maintain a record of all incident related activities. It’s always a good idea to not trust your memory in a high-stress situation. This allows you to have all the information necessary to analyze better, write better postmortems and create an effective playbook as a pre-emptive measure. With Squadcast, we use our automated timelines to understand the resolution activities while conducting postmortems.

Set-up an automated on-call rotation: If you haven’t already set this up, you can expect a downhill graph of motivation for all the engineers that do this today. It is highly likely that when you don’t have a rotation set up, the stress of incidents fall on just 1 or few.

It’s a load off your mind if you just knew beforehand when you’d have to go on-call. Rotations also help you assign appropriate load to everyone on the team.

Remember, being on-call is everyone’s responsibility.

Incident Resolution

Always Create Runbooks (with fallback options): It’s useful to create a knowledge base of all incident resolution information that one can refer to when similar incidents hit your service. This way, you don’t have to spend time figuring out the incident all over again.

Runbooks are especially useful to folks who are new to on-call or newer in your organization. It’s always good to have more information when you’re new.

Blameless Postmortems: Another great source of information is postmortems and post incident reviews. Not a lot of organizations follow through to finish a postmortem simply because it’s a long, tedious and sometimes stressful process. But the best way to ensure that an incident doesn’t occur again is to analyze why it happened in the first place and then making this information available for the entire team. In Squadcast, you can create postmortems of incidents from within the app and can be viewed by anyone on your team.

The unexpected and sudden shift to remote working introduces new risks. And while each organization needs to take its own unique circumstances into account, the aforementioned practices offer a step in the right direction in keeping operations both productive and proactive.