📢 Webinar Alert! Reliability Automation - AI, ML, & Workflows in Incident Management. Register Here
Blog
SRE
Transparency in Incident Response

Transparency in Incident Response

December 16, 2019
Transparency in Incident Response
In This Article:
Our Products
On-Call Management
Incident Response
Continuous Learning
Workflow Automation

An often overlooked bedrock of Site Reliability Engineering (SRE)


When your production systems are hit with a critical issue, you can trust your DevOps team, your Sysadmins or your SREs to get the system back on track. This is a no brainer.

And in turn, these folks need to be able to trust the rest of the team to let them do their jobs, be it engineering, customer support or product management. But where does this trust come from? It comes from understanding - the more you understand, the more you can trust. But when there is obscurity, it severely impedes understanding.

This is why transparency matters.

In most organizations, the pursuit of reliability is often blended with obscurity. There is strict access control and sometimes this means relevant people not having access to even basic observability metrics. This leads to increased stress when systems go down, with people pointing fingers & assigning blame to things they may not fully understand.

As a general rule of thumb, higher transparency not only results in a better incident management and response process but more importantly also increases trust between team members and gives them a way to calmly figure out what went wrong before fixing it for good.

In this article, we are going to outline how you can cultivate transparency in your team and benefit from it.

Evolution of transparency in Tech teams

Transparency, although not a traditional goal of tech teams while handling incidents has evolved over the years to become an auxiliary objective. The most recent developments in terms of enabling transparency are widespread adoption of incident management and alert notification tools that help you plan, track, and work faster and more efficiently in an ever-changing environment. By offering increased visibility into tasks and who owns these tasks, these tools facilitate better collaboration. But on the other hand, if transparency is regarded as one of the primary objectives of teams doing incident response, the productivity gains increase manifold.

Transparency as a primary objective

In order to make transparency as a primary objective, it is important to think about what would be the milestones in your journey. You can choose these milestones based on the four progressive levels of transparency we’ve seen many tech teams use as a reference. These levels are:

  • Level 1 - Engineering Transparency
    The first level of transparency is purely internal to the engineering team. Once you have selected the metrics that are the most crucial and their target range of values i.e. your Service Level Indicators (SLIs) and Service Level Objectives (SLOs) - you can then share these across the entire engineering team (instead of restricting only to specific folks) through (i) status pages private to your team, (ii) centralized incident timelines that are accessible to every developer, and (iii) opening up the incident response documentation like post-mortems, runbooks and other best practices. This level of transparency is gated and serves to help the engineering team collaborate better.
  • Level 2 - Organizational Transparency
    Taking it a notch higher, you can expose this same information to the entire organization including product, support and business teams. You can start to do this by first setting your SLOs in collaboration with customer facing teams. The outcome at this level of transparency is increased trust in the engineering team and better communication with external stakeholders like customers, partners, resellers, etc.  
  • Level 3 - Stakeholder Transparency
    The third level of transparency is where you expose your incident management practices and your SLOs to all external stakeholders such as your customers, partners, resellers, vendors or anybody else that you're working with. This can be achieved with a public SLO dashboard, public status pages, and open post-mortems. The benefits at this level of transparency is higher customer loyalty, and improved brand perception.
  • Level 4 - Universal Transparency
    The final level of transparency is the holy grail where you really bare all. It's where you are public about your metrics not only to existing stakeholders but also future potential stakeholders. This is the level at which many teams tend to live stream their response to outages. Businesses at this level can be very confident about their metrics constantly improving because they are being fully transparent about them.

For various metrics and events, you can choose the level of transparency you want to go with for those specific metrics/events.

Often we need to iterate on our SLOs before we settle on what works best for particular situations. So, it's crucial that this information is made transparent at least within the engineering team, making it easier to reflect and understand if these SLOs are indeed the right ones. When you are transparent about your SLOs, you also have a better understanding of the dependencies between these SLOs. This further allows you to have better policies around your error budgets, and have a good understanding of how these SLOs interact with each other.

That being said, just because you're transparent about your SLOs doesn't mean that everyone gets to have a say in what your SLOs should be. It just means that you're communicating what is important to people across the organization. Also, there is certainly an assumption that things get more complicated if you have to be transparent about SLOs. But that's far from the truth because if you want to be transparent, the idea is to make your SLOs really simple so that even non-engineering teams can understand them. Another myth about being transparent is that it slows down processes because everybody needs to understand the SLOs. On the contrary, processes are much more streamlined and actually more effective because being transparent removes any blind spots in terms of the metrics that you're tracking.

Effective Incident response is a team effort with the right tool

Once important metrics have been identified and their target levels are defined, it is imperative that the collection of these metrics is carefully handled. Different metrics may be monitored with the help of tools like Prometheus and Datadog to collect and visualize the data. When a metric goes outside its target range of values, these tools generate an alert. Any organization with well defined SLOs will feel the need for multiple monitoring tools to track the underlying metrics or SLIs. A proper incident management tool centralizes all these alerts from the different monitoring tools and does a lot more than just alerting. It allows your team to have a robust incident response plan in place, and helps teams perform retrospectives so that repeated incidents can be resolved faster. A well-designed, dynamic incident management tool can potentially save the day, with the ability to automate a number of different incident response activities.

Each incident will have unique requirements like the data to be verified, recorded and tracked, the runbooks and processes to be followed, the stakeholders to be notified, and the reports to be filed etc.

A holistic incident response tool can address all of these. While this level of flexibility allows for individualized workflows based on the type of incident, a well-designed technology solution for incident management can also aid in providing greater transparency for all incident types.

SLOs can be much more effective if the cycle that starts with creating the objectives, ends with evaluating them based on the SLO breaches that have happened. Reevaluation of service level objectives is a must to take corrective action either by refining the indicators and their target ranges or by making the services more robust. It is crucial to design service level objectives while keeping in mind that services will fail because they inevitably will.

Implementation of Transparency

Squadcast is an incident management tool that’s purpose-built for SRE. Its innovative design enables true transparency and minimizes friction in the incident response process. With transparency comes the ability to resolve incidents faster, create and iterate on SLOs and calculate error budgets to implement policies around them. This prevents re-occurrences of similar incidents and allows for faster innovation and enhanced customer satisfaction.

Squadcast helps with achieving transparency through the below inbuilt aspects of the platform.

One of the cornerstones of SRE is Transparency and status pages help you communicate the status of your services internally to other teams or externally to your customers at all times.

Status Page

   ❖  Public Status Page: You can configure your public-facing services and their dependent components and show their status in real-time directly within Squadcast itself. Customers can subscribe to real-time email updates by entering their contact information in the status page.

   ❖  Private Status Page: You can expose the status of your internal services privately to other internal teams. You can check who is working on an incident if the service is facing issues. You can also page teams responsible for specific services.

See all your configured SLOs on a single dashboard. Analyze breaches instantly with a quick snapshot of SLIs rolled up for all your services. Squadcast allows you to track Service Level Indicators (SLIs) like uptime, latency, throughput volume, availability, etc.. Set custom thresholds to get notified when breached. The SLO dashboard is accessible to all users of Squadcast.

SLO Dashboard
  • Collaboration with transparency

Communicate and collaborate to resolve incidents quickly irrespective of your location with the help of

   ❖  Virtual War Rooms: Incident-specific war rooms within the Squadcast platform enable real-time collaboration between all responders.

War Room

   ❖  Incident Timeline: All incident response activities are recorded in real time in chronological order. The incident timeline can be downloaded in CSV and PDF formats and can be shared with the rest of your team.

   ❖  ChatOps integration: Bidirectional integration with collaboration tools like Slack allows teams to track incident specific conversations on the war room. All responses from a Slack channel will be reflected on the Incident war room and vice versa.

Conclusion:

Doing incident management the SRE way helps you develop operational transparency across your organization. And you can choose different levels of transparency for different metrics and events that you may have. With a single source of truth for metrics, logs, events, traces, incident information and response, your team can be empowered to quickly access the information they need with sufficient context and collaborate to quickly resolve incidents.

With better collaboration and transparency, the overall reliability of your service improves significantly.

This article was originally a talk at SREcon'19 titled "Transparency—How Much Is Too Much". Slides available here.

We love your comments. What do you struggle with as a DevOps/SRE? Do you have ideas on how incident response could be done better in your organization?

We would be thrilled to hear from you! Leave us a comment or reach out over a DM via Twitter and let us know your thoughts.

Written By:
Amiya Adwitiya
Anusuya Kannabiran
Amiya Adwitiya
Anusuya Kannabiran
December 16, 2019
SRE
Share this blog:
In This Article:
Get reliability insights delivered straight to your inbox.
Get ready for the good stuff! No spam, no data sale and no promotion. Just the awesome content you signed up for.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
If you wish to unsubscribe, we won't hold it against you. Privacy policy.
Get reliability insights delivered straight to your inbox.
Get ready for the good stuff! No spam, no data sale and no promotion. Just the awesome content you signed up for.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
If you wish to unsubscribe, we won't hold it against you. Privacy policy.
Get the latest scoop on Reliability insights. Delivered straight to your inbox.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
If you wish to unsubscribe, we won't hold it against you. Privacy policy.
Squadcast is a leader in Incident Management on G2 Squadcast is a leader in Mid-Market IT Service Management (ITSM) Tools on G2 Squadcast is a leader in Americas IT Alerting on G2 Best IT Management Products 2024 Squadcast is a leader in Europe IT Alerting on G2 Squadcast is a leader in Enterprise Incident Management on G2 Users love Squadcast on G2
Squadcast is a leader in Incident Management on G2 Squadcast is a leader in Mid-Market IT Service Management (ITSM) Tools on G2 Squadcast is a leader in Americas IT Alerting on G2 Best IT Management Products 2024 Squadcast is a leader in Europe IT Alerting on G2 Squadcast is a leader in Enterprise Incident Management on G2 Users love Squadcast on G2
Squadcast is a leader in Incident Management on G2 Squadcast is a leader in Mid-Market IT Service Management (ITSM) Tools on G2 Squadcast is a leader in Americas IT Alerting on G2
Best IT Management Products 2024 Squadcast is a leader in Europe IT Alerting on G2 Squadcast is a leader in Enterprise Incident Management on G2
Users love Squadcast on G2
Copyright © Squadcast Inc. 2017-2024

Transparency in Incident Response

Dec 16, 2019
Last Updated:
November 20, 2024
Share this post:
Transparency in Incident Response

Transparency in incident response is often an overlooked bedrock of Site Reliability Engineering (SRE). In this blog, we talk about why transparency matters and how you can cultivate transparency in your team and benefit from the same.

Table of Contents:

    An often overlooked bedrock of Site Reliability Engineering (SRE)


    When your production systems are hit with a critical issue, you can trust your DevOps team, your Sysadmins or your SREs to get the system back on track. This is a no brainer.

    And in turn, these folks need to be able to trust the rest of the team to let them do their jobs, be it engineering, customer support or product management. But where does this trust come from? It comes from understanding - the more you understand, the more you can trust. But when there is obscurity, it severely impedes understanding.

    This is why transparency matters.

    In most organizations, the pursuit of reliability is often blended with obscurity. There is strict access control and sometimes this means relevant people not having access to even basic observability metrics. This leads to increased stress when systems go down, with people pointing fingers & assigning blame to things they may not fully understand.

    As a general rule of thumb, higher transparency not only results in a better incident management and response process but more importantly also increases trust between team members and gives them a way to calmly figure out what went wrong before fixing it for good.

    In this article, we are going to outline how you can cultivate transparency in your team and benefit from it.

    Evolution of transparency in Tech teams

    Transparency, although not a traditional goal of tech teams while handling incidents has evolved over the years to become an auxiliary objective. The most recent developments in terms of enabling transparency are widespread adoption of incident management and alert notification tools that help you plan, track, and work faster and more efficiently in an ever-changing environment. By offering increased visibility into tasks and who owns these tasks, these tools facilitate better collaboration. But on the other hand, if transparency is regarded as one of the primary objectives of teams doing incident response, the productivity gains increase manifold.

    Transparency as a primary objective

    In order to make transparency as a primary objective, it is important to think about what would be the milestones in your journey. You can choose these milestones based on the four progressive levels of transparency we’ve seen many tech teams use as a reference. These levels are:

    • Level 1 - Engineering Transparency
      The first level of transparency is purely internal to the engineering team. Once you have selected the metrics that are the most crucial and their target range of values i.e. your Service Level Indicators (SLIs) and Service Level Objectives (SLOs) - you can then share these across the entire engineering team (instead of restricting only to specific folks) through (i) status pages private to your team, (ii) centralized incident timelines that are accessible to every developer, and (iii) opening up the incident response documentation like post-mortems, runbooks and other best practices. This level of transparency is gated and serves to help the engineering team collaborate better.
    • Level 2 - Organizational Transparency
      Taking it a notch higher, you can expose this same information to the entire organization including product, support and business teams. You can start to do this by first setting your SLOs in collaboration with customer facing teams. The outcome at this level of transparency is increased trust in the engineering team and better communication with external stakeholders like customers, partners, resellers, etc.  
    • Level 3 - Stakeholder Transparency
      The third level of transparency is where you expose your incident management practices and your SLOs to all external stakeholders such as your customers, partners, resellers, vendors or anybody else that you're working with. This can be achieved with a public SLO dashboard, public status pages, and open post-mortems. The benefits at this level of transparency is higher customer loyalty, and improved brand perception.
    • Level 4 - Universal Transparency
      The final level of transparency is the holy grail where you really bare all. It's where you are public about your metrics not only to existing stakeholders but also future potential stakeholders. This is the level at which many teams tend to live stream their response to outages. Businesses at this level can be very confident about their metrics constantly improving because they are being fully transparent about them.

    For various metrics and events, you can choose the level of transparency you want to go with for those specific metrics/events.

    Often we need to iterate on our SLOs before we settle on what works best for particular situations. So, it's crucial that this information is made transparent at least within the engineering team, making it easier to reflect and understand if these SLOs are indeed the right ones. When you are transparent about your SLOs, you also have a better understanding of the dependencies between these SLOs. This further allows you to have better policies around your error budgets, and have a good understanding of how these SLOs interact with each other.

    That being said, just because you're transparent about your SLOs doesn't mean that everyone gets to have a say in what your SLOs should be. It just means that you're communicating what is important to people across the organization. Also, there is certainly an assumption that things get more complicated if you have to be transparent about SLOs. But that's far from the truth because if you want to be transparent, the idea is to make your SLOs really simple so that even non-engineering teams can understand them. Another myth about being transparent is that it slows down processes because everybody needs to understand the SLOs. On the contrary, processes are much more streamlined and actually more effective because being transparent removes any blind spots in terms of the metrics that you're tracking.

    Effective Incident response is a team effort with the right tool

    Once important metrics have been identified and their target levels are defined, it is imperative that the collection of these metrics is carefully handled. Different metrics may be monitored with the help of tools like Prometheus and Datadog to collect and visualize the data. When a metric goes outside its target range of values, these tools generate an alert. Any organization with well defined SLOs will feel the need for multiple monitoring tools to track the underlying metrics or SLIs. A proper incident management tool centralizes all these alerts from the different monitoring tools and does a lot more than just alerting. It allows your team to have a robust incident response plan in place, and helps teams perform retrospectives so that repeated incidents can be resolved faster. A well-designed, dynamic incident management tool can potentially save the day, with the ability to automate a number of different incident response activities.

    Each incident will have unique requirements like the data to be verified, recorded and tracked, the runbooks and processes to be followed, the stakeholders to be notified, and the reports to be filed etc.

    A holistic incident response tool can address all of these. While this level of flexibility allows for individualized workflows based on the type of incident, a well-designed technology solution for incident management can also aid in providing greater transparency for all incident types.

    SLOs can be much more effective if the cycle that starts with creating the objectives, ends with evaluating them based on the SLO breaches that have happened. Reevaluation of service level objectives is a must to take corrective action either by refining the indicators and their target ranges or by making the services more robust. It is crucial to design service level objectives while keeping in mind that services will fail because they inevitably will.

    Implementation of Transparency

    Squadcast is an incident management tool that’s purpose-built for SRE. Its innovative design enables true transparency and minimizes friction in the incident response process. With transparency comes the ability to resolve incidents faster, create and iterate on SLOs and calculate error budgets to implement policies around them. This prevents re-occurrences of similar incidents and allows for faster innovation and enhanced customer satisfaction.

    Squadcast helps with achieving transparency through the below inbuilt aspects of the platform.

    One of the cornerstones of SRE is Transparency and status pages help you communicate the status of your services internally to other teams or externally to your customers at all times.

    Status Page

       ❖  Public Status Page: You can configure your public-facing services and their dependent components and show their status in real-time directly within Squadcast itself. Customers can subscribe to real-time email updates by entering their contact information in the status page.

       ❖  Private Status Page: You can expose the status of your internal services privately to other internal teams. You can check who is working on an incident if the service is facing issues. You can also page teams responsible for specific services.

    See all your configured SLOs on a single dashboard. Analyze breaches instantly with a quick snapshot of SLIs rolled up for all your services. Squadcast allows you to track Service Level Indicators (SLIs) like uptime, latency, throughput volume, availability, etc.. Set custom thresholds to get notified when breached. The SLO dashboard is accessible to all users of Squadcast.

    SLO Dashboard
    • Collaboration with transparency

    Communicate and collaborate to resolve incidents quickly irrespective of your location with the help of

       ❖  Virtual War Rooms: Incident-specific war rooms within the Squadcast platform enable real-time collaboration between all responders.

    War Room

       ❖  Incident Timeline: All incident response activities are recorded in real time in chronological order. The incident timeline can be downloaded in CSV and PDF formats and can be shared with the rest of your team.

       ❖  ChatOps integration: Bidirectional integration with collaboration tools like Slack allows teams to track incident specific conversations on the war room. All responses from a Slack channel will be reflected on the Incident war room and vice versa.

    Conclusion:

    Doing incident management the SRE way helps you develop operational transparency across your organization. And you can choose different levels of transparency for different metrics and events that you may have. With a single source of truth for metrics, logs, events, traces, incident information and response, your team can be empowered to quickly access the information they need with sufficient context and collaborate to quickly resolve incidents.

    With better collaboration and transparency, the overall reliability of your service improves significantly.

    This article was originally a talk at SREcon'19 titled "Transparency—How Much Is Too Much". Slides available here.

    We love your comments. What do you struggle with as a DevOps/SRE? Do you have ideas on how incident response could be done better in your organization?

    We would be thrilled to hear from you! Leave us a comment or reach out over a DM via Twitter and let us know your thoughts.

    What you should do now
    • Schedule a demo with Squadcast to learn about the platform, answer your questions, and evaluate if Squadcast is the right fit for you.
    • Curious about how Squadcast can assist you in implementing SRE best practices? Discover the platform's capabilities through our Interactive Demo.
    • Enjoyed the article? Explore further insights on the best SRE practices.
    • Schedule a demo with Squadcast to learn about the platform, answer your questions, and evaluate if Squadcast is the right fit for you.
    • Curious about how Squadcast can assist you in implementing SRE best practices? Discover the platform's capabilities through our Interactive Demo.
    • Enjoyed the article? Explore further insights on the best SRE practices.
    • Get a walkthrough of our platform through this Interactive Demo and see how it can solve your specific challenges.
    • See how Charter Leveraged Squadcast to Drive Client Success With Robust Incident Management.
    • Share this blog post with someone you think will find it useful. Share it on Facebook, Twitter, LinkedIn or Reddit
    • Get a walkthrough of our platform through this Interactive Demo and see how it can solve your specific challenges.
    • See how Charter Leveraged Squadcast to Drive Client Success With Robust Incident Management
    • Share this blog post with someone you think will find it useful. Share it on Facebook, Twitter, LinkedIn or Reddit
    • Get a walkthrough of our platform through this Interactive Demo and see how it can solve your specific challenges.
    • See how Charter Leveraged Squadcast to Drive Client Success With Robust Incident Management
    • Share this blog post with someone you think will find it useful. Share it on Facebook, Twitter, LinkedIn or Reddit
    What you should do now?
    Here are 3 ways you can continue your journey to learn more about Unified Incident Management
    Discover the platform's capabilities through our Interactive Demo.
    See how Charter Leveraged Squadcast to Drive Client Success With Robust Incident Management.
    Share the article
    Share this blog post on Facebook, Twitter, Reddit or LinkedIn.
    We’ll show you how Squadcast works and help you figure out if Squadcast is the right fit for you.
    Experience the benefits of Squadcast's Incident Management and On-Call solutions firsthand.
    Compare our plans and find the perfect fit for your business.
    See Redis' Journey to Efficient Incident Management through alert noise reduction With Squadcast.
    Discover the platform's capabilities through our Interactive Demo.
    We’ll show you how Squadcast works and help you figure out if Squadcast is the right fit for you.
    Experience the benefits of Squadcast's Incident Management and On-Call solutions firsthand.
    Compare Squadcast & PagerDuty / Opsgenie
    Compare and see if Squadcast is the right fit for your needs.
    Compare our plans and find the perfect fit for your business.
    Learn how Scoro created a solid foundation for better on-call practices with Squadcast.
    Discover the platform's capabilities through our Interactive Demo.
    We’ll show you how Squadcast works and help you figure out if Squadcast is the right fit for you.
    Experience the benefits of Squadcast's Incident Management and On-Call solutions firsthand.
    We’ll show you how Squadcast works and help you figure out if Squadcast is the right fit for you.
    Learn how Scoro created a solid foundation for better on-call practices with Squadcast.
    We’ll show you how Squadcast works and help you figure out if Squadcast is the right fit for you.
    Discover the platform's capabilities through our Interactive Demo.
    Enjoyed the article? Explore further insights on the best SRE practices.
    We’ll show you how Squadcast works and help you figure out if Squadcast is the right fit for you.
    Experience the benefits of Squadcast's Incident Management and On-Call solutions firsthand.
    Enjoyed the article? Explore further insights on the best SRE practices.
    Written By:
    Share this post:
    Subscribe to our LinkedIn Newsletter to receive more educational content
    Subscribe now
    ant-design-linkedIN

    Subscribe to our latest updates

    Enter your Email Id
    Thank you! Your submission has been received!
    Oops! Something went wrong while submitting the form.
    FAQs
    More from
    Amiya Adwitiya
    A New Era for Squadcast
    A New Era for Squadcast
    December 12, 2022
    Announcing our $6M investment to double down on IT incident and Reliability needs
    Announcing our $6M investment to double down on IT incident and Reliability needs
    August 6, 2021
    Towards More Effective Incident Postmortems
    Towards More Effective Incident Postmortems
    April 27, 2020
    Learn how organizations are using Squadcast
    to maintain and improve upon their Reliability metrics
    Learn how organizations are using Squadcast to maintain and improve upon their Reliability metrics
    mapgears
    "Mapgears simplified their complex On-call Alerting process with Squadcast.
    Squadcast has helped us aggregate alerts coming in from hundreds...
    bibam
    "Bibam found their best PagerDuty alternative in Squadcast.
    By moving to Squadcast from Pagerduty, we have seen a serious reduction in alert fatigue, allowing us to focus...
    tanner
    "Squadcast helped Tanner gain system insights and boost team productivity.
    Squadcast has integrated seamlessly into our DevOps and on-call team's workflows. Thanks to their reliability...
    Alexandre Lessard
    System Analyst
    Martin do Santos
    Platform and Architecture Tech Lead
    Sandro Franchi
    CTO
    Squadcast is a leader in Incident Management on G2 Squadcast is a leader in Mid-Market IT Service Management (ITSM) Tools on G2 Squadcast is a leader in Americas IT Alerting on G2 Best IT Management Products 2022 Squadcast is a leader in Europe IT Alerting on G2 Squadcast is a leader in Mid-Market Asia Pacific Incident Management on G2 Users love Squadcast on G2
    Squadcast awarded as "Best Software" in the IT Management category by G2 🎉 Read full report here.
    What our
    customers
    have to say
    mapgears
    "Mapgears simplified their complex On-call Alerting process with Squadcast.
    Squadcast has helped us aggregate alerts coming in from hundreds of services into one single platform. We no longer have hundreds of...
    Alexandre Lessard
    System Analyst
    bibam
    "Bibam found their best PagerDuty alternative in Squadcast.
    By moving to Squadcast from Pagerduty, we have seen a serious reduction in alert fatigue, allowing us to focus...
    Martin do Santos
    Platform and Architecture Tech Lead
    tanner
    "Squadcast helped Tanner gain system insights and boost team productivity.
    Squadcast has integrated seamlessly into our DevOps and on-call team's workflows. Thanks to their reliability metrics we have...
    Sandro Franchi
    CTO
    Revamp your Incident Response.
    Peak Reliability
    Easier, Faster, More Automated with SRE.