Mean Time to Resolve (MTTR) –What It Is? and how to reduce it using Squadcast.

September 3, 2019
Share this post:
Mean Time to Resolve (MTTR) –What It Is? and how to reduce it using Squadcast.

Mean Time to Resolve (MTTR) is a performance metric that indicates the average time required to resolve an incident. Leverage Squadcast actions to reduce MTTR. Learn more.

Table of Contents:

    Once an incident is detected, taking the right actions automatically and immediately is the easiest step to make a sustainable and measurable improvement to your MTTR.

    Did you land here searching for a way to reduce MTTR as a DevOps/SRE or reliability engineer? If yes, then you are in the right place. If not, you should still read on if you care about the reliability of the system you are building.

    MTTR stands for Mean Time To Resolve is a widely used metric in the realm of systems reliability. However, people tend to interpret  MTTR differently. A temporary patch to get systems up and running may be considered a resolution in some teams, even if the root cause requires a more long-term fix. Regardless of its different definitions, MTTR is a crucial metric because its a measure of operational resilience and is closely linked to your uptime. And most importantly, there is a universal need to keep this number down as it has a direct impact on revenue and customer happiness.

    A recent study conducted by devops.com tries to measure the impact of downtime and the numbers are quite staggering

    • For the Fortune 1000, the average total cost of unplanned application downtime per year is $1.25 billion to $2.5 billion.
    • The average hourly cost of an infrastructure failure is $100,000 per hour.
    • The average cost of a critical application failure per hour is $500,000 to $1 million.

    It stands to reason, then, that engineering teams should strive to decrease their overall MTTR. But one of the biggest challenges that DevOps and IT teams face today is the inability to quickly take obvious mitigation actions when an incident has been detected - this, in turn, leads to increased TTR.

    The time taken to detect a problem or an incident depends on:

    1. A variety of logs, monitoring tools and other solutions in place
    2. The efficacy, and accessibility of these tools and
    3. Dependencies on other teams and systems.

    Once an incident is detected, taking the right actions automatically and immediately is the easiest step to make a sustainable and measurable improvement to your MTTR.

    Now this means not just alerting the right responder on time, but also triggering certain scripts and failsafes based on incident severity and context to minimize end-user impact.

    So, we thought - what if you could get push notification alerts for events, and you can just swipe those notifications to acknowledge and take basic mitigation actions. You don’t need to get to your laptop, or run stuff on the terminal, or log in to a bunch of other tools like CI/CD, Infra automation or Testing platforms. Sounds intriguing? Check out how it works.

    MTTR calculation example

    MTTR (Mean Time to Resolve) =  Total Time taken to resolve all the incidents at hand during a period of time /Incidents

    For instance: 

    A system has 4 incidents in a year, with a resolution time of 6 hours for the first incident, 8 hours for the second incident, 10 hours for the third incident, and 12 hours for the fourth incident. 

    Using MTTR = = Total Time taken to resolve all the incidents at hand during a period of time /Incidents

    MTTR = (6 hours + 8 hours + 10 hours + 12 hours)/4 incidents = 9 hours 

    MTTR Standard Value

    There is no standard value for MTTR (Mean Time To Resolve). Although a lower MTTR is preferred, it varies depending upon the systems, organization, and industry. More factors such as existing incident management processes, existing resources & capabilities along with complexity and criticality of the systems can affect the value of MTTR. 

    Mean Time to Repair vs Mean Time to Resolve

    Mean Time to Repair and Mean Time to Resolve are often interchangeably used, but have key differences. 

    Mean Time to Repair is the average time taken to restore a failed system. It is calculated as the total downtime experienced by the system during a period of time divided by the total incidents experienced. Whereas Mean Time To Resolve is the average time taken to resolve an incident once it has been reported either by a user or a system. It is calculated as Total Time taken to resolve all the incidents at hand during a period of time divided by incidents. 

    The Mean Time To Repair is a useful metric for assessing the effectiveness of your technical team’s efforts in resolving issues, while Mean Time To Resolve is a metric to assess your organizational processes and procedures related to incident management.

    MTTR vs. MTBF

    Businesses need to resolve any issues as soon as possible in order to keep their systems running smoothly and avoid any problems. Reactive metrics like Mean Time To Resolve (MTTR) and proactive metrics like Mean Time Between Failures (MTBF) are crucial for this purpose.

    Mean Time to Resolve (MTTR) is the amount of time it takes to resolve an incident after it has occurred, and Mean Time Between Failures (MTBF) is the average time interval between two failures of a system or equipment.

    Lower MTTR (Mean Time to Resolve) indicates issues are resolved faster and higher MTBF (Mean Time Before Failure) means a system is more reliable.

    How Do You Lower MTTR?

    There are several methods to reduce MTTR, including:

    • Defined Incident Response Plan: An incident response plan reduces MTTR by providing a framework for dealing with different types of incidents, and ensuring that clear roles and responsibilities are assigned.  
    • Incident Identification and Triage: Monitoring tools combined with clear incident reporting and triage procedures can quickly identify and prioritize incidents, thereby reducing the time taken to resolve. 
    • Automation: If you want to reduce your Mean Time To Resolve (MTTR), you can automate repetitive tasks and adopt self-healing systems that monitor and fix potential problems before they occur.
    • Create a Blameless Culture: Organizations that encourage knowledge sharing are more likely to benefit from the lessons learned from previous incidents.

    How to reduce MTTR using Squadcast Actions?

    Often, despite the DevOps/SRE/on-call team being alerted immediately about a major incident, and despite them knowing in a matter of seconds what actions need to be taken to minimize end-user impact, it still takes a few minutes and sometimes hours to recover due to human factors. This is especially true if the SRE/on-call engineer is outside work hours, or away from their computer.

    Needless to say, actual incident resolution can take many hours or even days depending on the triage time, access to key data/information, cooperation from other colleagues. But in cases like this, a quick recovery to a state where the end-user impact is inconsequential should be the only acceptable behavior. Empowering on-call teams to quickly take obvious and necessary actions can save the day (and most importantly, avoid those dreadful 3 AM calls!)

    This is why we built Squadcast Actions - a convenient and practical way to respond to incidents on time. At Squadcast, we obsess about improving the on-call experience and reducing the inherent stress of dealing with incidents.

    Squadcast Actions allow you to take actions directly from within the platform. You can take quick actions such as

    • Acknowledging or resolving an incident
    • Rebuilding a project
    • Rebooting a server
    • Rolling back a feature
    • Running custom scripts and much more

    All this with just a tap, thus making it easy to do tasks that are otherwise manual and repetitive. Or in other words - Reducing toil for your team.

    Error in java main activity - squadcast

    For instance, one of the actions that you can take is “Rebuilding CircleCI” projects directly from the incident page by clicking on the More actions button. (Note that in order do this, CircleCI Integration with Squadcast must be first completed)

    Rebuilding CircleCI - Squadcast

    You can also see the actions performed listed in chronological order as part of the Incident timeline. The incident timeline is intended to serve as your single source of truth of who did what and when, while the incident was live.

    Error in java main activity - squadcast

    Incident response on the go - Squadcast Actions on Mobile

    The best part about taking actions is doing it on the go - be it while you are enjoying a scrumptious meal with your colleagues at lunch or during your tiring commute to and fro from work.Our fully functional native apps on both Android and iOS platforms make it easy to respond to critical incidents with pre-defined actions.

    respond to critical incidents by Android and iOS apps

    Here’s a quick sneak peek

    Effective incident management not only requires sending the right information to the right on-call responders but also enabling your team with the right tools to act swiftly. Combining Squadcast with an existing incident management workflow allows DevOps/SRE professionals to efficiently track, analyze and resolve incidents.

    Enjoyed this? If you have come this far then you should definitely check out some cool new features that we are currently working on, available on our product road map.

    We love your comments. What do you struggle with as a DevOps/SRE? Do you have ideas on how incident response could be done better in your organization?

    We would love to hear from you! Leave us a comment or reach out over a DM via Twitter and let us know your thoughts.

    squadcast
    Written By:
    September 3, 2019
    September 3, 2019
    Share this post:
    Subscribe to our latest updates
    Enter your Email Id
    Thank you! Your submission has been received!
    Oops! Something went wrong while submitting the form.
    More from
    Anusuya Kannabiran
    A New Era for Squadcast
    A New Era for Squadcast
    December 12, 2022
    Towards More Effective Incident Postmortems
    Towards More Effective Incident Postmortems
    April 27, 2020
    Transparency in Incident Response
    Transparency in Incident Response
    December 16, 2019
    Learn how organizations are using Squadcast
    to maintain and improve upon their Reliability metrics
    Learn how organizations are using Squadcast to maintain and improve upon their Reliability metrics
    mapgears
    "Mapgears simplified their complex On-call Alerting process with Squadcast.
    Squadcast has helped us aggregate alerts coming in from hundreds...
    bibam
    "Bibam found their best PagerDuty alternative in Squadcast.
    By moving to Squadcast from Pagerduty, we have seen a serious reduction in alert fatigue, allowing us to focus...
    tanner
    "Squadcast helped Tanner gain system insights and boost team productivity.
    Squadcast has integrated seamlessly into our DevOps and on-call team's workflows. Thanks to their reliability...
    Alexandre Lessard
    System Analyst
    Martin do Santos
    Platform and Architecture Tech Lead
    Sandro Franchi
    CTO
    Squadcast is a leader in Incident Management on G2 Squadcast is a leader in Incident Management on G2 Users love Squadcast on G2 Best IT Management Products 2022 Squadcast is a leader in IT Service Management (ITSM) Tools on G2 Squadcast is a leader in IT Service Management (ITSM) Tools on G2 Squadcast is a leader in IT Service Management (ITSM) Tools on G2
    Squadcast awarded as "Best Software" in the IT Management category by G2 🎉 Read full report here.
    What our
    customers
    have to say
    mapgears
    "Mapgears simplified their complex On-call Alerting process with Squadcast.
    Squadcast has helped us aggregate alerts coming in from hundreds of services into one single platform. We no longer have hundreds of...
    Alexandre Lessard
    System Analyst
    bibam
    "Bibam found their best PagerDuty alternative in Squadcast.
    By moving to Squadcast from Pagerduty, we have seen a serious reduction in alert fatigue, allowing us to focus...
    Martin do Santos
    Platform and Architecture Tech Lead
    tanner
    "Squadcast helped Tanner gain system insights and boost team productivity.
    Squadcast has integrated seamlessly into our DevOps and on-call team's workflows. Thanks to their reliability metrics we have...
    Sandro Franchi
    CTO
    Revamp your Incident Response.
    Peak Reliability
    Easier, Faster, More Automated with SRE.
    Incident Response Mobility
    Manage incidents on the go with Squadcast mobile app for Android and iOS devices
    google playapple store
    Squadcast - On-call shouldn't suck. Incident response for SRE/DevOps, IT | Product Hunt Embed
    Squadcast is a leader in Incident Management on G2 Squadcast is a leader in Incident Management on G2 Users love Squadcast on G2 Best IT Management Products 2022 Squadcast is a leader in IT Service Management (ITSM) Tools on G2 Squadcast is a leader in IT Service Management (ITSM) Tools on G2 Squadcast is a leader in IT Service Management (ITSM) Tools on G2
    Squadcast - On-call shouldn't suck. Incident response for SRE/DevOps, IT | Product Hunt Embed
    Squadcast is a leader in IT Service Management (ITSM) Tools on G2 Squadcast is a leader in Incident Management on G2 Users love Squadcast on G2
    Best IT Management Products 2022 Squadcast is a leader in IT Service Management (ITSM) Tools on G2 Squadcast is a leader in IT Service Management (ITSM) Tools on G2
    Squadcast is a leader in IT Service Management (ITSM) Tools on G2
    Copyright © Squadcast Inc. 2017-2023