📢 Webinar Alert! Future-Proofing IT Operations: How Charter Enhanced Reliability with Squadcast. Register Here! 🌟

Curb alert noise for better productivity : How-To's and Best Practices

Nov 3, 2020
Last Updated:
Nov 3, 2020
Share this post:
Curb alert noise for better productivity : How-To's and Best Practices

On the quest to provide the best uptime, software platforms depend on complex interconnected microservices. This often leaves them vulnerable to cascading failures creating a massive deluge of alerts from monitoring tools when things go wrong. In this blog, we explore how Squadcast can be configured to curb alert noise for better productivity with the help of the most advanced deduplication features.

Table of Contents:

    Modern software platforms depend on complex interconnected microservices for smooth operation. As stakeholders rush to deliver consistent service to their customers, maintaining uptime and service stability becomes one of the most important factors for customer satisfaction. To provide the best on-time services, organisations end up creating a complex web of interconnected microservices (often as a collection of redundant services to ensure there is minimal downtime).

    However, this type of complex infrastructure has its own set of shortcomings. As interdependent systems experience breakdowns, often a cascade of failures occurs, triggering a massive deluge of alerts. In one of our previous blogs, we explored alert noise and how it can kill productivity and introduced you to Squadcast's deduplication rules. In this post we will take a look at service dependency based and status based deduplication rules, their benefits, use cases  and how to implement them.

    To reduce the amount of alert noise many monitoring tools have their own deduplication features. However, in the case of interdependent microservices a dedicated incident management platform that aggregates all alerts is always a better option. This comes as handy especially with organisations that are experiencing exponential growth and may have a haphazard web of interconnected services that are reliant on the handful of people who built it for it’s proper functioning.

    For the sake of your on-call team’s sanity it is vital that you have clear noise reduction rules in place. A fully featured incident management solution will have the capabilities to ensure that this tangled web of microservices is not bombarding your on-call team with unnecessary alerts. For instance, an ecommerce platform may employ tens if not hundreds of microservices for something as simple as searching for a book and then placing an order for it.

    Enter Squadcast, a super-customisable platform that helps you reduce alert noise. Be it too many alerts from the same critical service or alerts that are going out from dependent services.

    A typical on-call engineer’s time is too valuable to be wasted on inconsequential alerts. Squadcast gives granular control over alerts and configuration for interdependent services. The platform is built with the experience of serving organisations dealing with hundreds of requests per second.  Each organisation has an unique set of interdependent systems running on top of the other. Squadcast allows you to customise the level of alert noise you are comfortable with.

    How to configure Squadcast for Incident Status based Deduplication

    Regular deduplication rules in incident management platforms are in place to ensure that the on-call engineer is not overwhelmed by alerts coming from the same source in a short period of time. Our status based deduplication goes a step further by allowing granular control over the alerts being received. This includes checking the state of the incident status(whether “triggered”, “suppressed” ), and then deciding if deduplication should be done. This feature gives you that additional control of narrowing down the list of past incidents(based on the status they have) against which deduplication is to be considered.

    Below screenshots show how you can configure this feature to deduplicate based on the status of a single critical service.

    Screenshots

    Status based deduplication

    Benefits

    As stated earlier, Squadcast offers a very high level of granularity in the creation and application of deduplication rules. This includes creating complex regex expressions to cover any scenario and allowing dedupe based on the status (triggered, suppressed or acknowledged). For example, if the database service is affected it is possible to generate an alert based on the incident status for any subsequent failure.

    Use Cases

    • When deploying a feature you are bound to receive alerts, at times more than normal and you know why but there's no way to stop them. Creating a Maintenance mode would not suffice since it's for the entire service. However, if the alerts are coming only from a specific alert source mapped to the service, the best solution is status based deduplication.
    • Accurately diagnose problems with high-failure rate services.

    How to configure Squadcast for Service dependency based incident deduplication

    Service based dependencies

    When your platform relies on a variety of microservices working in tandem, we see dependencies springing up (that can lead to cascading failures if a critical service goes down). For instance, a typical design dependency for e-commerce sites features a central database service that is linked with a payment gateway and frontend service. Any failure in database would create two alerts:

    • One from the database monitoring alert source
    • One from payment monitoring alert source (cascading failure)

    With Squadcast’s service dependency based deduplication rules in place, only a single alert will be generated if the two services are configured to be interdependent.

    Once you have properly configured your incident management platform to combine the alerts for these two dependent services you will be able to deal with the problem without being drowned in alert noise.

    Below screenshots highlight deduplication rules configured for any interdependent service. In this case subsequent alerts are automatically deduplicated.

    Squadcast also gives you the flexibility to configure granular rules to combine alerts based on the JSON payload from your monitoring services.This high degree of control lets you prepare for novel scenarios as the dependency map of your organisation evolves. Once the dependent services are mapped in the platform deduplication can be enabled by ticking the checkbox as shown in the screenshot.

    Benefits

    Replicating the dependency map of your physical infrastructure to prevent spamming your on-call engineers with alerts.

    Use Cases

    • Improve uptime for Payment portals on ecommerce sites.
    • Diagnose problems with Mobile apps requiring external or third-party geo-location services.

    An incident management platform that responds proactively and helps you in learning from past failures is indispensable in today’s world of microservices and distributed computing. As customer expectations rise the pressure to have 99% and above uptime rises in step. Understandably, maintaining that very high level of uptime can have a significant human cost if not managed properly. Status based and service based deduplication are just two among many of Squadcast’s features to ensure that alert noise does not become unbearable and gives organisations the power to make the lives of on-call engineers better.

    We hope these practices help you reduce alert noise and improve your on-call experience. We’d love to hear from you on other best practices that can be followed to better on-call.

    Squadcast is an incident management tool that’s purpose-built for SRE. Your team can get rid of unwanted alerts, receive relevant notifications, work in collaboration using the virtual incident war rooms, and use automated tools like runbooks to eliminate toil.

    squadcast
    What you should do now
    • Schedule a demo with Squadcast to learn about the platform, answer your questions, and evaluate if Squadcast is the right fit for you.
    • Curious about how Squadcast can assist you in implementing SRE best practices? Discover the platform's capabilities through our Interactive Demo.
    • Enjoyed the article? Explore further insights on the best SRE practices.
    • Schedule a demo with Squadcast to learn about the platform, answer your questions, and evaluate if Squadcast is the right fit for you.
    • Curious about how Squadcast can assist you in implementing SRE best practices? Discover the platform's capabilities through our Interactive Demo.
    • Enjoyed the article? Explore further insights on the best SRE practices.
    • Get a walkthrough of our platform through this Interactive Demo and see how it can solve your specific challenges.
    • See how Charter Leveraged Squadcast to Drive Client Success With Robust Incident Management.
    • Share this blog post with someone you think will find it useful. Share it on Facebook, Twitter, LinkedIn or Reddit
    • Get a walkthrough of our platform through this Interactive Demo and see how it can solve your specific challenges.
    • See how Charter Leveraged Squadcast to Drive Client Success With Robust Incident Management
    • Share this blog post with someone you think will find it useful. Share it on Facebook, Twitter, LinkedIn or Reddit
    • Get a walkthrough of our platform through this Interactive Demo and see how it can solve your specific challenges.
    • See how Charter Leveraged Squadcast to Drive Client Success With Robust Incident Management
    • Share this blog post with someone you think will find it useful. Share it on Facebook, Twitter, LinkedIn or Reddit
    Written By:
    November 3, 2020
    November 3, 2020
    Share this post:
    Subscribe to our LinkedIn Newsletter to receive more educational content
    Subscribe now

    Subscribe to our latest updates

    Enter your Email Id
    Thank you! Your submission has been received!
    Oops! Something went wrong while submitting the form.
    FAQ
    More from
    Asutosh Sahoo
    Keep track of your on-call responsibilities
    Keep track of your on-call responsibilities
    August 19, 2020
    Configure an Intuitive Service Dashboard & Reduce Response Time
    Configure an Intuitive Service Dashboard & Reduce Response Time
    April 30, 2020
    Leverage JIRA with Squadcast throughout the incident lifecycle
    Leverage JIRA with Squadcast throughout the incident lifecycle
    March 30, 2020
    Learn how organizations are using Squadcast
    to maintain and improve upon their Reliability metrics
    Learn how organizations are using Squadcast to maintain and improve upon their Reliability metrics
    mapgears
    "Mapgears simplified their complex On-call Alerting process with Squadcast.
    Squadcast has helped us aggregate alerts coming in from hundreds...
    bibam
    "Bibam found their best PagerDuty alternative in Squadcast.
    By moving to Squadcast from Pagerduty, we have seen a serious reduction in alert fatigue, allowing us to focus...
    tanner
    "Squadcast helped Tanner gain system insights and boost team productivity.
    Squadcast has integrated seamlessly into our DevOps and on-call team's workflows. Thanks to their reliability...
    Alexandre Lessard
    System Analyst
    Martin do Santos
    Platform and Architecture Tech Lead
    Sandro Franchi
    CTO
    Squadcast is a leader in Incident Management on G2 Squadcast is a leader in Mid-Market IT Service Management (ITSM) Tools on G2 Squadcast is a leader in Americas IT Alerting on G2 Best IT Management Products 2022 Squadcast is a leader in Europe IT Alerting on G2 Squadcast is a leader in Mid-Market Asia Pacific Incident Management on G2 Users love Squadcast on G2
    Squadcast awarded as "Best Software" in the IT Management category by G2 🎉 Read full report here.
    What our
    customers
    have to say
    mapgears
    "Mapgears simplified their complex On-call Alerting process with Squadcast.
    Squadcast has helped us aggregate alerts coming in from hundreds of services into one single platform. We no longer have hundreds of...
    Alexandre Lessard
    System Analyst
    bibam
    "Bibam found their best PagerDuty alternative in Squadcast.
    By moving to Squadcast from Pagerduty, we have seen a serious reduction in alert fatigue, allowing us to focus...
    Martin do Santos
    Platform and Architecture Tech Lead
    tanner
    "Squadcast helped Tanner gain system insights and boost team productivity.
    Squadcast has integrated seamlessly into our DevOps and on-call team's workflows. Thanks to their reliability metrics we have...
    Sandro Franchi
    CTO
    Revamp your Incident Response.
    Peak Reliability
    Easier, Faster, More Automated with SRE.
    Incident Response Mobility
    Manage incidents on the go with Squadcast mobile app for Android and iOS devices
    google playapple store
    Squadcast is a leader in Incident Management on G2 Squadcast is a leader in Mid-Market IT Service Management (ITSM) Tools on G2 Squadcast is a leader in Americas IT Alerting on G2 Best IT Management Products 2022 Squadcast is a leader in Europe IT Alerting on G2 Squadcast is a leader in Enterprise Incident Management on G2 Users love Squadcast on G2
    Squadcast is a leader in Incident Management on G2 Squadcast is a leader in Mid-Market IT Service Management (ITSM) Tools on G2 Squadcast is a leader in Americas IT Alerting on G2
    Best IT Management Products 2022 Squadcast is a leader in Europe IT Alerting on G2 Squadcast is a leader in Enterprise Incident Management on G2
    Users love Squadcast on G2
    Copyright © Squadcast Inc. 2017-2024