Keep track of your on-call responsibilities

August 19, 2020
Share this post:
Keep track of your on-call responsibilities

On-call Engineers are the first line of defense when an outage occurs ensuring customer-impacting services are quickly noticed & resolved. Our latest blog outlines some of the use cases that can help to avoid the pitfalls in organization’s on-call rotation. Also, get access to our Free On-call Onboarding Checklist

Table of Contents:

    Typically an organization’s first step towards committing to reliability for customers and users is the practice of having an on-call rotation wherein on-call engineers are the first line of defense, ensuring customer-impacting outages are quickly noticed and resolved.

    At Squadcast we use our own on-call rotation using our platform that supports many custom on-call rotation types, including follow-the-sun, daily, weekly, or split shift rotations along with capabilities to create multiple scheduling layers within a single schedule.

    Below, we’ve highlighted some of the use cases highlighting how on-call responsibilities can be viewed with on-call schedule scripts that would help knowing who is on-call right now, along with knowing which Slack channel to send on-call notifications to for a particular schedule.

    Purpose of the script

    Get the list of people who are on-call for a given Squadcast Schedule across all the shifts, at the time of execution of the script and send it to a slack channel.

    Please note, this script doesn’t categorize users according to shift-name, i.e. doesn’t say which user is on-call for which shift.

    It makes use of the Who is On-Call API of Squadcast to get the list of people on-call for a given time.

    Script Inputs

    The script takes a configuration JSON file as its command-line argument which looks something like the following:

    {
    "REFRESH_TOKEN"
    :
    "Squadcast Refresh Token which will be used to be make requests to Public APIs"
    ,
    "SCHEDULE_NAME"
    :
    "Name of the schedule for which we are getting the on-call people"
    ,
    "SLACK_WEBHOOK_URL"
    :
    "Slack incoming webhook URL for the channel to which we want to send the on-call notification"

    }

    Steps

    1. Clone the repo

    Open a terminal window and execute the following command to clone the repo.

    $ git clone git@github.com:SquadcastHub/who-s-oncall-slack.git

    2. Getting the Refresh Token

    NOTE: If you are the Account Owner or an Admin of the Squadcast account, then follow the steps to obtain a refresh token. If you are a Stakeholder or a User, get in touch with the Account Owner or Admin to get the token.

    i. Login to your Squadcast Dashboard and go to the profile page.

    ii. Here, you’ll find an API Access Token. If you haven’t already created one, click on Generate new API token.

    iii. Copy the token generated to the value corresponding to REFRESH_TOKEN key in the params.json file.

    3. Getting the Schedule Name

    i. Select Schedules from the Sidebar within the platform

    ii. On the right side of the screen, you can see the list of different schedules created. Copy the name of the Schedule for which you want to get the list of on-call people to the params.json file.

    4. Getting the Slack Webhook URL

    i. Open your Slack app.

    ii. Scroll the sidebar on the left to get to the Apps section. Click on the ‘+’ button beside Apps.

    iii. Search “Webhook” in the search box. Click on the Add button of the Incoming WebHooks.

    iv. You’ll be redirected to a webpage which looks something like the picture below. Click on the Add to Slack button.

    v. You’ll be redirected to a screen where you’ll be asked to select which Slack channel the webhook will send messages to. Click on the Add Incoming WebHook Integration once done.

    vi. In the next screen, you’ll get your Webhook URL. Copy it to the params.json file.

    Now once we have the params.json file ready, next we’ll build the script and set-up the cron.

    NOTE : Building the Script

    You’ll need golang installed in your system in-order to build this script.

    If you don’t have it already, you can download here

    To build the script, go to the directory of the script and execute the command

    $ go build

    Once the build is successful, you’ll see a binary named oncall in the current working directory. To execute this binary, you’ll have to specify the path to the params.json file containing all the parameters we have configured in the previous steps. The command to run would be

    $ ./oncall <path_to_params.json>

    Now the script itself doesn’t take care of the configuration for when to execute this. That needs to be handled externally. So, in this blog, we’ll go over how you can set-up a cron-job in a linux system in-order to do so.

    5. Setting cron-job in Linux

    Let’s say that the on-call hand-off at an organization happens every Monday at 12pm. So, it makes sense to run this script at that time. In-order to do so, the steps would be:

    i. Open a terminal and execute the following command

    $ crontab -e

    ii. A file editor would open in the terminal. Paste the following line in the file and save it.

    0 12 * * 1 <path_to_binary> <path_to_params.json>

    Where

    • <path_to_binary>: absolute path of the script binary.
    • <path_to_params.json>: absolute path of the params.json file.

    For more details on how to use the crontab command, you can checkout its manpage by running the command

    $ man crontab

    You may also refer to these blogs as well.

    https://www.tecmint.com/create-and-manage-cron-jobs-on-linux/

    https://phoenixnap.com/kb/set-up-cron-job-linux

    NOTE: However you automate the invocation of the script, make sure that you don’t invoke the script very frequently. As a rule of thumb, avoid invoking the script more than once in a time window of 1 hour.

    Once the cron is setup, you’ll get notifications to the selected Slack channel about the new on-call users at the time of invocation of the script.

    Q&As

    Can I get notifications to other chat clients like Google Hangouts / Microsoft Teams?

    In the script, you just have to replace the NotifySlack function with your own written NotifyHangouts or NotifyTeams functions and you’re good to go. You’ll have to refer to the API documentation of the respective chat-ops platform for details regarding setting up.

    How do I  group the user names by shifts?

    Currently, our API doesn’t support grouping by shift names. Later when we support it in our API, we’ll update the script as well.

    Can I get on-call users for multiple schedules?

    You can simply have multiple params.json files with different configurations and configure the cron-job to call the script with multiple different configurations.

    For example, if you have 2 params.json files having absolute paths /home/params1.json & /home/params2.json with the binary at /home/oncall, then you can configure the crontab file as

    0 12 * * 1 /home/oncall /home/params1.json
    0 12 * * 1 /home/oncall /home/params2.json

    With this, you have a flexibility to do a lot of things:

    • You can send notifications for different schedules to the same / different Slack channels.
    • You can invoke the crons of different schedules at the same / different time.

    Roadmap

    As would be apparent by now, the notifications sent to Slack are done by polling our Squadcast API. It's not event-driven by some hook. Incase of event-hook driven notification, each time there is any change in schedule, an event would be generated and the hook would be notified upon which it would take action. Supporting event-hooks is part of our product roadmap.

    Once event hook notification is available the advantages are:

    i. The user won’t have to worry about when to run the script.

    ii. Incase of irregular sudden changes to the schedule, you can still get the on-call users notification. But in case of polling, it is done at regular intervals. Hence any irregular changes will be missed.

    Get the most out of on-call scheduling

    Streamline on-call management for any kind of rotation type or team with Squadcast. Here’s a simple checklist we created while first starting off our on-call on-boarding process. If you’re planning this for your on-call team, it’s best to get them involved from the start to pitch in and improve as it is being built. Access our Free On-call Onboarding Checklist here. Please note that you can choose to use this directly or tweak it to fit your current processes and needs.

    We hope these scripts enable you to formalize your on-call rotation process to make it as easy as possible for your team to respond to issues.We’d love to hear from you on other best practices that can be followed for an improved on-call experience.

    Squadcast is an incident management tool that’s purpose-built for SRE. Your team can get rid of unwanted alerts, receive relevant notifications, work in collaboration using the virtual incident war rooms, and use automated tools like runbooks to eliminate toil.

    squadcast
    Written By:
    August 19, 2020
    August 19, 2020
    Share this post:
    Subscribe to our LinkedIn Newsletter to receive more educational content
    Subscribe now

    Subscribe to our latest updates

    Enter your Email Id
    Thank you! Your submission has been received!
    Oops! Something went wrong while submitting the form.
    FAQ
    More from
    Asutosh Sahoo
    Curb alert noise for better productivity : How-To's and Best Practices
    Curb alert noise for better productivity : How-To's and Best Practices
    November 3, 2020
    Configure an Intuitive Service Dashboard & Reduce Response Time
    Configure an Intuitive Service Dashboard & Reduce Response Time
    April 30, 2020
    Leverage JIRA with Squadcast throughout the incident lifecycle
    Leverage JIRA with Squadcast throughout the incident lifecycle
    March 30, 2020
    Learn how organizations are using Squadcast
    to maintain and improve upon their Reliability metrics
    Learn how organizations are using Squadcast to maintain and improve upon their Reliability metrics
    mapgears
    "Mapgears simplified their complex On-call Alerting process with Squadcast.
    Squadcast has helped us aggregate alerts coming in from hundreds...
    bibam
    "Bibam found their best PagerDuty alternative in Squadcast.
    By moving to Squadcast from Pagerduty, we have seen a serious reduction in alert fatigue, allowing us to focus...
    tanner
    "Squadcast helped Tanner gain system insights and boost team productivity.
    Squadcast has integrated seamlessly into our DevOps and on-call team's workflows. Thanks to their reliability...
    Alexandre Lessard
    System Analyst
    Martin do Santos
    Platform and Architecture Tech Lead
    Sandro Franchi
    CTO
    Squadcast is a leader in Incident Management on G2 Squadcast is a leader in Mid-Market IT Service Management (ITSM) Tools on G2 Squadcast is a leader in Americas IT Alerting on G2 Best IT Management Products 2022 Squadcast is a leader in Europe IT Alerting on G2 Squadcast is a leader in Mid-Market Asia Pacific Incident Management on G2 Users love Squadcast on G2
    Squadcast awarded as "Best Software" in the IT Management category by G2 🎉 Read full report here.
    What our
    customers
    have to say
    mapgears
    "Mapgears simplified their complex On-call Alerting process with Squadcast.
    Squadcast has helped us aggregate alerts coming in from hundreds of services into one single platform. We no longer have hundreds of...
    Alexandre Lessard
    System Analyst
    bibam
    "Bibam found their best PagerDuty alternative in Squadcast.
    By moving to Squadcast from Pagerduty, we have seen a serious reduction in alert fatigue, allowing us to focus...
    Martin do Santos
    Platform and Architecture Tech Lead
    tanner
    "Squadcast helped Tanner gain system insights and boost team productivity.
    Squadcast has integrated seamlessly into our DevOps and on-call team's workflows. Thanks to their reliability metrics we have...
    Sandro Franchi
    CTO
    Revamp your Incident Response.
    Peak Reliability
    Easier, Faster, More Automated with SRE.
    Incident Response Mobility
    Manage incidents on the go with Squadcast mobile app for Android and iOS devices
    google playapple store
    google playapple store
    Squadcast - On-call shouldn't suck. Incident response for SRE/DevOps, IT | Product Hunt Embed
    Squadcast is a leader in Incident Management on G2 Squadcast is a leader in Mid-Market IT Service Management (ITSM) Tools on G2 Squadcast is a leader in Americas IT Alerting on G2 Best IT Management Products 2022 Squadcast is a leader in Europe IT Alerting on G2 Squadcast is a leader in Mid-Market Asia Pacific Incident Management on G2 Users love Squadcast on G2
    Squadcast is a leader in Incident Management on G2 Squadcast is a leader in Mid-Market IT Service Management (ITSM) Tools on G2 Squadcast is a leader in Americas IT Alerting on G2
    Best IT Management Products 2022 Squadcast is a leader in Europe IT Alerting on G2 Squadcast is a leader in Mid-Market Asia Pacific Incident Management on G2
    Users love Squadcast on G2
    Copyright © Squadcast Inc. 2017-2023