How to Implement Global View and High Availability for Prometheus

March 11, 2022
Share this post:
How to Implement Global View and High Availability for Prometheus
Table of Contents:

    Ensuring that systems run reliably is a critical function of a site reliability engineer. A big part of that is collecting metrics, creating alerts and graph data. It’s of the utmost importance to gather system metrics, from several locations and services, and correlate them to understand system functionality as well as to support troubleshooting.

    Prometheus, a Cloud Native Computing Foundation (CNCF) project, has become one of the most popular open source solutions for application and system monitoring. A single instance can handle millions of time series, but when systems grow, Prometheus needs the ablility to scale and handle the increased load. Because vertical scaling will eventually hit a limit, you require another solution.

    This article will guide you through transforming a simple Prometheus setup into a Thanos deployment. That setup will enable you to perform reliable queries to multiple Prometheus instances from a single endpoint, seamlessly handling a highly available Prometheus setup.

    Implement Global View and High Availability

    Thanos provides a set of components that can deliver a highly available metric system, with virtually unlimited storage capacity. It can be added on top of existing Prometheus deployments and provide capabilities like global query view, data backup and historical data access. Moreover, these features run independently of each other, which allows you to onboard Thanos features only when needed.

    Initial Cluster Setup

    You’ll be deploying Prometheus in a Kubernetes cluster, where you’ll simulate the desired scenario. The kind tool is a good solution to launch a Kubernetes cluster locally. You’ll use the following configuration.

    <p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/a7a270898062f1a006b654557d171ac4.js</p>

    With this configuration, you’re ready to launch the cluster.

    <p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/923f49ece38d8b6ed698e3cfc589ffa5.js</p>

    With the cluster up and running, you’ll check the installation to be sure you’re ready to launch Prometheus. You’ll need kubectl to interact with the Kubernetes cluster.

    <p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/5eb6eb7c3399a3fc732fa3db435d6c6a.js</p>

    <p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/ef4051008d4e5e3b952bf8d5351062b5.js</p>

    Initial Prometheus Setup

    Your goal is to deploy Thanos on top of an existing Prometheus installation and extend its functionality. With that in mind, you’ll start by launching three Prometheus servers. There are several reasons to have multiple Prometheus instances like sharding, high availability or query aggregation from multiple locations.

    For this scenario, let's imagine the following setup: You have one Prometheus server in a cluster in the United States and two replicas of Prometheus server in Europe that scrape the same targets.

    To deploy Prometheus, you’ll use the kube-prometheus-stack chart, and you’ll need Helm. After installing Helm, you’ll need to add the kube-prometheus-stack repository.

    <p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/2a25c8d58c0b1386a60d8d743afb82c6.js</p>

    Because in practice you only have one Kubernetes cluster, you’ll simulate multiple regions by deploying Prometheus in different namespaces. You’ll create namespaces for europe and another for united-states.

    <p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/d697f6ae1ea5ff6668c0f05e8d3e8884.js</p>

    Now that you have your regions, you’re ready to deploy Prometheus.

    <p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/9311e8dadc883ad1f216bb01bdcb9de2.js</p>

    <p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/0c7a57aca7c00999b2204affdb2f8ec3.js</p>

    Using the configurations above, you’ll deploy the Prometheus instances in each region.

    <p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/59a6a36714a016c5915156607036a239.js</p>

    You can now ensure your Prometheus is working as expected.

    <p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/c473840cb73c469d682c2a734ed7903d.js</p>

    You will be able to query any metrics on each individual instance, but unable to perform multicluster queries.

    Deploy Thanos Sidecars

    kube-prometheus-stack supports deploying Thanos as a sidecar, meaning it will be deployed alongside Prometheus itself. Thanos sidecar exposes Prometheus through the StoreAPI, a generic gRPC API that allows Thanos components to fetch metrics from various systems.

    <p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/d5127d19e0e94e5588f8805bd21ab445.js</p>

    <p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/0ac6cf7d900b2c325c311ca29d00bcd7.js</p>

    With the updated configuration, you’re ready to upgrade Prometheus.

    <p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/591fc4bfc0a8e5c32ceefec1899710be.js</p>

    You can check that the Prometheus pods have an extra container running alongside them.

    <p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/dac7f9f8707ecc89fa125e01bfd8b74f.js</p>

    Deploy Thanos Querier to Achieve Global View

    Querier implements the Prometheus HTTP v1 API to query data in a Thanos cluster via PromQL. It will allow you to fetch metrics from a single endpoint. It starts by gathering the data needed to evaluate a query from underlying StoreAPIs, evaluates the query and then returns the result.

    You leveraged kube-prometheus-stack to deploy Thanos sidecar. Unfortunately, that chart does not support other Thanos components. For that, you’ll take advantage of the Banzai Cloud Helm Charts repository. As before, you’ll start by adding the repository, the same way you did before.

    <p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/77b7733088c2eef8954b1bf01ccaac80.js</p>

    To simulate a central monitoring solution, you’ll create a monitoring namespace.

    <p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/64814b5a700be2c7dc1a3c4ea6ebc11c.js</p>

    The following configuration configures Thanos Querier and points it to the Prometheus instances.

    <p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/61e2dbc6969d7d07e7d5325d2d8cb90a.js</p>

    With the above configuration, you’re ready to deploy Querier.

    <p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/9746ad47549218d409484abb65eb6113.js</p>

    Using port-forward you can connect to your cluster. You can ensure that you can perform multicluster queries. When you deployed Prometheus, you set replicaExternalLabelName: "replica" and prometheusExternalLabelName: "cluster". The deduplication functionality will take advantage of those. By enabling it, you can ensure metrics from the europe cluster are deduplicated. That’s because Thanos assumes them to be from the same high-availability group. This happens because they have the same labels, except for the replica label.

    Deploy Thanos Query Frontend to Improve Readability

    The last piece of the puzzle will be to deploy Query Frontend, a service that can be put in front of queriers to improve readability. It is based on the Cortex Query Frontend component and supports features like splitting, retries, caching and slow query log.

    <p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/f770259555ca53c17e5e6575f3fa6db2.js</p>

    Updating the previous configuration to deploy Query Frontend, you now can update your setup.

    <p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/36f97e7c1824c2ad031da7ec9aeab301.js</p>

    Using port-forward again, you’ll be able to access Query Frontend.

    Query Frontend is the entry point to send queries to multiple Prometheus instances. Services that perform these types of queries, such as Grafana, should make them through Query Frontend.

    Conclusion

    In this article, you’ve gone through the steps required to go from a simple metrics-gathering solution to a global, highly available setup. In this setup, you leveraged Prometheus and Thanos in a Kuberentes cluster.

    You started by deploying Prometheus instances separately, simulating a multi-region setup, and then proceeded to add functionality incrementally. You started by injecting Thanos as a sidecar, implementing the Store API on top of Prometheus and paving the way to deploy Querier.

    With Querier you gathered data from the underlying Store APIs, evaluated queries and returned results. Lastly, you deployed Query Fronted, a component aimed at improving readability that supports features like splitting, retries, caching and slow query log.

    This setup allows you to run multi-replica Prometheus servers, in a highly available setup, and paves the way for more complex scenarios.

    Plug: Use Prometheus with Squadcast for Improved Monitoring and Reliability

    Squadcast is an incident management tool that’s purpose-built for SRE. Get rid of unwanted alerts, receive relevant notifications and integrate with popular ChatOps tools. Work in collaboration using virtual incident war rooms and use automation to eliminate toil.

    squadcast

    Written By:
    March 11, 2022
    March 11, 2022
    Share this post:
    Subscribe to our latest updates
    Enter your Email Id
    Thank you! Your submission has been received!
    Oops! Something went wrong while submitting the form.
    More from
    Ricardo Castro
    How important is Observability for SRE?
    How important is Observability for SRE?
    December 3, 2021
    How to improve your influence as an SRE
    How to improve your influence as an SRE
    November 10, 2021
    Going from Zero to SRE
    Going from Zero to SRE
    September 14, 2021
    Learn how organizations are using Squadcast
    to maintain and improve upon their Reliability metrics
    Learn how organizations are using Squadcast to maintain and improve upon their Reliability metrics
    mapgears
    "Mapgears simplified their complex On-call Alerting process with Squadcast.
    Squadcast has helped us aggregate alerts coming in from hundreds...
    bibam
    "Bibam found their best PagerDuty alternative in Squadcast.
    By moving to Squadcast from Pagerduty, we have seen a serious reduction in alert fatigue, allowing us to focus...
    tanner
    "Squadcast helped Tanner gain system insights and boost team productivity.
    Squadcast has integrated seamlessly into our DevOps and on-call team's workflows. Thanks to their reliability...
    Alexandre Lessard
    System Analyst
    Martin do Santos
    Platform and Architecture Tech Lead
    Sandro Franchi
    CTO
    Squadcast is a leader in Incident Management on G2 Squadcast is a leader in Incident Management on G2 Users love Squadcast on G2 Best IT Management Products 2022 Squadcast is a leader in IT Service Management (ITSM) Tools on G2 Squadcast is a leader in IT Service Management (ITSM) Tools on G2 Squadcast is a leader in IT Service Management (ITSM) Tools on G2
    Squadcast awarded as "Best Software" in the IT Management category by G2 🎉 Read full report here.
    What our
    customers
    have to say
    mapgears
    "Mapgears simplified their complex On-call Alerting process with Squadcast.
    Squadcast has helped us aggregate alerts coming in from hundreds of services into one single platform. We no longer have hundreds of...
    Alexandre Lessard
    System Analyst
    bibam
    "Bibam found their best PagerDuty alternative in Squadcast.
    By moving to Squadcast from Pagerduty, we have seen a serious reduction in alert fatigue, allowing us to focus...
    Martin do Santos
    Platform and Architecture Tech Lead
    tanner
    "Squadcast helped Tanner gain system insights and boost team productivity.
    Squadcast has integrated seamlessly into our DevOps and on-call team's workflows. Thanks to their reliability metrics we have...
    Sandro Franchi
    CTO
    Revamp your Incident Response.
    Peak Reliability
    Easier, Faster, More Automated with SRE.
    Incident Response Mobility
    Manage incidents on the go with Squadcast mobile app for Android and iOS devices
    google playapple store
    Squadcast - On-call shouldn't suck. Incident response for SRE/DevOps, IT | Product Hunt Embed
    Squadcast is a leader in Incident Management on G2 Squadcast is a leader in Incident Management on G2 Users love Squadcast on G2 Best IT Management Products 2022 Squadcast is a leader in IT Service Management (ITSM) Tools on G2 Squadcast is a leader in IT Service Management (ITSM) Tools on G2 Squadcast is a leader in IT Service Management (ITSM) Tools on G2
    Squadcast - On-call shouldn't suck. Incident response for SRE/DevOps, IT | Product Hunt Embed
    Squadcast is a leader in IT Service Management (ITSM) Tools on G2 Squadcast is a leader in Incident Management on G2 Users love Squadcast on G2
    Best IT Management Products 2022 Squadcast is a leader in IT Service Management (ITSM) Tools on G2 Squadcast is a leader in IT Service Management (ITSM) Tools on G2
    Squadcast is a leader in IT Service Management (ITSM) Tools on G2
    Copyright © Squadcast Inc. 2017-2023