How to Implement Global View and High Availability for Prometheus

Ensuring that systems run reliably is a critical function of a site reliability engineer. A big part of that is collecting metrics, creating alerts and graph data. It’s of the utmost importance to gather system metrics, from several locations and services, and correlate them to understand system functionality as well as to support troubleshooting.

Prometheus, a Cloud Native Computing Foundation (CNCF) project, has become one of the most popular open source solutions for application and system monitoring. A single instance can handle millions of time series, but when systems grow, Prometheus needs the ablility to scale and handle the increased load. Because vertical scaling will eventually hit a limit, you require another solution.

This article will guide you through transforming a simple Prometheus setup into a Thanos deployment. That setup will enable you to perform reliable queries to multiple Prometheus instances from a single endpoint, seamlessly handling a highly available Prometheus setup.

Implement Global View and High Availability

Thanos provides a set of components that can deliver a highly available metric system, with virtually unlimited storage capacity. It can be added on top of existing Prometheus deployments and provide capabilities like global query view, data backup and historical data access. Moreover, these features run independently of each other, which allows you to onboard Thanos features only when needed.

Initial Cluster Setup

You’ll be deploying Prometheus in a Kubernetes cluster, where you’ll simulate the desired scenario. The kind tool is a good solution to launch a Kubernetes cluster locally. You’ll use the following configuration.

<p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/a7a270898062f1a006b654557d171ac4.js</p>

With this configuration, you’re ready to launch the cluster.

<p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/923f49ece38d8b6ed698e3cfc589ffa5.js</p>

With the cluster up and running, you’ll check the installation to be sure you’re ready to launch Prometheus. You’ll need kubectl to interact with the Kubernetes cluster.

<p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/5eb6eb7c3399a3fc732fa3db435d6c6a.js</p>

<p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/ef4051008d4e5e3b952bf8d5351062b5.js</p>

Initial Prometheus Setup

Your goal is to deploy Thanos on top of an existing Prometheus installation and extend its functionality. With that in mind, you’ll start by launching three Prometheus servers. There are several reasons to have multiple Prometheus instances like sharding, high availability or query aggregation from multiple locations.

For this scenario, let's imagine the following setup: You have one Prometheus server in a cluster in the United States and two replicas of Prometheus server in Europe that scrape the same targets.

To deploy Prometheus, you’ll use the kube-prometheus-stack chart, and you’ll need Helm. After installing Helm, you’ll need to add the kube-prometheus-stack repository.

<p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/2a25c8d58c0b1386a60d8d743afb82c6.js</p>

Because in practice you only have one Kubernetes cluster, you’ll simulate multiple regions by deploying Prometheus in different namespaces. You’ll create namespaces for europe and another for united-states.

<p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/d697f6ae1ea5ff6668c0f05e8d3e8884.js</p>

Now that you have your regions, you’re ready to deploy Prometheus.

<p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/9311e8dadc883ad1f216bb01bdcb9de2.js</p>

<p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/0c7a57aca7c00999b2204affdb2f8ec3.js</p>

Using the configurations above, you’ll deploy the Prometheus instances in each region.

<p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/59a6a36714a016c5915156607036a239.js</p>

You can now ensure your Prometheus is working as expected.

<p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/c473840cb73c469d682c2a734ed7903d.js</p>

You will be able to query any metrics on each individual instance, but unable to perform multicluster queries.

Deploy Thanos Sidecars

kube-prometheus-stack supports deploying Thanos as a sidecar, meaning it will be deployed alongside Prometheus itself. Thanos sidecar exposes Prometheus through the StoreAPI, a generic gRPC API that allows Thanos components to fetch metrics from various systems.

<p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/d5127d19e0e94e5588f8805bd21ab445.js</p>

<p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/0ac6cf7d900b2c325c311ca29d00bcd7.js</p>

With the updated configuration, you’re ready to upgrade Prometheus.

<p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/591fc4bfc0a8e5c32ceefec1899710be.js</p>

You can check that the Prometheus pods have an extra container running alongside them.

<p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/dac7f9f8707ecc89fa125e01bfd8b74f.js</p>

Deploy Thanos Querier to Achieve Global View

Querier implements the Prometheus HTTP v1 API to query data in a Thanos cluster via PromQL. It will allow you to fetch metrics from a single endpoint. It starts by gathering the data needed to evaluate a query from underlying StoreAPIs, evaluates the query and then returns the result.

You leveraged kube-prometheus-stack to deploy Thanos sidecar. Unfortunately, that chart does not support other Thanos components. For that, you’ll take advantage of the Banzai Cloud Helm Charts repository. As before, you’ll start by adding the repository, the same way you did before.

‍

<p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/77b7733088c2eef8954b1bf01ccaac80.js</p>

To simulate a central monitoring solution, you’ll create a monitoring namespace.

<p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/64814b5a700be2c7dc1a3c4ea6ebc11c.js</p>

The following configuration configures Thanos Querier and points it to the Prometheus instances.

<p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/61e2dbc6969d7d07e7d5325d2d8cb90a.js</p>

With the above configuration, you’re ready to deploy Querier.

<p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/9746ad47549218d409484abb65eb6113.js</p>

Using port-forward you can connect to your cluster. You can ensure that you can perform multicluster queries. When you deployed Prometheus, you set replicaExternalLabelName: "replica" and prometheusExternalLabelName: "cluster". The deduplication functionality will take advantage of those. By enabling it, you can ensure metrics from the europe cluster are deduplicated. That’s because Thanos assumes them to be from the same high-availability group. This happens because they have the same labels, except for the replica label.

‍

Deploy Thanos Query Frontend to Improve Readability

The last piece of the puzzle will be to deploy Query Frontend, a service that can be put in front of queriers to improve readability. It is based on the Cortex Query Frontend component and supports features like splitting, retries, caching and slow query log.

<p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/f770259555ca53c17e5e6575f3fa6db2.js</p>

Updating the previous configuration to deploy Query Frontend, you now can update your setup.

<p>CODE: https://gist.github.com/ShubhanjanMedhi-dev/36f97e7c1824c2ad031da7ec9aeab301.js</p>

Using port-forward again, you’ll be able to access Query Frontend.

Query Frontend is the entry point to send queries to multiple Prometheus instances. Services that perform these types of queries, such as Grafana, should make them through Query Frontend.

Conclusion

In this article, you’ve gone through the steps required to go from a simple metrics-gathering solution to a global, highly available setup. In this setup, you leveraged Prometheus and Thanos in a Kuberentes cluster.

You started by deploying Prometheus instances separately, simulating a multi-region setup, and then proceeded to add functionality incrementally. You started by injecting Thanos as a sidecar, implementing the Store API on top of Prometheus and paving the way to deploy Querier.

With Querier you gathered data from the underlying Store APIs, evaluated queries and returned results. Lastly, you deployed Query Fronted, a component aimed at improving readability that supports features like splitting, retries, caching and slow query log.

This setup allows you to run multi-replica Prometheus servers, in a highly available setup, and paves the way for more complex scenarios.

Plug: Use Prometheus with Squadcast for Improved Monitoring and Reliability

Squadcast is an incident management tool that’s purpose-built for SRE. Get rid of unwanted alerts, receive relevant notifications and integrate with popular ChatOps tools. Work in collaboration using virtual incident war rooms and use automation to eliminate toil.

‍