Prometheus is a robust monitoring and alerting system widely used in cloud-native and Kubernetes environments. One of the critical features of Prometheus is its ability to create and trigger alerts based on metrics it collects from various sources. Additionally, you can analyze and filter the metrics to develop:
- Complex incident response algorithms
- Service Level Objectives
- Error budget calculations
- Post-mortem analysis or retrospectives
- Runbooks to resolve common failures.
In this article, we look at Prometheus alert rules in detail. We cover alert template fields, the proper syntax for writing a rule, and several Prometheus sample alert rules you can use as is. Additionally, we also cover some challenges and best practices in Prometheus alert rule management and response.
Summary of key Prometheus alert rules concepts
Before we go into more detail on writing Prometheus alert rules, let's quickly summarize the concepts that this article will cover.
| Concept |
Description |
| Alert Template Fields |
Prometheus has a number of required and optional fields for generating rules. |
| Alert Expression Syntax |
YAML is the format used to build rules. |
| Prometheus sample alert rules |
A list of examples of commonly-used Prometheus alert rules. |
| Limitations of Prometheus |
Inability to suppress alerts and increasing complexity at scale may pose some challenges. |
| Best Practices |
You should follow best practices around rule descriptions, testing, and deployment. |
| Incident Response Handling |
Prometheus can be used to facilitate the handling of incidents from detection to resolution. |
Alert template fields
Prometheus alert templates provide a way to define standard fields and behavior for multiple alerts. You can define these templates in the Prometheus configuration file. You can reuse templates across multiple alerts to keep your alert configuration clean, maintainable, and understandable.
The following are the main fields available in Prometheus alert templates:
Alert
This field specifies the alert's name. It identifies the alert and must be unique within a Prometheus instance.
Expr
This field specifies the Prometheus query expression that evaluates the alert condition. It is the most important field in an alert template, and you must specify it.
Labels
This field adds additional information to the alert. You can use it to specify the severity of the alert, the affected service or component, and any other relevant information.
Annotations
This field provides additional context and human-readable information about the alert. You can include a summary of the alert, a description of the issue, or any other relevant information.
For
This field specifies the duration for which the alert condition must be true before Prometheus triggers the alert.
Groups
This field groups multiple alerts together. A single alert condition in a group triggers all alerts in the same group.
Alert expression syntax
Prometheus uses the PromQL (Prometheus Query Language) to create alerting rules. The alert expression is the core of a Prometheus alert. You use PromQL to define the condition that triggers an alert. For example, the following expression triggers an alert if the average CPU utilization on a host exceeds 80% for 5 minutes:
avg(node_cpu{mode="system"}) > 80
Basic alert syntax
The basic syntax of an alert expression is as follows:
<metric_name>{<label_name>="<label_value>", ...} <operator> <value>
- The <metric_name> is the name of the metric being queried.
- The {<label_name>="<label_value>", ...} is an optional part of the query that specifies the labels that should be used to filter the metric.
- The <operator> is a mathematical operator, such as >, <, ==, etc.
- The <value> is the value that the metric must be compared against using the specified operator.
Advanced alert queries
For more complex scenarios, you can use functions, like avg, sum, min, max, etc., in the expression to aggregate the metrics and make more complex comparisons. For instance, the below query triggers an alert if the average rate of HTTP requests per second to the "api" service exceeds 50 for a 5-minute period.
avg(rate(http_requests_total{service="api"}[5m])) > 50
Other advanced features include:
Integrated full stack reliability management platform
Try for free
Drive better business outcomes with incident analytics, reliability insights, SLO tracking, and error budgets
Manage incidents on the go with native iOS and Android mobile apps
Seamlessly integrated alert routing, on-call, and incident response
Prometheus sample alert rules
We present examples that cover a variety of situations where you may want to produce alerts based on environment metrics. You can use them as-is, or adapted to fit your specific needs.
High CPU utilization alert
groups:
- name: example_alerts
rules:
- alert: HighCPUUtilization
expr: avg(node_cpu{mode="system"}) > 80
for: 5m
labels:
severity: critical
annotations:
summary: High CPU utilization on host {{ $labels.instance
}}
description: The CPU utilization on host {{
$labels.instance }} has exceeded 80% for 5 minutes.
Low disk space alert
groups:
- name: example_alerts
rules:
- alert: LowDiskSpace
expr: node_filesystem_free{fstype="ext4"} < 1e9
for: 5m
labels:
severity: critical
annotations:
summary: Low disk space on host {{ $labels.instance
}}
description: The free disk space on host {{
$labels.instance }} has dropped below 1G
High request error rate alert
groups:
- name: example_alerts
rules:
- alert: HighRequestErrorRate
expr: (sum(rate(http_requests_total{status="500"}[5m])) /
sum(rate(http_requests_total[5m]))) > 0.05
for: 5m
labels:
severity: critical
annotations:
summary: High request error rate
description: The error rate for HTTP requests has exceeded
5% for 5 minutes.
Node down alert
groups:
- name: example_alerts
rules:
- alert: NodeDown
expr: up == 0
for: 5m
labels:
severity: critical
annotations:
summary: Node {{ $labels.instance }} is down
description: Node {{ $labels.instance }} has been down for
5 minutes.
High memory utilization alert
groups:
- name: example_alerts
rules:
- alert: HighMemoryUtilization
expr: node_memory_MemTotal - node_memory_MemFree < 0.8 *
node_memory_MemTotal
for: 5m
labels:
severity: warning
annotations:
summary: High memory utilization on host {{
$labels.instance }}
description: The memory utilization on host {{
$labels.instance }} has exceeded 80% for 5 minutes.
High network traffic alert
groups:
- name: example_alerts
rules:
- alert: HighNetworkTraffic
expr: node_network_receive_bytes > 100e6
for: 5m
labels:
severity: warning
annotations:
summary: High network traffic on host {{
$labels.instance }}
description: The inbound network traffic on host {{
$labels.instance }} has exceeded 100 MB/s for 5 minutes.
Limitations of Prometheus
Like any tool, Prometheus has its own set of challenges and limitations.
Excessive alerts for noisy metrics
Prometheus alerts are based on metrics, and sometimes metrics can be noisy and difficult to interpret. This may lead to false positives or false negatives, which can be difficult to troubleshoot.
Scaling challenges
As the number of metrics and alerting rules increases, Prometheus becomes resource-intensive and may require additional scaling or optimization. Too many complex alerting rules can also become challenging to understand and troubleshoot. Additionally, Prometheus does not have built-in dashboards, so you have to use external dashboarding tools, like Grafana, for metric visualization.
Inability to detect dependent services
Prometheus alerts are based on metrics, but in some scenarios, a particular service metric depends on a different service behavior. In such cases, inaccuracy increases, and alerts become difficult to action.
No alert suppression
Prometheus does not have built-in alert suppression or deduplication. Depending on your configuration, you could have a high volume of alerts for non-critical issues. To mitigate this, users can use an additional component, such as Alertmanager, to group, deduplicate, and route alerts to the appropriate channel.
Limited integration with other tools
While you can integrate Prometheus with various notification channels, it does present limited integration opportunities with other monitoring and alerting tools. You may already have existing monitoring infrastructure that is incompatible with Prometheus.
Best practices for Prometheus alerts configuration
Despite some challenges, you can customize Prometheus to meet your organization's needs. Proper planning and configuration proactively identify and resolve issues before they become critical.
Here are some best practices to follow when using Prometheus alerting rules:
Create meaningful alert templates
Write alert templates and configurations that even new team members can understand. For example:
- Choose alert names that clearly describe the metric and scenario they monitor.
- Write descriptive annotations for each alert.
- Assign appropriate severity levels to your alerts, such as critical, warning, or info.
- Group related alerts together in a single alert group to improve manageability.
These best practices provide more context about the alert and improve response and troubleshooting time.
Set the appropriate alert frequency
Make sure the time window specified in the for clause of an alert is appropriate for the metric you are monitoring. A short time window may result in too many false positive alerts, while a long time window may delay detecting real issues. For example, some user actions may cause your application's CPU usage to spike quickly before subsiding again. You may not want to action every small spike.
Test Prometheus before deployment
Test your alert rules in a test environment before deploying them to production. This helps to ensure that the rules are working as expected and eliminates the risk of unintended consequences. Additionally, you can:
- Monitor the Prometheus Alertmanager to ensure it functions properly and handles alerts as expected.
- Regularly review and update your alert rules to ensure that they continue to accurately reflect your system state and incorporate environment changes.
- Use alert templates to reduce the amount of duplication in your alert rules, as duplication increases management complexity.
Use incident response systems
Automate alert handling where possible to reduce the time required to respond to alerts and to minimize human error. You can also use your Prometheus metrics and alerts for productive incident retrospectives or build runbooks to handle similar issues.
You can use tools like Squadcast to route alerts to applicable teams. Squadcast extends beyond basic incident response functionality to provide many other features like documenting retrospectives, tracking service level objectives (SLO), and error budgets.
Incident response handling
Your organization's incident response algorithms could be as simple as sending an email to your team letting them know that a failure is imminent. More complex alerts may trigger runbooks to automate the resolution process. For example, your ruleset could be defined to automatically scale services if a particular error budget exceeds a predefined threshold. Should the error rate continue to climb, a tool like Squadcast contacts the on-call administrator to step in and handle the incident.
Runbooks
It is crucial to build out proper runbooks for handling some of the more common issues. Administrators use runbooks to facilitate incident resolution or convert them into scripts to automate the process. For example, you may write a runbook on handling an issue where a specific web server starts to segfault randomly, causing a high rate of HTTP failures. The runbook includes information on where to look for the errors, and specifically what services you need to restart as a result.
The best time to develop these runbooks is during the post-mortem of the incident, also known as a retrospective. This is the time when incident managers determine what went well, what did not go well, and what action items the team can take to correct issues in the future.
Conclusion
As you can see, Prometheus is an excellent tool to alert on key metrics in cloud-native environments. Prometheus's flexible query language and integration capabilities make it a versatile solution for efficient monitoring and alerting at scale. Our Prometheus sample alert rules and best practices will surely assist you in fully utilizing the most comprehensive Kubernetes alerting tools available today.
Integrated full stack reliability management platform
Seamless Third-Party Integrations
Advanced Error Budget Tracking
Seamless Third-Party Integrations
Advanced Error Budget Tracking
