Monitoring and Alerting
rancher-monitoring application, you can quickly deploy leading open-source monitoring and alerting solutions onto your cluster.
Prometheus lets you view metrics from your Rancher and Kubernetes objects. Using timestamps, Prometheus lets you query and view these metrics in easy-to-read graphs and visuals, either through the Rancher UI or Grafana, which is an analytics viewing platform deployed along with Prometheus.
By viewing data that Prometheus scrapes from your cluster control plane, nodes, and deployments, you can stay on top of everything happening in your cluster. You can then use these analytics to better run your organization: stop system emergencies before they start, develop maintenance strategies, or restore crashed servers.
rancher-monitoring operator, introduced in Rancher v2.5, is powered by Prometheus, Grafana, Alertmanager, the Prometheus Operator, and the Prometheus adapter.
The monitoring application:
- Monitors the state and processes of your cluster nodes, Kubernetes components, and software deployments.
- Defines alerts based on metrics collected via Prometheus.
- Creates custom Grafana dashboards.
- Configures alert-based notifications via email, Slack, PagerDuty, etc. using Prometheus Alertmanager.
- Defines precomputed, frequently needed or computationally expensive expressions as new time series based on metrics collected via Prometheus.
- Exposes collected metrics from Prometheus to the Kubernetes Custom Metrics API via Prometheus Adapter for use in HPA.
See How Monitoring Works for an explanation of how the monitoring components work together.
Default Components and Deployments
By default, the monitoring application deploys Grafana dashboards (curated by the kube-prometheus project) onto a cluster.
It also deploys an Alertmanager UI and a Prometheus UI. For more information about these tools, see Built-in Dashboards.
Default Metrics Exporters
By default, Rancher Monitoring deploys exporters (such as node-exporter and kube-state-metrics).
These default exporters automatically scrape metrics for CPU and memory from all components of your Kubernetes cluster, including your workloads.
The monitoring application deploys some alerts by default. To see the default alerts, go to the Alertmanager UI and click Expand all groups.
Components Exposed in the Rancher UI
For a list of monitoring components exposed in the Rancher UI, along with common use cases for editing them, see this section.
Role-based Access Control
For information on configuring access to monitoring, see this page.
- Enable monitoring
- Uninstall monitoring
- Monitoring workloads
- Customizing Grafana dashboards
- Persistent Grafana dashboards
- Debugging high memory usage
Configuring Monitoring Resources in Rancher
The configuration reference assumes familiarity with how monitoring components work together. For more information, see How Monitoring Works.
Configuring Helm Chart Options
For more information on
rancher-monitoring chart options, including options to set resource limits and requests, see Helm Chart Options.
Windows Cluster Support
When deployed onto an RKE1 Windows cluster, Monitoring V2 will now automatically deploy a windows-exporter DaemonSet and set up a ServiceMonitor to collect metrics from each of the deployed Pods. This will populate Prometheus with
windows_ metrics that are akin to the
node_ metrics exported by node_exporter for Linux hosts.
To be able to fully deploy Monitoring V2 for Windows, all of your Windows hosts must have a minimum wins version of v0.1.0.
For more details on how to upgrade wins on existing Windows hosts, see Windows cluster support for Monitoring V2..
There is a known issue that K3s clusters require more than the allotted default memory. If you enable monitoring on a K3s cluster, set
prometheus.prometheusSpec.resources.memory.limit to 2500 Mi and
prometheus.prometheusSpec.resources.memory.request to 1750 Mi.
See Debugging High Memory Usage for advice and recommendations.