How to Set Up SLO Monitoring with Prometheus

Most teams alert on CPU, memory, and pod restarts — and still get woken up for things users never notice, while real degradations slip through. SLO monitoring flips that: you define what "good" means from the user's perspective, measure it continuously, and alert only when your error budget is actually burning. This guide walks through how to define SLOs and implement SLO monitoring in Prometheus end to end — SLIs, error budgets, recording rules, and multi-burn-rate alerts — the way the Google SRE workbook prescribes it, with working PromQL you can adapt today.

Step 1: Get the vocabulary straight — SLI vs SLO vs SLA

These three terms get conflated constantly, and the confusion leaks into bad alerting. The distinctions are simple:

• SLI (Service Level Indicator) — a measured signal about your service. It is a number you can compute from telemetry: the ratio of successful requests to total requests, or the fraction of requests served faster than 300ms. An SLI is a measurement, not a goal.
• SLO (Service Level Objective) — a target for that SLI over a window. "99.9% of requests succeed, measured over a rolling 28 days" is an SLO. It is an internal engineering commitment that drives alerting and prioritization.
• SLA (Service Level Agreement) — a contractual promise to a customer, usually with financial penalties attached. Your SLA should always be looser than your SLO, so your team notices and reacts long before lawyers do. If your SLA is 99.5%, your internal SLO might be 99.9%.

The dependency chain matters: you cannot set an SLO without an SLI to measure it against, and you should never sign an SLA that is tighter than an SLO you already meet. Everything downstream — error budgets, burn-rate alerts, the weekly reliability review — hangs off a well-chosen SLI.

Step 2: Choose good SLIs

A good SLI approximates the user's experience. Start with the golden signals and, for a typical request-driven service, reduce them to two SLIs that cover most of what users feel:

• Availability — the proportion of requests that succeed (no 5xx, no timeout).
• Latency — the proportion of requests served faster than a threshold, or a percentile like p95 under a target.

Where you measure matters as much as what you measure. Instrument at the load balancer or ingress — the service edge — not per pod. A pod that crash-loops behind a healthy load balancer with three other replicas is an operational event, not a user-facing one. If you measure per pod, your SLI reflects infrastructure churn; if you measure at the edge, it reflects what users actually got. Ingress-nginx, Envoy, and most API gateways expose exactly the request counters and latency histograms you need.

Keep the SLI count small. Two or three SLIs per user-facing service is plenty; ten SLIs means nobody owns any of them.

Step 3: Define the SLO and compute the error budget

Pick a target and a window. The standard, defensible starting point is 99.9% availability over a rolling 28 days. Twenty-eight days beats "calendar month" because it always covers the same number of weekends, making week-over-week comparisons honest.

The error budget is the inverse of the target: with 99.9%, you are allowed 0.1% failure. Over 28 days that is roughly 40 minutes of full unavailability per month — or a much longer stretch of partial degradation. The budget is the whole point of the exercise:

• Budget healthy? Ship faster, run riskier migrations, do chaos experiments.
• Budget nearly spent? Freeze risky launches and pay down reliability debt.

The error budget turns the eternal "velocity versus reliability" argument into arithmetic. Product and platform teams stop debating opinions and start reading a shared number. That only works if the budget has teeth — agree up front, in writing, what happens when it hits zero.

Resist 100% (or even 99.99%) targets. Every extra nine roughly multiplies the cost by ten, and your users are behind ISPs and Wi-Fi that fail more often than 99.99% anyway. Pick the loosest target that keeps users happy.

Step 4: Implement the SLIs in Prometheus

Assume your edge exports http_requests_total (with a status label) and ahttp_request_duration_seconds histogram. The availability SLI is a success ratio over a short rate window:

# Availability SLI: fraction of non-5xx requests
sum(rate(http_requests_total{job="api", status!~"5.."}[5m]))
/
sum(rate(http_requests_total{job="api"}[5m]))

For latency, use the histogram to get a percentile:

# p95 latency across all instances
histogram_quantile(0.95,
  sum(rate(http_request_duration_seconds_bucket{job="api"}[5m])) by (le)
)

For SLO math it is usually cleaner to track the error ratio (1 minus the SLI) and precompute it with recording rules at the window lengths the burn-rate alerts will need. Recording rules keep alert expressions cheap and readable:

groups:
  - name: slo-api-availability
    rules:
      - record: sli:http_requests:error_ratio_rate5m
        expr: |
          sum(rate(http_requests_total{job="api", status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{job="api"}[5m]))
      - record: sli:http_requests:error_ratio_rate30m
        expr: |
          sum(rate(http_requests_total{job="api", status=~"5.."}[30m]))
          /
          sum(rate(http_requests_total{job="api"}[30m]))
      - record: sli:http_requests:error_ratio_rate1h
        expr: |
          sum(rate(http_requests_total{job="api", status=~"5.."}[1h]))
          /
          sum(rate(http_requests_total{job="api"}[1h]))
      - record: sli:http_requests:error_ratio_rate6h
        expr: |
          sum(rate(http_requests_total{job="api", status=~"5.."}[6h]))
          /
          sum(rate(http_requests_total{job="api"}[6h]))

One subtlety: decide whether client errors (4xx) count against you. A spike of 401s from a broken client is not your outage; a spike of 429s caused by your own bad rate-limit config is. Encode that decision in the label matcher and document it next to the rule.

Step 5: Multi-burn-rate alerting — the part that actually cuts noise

This is the pattern that makes SLO monitoring worth the setup. Burn rate is how fast you are consuming the error budget relative to plan: a burn rate of 1 means you will spend exactly the whole budget by the end of the 28-day window; a burn rate of 14.4 means you would exhaust it in under two days. Concretely, burn rate is the observed error ratio divided by the allowed error ratio (1 minus the SLO target — 0.001 for 99.9%).

Instead of one static threshold, you run two alerts tuned to different failure modes:

• Fast burn (page): burn rate above 14.4 sustained over 1 hour. That pace consumes 2% of the monthly budget in a single hour — an acute outage. Wake someone up.
• Slow burn (ticket): burn rate above 6 sustained over 6 hours. That consumes about 5% of the budget — a gradual degradation like a slow memory leak or a flaky dependency. File a ticket; fix it during business hours.

Each alert also checks a shorter secondary window (5 minutes for the fast alert, 30 minutes for the slow one). The long window proves the burn is sustained and not a blip; the short window makes the alert stop firing quickly once the problem is fixed, instead of ringing for an hour of stale data. Compared to a naive "error rate above 1% for 5 minutes" alert, this multiwindow, multi-burn-rate setup pages you for everything that genuinely threatens the SLO and almost nothing else. Here it is as a PrometheusRule for the Prometheus Operator:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: api-slo-burn-rate
  namespace: monitoring
spec:
  groups:
    - name: api-slo-burn-rate
      rules:
        - alert: APIErrorBudgetBurnFast
          expr: |
            sli:http_requests:error_ratio_rate1h > (14.4 * 0.001)
            and
            sli:http_requests:error_ratio_rate5m > (14.4 * 0.001)
          for: 2m
          labels:
            severity: page
          annotations:
            summary: "API burning error budget at 14.4x — 2% of the 28-day budget per hour"
            description: "Sustained fast burn on the availability SLO. Treat as an active incident."
        - alert: APIErrorBudgetBurnSlow
          expr: |
            sli:http_requests:error_ratio_rate6h > (6 * 0.001)
            and
            sli:http_requests:error_ratio_rate30m > (6 * 0.001)
          for: 15m
          labels:
            severity: ticket
          annotations:
            summary: "API burning error budget at 6x — about 5% of the budget in 6 hours"
            description: "Slow, sustained degradation. Investigate within the working day."

The multipliers are not magic — they come straight from the SRE workbook and correspond to "2% of budget in 1 hour" and "5% of budget in 6 hours". If your SLO target changes, only the0.001 changes; the burn-rate multipliers stay the same.

Step 6: Dashboards and the weekly budget review

Alerts handle emergencies; dashboards handle decisions. Build one Grafana dashboard per SLO with three panels:

1. The SLI over time, with the SLO target drawn as a threshold line.
2. Current burn rate across the 1h and 6h windows.
3. Error budget remaining for the 28-day window — the number everyone actually cares about:

# Fraction of the 28-day error budget still remaining (1 = full, 0 = spent)
1 - (
  sum(increase(http_requests_total{job="api", status=~"5.."}[28d]))
  /
  sum(increase(http_requests_total{job="api"}[28d]))
) / 0.001

Then put the budget on a calendar. A 15-minute slot in the weekly ops review is enough: which SLOs burned budget this week, what caused it, and does anything change in next week's plan? If a budget is trending toward zero, that is the forum where a launch gets delayed — before the pager decides for you. SLOs that are never reviewed decay into dashboard wallpaper within a quarter.

Step 7: Avoid the common mistakes

• Alerting on causes instead of symptoms. High CPU, a restarted pod, one unhealthy replica — none of these are user pain. Page on the symptom (budget burn); keep cause-level signals as diagnostic dashboards for the responder.
• Per-pod or per-instance SLOs. Users talk to a service, not a pod. Aggregate at the edge; let the orchestrator handle individual replicas.
• 100% targets. A 100% SLO means zero error budget, which means every single failed request is technically an incident and change velocity goes to zero. Unachievable and unaffordable.
• Ignoring the budget. An SLO nobody consults before a risky deploy is decoration. The budget must gate real decisions or the whole system is theater.
• Too many SLOs. Start with availability and latency on your most user-facing service. Expand only after the review cadence sticks.

Where to go from here

A realistic rollout: instrument the edge, run the SLIs as recording rules for two weeks to learn your baseline, then set an SLO you already meet and turn on the burn-rate alerts. Tighten later if users need it. If you want experienced hands on this — from choosing SLIs to wiring Prometheus, Alertmanager, and Grafana into a review cadence your team actually keeps — that is exactly what our SRE and observability practice does. And because SLO rules are YAML, they belong in git next to your manifests, deployed through the same GitOps delivery pipeline as everything else — reviewed, versioned, and reproducible.