SREEXAMPLE SCENARIO
Introducing SLOs and burn-rate alerting
This is an illustrative scenario describing how we typically approach this kind of engagement. It does not describe a specific client, and no results are claimed.
Challenge
A common starting point: hundreds of alerts, frequent pages for non-issues, and no shared definition of 'reliable enough' between product and engineering.
Architecture
- OpenTelemetry instrumentation for key user journeys
- Prometheus recording rules for SLIs
- Multi-window burn-rate alerts
- Grafana dashboards per service SLO
Approach
- Identify the most important user journeys
- Define SLIs and initial SLOs with product owners
- Replace symptom-noise alerts with burn-rate alerts
- Agree on an error budget policy
Technology
OpenTelemetry · Prometheus · Grafana
Implementation
- Instrument request success and latency at service boundaries
- Create SLO dashboards and reviews
- Retire alerts that do not map to user impact
- Run incident reviews against the budget
Intended outcome
The aim is fewer, more meaningful pages and a shared, data-based way to decide between feature work and reliability work.
Lessons learned
- Start with a few SLOs and refine targets over time
- Burn-rate alerts need both short and long windows
- An error budget policy only works if agreed in advance