Alerting & on-call

Alerts are defined as queries plus a condition and an evaluation window. Route them to schedules, and escalate if nobody acknowledges.

A simple alert

rule: high-error-rate
query: sum(rate(http_requests_total{status=~"5.."}[5m]))
       / sum(rate(http_requests_total[5m]))
condition: > 0.02 for 10m
route: payments-oncall

Anomaly detection

Instead of a fixed threshold, let the platform learn the daily and weekly seasonality of a metric and alert on deviation. This is what keeps a nightly batch job from paging anyone.

Escalation policies

• Notify primary on-call via push and SMS.
• After 5 minutes unacknowledged, notify secondary.
• After 15 minutes, notify the team channel and the engineering manager.