Alert fatigue from “mostly harmless” anomalies happens when detection is technically correct but operationally irrelevant.????? The system is telling the truth too often about things no one needs to act on.

Here’s a structured way to think about it.

What makes an anomaly “mostly harmless”

These anomalies satisfy the detector but fail the operator value test:

1. Self-healing by design

   • Autoscaling blips

   • Retry storms that resolve within seconds

   • Leader re-elections

2. Expected under known conditions

   • Cache cold starts after deploys

   • Batch jobs causing predictable load spikes

   • Traffic shape changes from experiments

3. Low blast radius

   • Affects a tiny cohort

   • Impacts non-critical paths

   • Degraded-but-acceptable UX

4. No follow-up action exists

   • There is nothing the on-call can do better than “wait”

These alerts are true positives but false obligations.

Why they cause disproportionate fatigue

Alert fatigue isn’t linear—it’s compounding:

• Cognitive context switching: On-call must still load the system model to decide it’s harmless.

• Trust erosion: Engineers learn that “alerts ? urgency.”

• Signal masking: Real incidents hide in the noise floor.

• Policy drift: People start muting alerts ad hoc, often permanently.

This aligns with your earlier threads on observability SNR and trust erosion from small failures—harmless anomalies steadily destroy alert credibility.

Common anti-patterns that create them

1. Symptom-level alerting

   • Alerting on variance instead of user impact

   • Alerting on raw metrics rather than outcomes

2. Uniform severity mapping

   • Everything pages

   • No distinction between “investigate later” and “act now”

3. Single-signal alerts

   • One metric crossing a threshold

   • No corroboration required

4. Static thresholds in dynamic systems

   • Autoscaling + static limits = guaranteed noise

   • Design principles to reduce fatigue (without going blind)

     1. Actionability as a hard gate

     Before an alert exists, answer:

     What should the on-call do differently if this fires?

     If the answer is “monitor” or “wait,” it’s not a page.

     2. Multi-signal confirmation

     Require convergence:

   • Anomaly + user impact

   • Anomaly + sustained duration

   • Anomaly + correlated downstream effect

   • Harmless anomalies rarely correlate.

     3. Time as a filter, not a threshold

     Instead of:

   • “CPU > 80%”

   • Use:

   • “CPU > 80% and request latency degraded for 5 minutes”

   • Time distinguishes transients from incidents.

     4. Severity tied to blast radius

     Map alerts by who is harmed, not by how weird it looks:

   • Internal-only ? log / dashboard

   • Small cohort ? ticket

   • Revenue / safety ? page

   • 5. Explicit “known-noise” lanes

     Don’t suppress silently:

   • Label alerts as expected-but-monitored

   • Route to a non-paging channel

   • Periodically review for drift

   • If engineers say:

   • “Oh yeah, that one always fires”

   • “Just ignore it unless it keeps happening”

   • “It’s usually fine”

   • That alert is actively harming your incident response posture.

     If you want, next we can:

   • Turn this into an alert review checklist

   • Map it to error budgets / SLO-based alerting

   • Explore partial availability vs harmless anomalies

   • Or quantify alert fatigue as an operational metric

Just tell me where you want to go next.