There are two end states for a monitoring setup. In one, an alert fires and you stop what you’re doing, because alerts mean something. In the other, an alert fires and you archive it on muscle memory, because alerts are weather. Every monitoring system drifts toward the second state unless you actively fight it — and the fight is a design problem, not a discipline problem.

How Good Monitoring Goes Bad

Nobody sets out to build a spam cannon. It accretes. You add a check during an incident (“never again!”). You alert on a threshold that made sense that day. You monitor a thing because the dashboard had an empty slot. Each alert is individually defensible; collectively they train you to ignore them. The system cries wolf, and unlike the fable, the wolf shows up on page six of the notification history.

The One-Question Filter

For every alert, ask: “What action do I take when this fires?”

  • If the answer is a concrete action — disk is filling, certificate expires in a week, service is down — keep it.
  • If the answer is “hm, interesting” — that’s not an alert, that’s a graph. Put it on a dashboard you visit voluntarily.
  • If the answer is “nothing, it usually clears on its own” — delete it, and notice that you already knew.

An alert is a request for human attention. Human attention is the scarcest resource in the entire stack — more precious than disk, more precious than uptime. Spend it like it’s money.

Design Rules That Hold Up

Alert on symptoms, not causes. “The site is unreachable” matters. “CPU is at 90%” might be a problem or might be a machine doing its job. Symptom alerts catch failures you didn’t predict; cause alerts only catch the ones you did.

Give every alert a deadline, not a heartbeat. “Certificate expires in 7 days” is actionable all week. The same alert firing every hour for a week is one piece of information delivered 168 times, at a total cost of one habit of ignoring you.

Two severities are enough. “Wake me up” and “look at it tomorrow.” The moment there’s a middle tier, everything migrates into it and dies there.

The Test

Simple audit: scroll your last twenty alerts. Count how many led to a human doing something. If it’s under half, the monitoring isn’t protecting the system — it’s generating guilt with timestamps. Fix it the same way you’d fix any other broken service. It’s not virtue. It’s design.