Monitoring & alerting
Knowing before the customer does
Most systems are monitored in the sense that a dashboard exists. Far fewer will wake somebody when the thing that matters to a customer stops working.
In short
Useful monitoring alerts on symptoms a customer would notice — checkout failing, orders not reaching the warehouse, response times degrading — rather than on individual resources. The test is whether every alert that fires requires action. Once a monitor produces alerts people routinely ignore, it has stopped being monitoring and become noise, and the real outages hide inside it.
When it applies
Signals this is the work
If more than one of these is true today, the problem is usually further along than it looks from the outside.
Users report problems first
Your earliest signal is a complaint.
Alerts are muted
A channel exists and nobody reads it, because most of what arrives is not actionable.
Only infrastructure is watched
CPU and disk are monitored. Whether an order can actually be placed is not.
Diagnosis means guessing
When something breaks, the first hour goes on working out where, because nothing correlates.
Nobody owns the alert
It fires into a group. Everyone assumes someone else has it.
How we do it
The sequence
- 01
Define what "working" means
In business terms: an order can be placed, a payment clears, data reaches the ERP. That is what gets monitored.
- 02
Alert on symptoms, not causes
High CPU may be fine. Checkout failing is never fine. Alert on the second and use the first to explain it.
- 03
Give every alert an owner and a runbook
An alert with no named owner and no first step is a notification, not an alert.
- 04
Make logs and traces correlatable
Enough context to follow one request across services, so diagnosis is reading rather than guessing.
- 05
Tune ruthlessly
Every alert that fires without needing action gets fixed or deleted. Noise is the thing that kills monitoring.
- 06
Rehearse
Test that the alert path actually reaches a human, including at 3am on a weekend. Most on-call gaps are discovered during a real incident.
What we will not do
The refusals
- Add dashboards nobody looks at and call it observability.
- Alert on every metric available because the tool makes it easy.
- Report uptime from a check that only proves the homepage loads.
- Set up on-call without agreeing who is actually on it.
Questions
Before you ask
Tell us what isn't working.
One process, one system, one decision you are stuck on. We will come back with how we would approach it, what it would take, and whether it needs building at all.