Infrastructure monitoring that works

A monitoring dashboard shows every server green. CPU is comfortable, memory is fine, disks have space, and every host has responded to its ping for ninety consecutive days. Meanwhile the sales team cannot save a record, because a database connection pool is exhausted and the application is returning errors to every user who tries.

Nothing on that dashboard is wrong. The servers genuinely are up. The problem is that the dashboard answers a question nobody asked — is the hardware running — while the question that matters is whether the business can do its work. Those two are correlated often enough that the gap goes unnoticed for years, and then diverges on exactly the day it matters.

The business cost of this is not the outage. It is that the outage is discovered by a customer or an employee rather than by the system that exists to discover it, which adds however long that takes to every incident, and removes any possibility of catching the problem before it became one.

Up is not the same as working

Almost every monitoring deployment starts from the host: is it reachable, is it out of CPU, is the disk filling. These are worth collecting and they are the wrong things to alert on, because they answer a question about a machine rather than about a service.

The fourth row is the failure mode that recurs most across this whole series. A process that is alive and processing nothing produces no errors, consumes no unusual resources, and looks healthy by every host-level measure. It is the shape of a stopped integration, a stalled automation, a trunk that has deregistered, and a backup job that has silently not run.

That is why the most valuable alert most businesses do not have is an alert on absence: nothing has been processed in a period where something should have been. It is cheap, it catches a whole class of silent failure, and it cannot be derived from any host metric.

What to actually watch

Google's Site Reliability Engineering material gives the most widely used framing, and it is useful precisely because it is short. Four signals, defined from the perspective of the service rather than the machine.

The implicit failure in the errors definition deserves attention. A response can be a technical success and a business failure — the material gives the example of an HTTP 200 returned with the wrong content. Monitoring that counts status codes will report that service as perfectly healthy, which is the same category of blindness as counting registrations on a telephony platform while every call is silent.

Saturation is the one worth investing in, because it is the only forward-looking signal of the four. Latency, traffic and errors tell you about now. Saturation tells you that a disk will be full on Thursday, that a connection pool is running at ninety per cent, or that a channel count is approaching its ceiling — all of which are cheap to fix on Tuesday and expensive to fix on Thursday.

Alert on symptoms, investigate with causes

The distinction that separates a usable alerting setup from an unusable one is between what is broken and why. The SRE material frames this as symptoms versus causes, and notes that the boundary moves depending on where you sit: slow database reads are a symptom to a database engineer and a cause to a frontend engineer looking at a sluggish site.

The practical rule is that pages go on symptoms — the things users can feel — and causes go on dashboards, where they are consulted during an investigation. Alerting on causes produces alerts for conditions that may have no user impact at all, which is how alert fatigue begins.

The same material sets four conditions for anything that pages a human: it should trigger on a symptom, be actionable by human judgement rather than something a script should have handled, represent a novel problem rather than one seen repeatedly, and preserve the responder's capacity to treat it as urgent. That last condition is the one businesses violate first and notice last.

Alert fatigue is a design failure, not a discipline failure

When an on-call person starts ignoring alerts, the instinct is to treat it as a people problem. It is not. It is the predictable result of a system that pages on conditions which are frequently not actionable, and the fix is in the alert definitions rather than in the rota.

1. 1

   Every page must have an action

   If the documented response is to look at it and confirm it recovers, it should not be a page. Send it to a dashboard or a daily digest instead.

2. 2

   Alert on the user-visible condition, not its cause

   One page saying checkout is failing beats six saying a queue is deep, a pool is exhausted, latency is up, and three hosts are loaded — all describing one incident.

3. 3

   Use duration, not instantaneous values

   Alerting when CPU exceeds a threshold produces noise. Alerting when it has stayed there for fifteen minutes produces signal, because the transient spikes are filtered out by definition.

4. 4

   Route by severity deliberately

   Waking somebody is reserved for conditions that cannot wait until morning. Everything else is a ticket, and the difference is decided when the alert is written rather than at three in the morning.

5. 5

   Review what fired, on a schedule

   Any alert that fires regularly and is regularly ignored should be deleted or rewritten. Keeping it costs attention on every future incident.

Logs, metrics and traces do different jobs

These are collected by different tools and answer different questions, and treating them as interchangeable produces a setup that is expensive and still cannot answer the question during an incident.

For most growing businesses the correct emphasis is metrics for alerting and capacity, logs for investigation and audit, and traces only where a request genuinely crosses several services and the time is disappearing somewhere unclear. Buying distributed tracing before the architecture is distributed is a common and expensive mis-sequencing.

Monitor the things that are not servers

Monitoring coverage tends to follow whoever set it up. Where that was a systems administrator, servers are covered comprehensively and everything else is not. The gaps below account for a large share of incidents discovered by users rather than by monitoring.

• Certificate expiry. An expired TLS certificate is a total outage with a known date, which makes it the most avoidable incident there is — and it still happens routinely, because nothing was watching.
• Domain and DNS. Registration expiry, and whether resolution actually returns what it should from outside your network.
• Backup completion, and more importantly restore success. A backup job that reports success while producing an unusable archive is worse than one that fails loudly.
• Scheduled jobs and automations, alerted on absence rather than on error, for the reasons described above.
• Integrations and third-party dependencies, since an outage at a provider you depend on is your outage from a user's perspective.
• Trunk registration and channel utilisation where telephony is in scope, which the call centre article covers in more detail.
• The monitoring system itself. A monitoring platform that has stopped collecting reports perfect health, because health is inferred from the absence of alerts.

The last item is worth taking seriously. The failure mode is that monitoring stops and everything appears fine, which is indistinguishable from everything being fine right up until it is not. A dead man's switch — an external check that expects a regular heartbeat and alerts when it stops arriving — is the standard answer and costs almost nothing.

External checks see what users see

Monitoring that runs entirely inside your own network shares fate with your own network. If the circuit fails, the firewall misroutes, or DNS returns the wrong answer to the outside world, internal checks continue to pass while the service is unreachable to everyone who matters.

At least one check should run from outside the estate, against the same address a customer uses, exercising a request that reflects real use rather than fetching a static page. Where a login or a database read is part of the journey, the check should include it — a synthetic transaction is worth more than a hundred pings, because it fails when the business fails rather than when the network does.

Common mistakes

A representative scenario

A composite of situations we see repeatedly; no client detail is included.

A business runs monitoring across roughly forty hosts. Coverage is thorough at the host level and the dashboards are well maintained. The review is prompted by a two-hour outage that was reported by a customer.

The cause was a certificate expiry on a public endpoint. Nothing monitored certificates. Separately, the review found that a nightly export to a partner had not run for eleven days — the scheduled task had failed to start rather than failing during execution, so it produced no error and no alert. On-call staff described roughly forty alerts a night, most of which were CPU spikes that resolved themselves within minutes.

The remediation started with removing rather than adding. Alerts that fired regularly and were routinely ignored were deleted or converted to daily summaries, which cut overnight paging by most of its volume. Duration conditions were added to the thresholds that remained. Certificate and domain expiry monitoring was added with several weeks of warning. Absence alerting was added for scheduled jobs and integrations. One external synthetic check was created exercising a login and a database read from outside the network, and a heartbeat check was added so that the monitoring platform failing would itself raise an alert.

The number of alerts fell substantially and the number of real incidents caught before a customer noticed rose. Those two moving in opposite directions is the signature of monitoring that has started working.

Implementation checklist

• Alerts fire on user-visible symptoms; host metrics are collected for investigation but do not page.
• Latency, traffic, errors and saturation are tracked for each significant service.
• Absence of expected activity raises an alert, not only errors.
• Every alert that pages has a documented action; anything without one is a dashboard item or a digest.
• Thresholds require a duration rather than firing on an instantaneous value.
• Severity is assigned when the alert is written, and only what cannot wait until morning wakes anybody.
• At least one check runs from outside the estate against a real user journey, not a static page.
• Certificate and domain expiry are monitored with weeks of warning.
• Backups are monitored for restore success, not only for job completion.
• Scheduled jobs, automations and integrations are covered, with absence alerting.
• Authentication and administrative logs are separated from high-volume application logs and retained longer.
• The monitoring system has an external heartbeat so its own failure is detectable.
• Alerts that fire and are routinely ignored are reviewed on a schedule and deleted or rewritten.

Frequently asked questions

Conclusion

Monitoring is usually judged by how much it collects, and it should be judged by two numbers: how long a problem exists before anyone knows, and how long it takes to explain once they do. Collecting more host metrics improves neither. Watching what users experience, alerting on that alone, and keeping enough detail underneath to investigate improves both.

The clearest sign that a setup is working is counter-intuitive: fewer alerts and more incidents caught before a customer notices. If those two are moving in the same direction, something is wrong with the alert definitions rather than with the people responding to them.