Server monitoring, as commonly deployed, watches three things: processor use, memory use and disk space. They appear on every default dashboard and they are what people mean by server monitoring.
Two of the three are poor predictors of anything. High CPU usually means a server doing its job. High memory use on Linux usually means the operating system caching, which is correct behaviour and alarms people who have not seen it before.
Meanwhile the signals that genuinely precede outages — a certificate approaching expiry, a backup that silently stopped, time drifting, a disk reporting internal errors — are frequently not collected at all, because no default dashboard includes them.
Why the usual three mislead
| Metric | Reality |
|---|---|
| CPU utilisation | High is usually correct. Sustained saturation matters; a busy hour does not. |
| Memory used | On Linux, includes cache. The meaningful figure is memory available and swap activity. |
| Disk space used | Genuinely useful — but the useful form is time until full, not percentage. |
The disk row contains the general principle. Ninety percent full is not information: a disk at ninety percent that has been stable for a year is fine, and one at seventy percent that gained twenty points this week is an incident next Tuesday. The rate of change is the signal; the percentage is a snapshot.
The same reframing applies broadly. Alerting on a threshold produces alerts when a value crosses a line, which is frequently after the useful moment. Alerting on a trend produces alerts while there is still time to act, which is what monitoring is for.
The signals that actually precede outages
These are the ones worth adding, roughly in order of how often their absence causes a preventable incident.
| Signal | Catches |
|---|---|
| Certificate expiry dates | The outage that arrives on a scheduled date and takes every replica at once |
| Time since last successful backup | Backups that stopped silently, which is how backups usually fail |
| Time since last successful patch run | Update mechanisms that broke and stopped without erroring |
| Clock offset from NTP | Authentication, certificate validation and log correlation failures |
| Disk SMART attributes | Physical disks degrading before they fail |
| Service and unit failure state | Something that stopped and did not restart |
| Storage latency | The most common real bottleneck, and the least watched |
| Filesystem inode usage | A disk that is full while showing free space |
| Absence of expected data | Anything that stopped happening — the failure nothing else detects |
Three of these are worth expanding, because they cause the most damage relative to how easy they are to collect.
Certificate expiry
Certificates expire on a known date, which makes this the most preventable outage there is, and they still cause outages regularly. The reason is that no redundancy addresses it: certificates issued together expire together, so every replica fails simultaneously, at a moment nobody scheduled.
Monitor days remaining, on every certificate including internal ones, with an alert far enough ahead to renew calmly. Automated renewal is better, and monitoring is still required, because automated renewal also fails silently.
Absence, rather than errors
Most monitoring alerts on bad things happening. The more valuable and less common pattern is alerting on expected things not happening.
- A backup job that has not completed in 26 hours, rather than a backup job that reported an error.
- A nightly report that has not been generated.
- An integration that has not processed a record during business hours.
- A server that has not reported metrics for ten minutes.
The distinction matters because the common failure mode of automation is not failing loudly. It is stopping. A job that errors sends an error; a job that never runs sends nothing, and nothing is indistinguishable from everything being fine unless something is watching for it.
Storage latency
Disk latency is the constraint that most often produces the complaint that everything is slow with no obvious cause. It is not on default dashboards, it is shared between every workload on the host in a virtualised estate, and it degrades everything simultaneously in a way that looks like a general problem rather than a specific one.
Making the alert set worth responding to
An alert set nobody trusts is worse than no alerts, because it trains people to dismiss them, and the dismissal habit persists into the alert that mattered.
- 1
Every alert must have an action
If the answer to receiving it is to look and do nothing, it is a dashboard item. This single rule removes most alert noise.
- 2
Alert on trends and durations, not instantaneous values
CPU at 100% for one minute is a busy minute. CPU at 100% for thirty is a problem. The duration qualifier eliminates most false alarms without reducing coverage.
- 3
Separate urgent from scheduled
A disk filling in four days is real and is a ticket. Paging someone at 3am for it teaches them that pages are not urgent.
- 4
Write the response into the alert
What it means, what to check, who owns it. An alert that requires the recipient to already know the system is an alert only one person can action.
- 5
Review what fired and what was acted on
Monthly. Alerts that fire and are always dismissed should be deleted or fixed. Incidents that fired no alert are the gaps worth closing.
- 6
Monitor the monitoring
A monitoring system that stops reports nothing, including its own absence. Something outside it must notice.
A reasonable starting set
For a growing business with a modest estate, this is a defensible baseline. It is short deliberately.
- Host unreachable for more than a few minutes — urgent.
- Any certificate expiring within 30 days — scheduled, escalating as the date approaches.
- No successful backup in 26 hours — urgent, because the window to fix it before the next one is short.
- Filesystem projected to fill within 7 days — scheduled.
- Clock offset beyond a small threshold — scheduled.
- A service in failed state — urgent for anything customer-facing, scheduled otherwise.
- Disk reporting SMART errors — scheduled, and act on it before it becomes urgent.
- No metrics received from a host for 10 minutes — urgent, because it means you are blind rather than that everything is fine.
- Business outcome stopped — orders, calls, logins. Urgent, and the most valuable alert on the list.
The last item is the one most estates lack and the one that would catch the most damaging failures, because it detects the intrusion, the bad deployment, the stalled integration and the upstream outage without any of them having been anticipated.
Why is high CPU usage not a good alert?
Because high CPU usually means a server doing the work it exists to do. A busy hour is not a fault. What matters is sustained saturation over a period long enough to indicate the workload no longer fits, which is why duration qualifiers on CPU alerts eliminate most false alarms without reducing real coverage. The same applies to memory on Linux, where reported usage includes cache and the meaningful figures are memory available and swap activity.
What is the most preventable server outage?
Certificate expiry. It happens on a known date, and no amount of redundancy helps because certificates issued together expire together, so every replica fails at the same moment. Monitoring days remaining on every certificate, including internal ones, with enough lead time to renew calmly, prevents it. Automated renewal is better and still requires monitoring, because automated renewal also fails silently.
Why alert on the absence of things rather than on errors?
Because the common failure mode of automation is stopping rather than failing loudly. A job that errors sends an error; a job that never runs sends nothing, and nothing is indistinguishable from everything working unless something is specifically watching for the absence. Alerting on time since last successful backup, rather than on backup errors, is the clearest example — it catches the silent stop that error-based alerting cannot see.
Should disk alerts be based on percentage full?
No, on projected time to full. Ninety percent full and stable for a year is fine; seventy percent that gained twenty points this week is an incident next week. The rate of change is the signal and the percentage is a snapshot. This generalises: threshold alerts fire when a value crosses a line, which is often after the point at which action was easy, whereas trend alerts fire while there is still time to act.
What single alert catches the most serious problems?
A business outcome stopping — orders not completing, calls not connecting, logins not succeeding. It detects intrusions, bad deployments, stalled integrations and upstream provider outages without any of those having been anticipated, which resource-level alerts cannot do. It is also the alert most estates lack, because resource metrics arrive automatically from exporters while outcome metrics have to be deliberately instrumented.
Sources and further reading
- Infrastructure monitoring that works— the wider monitoring design this fits into
- Prometheus for business infrastructure— the collection layer, and why absence is detectable in a pull model
- Grafana dashboards that get used— where the causes belong, once the symptoms are alerting
- High availability for business systems— correlated failure, including the shared certificate expiry date
- Linux servers for business— the update mechanism whose silent failure this monitors for
- Data retention and archiving for growing businesses— the backups whose absence is the second alert on the list
- Prometheus metric and label naming conventions— collecting these signals consistently across an estate
Services This Relates To
Written by KYCONNECTS Engineering. Client names are withheld under confidentiality.