A business installs Grafana, connects a data source, imports a community dashboard, and puts it on a wall-mounted screen. For two weeks people glance at it. After a month nobody does, and when something breaks, nobody thinks to open it.
The dashboard was not badly built. It was built to satisfy a request for visibility, which is not a question, and so it answers nothing in particular. Forty panels of green graphs communicate that data is being collected. They do not tell anyone what to do.
The fix is not fewer panels or better colours. It is deciding what question each dashboard exists to answer, and removing everything that does not help answer it.
The business problem: visibility that nobody converts into action
Monitoring is bought after an outage that went unnoticed for too long. The value being sought is a shorter time between something going wrong and someone knowing about it.
A dashboard only shortens that interval if a human is looking at it at the moment of failure. In practice nobody is, which means an unread dashboard delivers none of the value the business paid for, while continuing to cost storage, query load and the belief that the problem is handled.
That last cost is the largest. A business that believes it has monitoring stops asking whether it would notice, and the answer stays untested until an outage answers it.
Dashboards and alerts are not the same tool
The distinction is drawn clearly in Google's Site Reliability Engineering book, and adopting it resolves most of the confusion about what belongs where. Alerts are for urgent, actionable symptoms that require an immediate human response. Dashboards are for subcritical and ongoing conditions reviewed manually.
The book is also direct about email alerts: they tend to become overrun with noise, and dashboards are the better home for information that does not need immediate action.
| Information | Mechanism | Reason |
|---|---|---|
| Orders are failing right now | Alert (page) | Urgent and actionable |
| A disk will fill in about four days | Alert (ticket) | Actionable, not urgent |
| CPU is at 70% | Dashboard | Not actionable on its own |
| Request latency over the last hour | Dashboard | Diagnostic during an incident |
| Capacity trend over six months | Dashboard | Input to planning |
| A node restarted | Depends | Alert if unexpected; dashboard if routine |
The reason column is the operative one. If the answer to seeing something is do nothing, it does not belong in an alert. Everything that pages a person and does not require them to act teaches them that pages can be ignored, which is the mechanism by which real pages get missed.
Alert on symptoms, not causes
The same source gives the principle that separates useful alerting from noisy alerting: monitoring should answer two distinct questions, what is broken and why. The first is the symptom, the second is the cause. For paging decisions, it recommends spending much more effort catching symptoms than causes, and worrying about causes only when they are very definite and very imminent.
Applied to a business system, the difference is concrete. High CPU is a cause. It might mean the system is under stress or it might mean a batch job is running as designed. Paging on it produces false alarms until people stop responding.
Checkout requests failing is a symptom. It is unambiguous, it is always worth waking someone for, and it catches every cause including the ones nobody predicted — CPU exhaustion, a full disk, an expired certificate, a dependency outage, a bad deployment.
- Symptom: users cannot log in. Cause: the LDAP connection pool is exhausted.
- Symptom: calls are not connecting. Cause: the SIP trunk has deregistered.
- Symptom: orders are not reaching the warehouse. Cause: the integration queue has stalled.
- Symptom: the site returns errors. Cause: the certificate expired at midnight.
Alert on the left column. Put the right column on the dashboard, where it will be read during the investigation the alert triggered. This is the whole relationship between the two tools.
What to measure: the four golden signals
For any request-serving system, the same book identifies four signals worth measuring above all others. They are a good default because between them they detect most failures without requiring the failure to have been anticipated.
| Signal | Definition | Detects |
|---|---|---|
| Latency | Time taken to service a request, with successful and failed requests distinguished | Degradation before outright failure |
| Traffic | Demand on the system, such as requests per second | Both unexpected load and unexpected silence |
| Errors | Rate of requests that fail explicitly, implicitly, or by policy | Correctness failures that latency alone misses |
| Saturation | How full the most constrained resource is | Exhaustion before it happens |
Two of these have subtleties that matter in practice.
Latency must separate successful from failed requests, because failures are frequently fast. A system returning errors instantly can show excellent average latency while being entirely broken, and the average will look better during the outage than before it.
Traffic dropping to zero is as significant as traffic spiking, and it is the one people forget to alert on. If your order rate goes to nought at 10am on a Tuesday, something upstream is broken, and no error will be recorded anywhere because no request arrived.
One dashboard, one question
The structural fix for the unread dashboard is to give each one a single stated purpose and hold it to that. Three types cover almost all real needs, and they are used by different people at different times.
The status dashboard
Question: is the business working right now? Audience: anyone, including non-technical staff. Should be readable in five seconds from across a room.
Business outcomes only — orders completing, calls connecting, logins succeeding. No CPU, no memory, no disk. If it has more than about eight panels it has stopped being a status dashboard and become a diagnostic one.
The diagnostic dashboard
Question: why is it broken? Audience: whoever is responding. Opened during an incident and at almost no other time.
This is where the causes live, and this dashboard is allowed to be dense. Arrange it in the order an engineer investigates — the four golden signals at the top, then dependencies, then resources — so that reading top to bottom follows the diagnostic path rather than requiring the responder to know where to look.
The capacity dashboard
Question: what will run out, and when? Audience: whoever plans and budgets. Reviewed monthly, never during an incident.
Long time ranges — ninety days minimum — with trends rather than current values. The useful output is a date, not a percentage: this disk fills in March, this link saturates at about forty more staff.
Building them
- 1
Write the question at the top of the dashboard
Literally, in a text panel. It disciplines every later decision about what to add, and it tells the next person what the dashboard is for.
- 2
Get the metrics right before drawing anything
Grafana queries; it does not store. If the business outcome you need is not instrumented, no panel arrangement will surface it. Instrumenting it is the work.
- 3
Start from business outcomes, then descend
Orders per minute before CPU. If the only failure you can detect is at the resource level, you will detect resource problems and miss application ones.
- 4
Use percentiles, not averages
p50, p95 and p99 for anything time-based. Averages conceal exactly the tail that constitutes the user-visible failure.
- 5
Give every panel a threshold or a comparison
A number with no reference point is not information. Is 340ms good? Nobody knows unless the panel shows the threshold or last week's line.
- 6
Delete panels nobody has used
Review quarterly and remove what has never been looked at during an incident. A dashboard is improved far more by deletion than by addition.
- 7
Treat imported dashboards as drafts
Community dashboards are a fast start and a poor finish. They are built for a generic deployment and typically show everything an exporter emits, which is the opposite of the discipline being described here.
Where dashboards do not help
Being honest about the limits saves money that would otherwise go into building more of them.
- Detecting failure out of hours. Nobody is watching at 3am. That is alerting's job, and no dashboard substitutes for it.
- Root cause in distributed systems. When a request crosses several services, aggregate charts show that something is slow but not where. That requires tracing.
- Understanding a single failed transaction. Metrics are aggregates. A specific customer's failed order is a logs question.
- Replacing an on-call rotation. A wall screen with nobody in front of it detects nothing.
Grafana supports alerting itself, with notifications to the usual destinations, so the alerting half of this does not require another product. It does require deciding what genuinely warrants waking someone — which is a judgement about the business rather than a configuration task.
Does Grafana store metrics?
No. Grafana is a query and visualisation layer that connects to external data sources — time-series databases, SQL and NoSQL databases, and other systems through its plugin framework. Metrics must be collected and stored elsewhere, commonly in Prometheus or a similar store. This matters when planning, because the collection and retention design is the substantial work and the dashboard is the presentation of it.
What should trigger an alert rather than appear on a dashboard?
Alerts should carry urgent, actionable symptoms that need immediate human response — orders failing, logins failing, calls not connecting. Dashboards should carry subcritical and ongoing conditions reviewed manually, including the resource-level causes an engineer consults while investigating. If the correct response to seeing something is to do nothing, it does not belong in an alert.
What are the four golden signals?
Latency, traffic, errors and saturation, as defined in Google's Site Reliability Engineering book. Latency is the time to service a request, measured separately for successful and failed requests. Traffic is demand on the system. Errors is the rate of requests that fail explicitly, implicitly or by policy. Saturation is how full the most constrained resource is. Together they detect most failures without the failure having been anticipated.
Why should latency be measured as percentiles rather than an average?
Because an average conceals the tail, and the tail is the user-visible failure. If ninety-five requests take 50ms and five take eight seconds, the mean is around 450ms and appears acceptable while five percent of users had an unusable experience. Charting p50, p95 and p99 makes that visible. Failed requests are also frequently fast, so a system returning instant errors can show an improving average while entirely broken.
Are community Grafana dashboards worth using?
As a starting point, yes; as a finished product, no. Imported dashboards are built for generic deployments and typically display everything the underlying exporter emits, which is the opposite of the one-dashboard-one-question discipline that makes a dashboard get used. Treat an import as a draft, decide what question it should answer, and delete everything that does not help answer it.
Sources and further reading
- Infrastructure monitoring that works— the collection layer and alert design this article builds on
- High availability for business systems— why noticing a failed node quickly is a separate problem from surviving it
- Infrastructure planning for growing businesses— what the capacity dashboard feeds into
- n8n workflow automation for modern businesses— automating the response once an alert is trustworthy enough to act on
- Monitoring Distributed Systems — Google SRE Book— the four golden signals, symptoms versus causes, and the dashboard-versus-alert distinction used throughout
- Grafana documentation: introduction— data source model and built-in alerting
- Prometheus documentation— the collection and storage layer most commonly sitting behind Grafana
Services This Relates To
Written by KYCONNECTS Engineering. Client names are withheld under confidentiality.