Alarms on custom metrics
Every custom metric can carry as many alarms as you want, each with its own threshold, its own window and its own severity. They open incidents and notify through the channels and policies you already use — the same machinery as a server CPU alarm, over a number only you can measure.
Add them from the metric’s page, or from Edit ▸ Alarms.
Building one
Section titled “Building one”An alarm reads as a sentence:
aggregate the metric over window, and if the result is comparison threshold, open a warning or critical incident.
| Part | Options |
|---|---|
| Aggregate | Count metrics: total. Level metrics: average, minimum, maximum, latest. |
| Window | 1 minute to 30 days |
| Comparison | below, at or below, above, at or above, equals |
| Threshold | any number, including 0 and negatives |
The aggregations offered follow the metric’s type, because the others would not mean anything: the total of a queue depth is not a number, and the average of individual upload events is not the question you asked.
What you can express
Section titled “What you can express”| You want to know | Type | Alarm |
|---|---|---|
| Nothing has been uploaded for 24 hours | count | total over 24h below 1 |
| Fewer than 100 orders in the last hour | count | total over 1h below 100 |
| More than 500 payment errors in 5 minutes | count | total over 5m above 500 |
| Queue depth peaked above 10,000 | level | maximum over 15m above 10000 |
| Free licence seats dropped below 5 | level | minimum over 1h below 5 |
| Average queue depth is sustained above 1,000 | level | average over 30m above 1000 |
| Stock is below the reorder point right now | level | latest over 5m below n |
| Last night’s ETL did not succeed | level | send 1/0; latest over 24h equals 0 |
| More than 1M API calls this month | count | total over 30d above 1000000 |
| Account balance went negative | level | latest over 1h below 0 |
The dead-man’s switch
Section titled “The dead-man’s switch”The one most people want first: “tell me if this stops happening.”
Total of
files.uploadedover 24 hours, below 1.
For a count metric, no values in the window genuinely means zero occurrences, so this fires — which is exactly right. For a level metric the same silence means we have no measurement, not the value is zero, so we leave the alarm alone rather than telling you a queue is empty when we simply cannot see it.
Escalating
Section titled “Escalating”Two alarms on one metric, with the same threshold and different windows:
- warning — total over 24h below 1
- critical — total over 72h below 1
A day of silence is worth knowing about; three days is worth waking someone.
You can escalate on severity instead — the same window with a more extreme threshold — or combine both.
Things worth knowing
Section titled “Things worth knowing”A metric that has never been sent to never alarms. A brand-new metric would otherwise trip its own dead-man’s switch before it ever had a chance to report.
Alarms look back at most 30 days, although values are kept for three years. You can chart a year and alarm on a month.
There are no default alarms. Nothing is attached to a metric until you attach it —
there is no threshold that would be meaningful for files.uploaded, queue.depth and
account.balance at the same time, so we do not invent one.
Changing a metric’s type may leave an alarm using an aggregation the new type does not offer — a total on a metric that is now a level. Those alarms are switched off for you to review rather than deleted or left running on a meaningless number.
When our store cannot be reached, alarms are left exactly as they are: nothing opens and nothing resolves. A store that did not answer is not a store reporting zero.