Incidents & alerts
When consensus moves a check between healthy and down, ZeroDrop opens or closes an incident. It then notifies the channels that are attached to that check. This page describes how to create channels and what ZeroDrop delivers.
Notification channels
A channel is a destination for alerts. You create a channel one time and attach it to as many checks as you want. There are two types.
| Type | Destination | Delivery |
|---|---|---|
| An email address | One formatted message for each event. | |
| Webhook | An https:// URL |
A JSON POST that your systems can act on. |
Create and verify a channel
- On the Channels page, select New channel. Then select email or webhook.
- Enter the address or the URL, and save the channel.
- Select Verify.
For an email address, ZeroDrop sends a confirmation link to it. The channel stays unverified until the recipient opens that link. For a webhook, ZeroDrop marks the channel verified immediately.
An email channel receives alerts only after this verification.
ZeroDrop refuses a destination it cannot use, and tells you why:
- A webhook URL must start with
https://orhttp://and give a host. - A webhook URL must not contain a user name or a password. Put the secret in the path or the query instead.
- A webhook URL must be reachable from the public internet. An address such as
localhostor10.0.0.5is refused. - An email address must be a bare address, without a display name.
Webhook URLs often contain secret tokens. ZeroDrop therefore stores the settings of every channel encrypted at rest.
Test a channel
To send one test notification now, select Send test on the Channels page. The result shows below the destination.
Use a test after you create a channel, and after you change a webhook URL. It shows that the destination accepts alerts, before a real problem occurs.
| Result | Meaning |
|---|---|
| Test sent | The destination accepted the notification. |
| Not sent | ZeroDrop did not try. The message gives the cause, usually an email address that is not verified. |
| Test failed | The destination refused the notification, or ZeroDrop could not reach it. The message gives the HTTP status. |
A test makes one attempt only. It does not retry, and it does not open an incident.
ZeroDrop must be able to reach a webhook URL from the public internet. A URL that points to a private or local address is refused.
An email channel must be verified first. ZeroDrop does not send mail to an address that did not confirm.
Attach a channel to a check
Open a check and select the channels to notify. A check without channels still records incidents, but it sends no alerts. If you detach a channel, the alerts for that check stop. The channel itself is not deleted.
What triggers a notification
Notifications follow the state transitions in Consensus & status.
| Event | Meaning | Contains |
|---|---|---|
| Fire | A check went down. An incident opened. | The failing locations. |
| Reminder | An incident is still open. Repeat reminders must be on. | The failing locations. |
| Resolve | The check recovered. The incident closed. | Nothing. |
| Monitoring blind | Too few locations are live for a verdict. | The live locations. |
| Monitoring restored | Visibility returned. | Nothing. |
By default ZeroDrop sends one message when the state changes. To get reminders while an incident stays open, turn on repeat reminders in Settings. ZeroDrop then notifies the channels of the check on the cadence that you select, until the incident resolves.
The webhook payload
A webhook receives a JSON POST. The body identifies the check, the event type,
and the locations. An incident event contains the failing locations. A blind or
restore event contains the live locations.
Delivery is at least once, so ZeroDrop can retry a webhook. Make your handler
idempotent with the incident_id and event fields as the key. You can use a
webhook to send alerts to Slack, PagerDuty, an on-call tool, or your own
automation.
A test notification has the event type test and no incident id. Read the
event field to keep tests out of your automation.
How delivery works
ZeroDrop delivers alerts from a durable queue. A slow or unreachable endpoint does not block monitoring. If ZeroDrop restarts, no alert is lost. ZeroDrop retries a failed delivery with increasing back-off for several hours. If a channel continues to fail, the owner of the account receives a separate email. A broken webhook or address is never silent.
The incident log
ZeroDrop records every fire and resolve event. Each check therefore keeps a history of when it was down and for how long. Blind and restore events are also in the log, as visibility events and not as incidents.
The Incidents page lists all incidents, with the newest first. Open an incident to see its detail page. The detail page shows:
- The time when the incident opened and resolved, and the total duration.
- The cause and the failing locations that breached consensus.
- A notification timeline. It lists every delivery for the incident, with the channel, the time, the result, and the response. The response is an HTTP status code or an error message.
Use the timeline to learn whether an alert reached you. If an alert did not arrive, the timeline gives the reason.
From an incident you can open the check that it belongs to. The detail page of a check also lists its past incidents.
Suppress expected downtime
Some work, such as a deploy or maintenance, makes a check fail on purpose. Schedule a maintenance window for that period. ZeroDrop then holds back the notifications.