Incidents & alerts
When consensus flips a check between healthy and down, ZeroDrop opens or closes an incident and notifies the channels attached to that check. This page covers setting up channels and what gets delivered.
Notification channels
A channel is a destination for alerts. You create channels once and attach them to as many checks as you like. Two types are supported:
| Type | Destination | Delivery |
|---|---|---|
| An email address | A formatted message per event. | |
| Webhook | An https:// URL |
A JSON POST your systems can act on. |
Creating and verifying a channel
- On the Channels page, choose New channel and pick email or webhook.
- Enter the address or URL and save.
- Verify the channel. ZeroDrop sends a test delivery so you can confirm it arrives before you rely on it. Email channels only ever receive alerts after they've been confirmed this way.
Webhook URLs often carry secret tokens, so ZeroDrop stores every channel's configuration encrypted at rest.
Attaching channels to a check
Open a check and choose which channels it should notify. A check with no attached channels still tracks incidents; it just stays silent. Detaching a channel stops that check's alerts without deleting the channel.
What triggers a notification
Notifications follow the state transitions described in Consensus & status:
| Event | Meaning | Carries |
|---|---|---|
| Fire | A check went down; an incident opened. | The failing locations. |
| Reminder | An incident is still open (only when repeat reminders are enabled). | The failing locations. |
| Resolve | The check recovered; the incident closed. | Nothing. |
| Monitoring blind | Too few locations are live to judge. | The live locations. |
| Monitoring restored | Visibility returned. | Nothing. |
By default you get one message when something changes, not a running commentary. If you'd rather be reminded while an incident drags on, turn on repeat reminders in Settings and ZeroDrop re-notifies the check's channels on the cadence you pick until it resolves.
The webhook payload
A webhook receives a JSON POST. The body identifies the check, the event type,
and the locations involved (failing locations for an incident, live locations
for a blind or restore event).
Delivery is at least once, so a webhook can be retried. Make your handler
idempotent by keying on the payload's incident_id and event. Use a webhook to
fan out into Slack, PagerDuty, an on-call tool, or your own automation.
How delivery works
Alerts are delivered from a durable queue, so a slow or briefly unreachable endpoint never blocks monitoring and nothing is lost if ZeroDrop restarts. A failed delivery is retried with growing back-off for several hours before it is given up on. If a channel keeps failing, the account owner gets an out-of-band email so a broken webhook or address is never silent.
The incident log
Every fire and resolve is recorded, so each check keeps a history of when it was down and for how long. Blind and restore events are logged too, but as visibility events rather than incidents.
The Incidents page lists them all, newest first. Open any incident for its detail page, which shows:
- When it opened and resolved, and the total duration.
- The cause and the failing locations that breached consensus.
- A notification timeline: every delivery for the incident, with the channel, the time sent, whether it succeeded or failed, and the response (an HTTP status code or the error). This is where you confirm an alert actually reached you, or find out why it didn't.
From the incident you can jump straight to the check it belongs to. A check's own detail page also lists its past incidents.
Suppressing expected downtime
Deploying, or doing maintenance you know will trip a check? Schedule a maintenance window so ZeroDrop holds back notifications for that period.