Delivery log and failures
Every notification Merge Pilot sends, to a route or as a personal email, is first stored as a delivery, then sent, retried when sending fails, and kept with each attempt. This page shows where to see them and what happens when one cannot be delivered.
Company page → Notifications. Company admins of the company and platform admins see the deliveries; Service health shows failures too.
How a delivery is sent
- Queued. One delivery per route that gets the event, and one per person for personal emails, stored at once, whatever happens next.
- Sent within a few seconds. One company's slow mail server or endpoint does not hold up other companies' notifications.
- Retried when an attempt fails, with a growing wait, at most 5 attempts in all:
| Attempt | Waits before it |
|---|---|
| 1 | None |
| 2 | 30 seconds |
| 3 | 1 minute |
| 4 | 2 minutes |
| 5 | 4 minutes |
The wait doubles each time, up to 5 minutes. After the fifth failed attempt, the delivery is failed and not tried again by itself.
A mail server's permanent refusal of a recipient (a 5xx reply, such as an unknown mailbox) is not retried: the delivery fails at once for that recipient. A temporary one (a 4xx reply, no answer, a broken connection) is retried. When some recipients of an email route accepted the message and others failed temporarily or were not reached, the retry goes only to those (see Email). A problem with the mail server itself, such as a refused sign-in, refuses no recipient (see Mail server problems are not refusals).
Each attempt uses the route as it is at that moment: its current URL, recipients, signing secret and mail server. Turning a route off stops new deliveries, and deliveries still waiting for it are skipped (the route was turned off). A route that was removed or moved to another channel gets nothing either: its waiting deliveries are skipped (the route was removed or moved to another channel). A skipped delivery is not retried and not announced as Notification not delivered. Route tests are the exception: they are sent to an inactive route too.
Statuses
| Status | Meaning |
|---|---|
pending | Waiting for its first or next attempt |
delivered | Accepted: by Slack (200), Teams or a webhook endpoint (any 2xx), or by the mail server for every recipient |
failed | Failed for good: 5 attempts, or a permanent refusal |
skipped | Not sent and never will be. The reason is the attempt's error. A route's delivery: the route was turned off, or the route was removed or moved to another channel (after it was queued). A personal email: the person no longer exists, the person was deactivated or left the company, the person has no valid email address, the person's role no longer gets these emails (for example an admin event to someone who is no longer a company admin), the person turned these emails off, or the company has no mail server |
Deliveries are kept for 7 days, whatever their status, then removed.
Recent deliveries of a route
A saved route of any channel shows Recent deliveries below its settings. Opening it lists the route's latest 50 deliveries, newest first (Nothing sent yet. when there are none, Could not load the recent deliveries. when they could not be read).
Delivery log
The Delivery log card at the bottom of the Notifications tab: Every notification sent in the last 7 days, to the company's routes and as personal emails, newest first. Send a failed one again once its route is fixed.
| Filter | Values |
|---|---|
| Status | Any status, Failed, Pending, Delivered, Skipped |
| Route | Any route, Personal emails, or one route |
| Event | Any event, test (route tests), or one event |
Refresh reads the log again; 20 deliveries per page, with Previous and Next.
Each delivery, in Recent deliveries and the log, shows:
- its status;
- the event;
- in the log, the route's name ((removed route) when the route is gone), or Personal email to <user> (<address>);
- when it was queued;
- N attempts, with , last answer <status> when a webhook endpoint answered, or , last error: <error> otherwise;
- for an email route, Delivered to N recipients: and Refused for good by N recipients: with the addresses;
- Send again, except while the delivery is
pending.
Errors say what went wrong without exposing your network: a mail server's reply is shown as it is, a network failure by its kind only (the endpoint refused the connection, the mail server did not answer in time …), and never the webhook URL, which can hold a secret. The meaning of each error is on Troubleshooting notifications.
Send again
Send again sends a delivered, failed or skipped delivery once more, right away (Sending again): the same message (for a webhook, the same body and delivery ID, with a new timestamp and signature; for an email route, to every recipient again). Its earlier attempts are kept and count towards the 5: a delivery that already had 5 attempts gets one more, and is failed again if that one fails. It is written to the audit log as notification_redelivered. The failed-deliveries banner and the routes' N failed badges are read again a few seconds later.
A pending delivery has no Send again: it is being sent, or it will be tried again by itself. A route's delivery whose route is now turned off, removed or moved to another channel has Send again disabled, as it would only be skipped: The route is turned off: turn it on to send this again. or The route was removed or moved to another channel: this cannot be sent again. (A route test can still be sent again to a route that is turned off.) Through the API, sending a pending delivery again moves its next attempt to now and keeps the recipients it already reached or that refused it; a delivery being sent at that moment is refused (the delivery is being sent right now: send it again when this attempt has finished).
Send test
Each saved route has Send test: it queues a test notice (Test notification: This is a test of the notification route <name>, sent by <you>. If you see it, the route works.) to that route, whether it is active or not and whatever events it takes (Test queued: its result shows under Recent deliveries). It is sent and retried like any delivery, as the event test, and its result shows under the route's Recent deliveries, which open and refresh by themselves for a few seconds, as do the failure banner and badges. A route not saved yet says Save the company to send a test to this route.
The test uses the route as saved. While the route has unsaved edits, Send test is disabled and says Save the company first: Send test uses the saved route. An email route is also sent with the saved mail server: while the SMTP configuration has unsaved edits, an email route's Send test is disabled and says Save the company first: Send test uses the saved mail server settings. Save the company, then test.
One route can be tested once every 30 seconds; a second test sooner is refused (the route was tested moments ago: try again in N seconds, the wait rounded up to whole seconds, as the API's Retry-After header gives it). Other routes can be tested meanwhile. A test can be sent only from a signed-in session, not with an API token.
A test is refused when the route cannot send: an email route without the mail server or a recipient, any other route without its URL. It is written to the audit log as notification_route_tested. A failed test is not announced as Notification not delivered.
Send test email under SMTP configuration is a different check: it tests the mail server, not a route (see Send test email).
When a delivery fails for good
A delivery that fails for good is made visible in three ways:
- Audit log:
notification_failed, naming the route (or the person, for a personal email), the event and the kind of failure (see Audit log). - The event Notification not delivered (
notification_failed) is sent to the company's other routes that take it, and emailed to company admins (see Events). It never goes to the route that failed, which would fail too, and the failing route's Stop here holds nothing back. It is sent at most once an hour per route and kind of failure. A failed route test, a failed personal email, or a failed Notification not delivered itself does not raise it. - Not even queued. Before a delivery is sent it is queued (stored for sending). When that fails, Merge Pilot tries again twice within a second or so. A route or person it still could not be queued for has no delivery in the log; the audit log records
notification_failedwith the kind of failure could not be queued, and nothing is announced. When nothing of an event could be queued, the event is not taken as sent: it is sent the next time it is found (a change approval's deadline, for example, at the watcher's next run). - Banner and counts. The Notifications tab shows a red banner, N notifications could not be delivered in the last 7 days. Each failure is also in the audit log (notification_failed)., with Show failed deliveries, which sets the delivery log to Failed, any route and any event, from the first page, and scrolls to it (each click does this again and reads the log again, even when Failed is already chosen). Each route with failures shows a red N failed badge on its row.
The kind of failure in the event and the audit log says what happened without naming addresses or server replies: the mail server refused 1 recipient for good, the mail server did not accept the message, even after retries, the mail server refused its settings or sign-in, the mail server could not be used, even after retries, the endpoint answered 500, or the attempt's error.
Service health
Insights → Service health has a Notification deliveries card listing each company whose deliveries failed for good in the last 7 days (<company>: N notifications could not be delivered in the last 7 days), with Open the delivery log for company admins and platform admins. People of a company see their own company only.
API
| Request | Who | Does |
|---|---|---|
GET /api/v1/companies/{id}/notification-deliveries | Company admins, platform admins | A page of the delivery log, newest first. Filters: status (pending, delivered, failed, skipped), route (a route ID), event (an event or test), channel (slack, teams, email, webhook, or person for personal emails), user (a person's ID: their personal emails); page, page_size. 400 for another value |
GET /api/v1/companies/{id}/notification-deliveries/failures | Company admins, platform admins | total failed deliveries since since (7 days), and by_route (route_id, count; route_id empty for personal emails) |
POST /api/v1/companies/{id}/notification-deliveries/{delivery id}/redeliver | Company admins, platform admins | Send again. 404 when the delivery is not the company's, 409 while it is being sent |
POST /api/v1/companies/{id}/notification-routes/{route id}/test | Company admins, platform admins | Send test with the saved route; answers the new delivery_id. Signed-in session only (403 with an API token). 404 when the route is not saved, 400 when it cannot send, 429 with Retry-After when the route was tested less than 30 seconds ago |
GET /api/v1/companies/{id}/notification-routes/inactive-recipients?email=… | Company admins, platform admins | Of the addresses given (repeat email, at most 200), those of deactivated people of the company |
GET /api/v1/health-checks/notifications | Signed-in users | The companies with failed deliveries in the last 7 days: company_id, company_slug, company_name, failed |
Each delivery has id, company_id, route_id (empty for a personal email), channel (slack, teams, email, webhook, person), user_id, username and to (personal emails), event, body (webhooks: the JSON sent), notice (Slack, Teams, email: title, message, color, company_name, url, reason, settings_url, at), status, attempts (each with at, status_code when a webhook endpoint answered, error when the attempt failed, and duration_ms), created_at, next_attempt_at while it waits, delivered_at once delivered, and for email routes delivered_to and refused_to.
Related pages
- Routes: which events go where.
- Channels: how each channel sends.
- Troubleshooting notifications: every error and what to do.
- Audit log:
notification_failed,notification_redelivered,notification_route_tested.
Video tutorial
Short videos of this feature working on real GitLab and Jira projects. English audio, English and Turkish subtitles. All video tutorials →