Troubleshooting notifications
This page lists why a notification does not arrive, what each delivery error means, and why your endpoint may refuse a signed webhook. How notifications are set up is described in Notifications.
Where to look first
- The red banner on the company page's Notifications tab: deliveries that failed for good in the last 7 days, with Show failed deliveries. A route with failures shows N failed on its row. Insights → Service health lists the companies with failed deliveries.
- The Delivery log at the bottom of the Notifications tab: every delivery of the last 7 days, to routes and as personal emails, with its status (
pending,delivered,failed,skipped), the number of attempts and the last answer or error. Filter it by status, route (or Personal emails) and event. Each route's Recent deliveries shows its own. See Delivery log and failures. Company admins and platform admins see them. - Send test on a route sends a test notice through it, and Send test email checks the mail server.
- The audit log:
notification_failedfor each delivery that failed for good,notification_personal_skippedwhen personal emails are not sent for want of a mail server.
What the log shows:
- No delivery listed for an event: the event never reached this route or person (Nothing is sent).
- A delivery listed as
pendingorfailed: sending failed (A delivery fails). - A personal email listed as
skipped: see A personal email does not arrive.
Nothing is sent
An event becomes a delivery only when a route or a person takes it. Check, in this order:
- Did the event happen? Most events are also written to the audit log (for example
mr_merged_without_approval,job_failed,connection_unhealthy). See Events for when each is sent. - Was it sent already? The same event about the same thing in the same state is sent once (for example Merge request blocked once per head commit and set of failing controls, Background job failed once an hour per connection and kind of failure). See Sent once per on Events.
- Is it held back? Events about imported merge requests, and merge requests left watched after their repository moved to change management, are never sent. Under What is announced: events of watched repositories are sent only with Watched repositories on (off by default); Direct changes off stops direct change events; New vulnerabilities on default branches sets the lowest severity announced (High and above by default), or Never. See What holds events back.
- Does a route match? The route must be Active, take the event (or Every event), and have no path patterns or one that matches the project's full path. A route with path patterns never takes an event that is not about a registered project, such as a connection problem. See Path patterns.
- Did a route above it stop the event? Every matching route gets the event, unless a matching route above it has Stop here. See How an event is routed.
- Is the route configured? A dot on the route's row: amber Channel not configured means its webhook URL or recipients are missing. Send test shows whether it can send.
- Is the email route skipped? An email route is passed over while the company has no mail server; the other routes still get the event.
A company with no active route sends nothing to channels.
A delivery fails
Each delivery is tried up to 5 times, with a growing wait (30 seconds, 1, 2 and 4 minutes). After the fifth failure, or at once when a mail server refuses a recipient for good, it is failed and not tried again by itself; it is written to the audit log as notification_failed and announced to the other routes as Notification not delivered. Send again sends it once more, right away. See Delivery log and failures.
| Last error in the delivery log | Why | What to do |
|---|---|---|
| slack webhook returned status N, teams webhook returned status N | Slack answered something else than 200, or Teams something else than a 2xx. Usually the incoming webhook was removed or the channel archived | Create a new incoming webhook in Slack or Teams, paste it in the route's Webhook URL, save, then Send again |
| the webhook's host "…" resolves to an address that is not allowed | Slack and Teams routes may not post to private network, loopback, link-local or cloud metadata addresses | Use the internet address Slack or Teams gave you |
| the webhook's host "…" could not be resolved | The Slack or Teams address does not resolve | Check the URL |
| the company has no mail server | The email route's company has no mail server any more | Set up SMTP configuration again (below) |
| <address>: RCPT TO: the mail server answered 550 … | The mail server refused that recipient for good (an unknown mailbox, a refused domain). Not retried; the other recipients still got the email (delivered to N of M recipients) | Fix or remove the address in the route's Recipients |
| the mail server could not be used: authentication: the mail server answered 535 … | The mail server refused the sign-in. No recipient was tried, and none counts as refused. Not retried | Check Username and Password (for Gmail and Google Workspace an app password); use Send test email (below) |
| the mail server could not be used: <step>: the mail server did not answer in time, … closed the connection, … could not be reached, … refused the connection | The mail server was not reachable or too slow at that step (connect, greeting, EHLO, TLS, STARTTLS, authentication, MAIL FROM). Retried | Check Host and Port, and that the server is up |
| the mail server could not be used: TLS: the mail server's TLS certificate is not trusted, … the TLS handshake with the mail server failed | The certificate is not valid for the Host, expired, or issued by your own CA; or the port does not speak TLS | Check TLS mode and Port (implicit TLS is usually 465, STARTTLS 587); add your CA in CA certificate |
| the mail server could not be used: STARTTLS: the mail server does not offer STARTTLS | TLS mode is STARTTLS, but the server does not offer it. Merge Pilot never falls back to plain text. Not retried | Choose the mode the server supports |
| the mail server could not be used: MAIL FROM: the mail server answered 4xx … | The mail server refused the sender for a while (for example 451 slow down). Retried; nobody counts as refused | Usually nothing. If it keeps failing, check that the server lets From address send |
| the mail server could not be used: settings: … | The saved settings cannot be used (for example the CA certificate is not PEM, or the from address is not valid). Not retried | Correct the SMTP configuration |
| the endpoint answered N | Your webhook endpoint answered a 4xx or 5xx | Look at your endpoint's logs; a 401 or 403 often means the signature check failed (below) |
| the endpoint answered N: redirects are not followed, use the final URL | Your endpoint answered a 3xx. The signed request goes only to the route's URL | Put the final URL in the route (for example https:// instead of http://, or with the trailing slash your server wants) |
| the endpoint's host name could not be resolved | The host name does not resolve | Check the URL |
| the endpoint refused the connection | Nothing listens on that address and port | Check the endpoint is running and reachable |
| the endpoint's TLS certificate is not trusted | The certificate is self-signed, expired, or not valid for the host | Use a certificate trusted for that host name |
| the endpoint did not answer in time | No answer within 10 seconds | Answer at once with 2xx and do the work afterwards |
| the endpoint's address is not allowed | Webhooks never go to link-local, multicast or cloud metadata addresses, and normally not to loopback addresses | Use the endpoint's network address |
| the endpoint could not be reached | Another network failure | Check the network between Merge Pilot and the endpoint |
| the route was removed or moved to another channel | The route changed after the delivery was queued | Nothing: the delivery belonged to the old route |
Errors never show the URL, because a webhook URL can hold a secret, and network failures are shown by their kind only.
Email
The email route cannot be saved
What you see: the route says Set up the mail server below first: an email route cannot be saved without it., and saving answers … set up the mail server (SMTP) before adding an email route.
Why: an email route needs the company's mail server.
What to do: fill in SMTP configuration on the same tab (at least Host, and usually Port, TLS mode, Auth mode, Username, Password and From address), then save the route. API tokens cannot change the mail server (API tokens cannot change smtp: use a signed-in session). See SMTP configuration.
Other refusals when saving a route: add at least one recipient, "…" is not an email address (use a plain address such as ops@example.com, not Ops <ops@example.com>).
The mail server cannot be removed
Clearing Host is refused while the company has email routes (invalid input: email routes need the mail server: remove them before the mail server). Remove the email routes first, or in the same save.
The mail server cannot be saved
The settings are refused when an auth mode is set with None (port 25) (except for localhost, 127.0.0.1 or ::1): the password would cross the network in the clear. Choose STARTTLS or Implicit TLS, or None (no auth) for a relay that needs no sign-in. Other refusals: a port outside 1 to 65535, a missing user name with authentication, a missing or invalid from address, a from name over 100 characters, a CA certificate that is not PEM. See SMTP configuration.
Send test email fails
| Result | What to do |
|---|---|
| sending failed: <host>:<port>: <step>: <reason> | The step where it failed and the mail server's reply or the kind of failure (see the table above and Send test email). Check host, port, TLS mode, CA certificate and credentials |
| set the SMTP host first | Fill in Host |
| A settings message (such as authentication without TLS) | The settings are not allowed: see The mail server cannot be saved |
| your account does not have an email address configured | The test goes to your own address: add an email to your account |
The test uses the settings as the form shows them, and takes at most 20 seconds. When Password is empty, the saved password is used only if Host, Port, Username, TLS mode, Auth mode and CA certificate are unchanged; otherwise type the password again. A test that passes checks the mail server, not a route: use the route's Send test for that. Every test is in the audit log (smtp_test_sent).
The saved password disappeared
Changing Host, Port, Username, TLS mode, Auth mode or CA certificate without typing the password again removes the saved password, so it is never sent to another server. Type it again and save.
A personal email does not arrive
Check, in this order (see Personal notifications):
- Does the event go to people at all, and to this person? Only some events do, to the assignee, the approvers when nobody is assigned, the author, company admins or the person concerned. The author is found only through the VCS account they linked in My profile, and only when exactly one person linked it.
- Does the company have a mail server? Without one, no personal email is sent (the audit log says so once a day:
notification_personal_skipped). - Is it turned off? In My profile → Notifications, or with the email's unsubscribe link (
notification_unsubscribedin the audit log). Connection recovered is off until turned on. - Is the person active, in the company, with a valid email address? Otherwise the delivery is
skipped, with the reason as its last error. - Does an email route already send to that address? Then the person gets the route's email only.
An unsubscribe link that says This unsubscribe link is not valid or has expired is older than 90 days, was changed, or no longer valid; change the setting in My profile instead.
Your endpoint rejects the signature
Merge Pilot signs every webhook delivery: X-MergePilot-Signature is sha256= followed by the hex HMAC-SHA256, keyed with the route's Signing secret, of the X-MergePilot-Timestamp header, a dot, and the raw body (see Verifying a request). A mismatch usually comes from one of these:
- The body was changed before checking. Compute the HMAC over the raw bytes received, before any JSON parsing or re-serialisation.
- The timestamp is left out. The signed message is
<timestamp>.<body>, not the body alone. - The secret differs. The signing secret is write-only and never shown again. Type a new one in the route and in your endpoint, and save. Each attempt is signed with the secret saved at the time of that attempt.
- The prefix or case. The header carries
sha256=and lower-case hex; compare the whole value, in constant time. - The timestamp check is too strict. Every attempt, including Send again, is signed with the time it is sent. Allow a few minutes for clock differences.
Send again sends the same body and delivery ID with a new timestamp and signature, so it can be used to test a fix. Use X-MergePilot-Delivery to ignore a delivery you already handled.