Skip to main content

Troubleshooting notifications

This page lists why a notification does not arrive, what each delivery error means, and why your endpoint may refuse a signed webhook. How notifications are set up is described in Notifications.

Where to look first​

  • The red banner on the company page's Notifications tab: deliveries that failed for good in the last 7 days, with Show failed deliveries. A route with failures shows N failed on its row. Insights → Service health lists the companies with failed deliveries.
  • The Delivery log at the bottom of the Notifications tab: every delivery of the last 7 days, to routes and as personal emails, with its status (pending, delivered, failed, skipped), the number of attempts and the last answer or error. Filter it by status, route (or Personal emails) and event. Each route's Recent deliveries shows its own. See Delivery log and failures. Company admins and platform admins see them.
  • Send test on a route sends a test notice through it, and Send test email checks the mail server.
  • The audit log: notification_failed for each delivery that failed for good, notification_personal_skipped when personal emails are not sent for want of a mail server.

What the log shows:

Nothing is sent​

An event becomes a delivery only when a route or a person takes it. Check, in this order:

  1. Did the event happen? Most events are also written to the audit log (for example mr_merged_without_approval, job_failed, connection_unhealthy). See Events for when each is sent.
  2. Was it sent already? The same event about the same thing in the same state is sent once (for example Merge request blocked once per head commit and set of failing controls, Background job failed once an hour per connection and kind of failure). See Sent once per on Events.
  3. Is it held back? Events about imported merge requests, and merge requests left watched after their repository moved to change management, are never sent. Under What is announced: events of watched repositories are sent only with Watched repositories on (off by default); Direct changes off stops direct change events; New vulnerabilities on default branches sets the lowest severity announced (High and above by default), or Never. See What holds events back.
  4. Does a route match? The route must be Active, take the event (or Every event), and have no path patterns or one that matches the project's full path. A route with path patterns never takes an event that is not about a registered project, such as a connection problem. See Path patterns.
  5. Did a route above it stop the event? Every matching route gets the event, unless a matching route above it has Stop here. See How an event is routed.
  6. Is the route configured? A dot on the route's row: amber Channel not configured means its webhook URL or recipients are missing. Send test shows whether it can send.
  7. Is the email route skipped? An email route is passed over while the company has no mail server; the other routes still get the event.

A company with no active route sends nothing to channels.

A delivery fails​

Each delivery is tried up to 5 times, with a growing wait (30 seconds, 1, 2 and 4 minutes). After the fifth failure, or at once when a mail server refuses a recipient for good, it is failed and not tried again by itself; it is written to the audit log as notification_failed and announced to the other routes as Notification not delivered. Send again sends it once more, right away. See Delivery log and failures.

Last error in the delivery logWhyWhat to do
slack webhook returned status N, teams webhook returned status NSlack answered something else than 200, or Teams something else than a 2xx. Usually the incoming webhook was removed or the channel archivedCreate a new incoming webhook in Slack or Teams, paste it in the route's Webhook URL, save, then Send again
the webhook's host "…" resolves to an address that is not allowedSlack and Teams routes may not post to private network, loopback, link-local or cloud metadata addressesUse the internet address Slack or Teams gave you
the webhook's host "…" could not be resolvedThe Slack or Teams address does not resolveCheck the URL
the company has no mail serverThe email route's company has no mail server any moreSet up SMTP configuration again (below)
<address>: RCPT TO: the mail server answered 550 …The mail server refused that recipient for good (an unknown mailbox, a refused domain). Not retried; the other recipients still got the email (delivered to N of M recipients)Fix or remove the address in the route's Recipients
the mail server could not be used: authentication: the mail server answered 535 …The mail server refused the sign-in. No recipient was tried, and none counts as refused. Not retriedCheck Username and Password (for Gmail and Google Workspace an app password); use Send test email (below)
the mail server could not be used: <step>: the mail server did not answer in time, … closed the connection, … could not be reached, … refused the connectionThe mail server was not reachable or too slow at that step (connect, greeting, EHLO, TLS, STARTTLS, authentication, MAIL FROM). RetriedCheck Host and Port, and that the server is up
the mail server could not be used: TLS: the mail server's TLS certificate is not trusted, … the TLS handshake with the mail server failedThe certificate is not valid for the Host, expired, or issued by your own CA; or the port does not speak TLSCheck TLS mode and Port (implicit TLS is usually 465, STARTTLS 587); add your CA in CA certificate
the mail server could not be used: STARTTLS: the mail server does not offer STARTTLSTLS mode is STARTTLS, but the server does not offer it. Merge Pilot never falls back to plain text. Not retriedChoose the mode the server supports
the mail server could not be used: MAIL FROM: the mail server answered 4xx …The mail server refused the sender for a while (for example 451 slow down). Retried; nobody counts as refusedUsually nothing. If it keeps failing, check that the server lets From address send
the mail server could not be used: settings: …The saved settings cannot be used (for example the CA certificate is not PEM, or the from address is not valid). Not retriedCorrect the SMTP configuration
the endpoint answered NYour webhook endpoint answered a 4xx or 5xxLook at your endpoint's logs; a 401 or 403 often means the signature check failed (below)
the endpoint answered N: redirects are not followed, use the final URLYour endpoint answered a 3xx. The signed request goes only to the route's URLPut the final URL in the route (for example https:// instead of http://, or with the trailing slash your server wants)
the endpoint's host name could not be resolvedThe host name does not resolveCheck the URL
the endpoint refused the connectionNothing listens on that address and portCheck the endpoint is running and reachable
the endpoint's TLS certificate is not trustedThe certificate is self-signed, expired, or not valid for the hostUse a certificate trusted for that host name
the endpoint did not answer in timeNo answer within 10 secondsAnswer at once with 2xx and do the work afterwards
the endpoint's address is not allowedWebhooks never go to link-local, multicast or cloud metadata addresses, and normally not to loopback addressesUse the endpoint's network address
the endpoint could not be reachedAnother network failureCheck the network between Merge Pilot and the endpoint
the route was removed or moved to another channelThe route changed after the delivery was queuedNothing: the delivery belonged to the old route

Errors never show the URL, because a webhook URL can hold a secret, and network failures are shown by their kind only.

Email​

The email route cannot be saved​

What you see: the route says Set up the mail server below first: an email route cannot be saved without it., and saving answers … set up the mail server (SMTP) before adding an email route.

Why: an email route needs the company's mail server.

What to do: fill in SMTP configuration on the same tab (at least Host, and usually Port, TLS mode, Auth mode, Username, Password and From address), then save the route. API tokens cannot change the mail server (API tokens cannot change smtp: use a signed-in session). See SMTP configuration.

Other refusals when saving a route: add at least one recipient, "…" is not an email address (use a plain address such as ops@example.com, not Ops <ops@example.com>).

The mail server cannot be removed​

Clearing Host is refused while the company has email routes (invalid input: email routes need the mail server: remove them before the mail server). Remove the email routes first, or in the same save.

The mail server cannot be saved​

The settings are refused when an auth mode is set with None (port 25) (except for localhost, 127.0.0.1 or ::1): the password would cross the network in the clear. Choose STARTTLS or Implicit TLS, or None (no auth) for a relay that needs no sign-in. Other refusals: a port outside 1 to 65535, a missing user name with authentication, a missing or invalid from address, a from name over 100 characters, a CA certificate that is not PEM. See SMTP configuration.

Send test email fails​

ResultWhat to do
sending failed: <host>:<port>: <step>: <reason>The step where it failed and the mail server's reply or the kind of failure (see the table above and Send test email). Check host, port, TLS mode, CA certificate and credentials
set the SMTP host firstFill in Host
A settings message (such as authentication without TLS)The settings are not allowed: see The mail server cannot be saved
your account does not have an email address configuredThe test goes to your own address: add an email to your account

The test uses the settings as the form shows them, and takes at most 20 seconds. When Password is empty, the saved password is used only if Host, Port, Username, TLS mode, Auth mode and CA certificate are unchanged; otherwise type the password again. A test that passes checks the mail server, not a route: use the route's Send test for that. Every test is in the audit log (smtp_test_sent).

The saved password disappeared​

Changing Host, Port, Username, TLS mode, Auth mode or CA certificate without typing the password again removes the saved password, so it is never sent to another server. Type it again and save.

A personal email does not arrive​

Check, in this order (see Personal notifications):

  1. Does the event go to people at all, and to this person? Only some events do, to the assignee, the approvers when nobody is assigned, the author, company admins or the person concerned. The author is found only through the VCS account they linked in My profile, and only when exactly one person linked it.
  2. Does the company have a mail server? Without one, no personal email is sent (the audit log says so once a day: notification_personal_skipped).
  3. Is it turned off? In My profile → Notifications, or with the email's unsubscribe link (notification_unsubscribed in the audit log). Connection recovered is off until turned on.
  4. Is the person active, in the company, with a valid email address? Otherwise the delivery is skipped, with the reason as its last error.
  5. Does an email route already send to that address? Then the person gets the route's email only.

An unsubscribe link that says This unsubscribe link is not valid or has expired is older than 90 days, was changed, or no longer valid; change the setting in My profile instead.

Your endpoint rejects the signature​

Merge Pilot signs every webhook delivery: X-MergePilot-Signature is sha256= followed by the hex HMAC-SHA256, keyed with the route's Signing secret, of the X-MergePilot-Timestamp header, a dot, and the raw body (see Verifying a request). A mismatch usually comes from one of these:

  • The body was changed before checking. Compute the HMAC over the raw bytes received, before any JSON parsing or re-serialisation.
  • The timestamp is left out. The signed message is <timestamp>.<body>, not the body alone.
  • The secret differs. The signing secret is write-only and never shown again. Type a new one in the route and in your endpoint, and save. Each attempt is signed with the secret saved at the time of that attempt.
  • The prefix or case. The header carries sha256= and lower-case hex; compare the whole value, in constant time.
  • The timestamp check is too strict. Every attempt, including Send again, is signed with the time it is sent. Allow a few minutes for clock differences.

Send again sends the same body and delivery ID with a new timestamp and signature, so it can be used to test a fix. Use X-MergePilot-Delivery to ignore a delivery you already handled.