Jobs
Merge Pilot does its work in the background, as jobs in a queue: evaluating a merge request, scanning it, opening a change approval in the tracker, recording a merge. Webhooks and actions in the UI only queue jobs, so a slow VCS or tracker never holds up a request. The Jobs page shows the queue, so you can see what is waiting, what is running and what failed.
Where to find it
Insights → Jobs.
| Role | What they can do |
|---|---|
| Every role | See the jobs of their own company |
| Platform admin | See the jobs of every company; the page shows the selected company's |
| Company admin, platform admin | Retry a job that is Failed or waiting to Retry. Other roles cannot: a job runs with the rights it was queued with (a bulk operation with those of the person who started it) |
The page
Counters
Six cards count the company's jobs by status. Each has a tooltip:
| Card | Counts |
|---|---|
| Total | Every job that is still kept (see How long jobs are kept) |
| Pending | Jobs waiting in the queue to be picked up |
| Processing | Jobs being worked on now, including jobs just picked up |
| Completed | Jobs that finished successfully |
| Failed | Jobs that used up all their attempts |
| Retry | Jobs that will run again automatically after a failure, or that wait for a VCS rate limit to reset |
Filters
- Search by job ID: matches part of the job ID or of the job type, ignoring case.
- Filter by status: All statuses, Pending, Processing (with jobs just picked up), Completed, Failed or Retry.
- Refresh reloads the list and the counters.
Columns
| Column | Shows |
|---|---|
| Job ID | The job's number, such as MP-42. Select it (or anywhere on the row) to open the job's details |
| Type | The job type (see Job types) |
| Status | Pending, Picked up (a worker just took it), Processing, Completed, Failed or Retry |
| MR | The merge request the job is about, when it is about one |
| Branch | The merge request's source branch |
| Result | For jobs that evaluate a merge request: the policy that covered it, the merge gate it left, and the change approval key |
| Retry | The attempt and the number of attempts allowed, such as 2/3, once the job has failed at least once |
| Created | When the job was queued |
The list can be sorted by Job ID, Type, Status and Created; it is newest first by default.
Job details
The details dialog shows:
- the status and the Priority: Approval (applying a change approval decision), Change (evaluating merge requests and opening change approvals), Scan (secret, dependency and workflow scans), Background (CI tracking, analyses, state syncs) or History import (the import of watched repositories' past merge requests, below everything live);
- Payload: what the job works on (Repo ID, MR, Source branch, Target branch, Action, and when present Merged by, Event type, Commit, Change type, Work items);
- Result: Policy, Merge gate, Change approval key and Actions taken;
- Error: the last error, the Attempt count and the time of the Next retry;
- Worker: which server instance holds the job (Pod ID, Worker ID, Claimed at, Heartbeat), while it is running;
- Timestamps: Created, Started, Completed and Duration.
A Retry button is shown for a failed or retrying job to company admins and platform admins.
How jobs run
- Kinds of work run side by side. Each kind of job has workers of its own: what a merge request's gate waits on (processing, evaluations, change approvals, merges, CI results); the secret, dependency and CI configuration scans; the container image and infrastructure scans; default-branch analyses; direct changes and head checks; and long background work (history imports, bulk operations, re-evaluations after a policy change). A burst of one kind, such as sixty merge requests' scans at once, never holds up another: a merge request opened during it is processed at once.
- Within a kind, jobs with a higher priority are picked up first; among equals, the oldest first. A live event joining a background job (a re-evaluation after a policy change) makes it live.
- One job per thing at a time. While a job for a merge request (or for another object) is waiting or running, a new event for the same thing does not queue a second job. A waiting job reads the latest state when it runs. For merge request evaluations, scans and webhook updates, an event that arrives while the job is running makes it run once more afterwards, so no event is missed. A scan whose merge request gets a newer commit while it runs is stopped (its result would be thrown away): it is listed as completed, and the newer commit's scan follows at once.
- One read for many. A merge request's changes at a commit, and the repository files at a commit, are read from the VCS once and shared by that commit's scans (a commit's files never change); merge requests asking for the same container image at once share one scan. Re-scan reads the changes again.
- Waiting for other work. A job that has to wait for another one (a push while the push before it is still being classified, a refresh of the inventory while an analysis of the project runs) is put back in line without using an attempt.
- Time limit. A job may run for at most 10 minutes (a container image scan or an analysis 60 minutes, an infrastructure check 30 minutes); then it counts as a failed attempt.
- Lost jobs. If the server instance running a job stops (a crash or a restart), the job is taken over after 2 minutes without a heartbeat and counts as a failed attempt. On a normal shutdown, running jobs get a short time to finish; a job interrupted by the shutdown is put back in the queue without using an attempt.
- Rate limits. When the VCS answers that its API rate limit is used up, the job waits in Retry until the limit resets (at most 2 hours) without using an attempt.
Retries and failure
A job that fails runs again automatically:
- by default a job runs at most 3 times in total;
- before each new attempt it waits, the wait doubling each time up to 5 minutes. With the defaults it waits 30 seconds before the second attempt and 1 minute before the third.
While it waits, the job's status is Retry, and the details show the last error and the time of the next attempt.
When the last attempt fails, the job is Failed. Then Merge Pilot:
- writes
job_failedto the audit log, with the job ID, type and error (on the merge request, when the job was about one); - sends the Background job failed notification to the company's notification routes that receive it, naming the kind of failure and linking to the jobs page (the error itself stays in the audit log), at most once an hour per connection and kind of failure. When the error shows that the VCS refuses Merge Pilot's credentials or its rate limit is used up, the connection's health records it and Connection problem is sent once instead (see Connection health);
- for a scan, marks the scan as failed for that commit on the merge request (a re-scan starts it again) instead of leaving it pending;
- for a change approval that could not be opened, shows the problem on the merge request. The periodic re-check of open merge requests tries again with a growing pause, and a creation that keeps failing for the same reason is reported only once.
A failed job is not lost work: the periodic re-check of open merge requests also recovers lost webhooks and queues missed work again.
Retrying by hand
Select the retry button on the row, or Retry in the job's details. The job goes back to Pending with its attempts reset, and runs as soon as it is picked up. This works for jobs in Failed and Retry status. Fix the cause first (for example a revoked VCS or tracker token), or the job fails again. Each retry is recorded in the audit log as job_retried.
How long jobs are kept
Completed and failed jobs are removed 7 days after they finished. Jobs that are waiting or running are kept until they end.
Job types
The Type column shows one of these.
Merge requests
| Type | What it does | What starts it |
|---|---|---|
mr_process | Matches the merge request to a change policy, brings an existing change approval up to date with new commits, starts the scans the policy needs, checks risk and work items, then evaluates the merge request | A merge request is opened, reopened or updated in the VCS; an evaluation or a change approval job finds new commits not processed yet; a re-check finds a lost event; open merge requests evaluated again after a policy change (through mr_requeue) or when their repository moves to change management; the history import of a watched repository; a platform admin reprocessing a merge request in Debug console |
mr_evaluate | Runs the policy's controls again and updates the merge gate and the status comment; queues a change approval when one is due | A finished scan or CI run, a code review, a vote, an accepted risk, a change to a vulnerability exception |
mr_requeue | Queues the company's open merge requests of one VCS connection for evaluation again, a batch at a time, change-managed repositories first and below live events | A change policy is created, edited, deleted or reordered, or a policy template is applied |
mr_merge | Records the merge and who merged, moves the change approval on, and flags a merge without approval, during a change freeze or against separation of duties | The VCS reports the merge; or a re-check or state sync finds a merge that was not reported |
mr_close | Records a merge request closed without merging and cancels its pending change approval | The VCS reports the close; or a re-check or state sync finds it |
mr_state_sync | Asks the VCS for the state of one connection's open merge requests (at most 300 a run, those checked longest ago first) and queues mr_merge or mr_close for those merged or closed meanwhile. Background priority; it waits while the VCS's rate limit is down to the share kept for live events | The Merge requests page when it opens (at most every 5 minutes per connection) and its Sync with <GitLab or GitHub> button (at most every minute); one per connection at a time |
mr_out_of_scope | Releases a tracked merge request that was moved to a target branch outside change management, as a close does | The merge request's target branch is changed to one Merge Pilot does not gate |
merge_group_check | Reports the merge gate's decision on a GitHub merge queue's commit: it passes only when the pull request was approved at the commit it was queued with | GitHub asks for checks on a merge queue group |
pipeline_tracking | Stores the merge request's CI runs for the CI control | CI events from the VCS (pipelines, jobs, workflows, checks, statuses); a re-check of open merge requests |
Scans
| Type | What it does | What starts it |
|---|---|---|
secret_scan | Scans the merge request for secrets, then evaluates it again | mr_process, when the policy's secret scan is on; a Re-scan on the merge request |
sca_scan | Looks up the changed dependency manifests for known vulnerabilities, licenses and package health, checks where new and changed dependencies come from, then evaluates the merge request again | mr_process, when the policy runs the dependency scan, licenses, package health or dependency sources; a Re-scan |
image_scan | Scans the container images the merge request starts referring to (Dockerfile FROM lines, Kubernetes manifests, Helm values, Compose files) for known vulnerabilities, then evaluates it again | mr_process, when the policy runs the container image scan; Re-scan |
iac_scan | Checks the infrastructure files the merge request touches for misconfigurations it introduces, then evaluates it again | mr_process, when the policy runs the infrastructure check; Re-scan |
workflow_scan | Checks the CI configuration the merge request adds or changes (GitHub Actions workflows, or GitLab CI configuration and the changed files it includes), then evaluates it again | mr_process, on GitHub and GitLab, when the policy runs the CI configuration check; a Re-scan |
Change approvals
| Type | What it does | What starts it |
|---|---|---|
create_change_approval | Opens the change approval in the tracker, assigns it and links it to the merge request | An evaluation finds the policy needs an approval; the re-check retries one that could not be opened |
change_approval_assign | Gives an open change approval that nobody could be assigned to the first person who may now take it, where the policy or the company says so | The periodic re-check of open merge requests |
change_approval_close | Records a decision made in the tracker (approved, rejected, or an approval withdrawn by moving it out of its approving status) after checking who made it, then evaluates the merge request | The tracker reports the change approval reached or left an approving status, or reached a rejecting one (Jira or the built-in tracker); a re-check finds a decision that was not reported |
change_approval_change | Handles a change approval that was deleted or moved in the tracker: a lost change approval is replaced | The tracker reports the change; a re-check finds it |
Repositories and projects
| Type | What it does | What starts it |
|---|---|---|
project_backfill | Imports a watched repository's past merge requests, merged within the company's import period, and records them as watched | A repository starts being watched while Import past merge requests (days) is set; Import past merge requests on a watched project |
open_mr_sweep | Lists a repository's open merge requests on the VCS (at most 2,000) and queues mr_process for those Merge Pilot does not follow yet, for example because their webhook was lost | The company's List open merge requests every (hours) interval (Repositories settings); a repository starts being watched or is put under change management |
project_analysis | Analyses the dependencies, vulnerabilities and licenses of a whole project | Run analysis on a project's Security tab |
vuln_refresh | Keeps a project's vulnerability inventory up to date: after a merge into the default branch it reads the manifests that changed; on the advisory check it looks the known dependencies up again without reading the repository | A merge into the default branch that touched dependency manifests (when the company analyses on merge); the company's advisory check interval; a direct change to a manifest on the default branch |
bulk_operation | Works through a bulk operation a few repositories at a time (watch, move to change management, move back to watching, stop watching), then queues the next batch | A bulk operation started on the Projects page |
events_sync | Updates the webhook of every registered repository of the company, a batch at a time | The company's Branch patterns or Receive push events setting changes |
branch_change | Finds out whether a push's commits came through merge requests, and records a change made without one (a direct change) as the company asks | The VCS reports a push to a watched branch |
branch_check | Compares each watched branch's head with the last one seen, to catch pushes whose webhook was missed, and checks the branch protection of a change-managed project | The company's branch head check interval |
Related pages
- Audit log:
job_failedentries. - Core concepts: change policies, change approvals and the merge gate.