Skip to main content

Jobs

Merge Pilot does its work in the background, as jobs in a queue: evaluating a merge request, scanning it, opening a change approval in the tracker, recording a merge. Webhooks and actions in the UI only queue jobs, so a slow VCS or tracker never holds up a request. The Jobs page shows the queue, so you can see what is waiting, what is running and what failed.

Where to find it​

Insights → Jobs.

RoleWhat they can do
Every roleSee the jobs of their own company
Platform adminSee the jobs of every company; the page shows the selected company's
Company admin, platform adminRetry a job that is Failed or waiting to Retry. Other roles cannot: a job runs with the rights it was queued with (a bulk operation with those of the person who started it)

The page​

Counters​

Six cards count the company's jobs by status. Each has a tooltip:

CardCounts
TotalEvery job that is still kept (see How long jobs are kept)
PendingJobs waiting in the queue to be picked up
ProcessingJobs being worked on now, including jobs just picked up
CompletedJobs that finished successfully
FailedJobs that used up all their attempts
RetryJobs that will run again automatically after a failure, or that wait for a VCS rate limit to reset

Filters​

  • Search by job ID: matches part of the job ID or of the job type, ignoring case.
  • Filter by status: All statuses, Pending, Processing (with jobs just picked up), Completed, Failed or Retry.
  • Refresh reloads the list and the counters.

Columns​

ColumnShows
Job IDThe job's number, such as MP-42. Select it (or anywhere on the row) to open the job's details
TypeThe job type (see Job types)
StatusPending, Picked up (a worker just took it), Processing, Completed, Failed or Retry
MRThe merge request the job is about, when it is about one
BranchThe merge request's source branch
ResultFor jobs that evaluate a merge request: the policy that covered it, the merge gate it left, and the change approval key
RetryThe attempt and the number of attempts allowed, such as 2/3, once the job has failed at least once
CreatedWhen the job was queued

The list can be sorted by Job ID, Type, Status and Created; it is newest first by default.

Job details​

The details dialog shows:

  • the status and the Priority: Approval (applying a change approval decision), Change (evaluating merge requests and opening change approvals), Scan (secret, dependency and workflow scans), Background (CI tracking, analyses, state syncs) or History import (the import of watched repositories' past merge requests, below everything live);
  • Payload: what the job works on (Repo ID, MR, Source branch, Target branch, Action, and when present Merged by, Event type, Commit, Change type, Work items);
  • Result: Policy, Merge gate, Change approval key and Actions taken;
  • Error: the last error, the Attempt count and the time of the Next retry;
  • Worker: which server instance holds the job (Pod ID, Worker ID, Claimed at, Heartbeat), while it is running;
  • Timestamps: Created, Started, Completed and Duration.

A Retry button is shown for a failed or retrying job to company admins and platform admins.

How jobs run​

  • Kinds of work run side by side. Each kind of job has workers of its own: what a merge request's gate waits on (processing, evaluations, change approvals, merges, CI results); the secret, dependency and CI configuration scans; the container image and infrastructure scans; default-branch analyses; direct changes and head checks; and long background work (history imports, bulk operations, re-evaluations after a policy change). A burst of one kind, such as sixty merge requests' scans at once, never holds up another: a merge request opened during it is processed at once.
  • Within a kind, jobs with a higher priority are picked up first; among equals, the oldest first. A live event joining a background job (a re-evaluation after a policy change) makes it live.
  • One job per thing at a time. While a job for a merge request (or for another object) is waiting or running, a new event for the same thing does not queue a second job. A waiting job reads the latest state when it runs. For merge request evaluations, scans and webhook updates, an event that arrives while the job is running makes it run once more afterwards, so no event is missed. A scan whose merge request gets a newer commit while it runs is stopped (its result would be thrown away): it is listed as completed, and the newer commit's scan follows at once.
  • One read for many. A merge request's changes at a commit, and the repository files at a commit, are read from the VCS once and shared by that commit's scans (a commit's files never change); merge requests asking for the same container image at once share one scan. Re-scan reads the changes again.
  • Waiting for other work. A job that has to wait for another one (a push while the push before it is still being classified, a refresh of the inventory while an analysis of the project runs) is put back in line without using an attempt.
  • Time limit. A job may run for at most 10 minutes (a container image scan or an analysis 60 minutes, an infrastructure check 30 minutes); then it counts as a failed attempt.
  • Lost jobs. If the server instance running a job stops (a crash or a restart), the job is taken over after 2 minutes without a heartbeat and counts as a failed attempt. On a normal shutdown, running jobs get a short time to finish; a job interrupted by the shutdown is put back in the queue without using an attempt.
  • Rate limits. When the VCS answers that its API rate limit is used up, the job waits in Retry until the limit resets (at most 2 hours) without using an attempt.

Retries and failure​

A job that fails runs again automatically:

  • by default a job runs at most 3 times in total;
  • before each new attempt it waits, the wait doubling each time up to 5 minutes. With the defaults it waits 30 seconds before the second attempt and 1 minute before the third.

While it waits, the job's status is Retry, and the details show the last error and the time of the next attempt.

When the last attempt fails, the job is Failed. Then Merge Pilot:

  • writes job_failed to the audit log, with the job ID, type and error (on the merge request, when the job was about one);
  • sends the Background job failed notification to the company's notification routes that receive it, naming the kind of failure and linking to the jobs page (the error itself stays in the audit log), at most once an hour per connection and kind of failure. When the error shows that the VCS refuses Merge Pilot's credentials or its rate limit is used up, the connection's health records it and Connection problem is sent once instead (see Connection health);
  • for a scan, marks the scan as failed for that commit on the merge request (a re-scan starts it again) instead of leaving it pending;
  • for a change approval that could not be opened, shows the problem on the merge request. The periodic re-check of open merge requests tries again with a growing pause, and a creation that keeps failing for the same reason is reported only once.

A failed job is not lost work: the periodic re-check of open merge requests also recovers lost webhooks and queues missed work again.

Retrying by hand​

Select the retry button on the row, or Retry in the job's details. The job goes back to Pending with its attempts reset, and runs as soon as it is picked up. This works for jobs in Failed and Retry status. Fix the cause first (for example a revoked VCS or tracker token), or the job fails again. Each retry is recorded in the audit log as job_retried.

How long jobs are kept​

Completed and failed jobs are removed 7 days after they finished. Jobs that are waiting or running are kept until they end.

Job types​

The Type column shows one of these.

Merge requests​

TypeWhat it doesWhat starts it
mr_processMatches the merge request to a change policy, brings an existing change approval up to date with new commits, starts the scans the policy needs, checks risk and work items, then evaluates the merge requestA merge request is opened, reopened or updated in the VCS; an evaluation or a change approval job finds new commits not processed yet; a re-check finds a lost event; open merge requests evaluated again after a policy change (through mr_requeue) or when their repository moves to change management; the history import of a watched repository; a platform admin reprocessing a merge request in Debug console
mr_evaluateRuns the policy's controls again and updates the merge gate and the status comment; queues a change approval when one is dueA finished scan or CI run, a code review, a vote, an accepted risk, a change to a vulnerability exception
mr_requeueQueues the company's open merge requests of one VCS connection for evaluation again, a batch at a time, change-managed repositories first and below live eventsA change policy is created, edited, deleted or reordered, or a policy template is applied
mr_mergeRecords the merge and who merged, moves the change approval on, and flags a merge without approval, during a change freeze or against separation of dutiesThe VCS reports the merge; or a re-check or state sync finds a merge that was not reported
mr_closeRecords a merge request closed without merging and cancels its pending change approvalThe VCS reports the close; or a re-check or state sync finds it
mr_state_syncAsks the VCS for the state of one connection's open merge requests (at most 300 a run, those checked longest ago first) and queues mr_merge or mr_close for those merged or closed meanwhile. Background priority; it waits while the VCS's rate limit is down to the share kept for live eventsThe Merge requests page when it opens (at most every 5 minutes per connection) and its Sync with <GitLab or GitHub> button (at most every minute); one per connection at a time
mr_out_of_scopeReleases a tracked merge request that was moved to a target branch outside change management, as a close doesThe merge request's target branch is changed to one Merge Pilot does not gate
merge_group_checkReports the merge gate's decision on a GitHub merge queue's commit: it passes only when the pull request was approved at the commit it was queued withGitHub asks for checks on a merge queue group
pipeline_trackingStores the merge request's CI runs for the CI controlCI events from the VCS (pipelines, jobs, workflows, checks, statuses); a re-check of open merge requests

Scans​

TypeWhat it doesWhat starts it
secret_scanScans the merge request for secrets, then evaluates it againmr_process, when the policy's secret scan is on; a Re-scan on the merge request
sca_scanLooks up the changed dependency manifests for known vulnerabilities, licenses and package health, checks where new and changed dependencies come from, then evaluates the merge request againmr_process, when the policy runs the dependency scan, licenses, package health or dependency sources; a Re-scan
image_scanScans the container images the merge request starts referring to (Dockerfile FROM lines, Kubernetes manifests, Helm values, Compose files) for known vulnerabilities, then evaluates it againmr_process, when the policy runs the container image scan; Re-scan
iac_scanChecks the infrastructure files the merge request touches for misconfigurations it introduces, then evaluates it againmr_process, when the policy runs the infrastructure check; Re-scan
workflow_scanChecks the CI configuration the merge request adds or changes (GitHub Actions workflows, or GitLab CI configuration and the changed files it includes), then evaluates it againmr_process, on GitHub and GitLab, when the policy runs the CI configuration check; a Re-scan

Change approvals​

TypeWhat it doesWhat starts it
create_change_approvalOpens the change approval in the tracker, assigns it and links it to the merge requestAn evaluation finds the policy needs an approval; the re-check retries one that could not be opened
change_approval_assignGives an open change approval that nobody could be assigned to the first person who may now take it, where the policy or the company says soThe periodic re-check of open merge requests
change_approval_closeRecords a decision made in the tracker (approved, rejected, or an approval withdrawn by moving it out of its approving status) after checking who made it, then evaluates the merge requestThe tracker reports the change approval reached or left an approving status, or reached a rejecting one (Jira or the built-in tracker); a re-check finds a decision that was not reported
change_approval_changeHandles a change approval that was deleted or moved in the tracker: a lost change approval is replacedThe tracker reports the change; a re-check finds it

Repositories and projects​

TypeWhat it doesWhat starts it
project_backfillImports a watched repository's past merge requests, merged within the company's import period, and records them as watchedA repository starts being watched while Import past merge requests (days) is set; Import past merge requests on a watched project
open_mr_sweepLists a repository's open merge requests on the VCS (at most 2,000) and queues mr_process for those Merge Pilot does not follow yet, for example because their webhook was lostThe company's List open merge requests every (hours) interval (Repositories settings); a repository starts being watched or is put under change management
project_analysisAnalyses the dependencies, vulnerabilities and licenses of a whole projectRun analysis on a project's Security tab
vuln_refreshKeeps a project's vulnerability inventory up to date: after a merge into the default branch it reads the manifests that changed; on the advisory check it looks the known dependencies up again without reading the repositoryA merge into the default branch that touched dependency manifests (when the company analyses on merge); the company's advisory check interval; a direct change to a manifest on the default branch
bulk_operationWorks through a bulk operation a few repositories at a time (watch, move to change management, move back to watching, stop watching), then queues the next batchA bulk operation started on the Projects page
events_syncUpdates the webhook of every registered repository of the company, a batch at a timeThe company's Branch patterns or Receive push events setting changes
branch_changeFinds out whether a push's commits came through merge requests, and records a change made without one (a direct change) as the company asksThe VCS reports a push to a watched branch
branch_checkCompares each watched branch's head with the last one seen, to catch pushes whose webhook was missed, and checks the branch protection of a change-managed projectThe company's branch head check interval