Bulk operations
A bulk operation watches, moves or stops many repositories of one VCS connection at once, in the background. It is how a company with hundreds or thousands of repositories starts watching them, and later moves them to change management, without doing it one repository at a time. Each repository is worked on and reported on its own: one that fails does not stop the others.
Watching a single repository from its page also runs as a bulk operation of one.
Kinds of operation
| Kind (label in the UI) | API kind | What it does to each repository | Started from | Connection permission needed (people who are not company admins) |
|---|---|---|---|---|
| Watch N repositories | watch | Watches a repository not in Merge Pilot yet (see Watching repositories) | Projects → Watch repositories; a repository's page → Watch | Set up change management |
| Add N repositories to change management | add_gated | Puts a repository not in Merge Pilot yet straight under change management, with the chosen setup steps | Projects → Add to change management | Set up change management |
| Move N repositories to change management | to_gated | Moves a watched project to change management, with the chosen setup steps and choice for open merge requests | Registered projects → choose → Move N to change management | Set up change management |
| Switch N repositories to watching | to_watching | Moves a project under change management back to watching | Registered projects → choose → Switch N to watching | Remove change management |
| Stop watching N repositories | unwatch | Stops watching a watched project | Registered projects → choose → Stop watching N | Remove change management |
Each repository is handled exactly as the single action on its page; see Change management for what is written and removed.
Choosing repositories
Repositories not in Merge Pilot yet
The Watch repositories and Add to change management dialogs list the connection's repositories and take a list of chosen repositories, or every repository matching a search that is not in Merge Pilot yet. See Watching a repository and Adding repositories straight to change management.
Registered projects
On Projects → Registered projects:
-
Check the projects in the list, or the box in the header (Choose the repositories on this page). Projects chosen one by one stay chosen while you search and filter.
-
When every project on the page is chosen and more match the filters, Choose all N matching chooses all of them: Merge Pilot counts them per connection and mode, and the operation finds them when it runs. When more than 10,000 match, the bar says N match: narrow the filters to choose them all (at most 10,000 at a time). Changing a filter or the search ends an "all matching" choice.
-
The bar shows N repositories chosen (or Every matching repository chosen (N)) and the moves that fit the choice:
- Move N to change management, counting the watched projects chosen;
- Switch N to watching, counting the projects under change management chosen;
- Stop watching N, counting the watched projects chosen;
- Clear.
A move is offered only when you may run it on every connection of the chosen projects (company admins always may).
Auditors cannot choose projects.
While an operation is running, the bar notes that An operation is still running: repositories it has not finished are skipped by another.
The dialog
The bulk dialog starts one operation per VCS connection of the chosen projects, and says how many it takes on each (12 on GitLab, 3 on GitHub). A connection with more than 10,000 chosen projects cannot start (More than 10,000 repositories of <connection>: narrow the choice.).
| Kind | The dialog shows |
|---|---|
| Move to change management | Merge Pilot writes the steps below to each repository and gates its merges by the change policies. What was recorded while the chosen projects were watched (While watched, N merge requests were recorded; M merged that change management would have held (as it decided then).); the setup steps for each connection (Written to each of N GitLab repositories (<connection>)), each with its optional steps to choose; Open merge requests: Only new merge requests and new commits are gated (the open ones keep passing) or Gate the open merge requests too: they are decided again now (see What happens to the open merge requests); the consent check box |
| Switch to watching | Merge Pilot stops gating them and only records their merge requests. Removed from each repository: GitLab's approval rules (branch protection, the bot membership and the approval settings stay, unless taken back below); GitHub's ruleset. The event delivery stays. Then the Also take back what change management created on GitLab check box (see below) |
| Stop watching | Merge Pilot stops following them. Removed from each repository: Merge Pilot's GitLab webhook (GitHub: nothing, the App's webhook stays). What was recorded stays; open merge requests are no longer followed. Then the same check box, for repositories that were once under change management |
Also take back what change management created on GitLab (off by default) removes, from each repository, the branch protections and the bot membership change management added, and puts back the approval settings it changed; what was there before stays. What GitLab refuses to take back is listed under the repository on the operation, to remove by hand (see What change management created).
Every dialog ends with It runs in the background, a few repositories at a time. You can follow and cancel it on this page; a repository that fails does not stop the others. and Start for N repositories.
When one connection's operation cannot start after another's did, the ones started run on and the page says which did not (Started for the other connections, but not for <connection>: …).
How an operation runs
- In the background, as a chain of jobs (
bulk_operationon the Jobs page). Each job works on up to 50 repositories or for about 7 minutes, then queues the next, so a restart picks up where it was. - A few at a time: by default 4 repositories of an operation at once, and at most two operations' jobs at once on one VCS connection (another waits).
- Skips what another operation holds: a repository waiting in, or being worked on by, another running operation of the same connection is skipped.
- Yields to live work: an operation uses less of the VCS's API rate limit than live merge requests. When the VCS answers that the limit is used up, the operation waits (Waiting for the VCS's API rate limit until <time>: the rest of its calls are kept for live merge requests.) and the repositories it was working on are tried again.
- With its creator's rights as they are now: each job checks the person who started the operation again. When their account was deleted or deactivated, they no longer belong to the company, they started it as a company admin and no longer are one, the connection was deleted, or the connection's rule no longer lets their role do it, the operation stops and its remaining repositories fail with the reason.
- "All matching" operations find their repositories when they start: Finding the matching repositories… shows meanwhile. For Watch and Add to change management, the repositories of the connection whose path contains the search and that are not in Merge Pilot yet; for the other kinds, the connection's projects whose name or path contains the search, in the mode the kind moves from.
Repository outcomes
| Outcome (label in the UI) | API status | Meaning |
|---|---|---|
| Waiting | pending | Not started yet |
| Running | running | Being worked on |
| Done | done | Done; warnings, such as an optional setup step that failed, are listed below the repository |
| Failed | failed | Not done, with the reason |
| Skipped | skipped | Nothing to do, with the reason: it is in another bulk operation that is still running, it is already under change management, it is already watched, it is watched: move it with Move to change management, or it is no longer in Merge Pilot |
| Cancelled | cancelled | Not started because the operation was cancelled |
Reasons a repository fails include this needs at least Maintainer on this repository in GitLab (you have Developer), link your <product> account in My profile, the repository is under change management already (watching a repository under change management), it is under change management: switch it to watching first (stopping watching a repository under change management), and the VCS's own error, such as a required setup step that was refused.
Following an operation
The Bulk operations box on Projects → Registered projects lists the operations of the last week and every running one, newest first, 20 at a time (Show older operations loads more). It refreshes itself while an operation runs, and the project list reloads when one finishes.
Each operation shows:
- its kind and number of repositories, Started by <name> <when>, and for an "all matching" operation what it matched (every repository matching "…", every repository not in Merge Pilot yet, every matching repository chosen);
- its status and the counts N done · N failed · N skipped · N cancelled of N, with a progress bar while it runs;
- why it stopped, when it did.
Select an operation to list its repositories, sorted by path, with a filter: All, Failed, Done, Skipped, Cancelled, Waiting. They load 200 at a time (Show more (N left)).
Operation statuses
| Status (label in the UI) | API status | Meaning |
|---|---|---|
| Queued | queued | Waiting for its first job |
| Running | running | Working through its repositories |
| Cancelling | cancelling | Cancel was asked: no new repository starts; those being worked on finish |
| Cancelled | cancelled | Ended by a cancel |
| Done | done | Every repository has an outcome. An "all matching" operation that found nothing to do (none matched, more than 10,000 matched, or the repositories could not be listed) also ends here, with the reason shown |
| Stopped | failed | It could not go on: it could not be queued, its job failed for good, its creator lost their rights, or no job was working on it any more after a restart (interrupted: no job is working on it any more (a restart or a lost job); start it again). Its unfinished repositories fail with the reason |
Who sees which operations
Company admins, platform admins and auditors see every operation of the company. Everyone else sees the operations they started.
Cancel
Cancel on a running operation stops it: no new repository is started, the ones being worked on finish, and the rest end as Cancelled. The person who started the operation and company admins may cancel it.
Retry failed
When an operation has finished and some repositories failed, Retry N failed starts a new operation over those repositories, with the same kind, setup steps, choice for open merge requests and target branches. The consent of the person who agreed to the setup steps in the first operation carries over. Fix the cause first (for example link your VCS account, or ask for the role the connection's rule needs).
Revert done
When an operation has finished, Revert N done starts a new operation that takes its successful repositories back:
| Operation | Reverted by | The confirmation asks |
|---|---|---|
| Watch | Stop watching | Stop watching N repositories? Their event delivery is removed; what was recorded stays. |
| Move to change management | Switch to watching | Switch N repositories back to watching? Their change management is removed as when switching one to watching: the gate rule goes, GitLab branch protection stays. |
| Add to change management | Switch to watching (they stay in Merge Pilot, watched) | The same as above |
Switch to watching and Stop watching operations cannot be reverted; move the repositories back with a new operation.
Retry and revert may be started by the operation's creator or a company admin, once it has finished. They need the connection permission of the new operation's kind, and each repository is checked against the rights of the person who starts them.
Permissions
Company admins and platform admins may run every kind. For everyone else:
- Starting: the VCS connection's rule for the kind must be on and name the person's Merge Pilot role (see Kinds of operation and VCS connection settings). Otherwise the request is refused (this connection does not let people set up change management through Merge Pilot, or your Merge Pilot role cannot set up change management through this connection; the same with remove change management).
- Each repository: when it runs, Merge Pilot checks through the person's linked VCS account that they hold at least the rule's minimum role on it. Repositories where they do not fail with the reason (for example this needs at least Maintainer on this repository in GitLab (you have Developer)); the others go ahead. Without a linked account, every repository fails (link your <product> account in My profile).
- Auditors cannot start, cancel, retry or revert operations.
Limits
| Limit | Value |
|---|---|
| Repositories in one operation | 10,000. More are refused (choose at most 10000 repositories at a time); an "all matching" operation that finds more ends without doing anything |
| Running operations per person | 3 per company by default (queued, running or cancelling). One more is refused: you already have as many operations running as allowed; wait for one to finish or cancel one |
| Repositories worked on at once | 4 per operation by default; at most two operations' jobs at once per VCS connection |
| Operations listed | Those of the last 7 days, and every running one |
Audit actions
| Action | When it is written |
|---|---|
bulk_operation_started | The operation started, with its kind and number of repositories |
bulk_operation_finished | It finished (done or stopped), with the counts |
bulk_operation_cancel_requested | Someone asked to cancel it |
bulk_operation_cancelled | It ended because it was cancelled, with the counts |
Each repository is also recorded with the action of the single move, as a success or a failure, with the operation's ID, its kind, the repository and its path, the error and (when setting up) the steps: project_watched (watch), cm_setup (add to change management), project_mode_changed (move to change management, switch to watching), project_unwatched (stop watching).
All entries name the person who started the operation, with the address they started it from. See the Audit log.
API
| Action | Request |
|---|---|
| Start | POST /api/v1/projects/bulk?company_id=…, answers 202 with the operation |
| List | GET /api/v1/projects/bulk?company_id=… with limit (default 20, at most 200) and active=true for running ones only; answers items and has_more |
| One operation and its repositories | GET /api/v1/projects/bulk/{op} with status, page and page_size (default 50, at most 200); answers operation, items and total |
| Cancel | POST /api/v1/projects/bulk/{op}/cancel |
| Retry failed | POST /api/v1/projects/bulk/{op}/retry-failed, answers 202 with the new operation |
| Revert done | POST /api/v1/projects/bulk/{op}/revert-done, answers 202 with the new operation |
The body of a start:
| Field | Meaning |
|---|---|
kind | watch, add_gated, to_gated, to_watching or unwatch. Required |
vcs_connection_id | The connection; empty for the company's default VCS connection |
repos | For watch and add_gated: the repositories, each {"repo_id": "…", "path": "…"} |
project_ids | For the other kinds: the projects |
select_all, q | Instead of repos or project_ids: every repository matching q, found when the operation runs |
steps, consent | For to_gated and add_gated: the setup steps and consent: true (see Setup steps and consent) |
gate_open | For to_gated: also gate the open merge requests now |
undo_setup | For to_watching and unwatch: also take back what change management created on GitLab |
target_branches | For watch: the target branch patterns of the repositories added (empty: the company's) |
An operation has id, company_id, vcs_connection_id, kind, status, total, counts (per repository status), steps, gate_open, target_branches, query, error, created_by, created_by_name, consented_by, cancelled_by, paused_until, created_at and finished_at. A repository of an operation has id, repo_id, project_id, path, status, error, warnings and finished_at.
Related pages
Video tutorial
Short videos of this feature working on real GitLab and Jira projects. English audio, English and Turkish subtitles. All video tutorials →