Skip to main content
Video tutorial · 0:57

Bulk operations

A bulk operation watches, moves or stops many repositories of one VCS connection at once, in the background. It is how a company with hundreds or thousands of repositories starts watching them, and later moves them to change management, without doing it one repository at a time. Each repository is worked on and reported on its own: one that fails does not stop the others.

Watching a single repository from its page also runs as a bulk operation of one.

Kinds of operation​

Kind (label in the UI)API kindWhat it does to each repositoryStarted fromConnection permission needed (people who are not company admins)
Watch N repositorieswatchWatches a repository not in Merge Pilot yet (see Watching repositories)Projects → Watch repositories; a repository's page → WatchSet up change management
Add N repositories to change managementadd_gatedPuts a repository not in Merge Pilot yet straight under change management, with the chosen setup stepsProjects → Add to change managementSet up change management
Move N repositories to change managementto_gatedMoves a watched project to change management, with the chosen setup steps and choice for open merge requestsRegistered projects → choose → Move N to change managementSet up change management
Switch N repositories to watchingto_watchingMoves a project under change management back to watchingRegistered projects → choose → Switch N to watchingRemove change management
Stop watching N repositoriesunwatchStops watching a watched projectRegistered projects → choose → Stop watching NRemove change management

Each repository is handled exactly as the single action on its page; see Change management for what is written and removed.

Choosing repositories​

Repositories not in Merge Pilot yet​

The Watch repositories and Add to change management dialogs list the connection's repositories and take a list of chosen repositories, or every repository matching a search that is not in Merge Pilot yet. See Watching a repository and Adding repositories straight to change management.

Registered projects​

On Projects → Registered projects:

  1. Check the projects in the list, or the box in the header (Choose the repositories on this page). Projects chosen one by one stay chosen while you search and filter.

  2. When every project on the page is chosen and more match the filters, Choose all N matching chooses all of them: Merge Pilot counts them per connection and mode, and the operation finds them when it runs. When more than 10,000 match, the bar says N match: narrow the filters to choose them all (at most 10,000 at a time). Changing a filter or the search ends an "all matching" choice.

  3. The bar shows N repositories chosen (or Every matching repository chosen (N)) and the moves that fit the choice:

    • Move N to change management, counting the watched projects chosen;
    • Switch N to watching, counting the projects under change management chosen;
    • Stop watching N, counting the watched projects chosen;
    • Clear.

    A move is offered only when you may run it on every connection of the chosen projects (company admins always may).

Auditors cannot choose projects.

While an operation is running, the bar notes that An operation is still running: repositories it has not finished are skipped by another.

The dialog​

The bulk dialog starts one operation per VCS connection of the chosen projects, and says how many it takes on each (12 on GitLab, 3 on GitHub). A connection with more than 10,000 chosen projects cannot start (More than 10,000 repositories of <connection>: narrow the choice.).

KindThe dialog shows
Move to change managementMerge Pilot writes the steps below to each repository and gates its merges by the change policies. What was recorded while the chosen projects were watched (While watched, N merge requests were recorded; M merged that change management would have held (as it decided then).); the setup steps for each connection (Written to each of N GitLab repositories (<connection>)), each with its optional steps to choose; Open merge requests: Only new merge requests and new commits are gated (the open ones keep passing) or Gate the open merge requests too: they are decided again now (see What happens to the open merge requests); the consent check box
Switch to watchingMerge Pilot stops gating them and only records their merge requests. Removed from each repository: GitLab's approval rules (branch protection, the bot membership and the approval settings stay, unless taken back below); GitHub's ruleset. The event delivery stays. Then the Also take back what change management created on GitLab check box (see below)
Stop watchingMerge Pilot stops following them. Removed from each repository: Merge Pilot's GitLab webhook (GitHub: nothing, the App's webhook stays). What was recorded stays; open merge requests are no longer followed. Then the same check box, for repositories that were once under change management

Also take back what change management created on GitLab (off by default) removes, from each repository, the branch protections and the bot membership change management added, and puts back the approval settings it changed; what was there before stays. What GitLab refuses to take back is listed under the repository on the operation, to remove by hand (see What change management created).

Every dialog ends with It runs in the background, a few repositories at a time. You can follow and cancel it on this page; a repository that fails does not stop the others. and Start for N repositories.

When one connection's operation cannot start after another's did, the ones started run on and the page says which did not (Started for the other connections, but not for <connection>: …).

How an operation runs​

  • In the background, as a chain of jobs (bulk_operation on the Jobs page). Each job works on up to 50 repositories or for about 7 minutes, then queues the next, so a restart picks up where it was.
  • A few at a time: by default 4 repositories of an operation at once, and at most two operations' jobs at once on one VCS connection (another waits).
  • Skips what another operation holds: a repository waiting in, or being worked on by, another running operation of the same connection is skipped.
  • Yields to live work: an operation uses less of the VCS's API rate limit than live merge requests. When the VCS answers that the limit is used up, the operation waits (Waiting for the VCS's API rate limit until <time>: the rest of its calls are kept for live merge requests.) and the repositories it was working on are tried again.
  • With its creator's rights as they are now: each job checks the person who started the operation again. When their account was deleted or deactivated, they no longer belong to the company, they started it as a company admin and no longer are one, the connection was deleted, or the connection's rule no longer lets their role do it, the operation stops and its remaining repositories fail with the reason.
  • "All matching" operations find their repositories when they start: Finding the matching repositories… shows meanwhile. For Watch and Add to change management, the repositories of the connection whose path contains the search and that are not in Merge Pilot yet; for the other kinds, the connection's projects whose name or path contains the search, in the mode the kind moves from.

Repository outcomes​

Outcome (label in the UI)API statusMeaning
WaitingpendingNot started yet
RunningrunningBeing worked on
DonedoneDone; warnings, such as an optional setup step that failed, are listed below the repository
FailedfailedNot done, with the reason
SkippedskippedNothing to do, with the reason: it is in another bulk operation that is still running, it is already under change management, it is already watched, it is watched: move it with Move to change management, or it is no longer in Merge Pilot
CancelledcancelledNot started because the operation was cancelled

Reasons a repository fails include this needs at least Maintainer on this repository in GitLab (you have Developer), link your <product> account in My profile, the repository is under change management already (watching a repository under change management), it is under change management: switch it to watching first (stopping watching a repository under change management), and the VCS's own error, such as a required setup step that was refused.

Following an operation​

The Bulk operations box on Projects → Registered projects lists the operations of the last week and every running one, newest first, 20 at a time (Show older operations loads more). It refreshes itself while an operation runs, and the project list reloads when one finishes.

Each operation shows:

  • its kind and number of repositories, Started by <name> <when>, and for an "all matching" operation what it matched (every repository matching "…", every repository not in Merge Pilot yet, every matching repository chosen);
  • its status and the counts N done · N failed · N skipped · N cancelled of N, with a progress bar while it runs;
  • why it stopped, when it did.

Select an operation to list its repositories, sorted by path, with a filter: All, Failed, Done, Skipped, Cancelled, Waiting. They load 200 at a time (Show more (N left)).

Operation statuses​

Status (label in the UI)API statusMeaning
QueuedqueuedWaiting for its first job
RunningrunningWorking through its repositories
CancellingcancellingCancel was asked: no new repository starts; those being worked on finish
CancelledcancelledEnded by a cancel
DonedoneEvery repository has an outcome. An "all matching" operation that found nothing to do (none matched, more than 10,000 matched, or the repositories could not be listed) also ends here, with the reason shown
StoppedfailedIt could not go on: it could not be queued, its job failed for good, its creator lost their rights, or no job was working on it any more after a restart (interrupted: no job is working on it any more (a restart or a lost job); start it again). Its unfinished repositories fail with the reason

Who sees which operations​

Company admins, platform admins and auditors see every operation of the company. Everyone else sees the operations they started.

Cancel​

Cancel on a running operation stops it: no new repository is started, the ones being worked on finish, and the rest end as Cancelled. The person who started the operation and company admins may cancel it.

Retry failed​

When an operation has finished and some repositories failed, Retry N failed starts a new operation over those repositories, with the same kind, setup steps, choice for open merge requests and target branches. The consent of the person who agreed to the setup steps in the first operation carries over. Fix the cause first (for example link your VCS account, or ask for the role the connection's rule needs).

Revert done​

When an operation has finished, Revert N done starts a new operation that takes its successful repositories back:

OperationReverted byThe confirmation asks
WatchStop watchingStop watching N repositories? Their event delivery is removed; what was recorded stays.
Move to change managementSwitch to watchingSwitch N repositories back to watching? Their change management is removed as when switching one to watching: the gate rule goes, GitLab branch protection stays.
Add to change managementSwitch to watching (they stay in Merge Pilot, watched)The same as above

Switch to watching and Stop watching operations cannot be reverted; move the repositories back with a new operation.

Retry and revert may be started by the operation's creator or a company admin, once it has finished. They need the connection permission of the new operation's kind, and each repository is checked against the rights of the person who starts them.

Permissions​

Company admins and platform admins may run every kind. For everyone else:

  • Starting: the VCS connection's rule for the kind must be on and name the person's Merge Pilot role (see Kinds of operation and VCS connection settings). Otherwise the request is refused (this connection does not let people set up change management through Merge Pilot, or your Merge Pilot role cannot set up change management through this connection; the same with remove change management).
  • Each repository: when it runs, Merge Pilot checks through the person's linked VCS account that they hold at least the rule's minimum role on it. Repositories where they do not fail with the reason (for example this needs at least Maintainer on this repository in GitLab (you have Developer)); the others go ahead. Without a linked account, every repository fails (link your <product> account in My profile).
  • Auditors cannot start, cancel, retry or revert operations.

Limits​

LimitValue
Repositories in one operation10,000. More are refused (choose at most 10000 repositories at a time); an "all matching" operation that finds more ends without doing anything
Running operations per person3 per company by default (queued, running or cancelling). One more is refused: you already have as many operations running as allowed; wait for one to finish or cancel one
Repositories worked on at once4 per operation by default; at most two operations' jobs at once per VCS connection
Operations listedThose of the last 7 days, and every running one

Audit actions​

ActionWhen it is written
bulk_operation_startedThe operation started, with its kind and number of repositories
bulk_operation_finishedIt finished (done or stopped), with the counts
bulk_operation_cancel_requestedSomeone asked to cancel it
bulk_operation_cancelledIt ended because it was cancelled, with the counts

Each repository is also recorded with the action of the single move, as a success or a failure, with the operation's ID, its kind, the repository and its path, the error and (when setting up) the steps: project_watched (watch), cm_setup (add to change management), project_mode_changed (move to change management, switch to watching), project_unwatched (stop watching).

All entries name the person who started the operation, with the address they started it from. See the Audit log.

API​

ActionRequest
StartPOST /api/v1/projects/bulk?company_id=…, answers 202 with the operation
ListGET /api/v1/projects/bulk?company_id=… with limit (default 20, at most 200) and active=true for running ones only; answers items and has_more
One operation and its repositoriesGET /api/v1/projects/bulk/{op} with status, page and page_size (default 50, at most 200); answers operation, items and total
CancelPOST /api/v1/projects/bulk/{op}/cancel
Retry failedPOST /api/v1/projects/bulk/{op}/retry-failed, answers 202 with the new operation
Revert donePOST /api/v1/projects/bulk/{op}/revert-done, answers 202 with the new operation

The body of a start:

FieldMeaning
kindwatch, add_gated, to_gated, to_watching or unwatch. Required
vcs_connection_idThe connection; empty for the company's default VCS connection
reposFor watch and add_gated: the repositories, each {"repo_id": "…", "path": "…"}
project_idsFor the other kinds: the projects
select_all, qInstead of repos or project_ids: every repository matching q, found when the operation runs
steps, consentFor to_gated and add_gated: the setup steps and consent: true (see Setup steps and consent)
gate_openFor to_gated: also gate the open merge requests now
undo_setupFor to_watching and unwatch: also take back what change management created on GitLab
target_branchesFor watch: the target branch patterns of the repositories added (empty: the company's)

An operation has id, company_id, vcs_connection_id, kind, status, total, counts (per repository status), steps, gate_open, target_branches, query, error, created_by, created_by_name, consented_by, cancelled_by, paused_until, created_at and finished_at. A repository of an operation has id, repo_id, project_id, path, status, error, warnings and finished_at.

Video tutorial

Short videos of this feature working on real GitLab and Jira projects. English audio, English and Turkish subtitles. All video tutorials →

Move Many Repositories at OnceYouTube ↗