fix(private-org-sync): process repos in parallel with configurable workers#5030
fix(private-org-sync): process repos in parallel with configurable workers#5030petr-muller wants to merge 5 commits intoopenshift:mainfrom
Conversation
|
Pipeline controller notification For optional jobs, comment This repository is configured in: automatic mode |
openshift-ci
bot
added
the
do-not-merge/work-in-progress
|
Skipping CI for Draft Pull Request. |
|
Important Review skippedAuto reviews are limited based on label configuration. 🚫 Excluded labels (none allowed) (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: petr-muller The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
…branch Previously, mirror() called ls-remote for both source and destination on every branch, resulting in 2*N network calls per repo where N is the number of branches. Since ls-remote --heads returns ALL branches at once, we can call it once per repo and pass the results to mirror(). This moves git init, remote setup, and ls-remote calls from mirror() into the main loop where repos are already grouped, reducing network calls from 2*N to 2 per repo. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
petr-muller
force-pushed
the
git-sync-speedup-7
branch
from
f4b25d1 to
d7b1783
Compare
Extract the per-repo sync logic from main() into a syncRepo method on gitSyncer. This makes the repo initialization, ls-remote batching, and branch mirroring flow independently testable. Add TestSyncRepo with 6 test cases covering: branches in sync, one branch needing sync, dst/src ls-remote failures (with and without failOnNonexistentDst), and init failure. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…errors Wrap gitExec with withRetryOnTransientError to transparently retry any git command (fetch, push, ls-remote, etc.) up to 3 times with a 5-second delay when transient network errors are detected (DNS resolution failures, connection timeouts/refused/reset, server 5xx errors). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Instead of querying source and destination branch heads sequentially, run both ls-remote calls concurrently using goroutines. This saves one network round-trip per repo. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…rkers Extract per-repo sync logic into a syncRepo function and process repos concurrently using a worker pool. The --parallelism flag (default 4) controls the number of concurrent workers. Each worker gets its own copy of the gitSyncer struct, avoiding races on the mutable logger field. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
petr-muller
force-pushed
the
git-sync-speedup-7
branch
from
d7b1783 to
908d58a
Compare
2 participants
Processes repos in parallel using a configurable worker pool (
--parallelismflag, default 4). Each repo is independent (separate git dir, separate remotes, separate branches), so this is safe to parallelize.Stacked on #5029.