Git Platforms Synchronization

Synchronize branches of repositories from a Git platform to another (with rebase, merge unsupported).

Principle:

• Check repositories to synchronise using API Git platform according includes/excludes pattern, create it/them on destination if not exist
• Check branches according includes/excludes and retrieve last commit
• For each branches if last commit different, clone repository from origin and push branches to destination (with tags)
• If no branches synchronized, push tags if number differ between origin and destination
• Write synchronization synthesis

Supported platforms:

• Bitbucket
• Gerrit Code Review (with limitation)
• Gitea
• GitHub
• GitLab

Usage

Dependencies installation (once)

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

For more fined dependencies installed, you could use one or multiple pip install with: modules/requirements-[bitbucket|gerrit|gitea|github|gitlab].txt

Execution program

python3 git_platforms_synchro.py \
    --from-url https://api.github.com --from-org spring-projects \
    --to-url http://localhost:3000 --to-type gitea --to-login foo --to-password bar --to-org MyOrg \
    --repos-include spring-petclinic,spring-ai-examples \
    --branches-include "main,spring-ai*"

Options

usage: git_platforms_synchro.py [-h] --from-url FROM_URL [--from-login FROM_LOGIN] [--from-password FROM_PASSWORD] --from-org FROM_ORG [--from-type FROM_TYPE] [--from-proxy FROM_PROXY]
                                [--from-disable-ssl-verify] --to-url TO_URL --to-login TO_LOGIN [--to-password TO_PASSWORD] --to-org TO_ORG [--to-type TO_TYPE] [--to-proxy TO_PROXY]
                                [--to-disable-ssl-verify] [--to-description-prefix TO_DESCRIPTION_PREFIX] [--repos-include REPOS_INCLUDE] [--repos-exclude REPOS_EXCLUDE]
                                [--branches-include BRANCHES_INCLUDE] [--branches-exclude BRANCHES_EXCLUDE] [-d] [-l LOG_LEVEL]

Git Platforms Synchronization

options:
  -h, --help            show this help message and exit
  --from-url FROM_URL   Git "from" platform API URL (Required)
  --from-login FROM_LOGIN
                        Git "from" login or token.
  --from-password FROM_PASSWORD
                        Git "from" password.
  --from-org FROM_ORG   Git "from" organization/user/project.
  --from-type FROM_TYPE
                        Git "from" type (Bitbucket, Gerrit, Gitea, GitLab, GitHub, ... ; To use when cannot be detected from URL).
  --from-proxy FROM_PROXY
                        Git "from" proxy (with credentials if needed).
  --from-disable-ssl-verify
                        Git "from" disable SSL verification.
  --to-url TO_URL       Git "to" platform API URL (Required)
  --to-login TO_LOGIN   Git "to" login or token (Required).
  --to-password TO_PASSWORD
                        Git "to" password.
  --to-org TO_ORG       Git "to" organization/user/project.
  --to-type TO_TYPE     Git "to" type (Bitbucket, Gitea, Gerrit, GitLab, GitHub, ... ; To use when cannot be detected from URL).
  --to-proxy TO_PROXY   Git "to" proxy (with credentials if needed).
  --to-disable-ssl-verify
                        Git "to" disable SSL verification.
  --to-description-prefix TO_DESCRIPTION_PREFIX
                        Description prefix for Git "to" repositories creation.
  --repos-include REPOS_INCLUDE
                        Repositories names patterns to include (comma separated).
  --repos-exclude REPOS_EXCLUDE
                        Repositories names patterns to exclude (comma separated).
  --branches-include BRANCHES_INCLUDE
                        Branches names patterns to include (comma separated).
  --branches-exclude BRANCHES_EXCLUDE
                        Branches names patterns to exclude (comma separated).
  -d, --dry-run         Dry-run : Just analyse which branches should be synchronized, without doning it really.
  -l LOG_LEVEL, --log-level LOG_LEVEL
                        Log level (DEBUG, INFO, WARNING, ERROR, CRITICAL)

Known issues

On Gerrit, when a repository is created and mirrored with git push --porcelain --mirror -- sync-to, access permission error:

[remote rejected] (prohibited by Gerrit: project state does not permit write)

Development

Unit tests

Dependencies installation (once):

pip install -r tests/requirements.txt

Execution (with code coverage support for some subprocessed tests):

pytest

# Or with code coverage
coverage run --concurrency=thread --parallel-mode -m pytest
coverage combine
coverage html

Utilities

To instantiate a Gitea accessible on http://my.gitea.local:3000 via proxy (login/password= evil/live, port 8000) or natively on http://localhost:3000, use this compose.yaml runnable via docker compose up:

# Before run, create directories: mkdir config data
services:
  # Gitea platform
  my.gitea.local:
    image: gitea/gitea:latest-rootless
    environment:
      GITEA__server__ROOT_URL: http://my.gitea.local:3000
      # GITEA__server__ROOT_URL: http://localhost:3000
    restart: always
    volumes:
      - ./data:/var/lib/gitea
      - ./config:/etc/gitea
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "3000:3000"
      - "2222:2222"

  # Proxy server
  proxy:
    image: monokal/tinyproxy
    environment:
      BASIC_AUTH_USER: evil
      BASIC_AUTH_PASSWORD: live
    ports:
      - '8000:8888/tcp'
    command: ANY

To instantiate a Gerrit accessible on http://localhost:8080, use this compose.yaml runnable via docker compose up:

services:
  gerrit:
    image: gerritcodereview/gerrit
    volumes:
       - git-volume:/var/gerrit/git
       - index-volume:/var/gerrit/index
       - cache-volume:/var/gerrit/cache
    ports:
       - "29418:29418"
       - "8080:8080"

volumes:
  git-volume:
  index-volume:
  cache-volume: