Addressing Secret leaks, on-demand

Across an Entire Owned Remote Git Repositories

Web Admin

— 6 min read
Addressing Secret leaks, on-demand

Discovering that hardcoded API keys, database passwords, or corporate credentials have leaked into your source control requires rapid, aggressive remediation. When secrets are scattered across dozens of repositories and multiple git branches, handling it manually is impossible.

This article details a battle-tested blueprint for an on-demand emergency remediation pipeline engineered inside a Linux (Debian) environment. We will cover the exact workflow to programmatically download every repository, scan them with gitleaks, parse the findings to overwrite history with REDACTED across all branches and timelines, and successfully force-push the clean results back to GitLab.

Technical Context & Real-World Challenges

During an actual emergency execution, standard documentation quickly falls apart due to environment edge cases. This architectural blueprint accounts for several specific obstacles encountered during a real remediation run:

  • Environmental Context: Run completely inside Development Environment (example Linux, Debian).
  • The False-Positive Loop: gitleaks scans the workspace root and flags its own report.json output as a leak. We fix this by isolating the workspace directory completely outside of the script repository tree.
  • Permissions Blockades: Forcing modifications down protected server branches defaults to a 403 Forbidden error. We map the explicit path to unprotect these targets inside GitLab.
  • Future Hardening: During the rewriting phase, we dynamically inject a robust .gitignore boilerplate to prevent heavy web application environments (like Node.js, Python, or PHP frameworks) from leaking environment variables or certificates again.

Phase 1: Environment Provisioning (Debian Linux here)

Before processing repositories, ensure your Debian environment inside Windows Subsystem for Linux has the core tools and binaries installed.

1. Install Gitleaks & Git-Filter-Repo

Do not rely on outdated apt repositories. Fetch the latest gitleaks pre-compiled binary directly, and install git-filter-repo via Python's package manager:

# Update local packages
sudo apt update && sudo apt install -y curl git jq python3-pip python3-venv

# Download and install latest, but stable and pinned Gitleaks version
GITLEAKS_VERSION="8.21.0"
curl -sSLo gitleaks.tar.gz "https://github.com/gitleaks/gitleaks/releases/download/v${GITLEAKS_VERSION}/gitleaks_${GITLEAKS_VERSION}_linux_x64.tar.gz"
tar -xzf gitleaks.tar.gz gitleaks
sudo mv gitleaks /usr/local/bin/
rm gitleaks.tar.gz

# Install git-filter-repo globally
pip3 install git-filter-repo --break-system-packages

2. Isolate the Workspace Directory

To prevent gitleaks from detecting its own JSON reports, do not run scans inside your active scripts directory. Establish an isolated workspace directory at a completely neutral system path (e.g., /tmp/secure_remediation_vault).

Phase 2: The Two-Script Solution

Our architecture splits the solution into two scripts to ensure precision:

  1. Script 1 (scan_fleet.sh): Handles recursive discovery, cloning, and outputs a target report.
  2. Script 2 (remediate_fleet.sh): Evaluates the report, rewrites history comprehensively, amends local hygiene files, and handles remote delivery synchronization.

Script 1: Fleet Discovery and Secret Extraction

This script loops through your GitLab instance using your Personal Access Token (PAT), clones repositories into the isolated vault, and runs a comprehensive gitleaks check across full repository histories.

#!/usr/bin/env bash
set -euo pipefail

GITLAB_URL="https://gitlab.com"
API_TOKEN="your_gitlab_pat_token"
ORG_ID="your_gitlab_group_or_user_id"

# ISOLATION: Workspace paths are kept out of this script directory to prevent false-positives
WORKSPACE_DIR="/tmp/secure_remediation_vault"
REPORTS_DIR="${WORKSPACE_DIR}/gitleaks_reports"

mkdir -p "$WORKSPACE_DIR"
mkdir -p "$REPORTS_DIR"

echo "=== PHASE 1: Fetching Repository Catalog from GitLab ==="
# Fetch projects via GitLab API
REPOS=$(curl -s --header "PRIVATE-TOKEN: ${API_TOKEN}" "${GITLAB_URL}/api/v4/groups/${ORG_ID}/projects?per_page=100" | jq -r '.[].ssh_url_to_repo')

cd "$WORKSPACE_DIR"

for repo_url in $REPOS; do
    repo_name=$(basename "$repo_url" .git)
    echo "Processing Catalog Destination: ${repo_name}"
    
    if [ ! -d "$repo_name" ]; then
        # Cloning all branches and full historical timelines via a mirror clone
        git clone --mirror "$repo_url" "$repo_name"
    fi
    
    echo "Scanning history for ${repo_name}..."
    # Scan the mirrored repository directory directly 
    gitleaks detect --source="./${repo_name}" --report-path="${REPORTS_DIR}/${repo_name}_report.json" || true
done

echo "=== Phase 1 Complete. Individual reports compiled safely in ${REPORTS_DIR} ==="

Script 2: Comprehensive Remediation & Synchronization

This script reads the respective parsed JSON reports generated by Phase 1. It extracts the raw values of compromised secrets, formats a replacements rule file, executes git-filter-repo across all branches and history tags, appends standard tracking ignore rules, and triggers forced remote synchronization updates. The key feature of the script is it looks up the scanned report json to identify fix the secrets instead of hard-coded values. Which however, can be added to be addressed as well. It also adds typical files and directories to .gitignore which shouldn't be pushed to remote at all, but inadvertently, sometimes, gets pushed to remote repo.

#!/usr/bin/env bash
set -euo pipefail

WORKSPACE_DIR="/tmp/secure_remediation_vault"
REPORTS_DIR="${WORKSPACE_DIR}/gitleaks_reports"
GITLAB_URL="https://gitlab.com"
API_TOKEN="your_gitlab_pat_token"

if [ ! -d "$WORKSPACE_DIR" ]; then
    echo "Workspace vault directory missing. Execute the discovery sequence first."
    exit 1
fi

cd "$WORKSPACE_DIR"

for repo_dir in */ ; do
    [ -d "$repo_dir" ] || continue
    repo_name="${repo_dir%/}"
    
    # Skip processing our report directory tracking structure
    if [ "$repo_name" == "gitleaks_reports" ]; then continue; fi
    
    report_file="${REPORTS_DIR}/${repo_name}_report.json"
    
    echo "--------------------------------------------------------"
    echo "Remediating Target Repository Timeline: ${repo_name}"
    echo "--------------------------------------------------------"
    
    cd "$repo_name"
    
    # Check if any legitimate leaks were documented
    if [ -f "$report_file" ] && [ "$(jq '. | length' "$report_file")" -gt 0 ]; then
        echo "Leaks found. Extracting matching strings..."
        
        # Build text replacement file mapping secret strings to 'REDACTED'
        expressions_file="../${repo_name}_expressions.txt"
        rm -f "$expressions_file"
        
        jq -r '.[].Secret' "$report_file" | sort -u | while read -r secret_string; do
            if [ -z "$secret_string" ]; then continue; fi
            echo "${secret_string}==>REDACTED" >> "$expressions_file"
        done
        
        echo "Executing total history scrub via git-filter-repo..."
        # Rewrites ALL commits on ALL branches simultaneously
        git filter-repo --replace-text "$expressions_file" --force
        rm -f "$expressions_file"
        
        # Restore remote target because git-filter-repo removes origins for safety
        git remote add origin "${GITLAB_URL}/your_group_or_user/${repo_name}.git" || true
    else
        echo "Clean validation scan report. Skipping filtration rewrite steps."
        git remote add origin "${GITLAB_URL}/your_group_or_user/${repo_name}.git" || true
    fi

    # RESTORE WORKSPACE HYGIENE: Update .gitignore with production best practices
    # We must briefly convert bare repos to normal or handle metadata. If working tree exists:
    echo "Injecting hardened .gitignore patterns..."
    cat << 'EOF' >> .gitignore

# Emergency Secret Remediation Hardening Boilerplate
.env
.env.*
*.log
*.pem
*.key
*.crt
*.p12
/certs/
/config/jwt/
/vendor/
/node_modules/
EOF
    
    # Stage amendments and commit changes into the rewritten lineage
    git add .gitignore
    git commit -m "chore: harden repository workspace safety and .gitignore patterns" || true

    echo "Pushing purged data assets back to GitLab..."
    git push origin --all --force
    git push origin --tags --force
    
    cd ..
done

echo "=== Emergency Fleet Recovery Script Finalized ==="

Phase 3: Addressing Server Permissions & Automation

When Script 2 attempts to rewrite remote tracking history with git push origin --all --force, GitLab will block the operation with a branch protection error for default branches (like main or master).

The GitLab UI Flow

To bypass this manual configuration blocker:

  1. Open your target group or repository space in the GitLab UI.
  2. Navigate to the left sidebar margin: Settings (
    ightarrow) Repository.
  3. Locate the Protected branches configuration section and select Expand.
  4. Scroll down to find your restricted main tracking entries (e.g., main).
  5. Click Unprotect to temporarily allow administrative forced push access.
  6. Rerun Script 2. The script will now rewrite all server records across those branches smoothly.
  7. Return to the configuration layout immediately and restore branch protection constraints.

Automating Protections with GitLab API (Alternative)

If you have dozens of repositories, clicking through the UI is tedious. You can automate toggling branch protections before and after running Script 2 using the GitLab API:

# To UNPROTECT a branch (Run before Script 2)
curl --request DELETE --header "PRIVATE-TOKEN: ${API_TOKEN}"      "${GITLAB_URL}/api/v4/projects/${PROJECT_ID}/protected_branches/main"

# To RE-PROTECT a branch (Run immediately after Script 2)
curl --request POST --header "PRIVATE-TOKEN: ${API_TOKEN}"      "${GITLAB_URL}/api/v4/projects/${PROJECT_ID}/protected_branches?name=main&push_access_level=40&merge_access_level=40"

(Note: 40 corresponds to Maintainer access rules in GitLab).

Team Instructions: Recovering from a Force Push

Because you rewrote the history tracking elements on the remote server, your engineering team will experience errors the next time they attempt to run git pull. Their local commit tracking history no longer matches the server.

Provide your development team with these exact commands to safely reconcile their workspaces without re-introducing the deleted secrets:

# WARNING: Do NOT run 'git pull' or 'git merge'. It will re-upload the leaked secret!
# Run these steps instead:

# 1. Fetch the newly cleaned history from the remote tracking host
git fetch origin

# 2. Reset your active local branch to match the remote server exactly
# (CAUTION: This discards uncommitted changes. Stash them first if necessary via 'git stash')
git reset --hard origin/$(git branch --show-current)

Verification Strategy

Once the pipeline scripts conclude operations, verify your history is clean by forcing an on-demand validation check.

Clear your local cache paths completely, download a fresh copy of any target repository, and rerun a targeted gitleaks execution profile:

cd /tmp
git clone https://gitlab.com/your-org/your-remediated-repo.git
gitleaks detect --source="./your-remediated-repo" -v

The validation terminal output should read:
INF no leaks found

Your repositories are now clean, your histories are completely purged of credentials, your hygiene files are updated, and your team's tracking targets are securely locked down against future leaks.

Read more