Completely Removing Secrets from Git History

What I Learned

Sensitive information (API keys, private keys, configuration files, etc.) that has been accidentally committed continues to remain in Git’s history even with normal deletion. By using git filter-repo, you can rewrite Git history and completely remove sensitive information. However, removing from history does not make already leaked credentials safe, so you must always revoke and reissue the relevant credentials.

Details

Background

Previously, methods using git filter-branch were introduced, but warnings are now issued in the official documentation, and when creating new procedures, the use of git filter-repo is recommended.

Tools Used

  • git
  • git filter-repo

Precautions Before Work

  • This procedure is a destructive operation that rewrites history
  • When implementing in shared repositories, notify stakeholders in advance
  • After work, request all users to re-clone

Case-by-Case Procedures

Case 1: Completely Remove a Specific File from History

1. Mirror Clone

1git clone --mirror REMOTE_URL REPO_NAME.git
2cd REPO_NAME.git

Mirror cloning retrieves all branches, tags, and references. To accurately reflect the results of rewriting history to the remote, this procedure uses mirror cloning.

2. Remove Sensitive Files from History

1git filter-repo \
2  --path SENSITIVE_PATH_1 \
3  --path SENSITIVE_PATH_2 \
4  --invert-paths
  • SENSITIVE_PATH_n: repository-relative path of files to remove from history (multiple can be specified)

3. Reflect to Remote

1git push --force --all
2git push --force --tags

Case 2: Remove/Replace Sensitive Strings (API Keys, etc.) from History

1. Mirror Clone

1git clone --mirror REMOTE_URL REPO_NAME.git
2cd REPO_NAME.git

Normal cloning doesn’t include some references, so use mirror cloning for history modification work.

2. Create Replacement Rules File

1cat > replace-rules.txt << 'EOF'
2regex:SECRET_PATTERN_1==>***REDACTED***
3regex:SECRET_PATTERN_2==>***REDACTED***
4EOF

Replacement rule format: [search mode]:[pattern to search]==>[replacement string]

  • Specify regex etc. for search mode
  • When using regular expressions, verify carefully to avoid unintended replacements

3. Rewrite History

1git filter-repo --replace-text replace-rules.txt

4. Reflect to Remote

1git push --force --all
2git push --force --tags

Case 3: Process Multiple Sensitive Files and Strings Simultaneously

 1git clone --mirror REMOTE_URL REPO_NAME.git
 2cd REPO_NAME.git
 3
 4git filter-repo \
 5  --path SENSITIVE_PATH_1 \
 6  --path SENSITIVE_PATH_2 \
 7  --invert-paths \
 8  --replace-text replace-rules.txt
 9
10git push --force --all
11git push --force --tags

What You Must Do After Work

  • Notify stakeholders of the impact scope
  • Have all local repositories re-cloned
  • Revoke and reissue all credentials that may have been leaked

Important Points

  • Always invalidate and reissue sensitive information even after removing it from history
  • git push --force destroys history, so notify the team beforehand
  • Don’t include sensitive information in repositories; manage them with .env files etc.
  • Always add .env and files containing secrets to .gitignore
  • Enable GitHub Secret Scanning for early detection of sensitive information

References