
It was supposed to be a quick audit.
A client had asked me to sweep a private repo before they handed access to a new contractor. Routine. I cloned it, ran a scanner over the working tree, and started reading. Five minutes in, the scanner lit up red on a config helper.
A reusable credential. Plaintext. In the repo.
I felt that familiar little spike of righteous sysadmin adrenaline — who the hell committed THIS — opened the blame, and read the author line.
It was me.
The investigation
Here’s the part everyone gets wrong, and the part I almost got wrong too.
My first instinct was to delete the line, commit “remove secret,” and move on. Crisis averted, right?
Wrong. The file in the working tree is the least important place that secret lives. Git is a time machine. Deleting it today does nothing about the dozens of commits where it sits, fat and happy, in the history. Anyone with git log and ten seconds can walk straight to it.
I checked how deep it went:
git log --all --oneline -S 'the_secret_string' -- path/to/config
git rev-list --all --count
It went deep. The credential had been in the repo for months, across dozens of commits, surviving file renames and a refactor. Deleting the file would leave it perfectly intact in every parent commit.
the working tree (what you see)
│
▼
┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐
│ commit │◄──│ commit │◄──│ commit │◄──│ commit │
│ HEAD │ │ ~12 │ │ ~31 │ │ ~40 │
│ ✗ gone │ │ ▓ secret│ │ ▓ secret│ │ ▓ secret│
└─────────┘ └─────────┘ └─────────┘ └─────────┘
▲ └──────────────┬──────────────┘
you "fixed" it still right here, forever
The “aha”
The realization isn’t “I need to scrub this.” It’s earlier and uglier than that:
A secret is compromised the instant it’s committed. Not when the repo goes public. Not when someone clones it. At commit time. Every backup, every fork, every laptop that ever pulled, every CI cache — assume the secret is out. You cannot un-ring that bell by editing history.
So scrubbing history is necessary, but it is not step one. Step one is to make the leaked value worthless.
The fix
1. Rotate, immediately. Before anything else, I logged into the provider, revoked the credential, and issued a fresh one. The old value is now a dead string. Whoever has it has nothing.
2. Purge it from history. With the value already dead, I scrubbed it so it stops scaring the next auditor (me, in six months). git filter-repo is the modern tool; BFG works too.
# install once: pip install git-filter-repo
echo 'the_secret_string==>REDACTED' > replacements.txt
git filter-repo --replace-text replacements.txt
# then force the rewritten history out
git push --force --all
git push --force --tags
Every collaborator re-clones after this. A force-push that rewrites history will wreck anyone’s local copy — warn them first.
3. Move secrets out of the repo for good. The config now reads from the environment, injected at runtime from a secrets manager. Nothing sensitive touches the tree.
# .env stays OUT of git
echo '.env' >> .gitignore
# app reads from the environment, value lives in the secrets manager
export API_TOKEN="$(vault kv get -field=token secret/app)"
4. Make it impossible to repeat. A pre-commit hook scans every staged change so a secret never reaches a commit in the first place.
# .pre-commit-config.yaml
# - repo: https://github.com/gitleaks/gitleaks
# rev: v8.18.0
# hooks: [{ id: gitleaks }]
pre-commit install
gitleaks detect --source . --verbose
Why it happened
No villain here. Just the most ordinary mistake in the trade.
Months ago, mid-deploy, I needed the thing to work now. I hardcoded the credential “just to test,” told myself I’d pull it out before committing, got interrupted, and committed everything with a git add .. The TODO never came back. The repo was private, so it felt safe — and “private” is exactly the lie that lets these things rot in history for half a year.
Private isn’t a control. It’s a setting someone can flip.
Takeaways
- A secret is burned the moment it’s committed. Rotate first, always. Treat the value as public from commit time forward — because effectively it is.
- Deleting the file is not remediation. The credential lives in every old commit. You must rewrite history (
git filter-repo/ BFG) and force-push, or it’s still there. - Secrets belong in a manager, not the tree. Env-injected values,
.gitignoreyour.env, and never let credentials and code share a home. - Automate the catch. A
gitleaks/trufflehogpre-commit hook stops the next leak before it’s a commit — humans forget, hooks don’t. - “Private repo” is not a security boundary. Audit your own repos like you’d audit a stranger’s. The author line might surprise you.