Design a secrets scanner

Design the scanner that finds leaked credentials in code and git history: provider patterns with checksums, Shannon entropy as a gate rather than a verdict, keyed fingerprints instead of stored secrets, suppressions that leave an audit trail, incremental history scans that never skip a commit, and safe verification of whether a key is still live.

● Medium🔒 PROBy ·Design problem·32 min read·Intermediate → Advanced
secrets scanningShannon entropygitleakstrufflehoggit history scanTypeScript

01Understand the problem

✓What you will learn
  • Combine provider patterns (prefix plus checksum) with Shannon entropy for unknown formats
  • Store a keyed fingerprint, never the secret, so findings dedupe without becoming a second leak
  • Scan git history incrementally without skipping commits pushed during a scan
  • Verify liveness with a read-only identity call, bounded and cached, without turning the scanner into an exfiltration tool
◎Interview radar

What they probe: how you keep false positives low enough that developers do not mute the tool, why entropy alone fails, where the secret lives after a hit (it must not land in logs or the findings database), how history scanning stays incremental, and how verification avoids calling an attacker's endpoint. Where candidates slip: storing raw matches, a global regex reused across calls, scanning only HEAD, verifying against a URL found in the same file.

A secrets scanner reads text (a commit, a file, a CI log) and reports strings that look like credentials: cloud keys, API tokens, private keys, passwords in config. Tools such as gitleaks and trufflehog do this. The hard part is not matching; it is precision. A scanner that flags every hex string gets ignored within a week.

  1. scanText(file, text) returns candidates with detector id, line and a confidence grade.
  2. Known providers are matched by prefix and structure, and by checksum where the format has one.
  3. Unknown formats are caught by an assignment pattern (password = "...") gated by entropy.
  4. Findings leave the process with a fingerprint (HMAC of the secret) and a redacted hint; the raw secret is never persisted.
  5. History scans process only commits not reachable from the last scanned tips, and checkpoint after success.
  6. Optional verification calls the provider's fixed identity endpoint, with a concurrency cap and a timeout.
NumberValueSo what
Monorepo history400k commits, 30 GB of diffsA full rescan takes hours: incremental is a requirement, not an optimisation
Pre-receive hook budgetabout 2 s per pushPrefilter with literal keywords before any regex runs
Detectors100 to 800 patternsMost lines contain none of the keywords, so most regexes never run
Precision targetabove 90 percent on high-confidence findingsBelow that, developers add blanket suppressions

This is an advanced module

Problems: detection & enforcement components is part of the Pro pass. 4 more sections — the deep dives, the final design, follow-ups and the test — unlock with it.

Already a cyart.io or studio.cyart.io member? Your pass code is in your Studio account.