Design a secrets scanner
Design the scanner that finds leaked credentials in code and git history: provider patterns with checksums, Shannon entropy as a gate rather than a verdict, keyed fingerprints instead of stored secrets, suppressions that leave an audit trail, incremental history scans that never skip a commit, and safe verification of whether a key is still live.
01Understand the problem
- Combine provider patterns (prefix plus checksum) with Shannon entropy for unknown formats
- Store a keyed fingerprint, never the secret, so findings dedupe without becoming a second leak
- Scan git history incrementally without skipping commits pushed during a scan
- Verify liveness with a read-only identity call, bounded and cached, without turning the scanner into an exfiltration tool
What they probe: how you keep false positives low enough that developers do not mute the tool, why entropy alone fails, where the secret lives after a hit (it must not land in logs or the findings database), how history scanning stays incremental, and how verification avoids calling an attacker's endpoint. Where candidates slip: storing raw matches, a global regex reused across calls, scanning only HEAD, verifying against a URL found in the same file.
A secrets scanner reads text (a commit, a file, a CI log) and reports strings that look like credentials: cloud keys, API tokens, private keys, passwords in config. Tools such as gitleaks and trufflehog do this. The hard part is not matching; it is precision. A scanner that flags every hex string gets ignored within a week.
scanText(file, text)returns candidates with detector id, line and a confidence grade.- Known providers are matched by prefix and structure, and by checksum where the format has one.
- Unknown formats are caught by an assignment pattern (
password = "...") gated by entropy. - Findings leave the process with a fingerprint (HMAC of the secret) and a redacted hint; the raw secret is never persisted.
- History scans process only commits not reachable from the last scanned tips, and checkpoint after success.
- Optional verification calls the provider's fixed identity endpoint, with a concurrency cap and a timeout.
| Number | Value | So what |
|---|---|---|
| Monorepo history | 400k commits, 30 GB of diffs | A full rescan takes hours: incremental is a requirement, not an optimisation |
| Pre-receive hook budget | about 2 s per push | Prefilter with literal keywords before any regex runs |
| Detectors | 100 to 800 patterns | Most lines contain none of the keywords, so most regexes never run |
| Precision target | above 90 percent on high-confidence findings | Below that, developers add blanket suppressions |
This is an advanced module
Problems: detection & enforcement components is part of the Pro pass. 4 more sections — the deep dives, the final design, follow-ups and the test — unlock with it.
Already a cyart.io or studio.cyart.io member? Your pass code is in your Studio account.