Design a bot detection system

A bot here is any client driven by software rather than a person: scrapers, credential stuffers, scalpers buying limited stock, fake sign-up farms. Some bots are wanted (search crawlers, uptime monitors, partner APIs), so the system must also verify good bots, usually by reverse-then-forward DNS of the crawler IP or a published IP list.

Functional requirements

1. Score every HTTP request to protected routes (login, sign-up, checkout, search) for likelihood of automation.
2. Act on the score per route: allow, silently challenge, interactively challenge, rate limit, or block.
3. Let verified good bots through and let product teams set per-route policy without a deploy.
4. Feed outcomes (challenge results, fraud labels, support reports) back into rules and models.

Non-functional requirements

1. 300k requests/s peak across 200 edge points of presence (PoPs).
2. Edge decision adds < 1 ms p99; the site must not slow down for humans.
3. False positive rate on humans < 0.1 % on checkout, because a blocked buyer is lost revenue.
4. Collect the minimum personal data, with documented retention.

Back of the envelope

peak requests              300,000 / s across 200 PoPs  = 1,500 / s per PoP on average
edge scoring CPU           0.5 ms per request
  -> 300,000 x 0.5 ms      = 150 core-seconds per second = 150 cores fleet-wide
request metadata to async  400 B per request (JA4, H2 fp, IP, ASN, route, status, cookie id)
  -> 300,000 x 400 B       = 120 MB/s ≈ 10.4 TB/day raw
aggregate state            50 M distinct IPs/day x 100 B ≈ 5 GB  (fits in memory per region)
model push to edge         every 5 min, a few MB of lists + weights

Signal families

JA4 in one paragraph

JA3 (2017) hashed the raw ClientHello fields with MD5. When Chrome began randomising TLS extension order in early 2023, every Chrome connection got a different JA3. JA4 (FoxIO, 2023) fixes that by sorting ciphers and extensions before hashing and by putting readable fields up front. A JA4 has three parts separated by underscores: a readable prefix, a truncated SHA-256 of the sorted cipher list, and a truncated SHA-256 of the sorted extensions plus signature algorithms.

JA4 = t13d1516h2_8daaf6152771_e5627efa2ab1
       | | | | | |  |            |
       | | | | | |  |            +-- sha256(sorted extensions + sig algs)[:12]
       | | | | | |  +--------------- sha256(sorted cipher suites)[:12]
       | | | | | +------------------ ALPN first and last char: "h2"
       | | | | +-------------------- 16 extensions
       | | | +---------------------- 15 cipher suites
       | | +------------------------ d = SNI is a domain (i = IP, no SNI)
       | +-------------------------- TLS 1.3
       +---------------------------- t = TCP (q = QUIC)

request:  User-Agent: Chrome/131 on Windows
          JA4 seen:   t13d1312h2_...   (a Go net/http stack, not Chrome)
verdict:  claimed browser and TLS stack disagree -> strong automation signal

Two scoring tiers

• Edge (synchronous): features available from this request alone plus pushed state: JA4 and HTTP/2 fingerprint, UA consistency, IP and ASN reputation, cookie validity, a small gradient-boosted model or weighted rules. Budget 0.5 ms CPU, no network calls.
• Async (regional stream): aggregates over windows: requests per JA4 per ASN per minute, login failures per fingerprint, distinct accounts per device cookie, challenge pass rates. Produces lists (fingerprint plus ASN pairs to challenge, IP ranges to rate limit) and refreshed weights pushed to the edge every few minutes.
• Session cookie: once a client passes a challenge, the edge issues a signed, short-lived cookie (HMAC over cookie ID, JA4, IP prefix, expiry) so it is not challenged on every request. Binding to JA4 stops a solved cookie being replayed from a different stack.

The challenge ladder

A block teaches the attacker exactly what tripped. A ladder costs bots more than humans and gives you labels at each rung.

Thresholds are per route: search can tolerate more automation than checkout. Private Access Tokens (the Privacy Pass protocol, RFC 9576 to 9578) let a device attest "a real device solved a check" through an issuer without revealing identity, which removes the CAPTCHA for most Apple users.

Edge scoring logic

type Action = "allow" | "silent" | "interactive" | "block";

interface EdgeFeatures {
  ja4: string;
  uaFamily: "chrome" | "firefox" | "safari" | "other";
  asnType: "residential" | "mobile" | "hosting" | "unknown";
  ipRep: number;            // 0 (clean) .. 1 (known bad), from pushed list
  hasValidClearance: boolean;
}

const JA4_RE = /^[tqd][0-9a-z]{2}[di][0-9]{4}[0-9a-z]{2}_[0-9a-f]{12}_[0-9a-f]{12}$/;

export function score(f: EdgeFeatures, knownBrowserJa4: Map<string, string>, listed: Set<string>): number {
  if (!JA4_RE.test(f.ja4)) return 90;                       // malformed or missing: treat as hostile
  if (f.hasValidClearance) return 10;
  let s = 20 * f.ipRep;
  if (f.asnType === "hosting") s += 25;
  const family = knownBrowserJa4.get(f.ja4);                 // ja4 -> browser family it belongs to
  if (family === undefined) s += 15;                          // unseen stack: new browser build or a tool
  else if (f.uaFamily !== "other" && family !== f.uaFamily) s += 40; // claims a browser, TLS says otherwise
  if (listed.has(f.ja4 + "|" + f.asnType)) s += 30;          // async tier flagged this pair
  return Math.min(100, Math.max(0, Math.round(s)));
}

export function decide(s: number, route: { silent: number; interactive: number; block: number }): Action {
  if (s >= route.block) return "block";
  if (s >= route.interactive) return "interactive";
  if (s >= route.silent) return "silent";
  return "allow";
}

Privacy

• Behavioural and device signals are personal data under GDPR when tied to a user. Collect them for security under legitimate interest, document it, and keep raw pointer traces only as derived features, not recordings.
• Hash IPs with a rotating keyed hash in the analytics tier; keep raw IPs at the edge for minutes, not months.
• Retain request metadata 30 days for training and investigations, aggregates longer.

Evasion and how the design answers it

Failure modes

Labels and feedback loops

• Weak labels arrive fast: challenge pass or fail, PoW completed, cookie reuse patterns.
• Strong labels arrive slowly: accounts later confirmed fraudulent, chargebacks on scalped orders, users reporting takeover.
• Selection bias: blocked traffic never shows its true outcome. Keep a small random holdout (for example 1 % of mid-band traffic is challenged instead of blocked) so you can measure what you would have caught.
• Adversarial drift: watch the score distribution per route daily; a sudden shift towards the middle bands usually means a new evasion.

Trade-offs to state