Design a payment fraud scoring service

A merchant or payment processor sends an authorisation request (card, amount, merchant, device, IP) and must answer approve, decline or challenge (for example 3-D Secure, the card scheme's step-up for online payments) within the card network's time limit. Fraud scoring sits inside that window. A chargeback is the cardholder disputing a transaction through their bank; fraud chargebacks are the main ground truth and can arrive up to around 120 days later under common scheme rules.

Functional requirements

1. Score every authorisation and return approve, decline, review or step-up (3-D Secure) with reason codes.
2. Support rules authored by fraud analysts without a deploy, and ML models trained by data scientists.
3. Compute velocity features over windows from 1 minute to 30 days.
4. Run new models and rules in shadow mode and learn from chargebacks and issuer fraud reports.

Non-functional requirements

1. 10k transactions/s peak, 2k average.
2. 50 ms p99 end to end inside our service.
3. 99.99 % availability, with a defined fallback when scoring is slow or down.
4. Training data is point-in-time correct (no feature from the future).

Back of the envelope

average 2,000 tps x 86,400 s     = 172.8 M transactions / day
peak    10,000 tps
entities per txn                 card, account, device, IP, email, merchant = 6 keys
online feature reads             one batched multi-get per txn -> 10,000 calls / s, 60,000 keys / s
online store size                200 M entities x 1 kB = 200 GB, x3 replicas = 600 GB in memory
decision log                     5 kB per txn (features + scores + decision)
  -> 172.8 M x 5 kB              = 864 GB / day, about 315 TB / year before compression
chargeback rate                  ~0.1 % of transactions -> ~173k fraud labels / day at most

Spending 50 ms

Per-stage p99s do not add exactly (tails rarely coincide), but budgeting as if they do gives a safe plan. Everything slow, such as writing the decision log, updating aggregates, opening cases, happens after the response, asynchronously.

Velocity features

Velocity is how much activity an entity has had recently: transactions per card in the last 10 minutes, total amount per card in 24 hours, distinct cards per device in 1 hour, distinct merchants per card in 24 hours. Fraud is bursty (a stolen card is drained fast), so these are among the strongest features.

• Streaming aggregates (Flink or similar) consume the transaction stream and maintain bucketed counters per entity per window, written to the online store. Lag is typically a second or two.
• The current transaction is not in the stream yet, and neither are ones from the last second. So at score time, read the stored buckets and add the current transaction in-request. For card testing (dozens of small authorisations a second), also keep a tiny synchronous counter per card with an atomic increment.
• Distinct counts (cards per device) use HyperLogLog sketches; exact sets are too big.

// Sliding-window velocity from per-minute buckets kept in the online store.
// buckets: map of epoch-minute -> { count, amountMinor } for one entity, pruned beyond 24 h.
interface Bucket { count: number; amountMinor: number }

const MAX_WINDOW_MIN = 24 * 60;

export function velocity(
  buckets: ReadonlyMap<number, Bucket>,
  nowMs: number,
  windowMin: number,
  current: { amountMinor: number },
): { count: number; amountMinor: number } {
  if (!Number.isInteger(windowMin) || windowMin < 1 || windowMin > MAX_WINDOW_MIN) {
    throw new RangeError("windowMin must be an integer in 1..1440");
  }
  if (!Number.isSafeInteger(current.amountMinor) || current.amountMinor < 0) {
    throw new RangeError("amountMinor must be a non-negative integer");
  }
  const nowMin = Math.floor(nowMs / 60_000);
  let count = 1;                                   // include the transaction being scored
  let amountMinor = current.amountMinor;
  for (let m = nowMin - windowMin + 1; m <= nowMin; m++) {   // at most 1,440 iterations
    const b = buckets.get(m);
    if (b) { count += b.count; amountMinor += b.amountMinor; }
  }
  return { count, amountMinor };
}

Rules and ML as an ensemble

Analysts get a rule language with backtesting (run the rule over last month's labelled data before enabling) and every rule carries an owner and an expiry date. Rules that never expire are how a fraud system ends up declining good customers for an attack that ended two years ago.

Training without leakage

A model trained on features computed today for a transaction from last month sees information the scorer did not have, such as later transactions on the same card. That is label leakage, and it makes offline metrics look far better than production. The fix is to train on the decision log: the exact features served at decision time. For new features, the offline store must support point-in-time joins (feature value as of the transaction timestamp).

Shadow mode, champion/challenger, holdouts

• Shadow mode: the challenger model scores every live transaction alongside the champion; only the champion's score is used. Compare score distributions, decline rates and, as labels mature, catch rates. Shadow costs inference capacity, not risk.
• Champion/challenger: route a small hashed share of traffic (for example 5 % by card) to the challenger's decisions once shadow results look good.
• Holdout: you never learn whether a declined transaction was fraud. Approve a tiny random share of transactions the model would decline (under a strict amount cap) to get unbiased labels. Without it, each model is trained only on what the previous model let through.

Chargeback feedback

• Issuer fraud reports (for example Visa TC40 data) arrive in days and flag transactions the cardholder reported, even without a dispute.
• Chargebacks arrive over weeks to months. Labels are only "mature" for transactions older than about 90 to 120 days, so evaluate models on matured windows and use early signals for monitoring.
• Friendly fraud (a real customer disputing a purchase they made) pollutes labels. Keep reason codes and treat them separately.
• Join labels back to the decision log by transaction ID and write them to the offline store for the next training run.

Failure modes

Trade-offs to state