Design an email security gateway

An email security gateway (SEG) is an SMTP relay that receives a company's inbound mail first, inspects it, and forwards what is safe to the mailbox provider (Exchange Online, Google Workspace). The company points its MX record (the DNS record that tells senders where to deliver mail for a domain) at the gateway.

Functional requirements

1. Receive all inbound mail for the company's domains over SMTP and deliver clean mail to the mailbox provider.
2. Authenticate senders (SPF, DKIM, DMARC, ARC) and detect spoofing, lookalike domains and business email compromise (BEC).
3. Rewrite URLs for time-of-click checks and sandbox unknown attachments.
4. Quarantine suspicious mail with user and admin release, and remediate mail found malicious after delivery.

Non-functional requirements

1. 100,000 mailboxes; added delivery latency < 30 s p95 for mail without attachments needing detonation.
2. Never lose an accepted message: once we answer 250 OK, we own it.
3. Phishing catch rate high, but false positive rate on business mail < 0.01 %.
4. Scale with bursts (Monday 9 a.m., marketing blasts).

Back of the envelope

mailboxes                100,000
inbound per mailbox      100 messages / day (incl. spam)   = 10 M messages / day
average                  10 M / 86,400 s                   ≈ 116 messages / s
peak (x5)                                                  ≈ 580 messages / s
average size             75 kB                             -> 750 GB / day, ~70 MB/s at peak
messages with attachments  20 %                            = 2 M / day
unknown after hash cache + static AV   5 % of those        = 100,000 detonations / day
detonation rate          100,000 / 86,400                  ≈ 1.16 / s
sandbox time             180 s per sample
concurrent VMs           1.16 x 180                        ≈ 208 on average, ~1,040 at peak (x5)
spool (durable queue)    1 h of peak backlog: 580 x 3,600 x 75 kB ≈ 157 GB

1. Connection: TLS (STARTTLS), connecting IP reputation and DNS blocklists, per-IP connection and rate limits. Reject here with a 5xx and the sender's own server generates any bounce.
2. Envelope: valid recipient check (RCPT TO against the directory, so mail to non-existent users is refused, not bounced later), size limits.
3. Spool: write the full message to a replicated durable queue and only then answer 250 OK.
4. Authentication: SPF, DKIM, DMARC, ARC; results stamped into an Authentication-Results header.
5. Content and threat: spam and phishing classifiers, BEC and impersonation checks, URL analysis and rewriting, attachment static analysis, sandbox for unknowns.
6. Policy and delivery: deliver, deliver with banner, quarantine, or drop; relay to the mailbox provider over TLS on a connector that accepts only our IPs.

SPF, DKIM, DMARC: what each one proves

; DNS published by the sender domain
example.com.                  TXT "v=spf1 include:_spf.google.com ip4:203.0.113.0/24 -all"
s1._domainkey.example.com.    TXT "v=DKIM1; k=rsa; p=MIIBIjANBgkqh...IDAQAB"
_dmarc.example.com.           TXT "v=DMARC1; p=reject; rua=mailto:dmarc@example.com; adkim=r; aspf=r"

; Header our gateway adds after evaluation (RFC 8601)
Authentication-Results: mx1.gw.cyart.io;
  spf=pass smtp.mailfrom=bounce.example.com;
  dkim=pass header.d=example.com header.s=s1;
  dmarc=pass (p=reject) header.from=example.com

; A spoof that passes SPF but fails DMARC
MAIL FROM:<x@attacker.net>       -> spf=pass smtp.mailfrom=attacker.net
From: "CEO" <ceo@cyart.io>       -> dkim=none, no aligned domain
                                  -> dmarc=fail (p=reject) header.from=cyart.io

Alignment is the key idea: with relaxed alignment (aspf=r, adkim=r), bounce.example.com aligns with example.com because they share an organisational domain. The attacker in the last example passes SPF for their own domain, but nothing aligns with cyart.io, so DMARC fails and the domain owner's p=reject tells us to refuse it.

URL rewriting and time-of-click

Attackers send links that are clean at delivery and weaponise them hours later, after the gateway has scanned them. So the gateway rewrites every URL to point at its own click service, which checks the destination when the user clicks: reputation, a fresh crawl in an isolated browser, lookalike detection, and the policy for that user.

original : https://docs.vendor-invoices.com/inv/8841
rewritten: https://click.gw.cyart.io/v2/r?u=aHR0cHM6Ly9kb2NzLnZlbmRvci1pbnZvaWNlcy5jb20vaW52Lzg4NDE
           &m=4f2a9c...  (message id)  &r=jdoe  &sig=HMAC-SHA256(key, u|m|r)[:16 bytes, base64url]

click service on GET:
  1. verify sig; reject if invalid       -> never an open redirect
  2. decode u, re-check verdict (cache 5 min), detonate in isolated browser if unknown
  3. clean   -> 302 to original
     unknown -> interstitial warning or open in remote browser isolation
     bad     -> block page, log click by (message, recipient), alert SOC

Attachment sandboxing

• Cheap first: hash lookup against known good and known bad, then static analysis (file type by magic bytes, macros, embedded objects, archive contents). Most attachments never reach the sandbox.
• Detonate unknowns in instrumented VMs (Windows with Office, a PDF reader, a browser) for about 3 minutes, watching for process launches, network calls and file drops. Sized above: around 200 VMs on average, 1,000 at peak, so autoscale and cap per tenant.
• Delay trade-off: holding the whole message for 3 minutes annoys users. Dynamic delivery sends the body immediately with a placeholder and attaches the file once clean.
• Password-protected archives cannot be inspected; try passwords found in the message body, otherwise quarantine with a banner or block by policy.

Quarantine and post-delivery remediation

• Quarantine is a store keyed by message, with the verdict, reasons and the original raw message, kept 30 days.
• Users get a daily digest and can release low-risk categories (bulk, spam); phishing and malware need admin release.
• When a verdict changes after delivery (a URL turns bad, a sandbox result arrives late, a user reports phishing), the gateway pulls the message from mailboxes through the provider's API and records who had opened it or clicked.

False positives without weakening the filter

• Allow-list entries must require authentication: "allow vendor.com when DMARC passes", never "allow From: vendor.com", which any spoofer can write.
• Release requests and "not junk" reports feed a review queue and the classifiers, with per-tenant thresholds.
• Track false positives per rule and per model version; a rule that quarantines invoices from one supplier gets fixed, not allow-listed globally.
• Show users why in the quarantine view ("external sender impersonating an executive") so they can judge.

Failure modes

Trade-offs to state