Design a secure file sharing service
Design a file sharing service for companies: chunked resumable uploads up to 50 GB, encryption options from server-side KMS to client-side keys in the URL fragment, scoped and expiring share links, malware scanning, DLP hooks and an audit trail, with the trade-offs between privacy and inspection stated plainly.
01Understand the problem and size it
- Choose between server-side, client-side and link-key encryption and say what each one costs in scanning and DLP
- Design share links with unguessable tokens, scope, expiry, revocation and download limits
- Handle large files with chunked, resumable uploads and per-chunk integrity
- Plug in malware scanning, DLP and audit without blocking the upload path
What they probe: how a share link is made unguessable and revocable, who can decrypt files, how scanning works if files are client-side encrypted, how a 50 GB upload survives a dropped connection, what is logged. Where candidates slip: sequential or short link IDs, permanent public URLs to object storage, claiming end-to-end encryption and DLP scanning at the same time, scanning synchronously in the upload request.
Functional requirements
- Users upload files and folders up to 50 GB each, resumable across network drops.
- Users share by link with scope (anyone with the link, signed-in members of the organisation, named people), expiry, optional password and download limit; links can be revoked.
- Files are scanned for malware and checked by DLP policy before anyone else can download them.
- Every upload, share, access and admin action is audited.
Non-functional requirements
- 10 M users, 2 M uploads a day, average 20 MB.
- Upload throughput limited by the client's link, not by us.
- Link tokens unguessable; no file reachable without an authorisation check.
- Optional mode where the service cannot read files, for the most sensitive tenants.
Back of the envelope
uploads 2 M / day x 20 MB = 40 TB / day ingest
average bandwidth 40 TB / 86,400 s ≈ 463 MB/s ≈ 3.7 Gbit/s, ~11 Gbit/s peak (x3)
storage growth 40 TB x 365 ≈ 14.6 PB / year before expiry and dedup
chunk size 8 MiB
largest file 50 GB / 8 MiB ≈ 5,961 parts (S3-style limit: 10,000 parts, min 5 MiB except last)
scan jobs 2 M / day ≈ 23 / s average
share links 5 M active x 300 B = 1.5 GB of metadata
audit events ~20 M / day x 500 B = 10 GB / day02High-level design
- ACTOR
- EDGE
- SERVICE
- DATA
- STREAM
- Live flow
- Lookup / async
Encryption options
| Mode | Who holds the key | Malware scan and DLP | Recovery | Fits |
|---|---|---|---|---|
| Server-side envelope encryption | Service, per-file data key wrapped by KMS | Full, on the server | Admin can recover | Default for most companies |
| Customer-managed keys (BYOK/HYOK) | Customer's KMS key wraps the data keys | Full, while the customer key is available | Customer can revoke all access | Regulated tenants |
| Client-side, keys shared with recipients | Only users' devices; data key wrapped to each recipient's public key | Only on the client, or not at all | Lost keys mean lost files | Zero-access tenants |
| Link key in the URL fragment | Whoever has the full link | None on the server | None | One-off anonymous sends |
The URL fragment trick works because the part after # is never sent to the server in an HTTP request. A link like https://share.cyart.io/s/Q7m...#k=3vT... lets the browser fetch ciphertext by the path and decrypt with the key in the fragment. Anyone holding the link holds the key, so the link itself must be treated as the secret.
Chunked, resumable upload
POST /v1/uploads { name, size: 21474836480, sha256, folderId }
-> 201 { uploadId, chunkSize: 8388608, parts: 2560 }
POST /v1/uploads/{id}/parts:sign { partNumbers: [1..100] }
-> { urls: [{ part: 1, url: "https://store.../part1?X-Amz-Expires=900&X-Amz-Signature=..." }, ...] }
PUT <presigned url> (8 MiB body, Content-MD5 / checksum header) -> ETag
GET /v1/uploads/{id} -> { uploadedParts: [1,2,...,1874] } # resume after a drop
POST /v1/uploads/{id}:complete { parts: [{ part, etag, sha256 }] }
-> 202 { fileId, state: "scanning" }- Chunks go directly to object storage with presigned URLs valid 15 minutes, so API servers never proxy 40 TB a day.
- The client uploads several parts in parallel, records which succeeded, and resumes by asking which parts the server has.
- On complete, the server verifies part checksums and the whole-file hash, then moves the file to
scanning. Incomplete uploads are aborted after 24 h so orphan parts do not cost storage. - For client-side encryption, each chunk is encrypted with AES-256-GCM under the file key, with a nonce built from a random prefix, the chunk index and a last-chunk flag (the STREAM construction), so chunks cannot be reordered, dropped or the file truncated without detection.
03Deep dives: share links, scanning, DLP, audit
Share links
import { createHash, randomBytes, timingSafeEqual } from "node:crypto";
type Scope = "anyone" | "org" | "named";
interface ShareLink {
tokenHash: Buffer; fileId: string; scope: Scope; allowed: string[]; // emails for "named"
expiresAt: number; maxDownloads: number | null; downloads: number; revoked: boolean; orgId: string;
}
export function newToken(): { token: string; tokenHash: Buffer } {
const token = randomBytes(16).toString("base64url"); // 128 bits, 22 chars
return { token, tokenHash: createHash("sha256").update(token).digest() };
}
export function authorise(
link: ShareLink | undefined, token: string,
viewer: { email?: string; orgId?: string } | null, now: number,
): { ok: true } | { ok: false; status: 404 | 403 } {
if (!/^[A-Za-z0-9_-]{22}$/.test(token) || !link) return { ok: false, status: 404 };
const h = createHash("sha256").update(token).digest();
if (!timingSafeEqual(h, link.tokenHash)) return { ok: false, status: 404 };
if (link.revoked || now >= link.expiresAt) return { ok: false, status: 404 }; // do not reveal it existed
if (link.maxDownloads !== null && link.downloads >= link.maxDownloads) return { ok: false, status: 404 };
if (link.scope === "org" && viewer?.orgId !== link.orgId) return { ok: false, status: 403 };
if (link.scope === "named" && !(viewer?.email && link.allowed.includes(viewer.email.toLowerCase()))) {
return { ok: false, status: 403 };
}
return { ok: true }; // caller increments downloads atomically, logs, then issues a 5-minute presigned GET
}- 128-bit random tokens, stored as hashes, so a database leak does not leak working links.
- The link points at the API, never at object storage. After the check, the API returns a presigned GET valid for minutes, so revocation takes effect quickly.
- Download limits need an atomic increment (conditional update) or two concurrent downloads both pass a limit of one.
- Optional link passwords are hashed with Argon2id and rate limited per link.
Malware scanning
- A completed upload emits an event; scanners fetch the object, check the hash against known bad, run AV engines and unpack archives with limits (depth, file count, expanded size) to defuse zip bombs.
- Until the verdict, the file is visible to its owner but not downloadable by others. Scan at about 23 files a second average, with large files in their own queue so one 50 GB file does not delay everyone.
- Re-scan popular shared files when signatures update, because a file clean yesterday can be detected today.
- Encrypted archives and client-side encrypted files cannot be scanned: policy decides (block external sharing, warn recipients, or allow for zero-access tenants with that risk accepted in writing).
DLP hooks
- Run content inspection after the scan: patterns with validation (card numbers checked with the Luhn algorithm, national ID formats), document fingerprints, sensitivity labels already in the file's metadata.
- Evaluate at share time too: the same file can be fine to share internally and blocked to an external address. Policy outcome: allow, warn and require a reason, require approval, or block.
- Offer a webhook or ICAP interface so a tenant's existing DLP engine can return the verdict; time out to a default the tenant chooses.
Audit
Log upload, share created or changed, link accessed (token ID, never the token; IP; user agent; scope result), download, revoke, admin key actions. Send to an append-only store and the tenant's SIEM. Access logs for anonymous links are the only way to answer "who downloaded our contract", so they are a product feature, not just compliance.
A permanent public URL to the bucket bypasses expiry, revocation, download limits, DLP and audit at once. Every download goes through an authorisation check that issues a presigned URL valid for minutes.
04Failure modes and trade-offs
| Failure | Behaviour | Why |
|---|---|---|
| Scanner backlog | Files stay in scanning; owner sees status; never skip to clean | An unscanned file shared widely is the attack |
| KMS unavailable | Cached data keys for in-flight downloads; new decrypts fail closed | No plaintext fallback |
| Upload interrupted at 70 % | Client resumes from the server's part list | No re-upload of 35 GB |
| Link leaked publicly | Owner revokes; existing presigned URLs expire within 5 minutes; audit shows who fetched | Short presign lifetime bounds the window |
| DLP webhook slow | Share waits up to a timeout, then applies the tenant's default (usually block external) | Fail to the tenant's chosen side, explicitly |
Trade-offs to state
| Choice | Gain | Cost |
|---|---|---|
| Server-side encryption | Scanning, DLP, search, previews, recovery | Service and its admins can read files |
| Client-side encryption | Service cannot read; breach exposes ciphertext | No server scanning or DLP, no previews, key loss is data loss |
| Short presigned URLs | Revocation is fast | More API calls for long downloads; clients re-request |
| Blocking share until scanned | No malware spread via links | Delay before a fresh upload can be shared |
| Dedup across users | Storage savings | Leaks whether a file exists elsewhere; never dedup encrypted or across tenants |
"End-to-end encrypted" and "we scan every file for malware and DLP on the server" cannot both be true. Offer them as tenant modes and say which controls each mode gives up.
05Final design, follow-ups and self-test
"Uploads go straight to object storage in 8 MiB presigned parts, in parallel and resumable, verified by checksum on completion, so 40 terabytes a day never touches the API tier. Each file gets its own data key wrapped by KMS, which lets us scan for malware and run DLP before anyone else can download it. Share links are 128-bit random tokens stored hashed, with scope, expiry, revocation and download limits checked on every download, and the API only ever returns a presigned URL valid for a few minutes. For tenants who need zero access, we offer client-side encryption and say plainly that server-side scanning and DLP go away in that mode."
Q1How would you let a recipient outside the company receive a zero-access file?+
Wrap the file key to a key derived from a one-time secret sent out of band, or put the key in the URL fragment and send the link through a separate channel. Either way the link or secret is the credential.
Q2How do you prevent a user abusing the service to host malware for phishing?+
Scan before shareable, rate limit anonymous links per account, report-abuse links on download pages, reputation for new accounts, and take down by revoking links, which works because every download goes through the API.
Q3Why store link tokens hashed if they are random?+
So a read-only database leak (backup, replica, SQL injection) does not hand out working links. The token only exists in the URL the user holds.
Q4How do you make downloads of 50 GB files resumable?+
Range requests on presigned GETs; the client re-requests a fresh presigned URL through the API when one expires, which re-checks authorisation.
A tenant wants files encrypted so the service cannot read them, and also wants server-side DLP to block credit card numbers being shared externally. What is the honest design answer?
Related lessons
- Data protection: KMS, envelope encryption, tokenisationSecEng · Cloud & Platform SecurityPro
- Cryptography you actually needSecEng · Foundations of Security EngineeringFree
- Design a key management serviceHLD · Design problems: Infrastructure & dataPro
- The round, the framework, the numbersHLD · The security HLD roundFree