Design a secure file upload service
Design the upload pipeline behind a product that accepts files from the public: direct-to-storage uploads into a quarantine bucket, type detection by content, sandboxed scanning and re-encoding, safe storage keys, and serving user content from a separate domain so an uploaded file can never run as your site.
01Understand the problem and size it
- Upload directly to object storage with constrained presigned requests into a quarantine area
- Decide a file's type by its content, not its name or the client's
Content-Type - Process files in a sandbox: scan, re-encode images, limit archives and decompression
- Serve user files from a separate domain with headers that stop them executing as your site
What they probe: where bytes flow, how big files are bounded, what stops an uploaded HTML or SVG file running script on your origin, how image parsers are kept from becoming the breach, what happens to a file before scanning finishes. Where candidates slip: trusting the file extension, using the user's file name as a storage path, serving uploads from the main domain, scanning in the web server process.
Accepting files from the public means running untrusted bytes through parsers (image libraries, PDF readers, unzip) and then handing those bytes to other users' browsers. Each step has a classic failure: path traversal through file names, stored XSS through HTML or SVG uploads, polyglot files that are valid as two types at once, decompression bombs, and memory-corruption bugs in image libraries such as the 2016 ImageMagick flaws known as ImageTragick.
Functional requirements
- Signed-in users upload images and documents (PDF, Office) up to 50 MB.
- Files are scanned and normalised before anyone else can view them.
- Images are shown inline; documents are downloaded.
- Owners can delete files; admins can remove abusive content.
Non-functional requirements
- 2 M uploads a day, average 3 MB.
- A file is available within 30 s p95 of upload.
- No uploaded content ever executes in the context of the main site.
- Processing compromise does not reach the rest of the system.
uploads 2,000,000 / day / 86,400 s ≈ 23 / s average, ~116 / s peak (x5)
volume 2 M x 3 MB = 6 TB / day in
processing ~2 s per file (scan + re-encode)
-> concurrent workers 23 x 2 ≈ 46 average, 116 x 2 ≈ 232 at peak
API tier bandwidth ~0: bytes go straight to object storage
quarantine retention 24 h for files that never pass02High-level design
- Client calls
POST /uploadswith the intended size and type. The API checks quota and returns a presigned POST to the quarantine bucket with a server-generated key (a random ID), acontent-length-rangecondition up to 50 MB, and a 5-minute expiry. - The client uploads directly to storage. The API never handles the bytes.
- An object-created event goes to a queue. A worker fetches the file into an isolated sandbox.
- The worker detects the type from magic bytes and a real parse, checks it against an allow list, scans it, and re-encodes images (which drops metadata and anything hidden after the image data).
- The cleaned file is written to the clean bucket; the database row moves from
pendingtoready. Failures move torejectedwith a reason the user can see. - Downloads go through the API, which checks permission and returns a short presigned GET, or through the user-content domain.
| Never trust | Because | Use instead |
|---|---|---|
| File name | Path traversal (../../), control characters, misleading extensions | Random storage key; original name kept as sanitised metadata only |
Client Content-Type | The client sets it | Detected type from content |
| Extension | invoice.pdf.html or a renamed executable | Detected type, and served type set by you |
03Deep dives: processing safely and serving safely
Processing in a sandbox
- Workers run in a sandbox (gVisor, Firecracker microVMs or similar) with no network access except to fetch and write their one object, a read-only filesystem, and CPU, memory and time limits.
- Check dimensions before decoding: a small PNG can declare 50,000 x 50,000 pixels, which is 2.5 billion pixels and about 10 GB of RGBA memory when decoded. Reject above a pixel cap.
- Archives: cap file count, nesting depth and total uncompressed size, and stop at a compression-ratio limit. A zip bomb of a few kilobytes can expand to gigabytes.
- Re-encoding an image to a fresh PNG or JPEG keeps the picture and drops everything else: EXIF location data, trailing payloads, polyglot structures.
- Documents: run antivirus and, where the risk justifies it, content disarm and reconstruction (CDR), which rebuilds the document without macros or active content.
PNG header declares 50,000 x 50,000 pixels
decoded size 50,000 x 50,000 x 4 bytes (RGBA) = 10,000,000,000 B ≈ 10 GB
file on disk a few hundred kB (compresses extremely well)
rule read the header, reject if width x height > 40 M pixels, before decodingServing user content
| Control | What it stops |
|---|---|
| Separate registrable domain for user content (as large providers do with dedicated user-content domains) | An uploaded HTML or SVG file running script with your main site's cookies and origin |
Content-Disposition: attachment for anything that is not a known-safe image | Browser rendering documents inline |
X-Content-Type-Options: nosniff and an explicit Content-Type | Browsers guessing that a text file is HTML |
Content-Security-Policy: sandbox on user-content responses | Script execution even if a file is rendered |
| SVG converted to PNG, or served only as attachment | SVG is XML that can contain script |
Serving uploads from files.example.com still shares cookies scoped to example.com and is same-site for many browser rules. Use a different registrable domain, such as example-usercontent.com.
04Failure modes and trade-offs
| Failure | Behaviour | Why |
|---|---|---|
| Scanner backlog | Files stay pending; UI says "processing"; queue autoscaling on depth | Never serve unscanned files to avoid a delay |
| Scanner signature update fails | Alert; keep processing with the last signatures; re-scan recent files when fixed | Stale signatures are better than none |
| Worker compromise via parser bug | Sandbox with no network and no credentials beyond one object | The parser is the most likely breach point |
| Malware found after release | Re-scan on signature updates for recent files; quarantine and notify | Detection improves after upload |
| Abandoned uploads | Lifecycle rule deletes quarantine objects after 24 h | Cost and clutter |
| Choice | Gain | Cost |
|---|---|---|
| Re-encode images | Removes hidden payloads and metadata | Slight quality loss; animated formats need care |
| Block until scanned | Nothing unsafe is ever served | Seconds of delay before a file is visible |
| CDR for documents | Strips macros and active content | Can break complex documents; costs CPU or licence fees |
05Final design, follow-ups and self-test
"Clients upload straight to a quarantine bucket with a presigned POST that fixes the key, caps the size at 50 megabytes and expires in five minutes, so the API never touches the bytes and the user never names the storage path. An event queues the file for a sandboxed worker with no network, which decides the type from the content, checks pixel dimensions before decoding, scans, and re-encodes images to strip anything hidden. Only then does it land in the clean bucket. Everything is served from a separate registrable domain with attachment, nosniff and a sandbox CSP, so even a file we got wrong cannot run as our site."
Q1How do you handle a 5 GB video upload?+
Multipart upload with presigned parts, resumable, with the size limit enforced by the API when it issues parts and checked again on completion. Processing runs as a longer job with progress reporting.
Q2Why not scan in the web server as the file arrives?+
It puts a complex parser and large memory use in the process that holds user sessions and database credentials, and it ties request threads to slow work.
Q3What is a polyglot file?+
A file that is valid as two formats at once, such as an image that is also a valid script or archive. Detecting by one format and serving it where another interpretation applies is how it attacks; re-encoding destroys it.
Users can upload SVG avatars, which are served from `https://example.com/avatars/`. What is the main risk?