Design a secure file upload service

Design the upload pipeline behind a product that accepts files from the public: direct-to-storage uploads into a quarantine bucket, type detection by content, sandboxed scanning and re-encoding, safe storage keys, and serving user content from a separate domain so an uploaded file can never run as your site.

● MediumFREEBy ·Design problem·13 min read·Beginner → Intermediate
file uploadmalware scanningcontent disarmpresigned URLspolyglot filesuser content domain

01Understand the problem and size it

✓What you will learn
  • Upload directly to object storage with constrained presigned requests into a quarantine area
  • Decide a file's type by its content, not its name or the client's Content-Type
  • Process files in a sandbox: scan, re-encode images, limit archives and decompression
  • Serve user files from a separate domain with headers that stop them executing as your site
◎Interview radar

What they probe: where bytes flow, how big files are bounded, what stops an uploaded HTML or SVG file running script on your origin, how image parsers are kept from becoming the breach, what happens to a file before scanning finishes. Where candidates slip: trusting the file extension, using the user's file name as a storage path, serving uploads from the main domain, scanning in the web server process.

Accepting files from the public means running untrusted bytes through parsers (image libraries, PDF readers, unzip) and then handing those bytes to other users' browsers. Each step has a classic failure: path traversal through file names, stored XSS through HTML or SVG uploads, polyglot files that are valid as two types at once, decompression bombs, and memory-corruption bugs in image libraries such as the 2016 ImageMagick flaws known as ImageTragick.

Functional requirements

  1. Signed-in users upload images and documents (PDF, Office) up to 50 MB.
  2. Files are scanned and normalised before anyone else can view them.
  3. Images are shown inline; documents are downloaded.
  4. Owners can delete files; admins can remove abusive content.

Non-functional requirements

  1. 2 M uploads a day, average 3 MB.
  2. A file is available within 30 s p95 of upload.
  3. No uploaded content ever executes in the context of the main site.
  4. Processing compromise does not reach the rest of the system.
Sizingtxt
uploads           2,000,000 / day / 86,400 s     ≈ 23 / s average, ~116 / s peak (x5)
volume            2 M x 3 MB                      = 6 TB / day in
processing        ~2 s per file (scan + re-encode)
  -> concurrent workers   23 x 2 ≈ 46 average, 116 x 2 ≈ 232 at peak
API tier bandwidth        ~0: bytes go straight to object storage
quarantine retention      24 h for files that never pass
▭On the whiteboard · Scope3 points
01
FR
upload images + docs to 50 MB | scan + normalise before visible | inline images, downloaded docs | delete
02
NFR
2 M/day, 6 TB/day | ready < 30 s p95 | no execution on main origin | sandboxed processing
03
out
sharing and permissions model, end-to-end encryption

02High-level design

  1. Client calls POST /uploads with the intended size and type. The API checks quota and returns a presigned POST to the quarantine bucket with a server-generated key (a random ID), a content-length-range condition up to 50 MB, and a 5-minute expiry.
  2. The client uploads directly to storage. The API never handles the bytes.
  3. An object-created event goes to a queue. A worker fetches the file into an isolated sandbox.
  4. The worker detects the type from magic bytes and a real parse, checks it against an allow list, scans it, and re-encodes images (which drops metadata and anything hidden after the image data).
  5. The cleaned file is written to the clean bucket; the database row moves from pending to ready. Failures move to rejected with a reason the user can see.
  6. Downloads go through the API, which checks permission and returns a short presigned GET, or through the user-content domain.
Never trustBecauseUse instead
File namePath traversal (../../), control characters, misleading extensionsRandom storage key; original name kept as sanitised metadata only
Client Content-TypeThe client sets itDetected type from content
Extensioninvoice.pdf.html or a renamed executableDetected type, and served type set by you
▭On the whiteboard · Core3 points
01
ingest
presigned POST, server key, size range, 5 min, quarantine bucket
02
process
queue → sandbox worker: detect, allow list, scan, re-encode
03
serve
clean bucket, separate domain, attachment for documents

03Deep dives: processing safely and serving safely

Processing in a sandbox

  • Workers run in a sandbox (gVisor, Firecracker microVMs or similar) with no network access except to fetch and write their one object, a read-only filesystem, and CPU, memory and time limits.
  • Check dimensions before decoding: a small PNG can declare 50,000 x 50,000 pixels, which is 2.5 billion pixels and about 10 GB of RGBA memory when decoded. Reject above a pixel cap.
  • Archives: cap file count, nesting depth and total uncompressed size, and stop at a compression-ratio limit. A zip bomb of a few kilobytes can expand to gigabytes.
  • Re-encoding an image to a fresh PNG or JPEG keeps the picture and drops everything else: EXIF location data, trailing payloads, polyglot structures.
  • Documents: run antivirus and, where the risk justifies it, content disarm and reconstruction (CDR), which rebuilds the document without macros or active content.
Why you check dimensions firsttxt
PNG header declares 50,000 x 50,000 pixels
decoded size          50,000 x 50,000 x 4 bytes (RGBA) = 10,000,000,000 B ≈ 10 GB
file on disk          a few hundred kB (compresses extremely well)
rule                  read the header, reject if width x height > 40 M pixels, before decoding

Serving user content

ControlWhat it stops
Separate registrable domain for user content (as large providers do with dedicated user-content domains)An uploaded HTML or SVG file running script with your main site's cookies and origin
Content-Disposition: attachment for anything that is not a known-safe imageBrowser rendering documents inline
X-Content-Type-Options: nosniff and an explicit Content-TypeBrowsers guessing that a text file is HTML
Content-Security-Policy: sandbox on user-content responsesScript execution even if a file is rendered
SVG converted to PNG, or served only as attachmentSVG is XML that can contain script
!A subdomain is not a separate domain

Serving uploads from files.example.com still shares cookies scoped to example.com and is same-site for many browser rules. Use a different registrable domain, such as example-usercontent.com.

04Failure modes and trade-offs

FailureBehaviourWhy
Scanner backlogFiles stay pending; UI says "processing"; queue autoscaling on depthNever serve unscanned files to avoid a delay
Scanner signature update failsAlert; keep processing with the last signatures; re-scan recent files when fixedStale signatures are better than none
Worker compromise via parser bugSandbox with no network and no credentials beyond one objectThe parser is the most likely breach point
Malware found after releaseRe-scan on signature updates for recent files; quarantine and notifyDetection improves after upload
Abandoned uploadsLifecycle rule deletes quarantine objects after 24 hCost and clutter
ChoiceGainCost
Re-encode imagesRemoves hidden payloads and metadataSlight quality loss; animated formats need care
Block until scannedNothing unsafe is ever servedSeconds of delay before a file is visible
CDR for documentsStrips macros and active contentCan break complex documents; costs CPU or licence fees

05Final design, follow-ups and self-test

▭On the whiteboard · Final design3 points
01
ingest
presigned POST to quarantine, random key, 50 MB range, 5 min expiry
02
process
sandboxed, no network: magic-byte type, allow list, pixel and archive caps, AV, re-encode
03
serve
clean bucket, separate user-content domain, attachment + nosniff + CSP sandbox
“How to say it in the room

"Clients upload straight to a quarantine bucket with a presigned POST that fixes the key, caps the size at 50 megabytes and expires in five minutes, so the API never touches the bytes and the user never names the storage path. An event queues the file for a sandboxed worker with no network, which decides the type from the content, checks pixel dimensions before decoding, scans, and re-encodes images to strip anything hidden. Only then does it land in the clean bucket. Everything is served from a separate registrable domain with attachment, nosniff and a sandbox CSP, so even a file we got wrong cannot run as our site."

Q1How do you handle a 5 GB video upload?+

Multipart upload with presigned parts, resumable, with the size limit enforced by the API when it issues parts and checked again on completion. Processing runs as a longer job with progress reporting.

Q2Why not scan in the web server as the file arrives?+

It puts a complex parser and large memory use in the process that holds user sessions and database credentials, and it ties request threads to slow work.

Q3What is a polyglot file?+

A file that is valid as two formats at once, such as an image that is also a valid script or archive. Detecting by one format and serving it where another interpretation applies is how it attacks; re-encoding destroys it.

Test yourself

Users can upload SVG avatars, which are served from `https://example.com/avatars/`. What is the main risk?

Finished this one?
Tick it and it shows as done on the roadmap and the course.