Design a file upload validator

Q: Why is checking the extension not enough?

The name is user input. shell.php.png passes a naive check, and content can be anything regardless of the name. Content decides; the extension only has to agree.

Design the pipeline every upload passes through: size limits enforced while streaming, type decided by magic bytes rather than extension or Content-Type, zip bomb checks on the central directory including overlapping entries, image re-encoding to kill polyglots and metadata, storage outside the web root under random names, and an asynchronous malware scan with explicit states that gate downloads.

● MediumFREEDesign problem·26 min read·Intermediate → Advanced

file upload securitymagic byteszip bombpolyglot filesmalware scanningunrestricted file upload

01Understand the problem

✓What you will learn

Decide file type from magic bytes and cross-check the extension; treat Content-Type as a hint
Detect zip bombs with ratio, size and entry limits plus an overlap check
Re-encode images to strip metadata and break polyglots
Model the malware scan as states, and serve only clean files from a safe origin

◎Interview radar

What they probe: what you trust (nothing the client says), where the file lands, what stops shell.php.png, how you handle a 42 KB zip that expands to petabytes, and what users see while a scan runs. Where candidates slip: trusting the extension or Content-Type, saving under the original name in a web-served folder, scanning synchronously in the request, serving uploads inline from the main domain.

Unrestricted file upload turns into remote code execution (a script dropped where the server executes it), stored XSS (an HTML or SVG file served from your origin), denial of service (decompression bombs) or malware distribution to other users. The validator is a pipeline: each stage removes one class of problem and the file only becomes downloadable at the end.

Reject at the edge once the stream passes maxBytes (for example 20 MB); never buffer an unbounded body.
Sniff the first bytes; the kind must be on the allow-list for this feature.
The extension must match the sniffed kind; the client's Content-Type is logged, never used.
Zips pass central directory checks; images are re-encoded.
Store under a random key in a private bucket in state pending; a scanner moves it to clean or infected; only clean files are served.

Kind	Magic bytes (hex)	Notes
PNG	89 50 4E 47 0D 0A 1A 0A	8-byte signature
JPEG	FF D8 FF	Start of image marker plus the next marker
GIF	47 49 46 38 37 61 or ...39 61	ASCII GIF87a or GIF89a
WebP	52 49 46 46 at 0, 57 45 42 50 at 8	RIFF container, WEBP form type
PDF	25 50 44 46 2D	ASCII %PDF-
ZIP (and docx, xlsx, jar)	50 4B 03 04, or 50 4B 05 06 when empty	ASCII PK; Office files are zips

02Interfaces first

The contractts

export type Kind = "png" | "jpeg" | "gif" | "webp" | "pdf" | "zip";

export interface ZipLimits { maxEntries: number; maxTotalUncompressed: number; maxRatio: number }
export interface UploadPolicy { allowed: readonly Kind[]; maxBytes: number; zip: ZipLimits }

export interface IncomingFile { originalName: string; declaredType: string; bytes: Buffer }

export type Verdict =
  | { ok: true; kind: Kind; sha256: string; storageKey: string; warnings: string[] }
  | { ok: false; reason: string };

export type ScanState = "pending" | "scanning" | "clean" | "infected" | "failed";

export interface FileRow {
  id: string; ownerId: string; storageKey: string; kind: Kind; size: number; sha256: string;
  displayName: string;          // for UI only, never part of a path
  state: ScanState; attempts: number; version: number;
}

export interface FileStore {
  get(id: string): Promise<FileRow | undefined>;
  /** Compare-and-set on version, so two scanner workers cannot both decide a file's fate. */
  updateIf(id: string, version: number, patch: Partial<FileRow>): Promise<boolean>;
}

iThree signals, one source of truth

The extension is chosen by the user, the Content-Type by the user's browser (or the attacker's script), the magic bytes by the content. Decide on content, require the extension to agree so users are not surprised, and log Content-Type mismatches as a signal. When you serve the file, set Content-Type from the sniffed kind.

03Sniffing, validation and the zip inspector

Sniff and validatets

import { createHash, randomBytes } from "node:crypto";
import { extname } from "node:path";

const at = (b: Buffer, off: number, sig: readonly number[]) => b.length >= off + sig.length && sig.every((x, i) => b[off + i] === x);
const ascii = (s: string) => [...s].map((c) => c.charCodeAt(0));

export function sniff(b: Buffer): Kind | null {
  if (at(b, 0, [0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a])) return "png";
  if (at(b, 0, [0xff, 0xd8, 0xff])) return "jpeg";
  if (at(b, 0, ascii("GIF87a")) || at(b, 0, ascii("GIF89a"))) return "gif";
  if (at(b, 0, ascii("RIFF")) && at(b, 8, ascii("WEBP"))) return "webp";
  if (at(b, 0, ascii("%PDF-"))) return "pdf";
  if (at(b, 0, [0x50, 0x4b, 0x03, 0x04]) || at(b, 0, [0x50, 0x4b, 0x05, 0x06])) return "zip";
  return null;
}

const EXTENSIONS: Record<Kind, readonly string[]> = {
  png: [".png"], jpeg: [".jpg", ".jpeg"], gif: [".gif"], webp: [".webp"], pdf: [".pdf"], zip: [".zip"],
};
export const SERVE_TYPE: Record<Kind, string> = {
  png: "image/png", jpeg: "image/jpeg", gif: "image/gif", webp: "image/webp", pdf: "application/pdf", zip: "application/zip",
};

export function validate(f: IncomingFile, policy: UploadPolicy): Verdict {
  if (f.bytes.length === 0) return { ok: false, reason: "empty file" };
  if (f.bytes.length > policy.maxBytes) return { ok: false, reason: "too large" };   // the edge should already have cut the stream
  const kind = sniff(f.bytes);
  if (!kind || !policy.allowed.includes(kind)) return { ok: false, reason: "content type not allowed" };
  const ext = extname(f.originalName).toLowerCase();                                // "shell.php.png" -> ".png", "x.png.php" -> ".php"
  if (!EXTENSIONS[kind].includes(ext)) return { ok: false, reason: "extension " + (ext || "(none)") + " does not match " + kind + " content" };
  const warnings: string[] = [];
  if (f.declaredType.toLowerCase() !== SERVE_TYPE[kind]) warnings.push("client Content-Type ignored: " + f.declaredType);
  if (kind === "zip") {
    const z = inspectZip(f.bytes, policy.zip);
    if (!z.ok) return { ok: false, reason: z.reason };
  }
  return {
    ok: true, kind, warnings,
    sha256: createHash("sha256").update(f.bytes).digest("hex"),
    storageKey: "quarantine/" + randomBytes(16).toString("hex"),                    // random, no user input in the key
  };
}

Zip inspectorts

const EOCD_SIG = 0x06054b50, CDH_SIG = 0x02014b50, LFH_SIG = 0x04034b50;
const EOCD_LEN = 22, CDH_LEN = 46, LFH_LEN = 30;

type ZipCheck = { ok: true; entries: number; totalUncompressed: number } | { ok: false; reason: string };

function safeEntryName(name: string): boolean {
  if (name.length === 0 || name.length > 255) return false;
  if (name.includes("\0") || name.includes("\\") || name.startsWith("/") || /^[A-Za-z]:/.test(name)) return false;
  if (/\.(zip|jar|7z|rar|gz|tgz|tar)$/i.test(name)) return false;                  // no nested archives
  return !name.split("/").includes("..");                                            // zip slip
}

/** Cheap pre-filter on declared sizes. Extraction must still count real inflated bytes, because headers can lie. */
export function inspectZip(b: Buffer, lim: ZipLimits): ZipCheck {
  let eocd = -1;
  const stop = Math.max(0, b.length - EOCD_LEN - 0xffff);                           // the comment is at most 65,535 bytes
  for (let i = b.length - EOCD_LEN; i >= stop; i--) {
    if (b.readUInt32LE(i) === EOCD_SIG && i + EOCD_LEN + b.readUInt16LE(i + 20) === b.length) { eocd = i; break; }
  }
  if (eocd === -1) return { ok: false, reason: "no end of central directory record" };
  const onDisk = b.readUInt16LE(eocd + 8), total = b.readUInt16LE(eocd + 10);
  const cdSize = b.readUInt32LE(eocd + 12), cdOffset = b.readUInt32LE(eocd + 16);
  if (b.readUInt16LE(eocd + 4) !== 0 || b.readUInt16LE(eocd + 6) !== 0 || onDisk !== total) return { ok: false, reason: "multi-disk archives are not supported" };
  if (total === 0xffff || cdSize === 0xffffffff || cdOffset === 0xffffffff) return { ok: false, reason: "ZIP64 is not supported" };
  if (total > lim.maxEntries) return { ok: false, reason: "too many entries" };
  if (cdOffset + cdSize !== eocd) return { ok: false, reason: "central directory is not where the EOCD says" };

  const spans: { start: number; end: number }[] = [];
  let p = cdOffset, totalU = 0;
  for (let n = 0; n < total; n++) {
    if (p + CDH_LEN > eocd || b.readUInt32LE(p) !== CDH_SIG) return { ok: false, reason: "corrupt central directory" };
    const flags = b.readUInt16LE(p + 8), method = b.readUInt16LE(p + 10);
    const csize = b.readUInt32LE(p + 20), usize = b.readUInt32LE(p + 24);
    const nameLen = b.readUInt16LE(p + 28), extraLen = b.readUInt16LE(p + 30), commentLen = b.readUInt16LE(p + 32);
    const local = b.readUInt32LE(p + 42);
    const nameEnd = p + CDH_LEN + nameLen;
    if (nameEnd > eocd) return { ok: false, reason: "corrupt central directory" };
    const name = b.toString("utf8", p + CDH_LEN, nameEnd);
    if (flags & 0x1) return { ok: false, reason: "encrypted entries cannot be scanned" };
    if (method !== 0 && method !== 8) return { ok: false, reason: "compression method " + method + " not allowed" };
    if (!safeEntryName(name)) return { ok: false, reason: "unsafe entry name " + JSON.stringify(name) };
    if (usize > 0 && (csize === 0 || usize / csize > lim.maxRatio)) return { ok: false, reason: "compression ratio over " + lim.maxRatio + ":1" };
    totalU += usize;
    if (totalU > lim.maxTotalUncompressed) return { ok: false, reason: "declared uncompressed total over limit" };
    if (local + LFH_LEN > cdOffset || b.readUInt32LE(local) !== LFH_SIG) return { ok: false, reason: "entry points outside the data area" };
    spans.push({ start: local, end: local + LFH_LEN + nameLen + csize });            // a lower bound on where the entry ends
    p = nameEnd + extraLen + commentLen;
  }
  if (p !== eocd) return { ok: false, reason: "central directory size mismatch" };
  spans.sort((x, y) => x.start - y.start);
  for (let i = 0; i < spans.length; i++) {
    const next = i + 1 < spans.length ? spans[i + 1].start : cdOffset;
    if (spans[i].end > next) return { ok: false, reason: "overlapping entries (non-recursive zip bomb)" };
  }
  return { ok: true, entries: total, totalUncompressed: totalU };
}

▭On the whiteboard · Design choices4 points

type

magic bytes decide; extension must agree; Content-Type only logged; served Content-Type comes from the sniffed kind

zip

EOCD must end the file; per-entry ratio, total size, entry count, names, encryption, method; entries may not overlap

storage

random key in a private bucket, never the original name, never under the web root

serving

only clean files, from a separate domain, with nosniff and attachment disposition

04Bombs, images, scanning states and tests

A zip bomb is a small archive that expands enormously. The recursive kind (42.zip, about 42 KB, nested archives totalling about 4.5 PB) is stopped by refusing nested archives. The non-recursive kind (David Fifield, 2019) makes many central directory entries point at the same compressed data, so one deflate stream is counted again and again; about 42 KB expands to 5.5 GB and 10 MB to 281 TB. The overlap check catches it because entries cannot legitimately share bytes.

Re-encoding and scan statests

import sharp from "sharp";

/** Decode with a pixel cap (a 1 KB PNG can claim 65535 x 65535 pixels, about 17 GB of RGBA), re-encode, drop metadata. */
export async function reencodeImage(input: Buffer): Promise<Buffer> {
  const img = sharp(input, { limitInputPixels: 40_000_000, failOn: "error" });
  const meta = await img.metadata();                                   // reads the header only
  if (!meta.width || !meta.height || meta.width > 8000 || meta.height > 8000) throw new Error("image dimensions not allowed");
  return img.rotate()                                                  // apply EXIF orientation before it is stripped
    .resize({ width: 4096, height: 4096, fit: "inside", withoutEnlargement: true })
    .webp({ quality: 85 })                                             // sharp writes no EXIF, GPS or ICC metadata unless asked
    .toBuffer();
}

const NEXT: Record<ScanState, readonly ScanState[]> = {
  pending: ["scanning"],
  scanning: ["clean", "infected", "failed", "pending"],               // back to pending when a worker's lease expires
  failed: ["pending"],                                                 // retried with back-off up to a maximum
  clean: [],
  infected: [],
};

export async function transition(store: FileStore, id: string, to: ScanState): Promise<FileRow> {
  const row = await store.get(id);
  if (!row) throw new Error("no such file");
  if (!NEXT[row.state].includes(to)) throw new Error("illegal transition " + row.state + " -> " + to);
  const patch = { state: to, attempts: to === "scanning" ? row.attempts + 1 : row.attempts };
  if (!(await store.updateIf(id, row.version, patch))) throw new Error("concurrent update; reload and retry");
  return { ...row, ...patch, version: row.version + 1 };
}

export const canDownload = (row: FileRow, requesterId: string) => row.state === "clean" && row.ownerId === requesterId;

Serving header or rule	Why
Separate origin, e.g. usercontent.example.net	An HTML or SVG upload cannot read your main site's cookies or DOM
`Content-Type` from the sniffed kind	Never echo the client's value
`X-Content-Type-Options: nosniff`	Stops browsers guessing a script type from content
`Content-Disposition: attachment; filename="..."`	Downloads rather than renders; encode the name with RFC 6266 rules
Short-lived signed URLs	Access checks happen at issue time; links expire
Only `clean` state	Pending and failed files are visible to the owner as status only

!A scan is a signal, not a guarantee

Signature scanners such as ClamAV miss new malware, and encrypted archives are opaque to them, which is why they are refused. Scan asynchronously with a timeout, treat failed as not clean, rescan stored files when signatures update, and keep re-encoding and safe serving as the main defences.

vitestts

import { describe, it, expect } from "vitest";

function makeZip(files: { name: string; data: Buffer; usize?: number }[], overlap = false): Buffer {
  const locals: Buffer[] = [], central: Buffer[] = [];
  let offset = 0;
  for (const f of files) {
    const name = Buffer.from(f.name);
    const lfh = Buffer.alloc(LFH_LEN);
    lfh.writeUInt32LE(LFH_SIG, 0); lfh.writeUInt16LE(20, 4); lfh.writeUInt32LE(f.data.length, 18);
    lfh.writeUInt32LE(f.usize ?? f.data.length, 22); lfh.writeUInt16LE(name.length, 26);
    const cdh = Buffer.alloc(CDH_LEN);
    cdh.writeUInt32LE(CDH_SIG, 0); cdh.writeUInt16LE(20, 4); cdh.writeUInt16LE(20, 6); cdh.writeUInt16LE(f.usize ? 8 : 0, 10);
    cdh.writeUInt32LE(f.data.length, 20); cdh.writeUInt32LE(f.usize ?? f.data.length, 24); cdh.writeUInt16LE(name.length, 28);
    cdh.writeUInt32LE(overlap ? 0 : offset, 42);
    locals.push(lfh, name, f.data);
    central.push(cdh, name);
    offset += LFH_LEN + name.length + f.data.length;
  }
  const cd = Buffer.concat(central);
  const eocd = Buffer.alloc(EOCD_LEN);
  eocd.writeUInt32LE(EOCD_SIG, 0); eocd.writeUInt16LE(files.length, 8); eocd.writeUInt16LE(files.length, 10);
  eocd.writeUInt32LE(cd.length, 12); eocd.writeUInt32LE(offset, 16);
  return Buffer.concat([...locals, cd, eocd]);
}

const lim = { maxEntries: 1000, maxTotalUncompressed: 100_000_000, maxRatio: 100 };
const policy: UploadPolicy = { allowed: ["png", "pdf", "zip"], maxBytes: 20_000_000, zip: lim };
const png = Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a, 0, 0, 0, 13]);
const reason = (v: Verdict) => (v.ok ? "ok" : v.reason);

describe("upload validator", () => {
  it("decides on content, not on the name or the client's type", () => {
    expect(validate({ originalName: "cat.png", declaredType: "text/html", bytes: png }, policy).ok).toBe(true);
    expect(reason(validate({ originalName: "cat.png.php", declaredType: "image/png", bytes: png }, policy))).toMatch(/extension/);
    expect(reason(validate({ originalName: "x.png", declaredType: "image/png", bytes: Buffer.from("<?php system($_GET[c]);") }, policy))).toBe("content type not allowed");
  });
  it("accepts an ordinary zip", () => {
    expect(inspectZip(makeZip([{ name: "a.txt", data: Buffer.from("hello") }, { name: "b/c.txt", data: Buffer.from("world") }]), lim).ok).toBe(true);
  });
  it("refuses a high ratio, zip slip and overlapping entries", () => {
    expect(inspectZip(makeZip([{ name: "big.txt", data: Buffer.alloc(10), usize: 10_000 }]), lim).ok).toBe(false);
    expect(inspectZip(makeZip([{ name: "../../etc/cron.d/x", data: Buffer.from("x") }]), lim).ok).toBe(false);
    const bomb = makeZip([{ name: "a", data: Buffer.alloc(50) }, { name: "b", data: Buffer.alloc(50) }], true);
    expect(inspectZip(bomb, lim)).toEqual({ ok: false, reason: "overlapping entries (non-recursive zip bomb)" });
  });
});

05Extensions, follow-ups, how to say it

Documents: convert office files and PDFs to PDF/A or images in a sandbox (gVisor, a locked-down container) to strip macros and active content; never open them on the web tier.
Content disarm and reconstruction (CDR): rebuild files from parsed content instead of trying to detect badness.
Deduplication: the SHA-256 lets you skip rescanning identical content and block known-bad hashes instantly.
Quotas: per-user byte and count quotas stop storage exhaustion; presigned direct-to-bucket uploads keep big bodies off your servers, with the same validation run by a bucket event.

Q1Why is checking the extension not enough?+

The name is user input. shell.php.png passes a naive check, and content can be anything regardless of the name. Content decides; the extension only has to agree.

Q2Why re-encode images that already passed the magic-byte check?+

A polyglot can be a valid PNG and valid HTML, JavaScript or a zip at the same time, and images carry EXIF with GPS data. Decoding to pixels and encoding fresh keeps only the picture.

Q3What do users see while the scan runs?+

The file appears with a pending status to its owner only. Downloads and sharing are disabled until the state is clean; infected files are quarantined and the owner is told.

“How to say it in the room

"The stream is cut at the size limit at the edge. Type comes from magic bytes against a per-feature allow-list, the extension must agree, and the client's Content-Type is only logged. Zips must have an EOCD ending the file and pass entry count, per-entry ratio, total size, name and overlap checks, and extraction counts real bytes anyway. Images are decoded with a pixel cap and re-encoded without metadata. Files land under random keys in a private bucket in a pending state; a scanner moves them through a compare-and-set state machine, and only clean files are served, from a separate origin with nosniff and attachment disposition."

Test yourself

An attacker uploads `invoice.pdf` whose bytes start with `PK\x03\x04` and whose request says `Content-Type: application/pdf`. With PDFs and zips both allowed, what should the validator conclude?

Finished this one?

Tick it and it shows as done on the roadmap and the course.