Input validation and canonicalisation
Turn untrusted input into typed, trusted values at the boundary: canonicalise first (Unicode, percent-encoding, paths), then validate against an allow-list, then hand the rest of the program a type that cannot be built any other way.
01Understand the problem
- Separate validation, sanitisation and output encoding, and know which one fixes which bug
- Canonicalise before validating, so the value you checked is the value you use
- Contain user-supplied paths inside a root, including symlinks
- Use parse, don't validate: return a branded type that only the parser can create
What they probe: the order of operations (decode, normalise, validate, use), why deny-lists fail, how ../ sneaks past a check through encoding or Unicode, and where validation lives in the code. Where candidates slip: validating then decoding, startsWith(root) without a separator, Python re.match with $, regexes that backtrack catastrophically, and validation copied into every handler instead of done once at the edge.
Most injection and traversal bugs share one shape: the program checks one representation of a value and then uses a different one. A filename is checked as %2e%2e%2fsecret and opened as ../secret. A username is checked as sysadmin (fullwidth letters) and stored after NFKC as sysadmin. Canonicalisation means reducing every equivalent spelling to one form; validation means checking that one form against rules. Doing them in the wrong order is the vulnerability.
- Decode transport encodings exactly once (percent-encoding, base64, JSON).
- Normalise: Unicode NFC or NFKC, case-fold identifiers, resolve paths.
- Validate the canonical value against an allow-list: type, length, character set, range, format.
- Convert it into a domain type and pass only that type inward.
- Encode for the sink at the moment of use (HTML, SQL parameters, shell argv), whatever validation already did.
| Technique | What it does | What it does not do |
|---|---|---|
| Validation | Rejects input that is not in the expected set | Make a valid string safe for HTML or SQL; O'Brien is a valid surname |
| Sanitisation | Changes input to remove dangerous parts | Guarantee the result is what the user meant; it can create new payloads (<scr<script>ipt>) |
| Output encoding | Makes a value inert in one specific context | Help in a different context; HTML escaping does nothing for a shell |
| Parameterisation | Keeps data and code in separate channels (SQL placeholders, argv arrays) | Validate business rules such as ranges |
02Interfaces first
Validation is a function from unknown to either a typed value or a precise error. Making the success type branded means the compiler stops anyone constructing a Username without going through the parser, so the check cannot be forgotten further in.
declare const brand: unique symbol;
export type Branded<T, B extends string> = T & { readonly [brand]: B };
export type Username = Branded<string, "Username">;
export type SafeRelPath = Branded<string, "SafeRelPath">;
export type Result<T> =
| { ok: true; value: T }
| { ok: false; field: string; reason: string };
export interface Parser<T> {
parse(input: unknown, field: string): Result<T>;
}
export class InputError extends Error {
constructor(readonly field: string, readonly reason: string) {
super(field + ": " + reason);
this.name = "InputError";
}
}The caller turns an InputError into a 400 with a stable, non-sensitive message such as username: must be 3 to 32 characters. The reason must describe the rule, never echo the raw input back, which would turn the error page into a reflection point for XSS or log injection.
03Canonicalise, then validate
A username parser shows the full order: reject the wrong type, bound the length before expensive work (normalisation of a 10 MB string is a denial of service), normalise with NFKC so compatibility characters collapse, case-fold, then check against an allow-list pattern anchored at both ends.
const USERNAME_RE = /^[a-z0-9](?:[a-z0-9_-]{1,30})[a-z0-9]$/; // 3..32, ASCII only, no leading or trailing symbol
const RESERVED = new Set(["admin", "root", "support", "security", "postmaster"]);
export const username: Parser<Username> = {
parse(input, field) {
if (typeof input !== "string") return { ok: false, field, reason: "must be a string" };
if (input.length > 128) return { ok: false, field, reason: "too long" }; // cheap bound before normalising
const canonical = input.normalize("NFKC").toLowerCase().trim();
if (!USERNAME_RE.test(canonical)) return { ok: false, field, reason: "3 to 32 of a-z, 0-9, _ or -, starting and ending with a letter or digit" };
if (RESERVED.has(canonical)) return { ok: false, field, reason: "reserved" };
return { ok: true, value: canonical as Username };
},
};
/** Percent-decode exactly once and refuse anything still encoded, so no later layer can decode it again. */
export function decodeOnce(raw: string, field: string): string {
let s: string;
try { s = decodeURIComponent(raw); } catch { throw new InputError(field, "invalid percent-encoding"); }
if (/%[0-9a-fA-F]{2}/.test(s)) throw new InputError(field, "double-encoded input");
return s;
}Paths need the same discipline plus the filesystem. Resolve the user's path against a root, compare with path.relative (not string prefixes, since /srv/data-evil starts with /srv/data), then resolve symlinks with realpath and compare again, because a symlink inside the root can point outside it.
import path from "node:path";
import { realpath } from "node:fs/promises";
function isInside(root: string, candidate: string): boolean {
const rel = path.relative(root, candidate);
return rel !== "" && rel !== ".." && !rel.startsWith(".." + path.sep) && !path.isAbsolute(rel);
}
/** Returns the real path of an existing file under root, or throws InputError. */
export async function resolveInside(root: string, userPath: string): Promise<string> {
if (userPath.length === 0 || userPath.length > 1024) throw new InputError("path", "length out of range");
if (userPath.includes("\0")) throw new InputError("path", "NUL byte");
const normalised = userPath.normalize("NFC"); // fullwidth or decomposed forms stay distinct names; NFC keeps one spelling
const rootReal = await realpath(root);
const lexical = path.resolve(rootReal, normalised); // collapses "a/../b"; an absolute userPath replaces root here
if (!isInside(rootReal, lexical)) throw new InputError("path", "outside the allowed directory");
let real: string;
try { real = await realpath(lexical); } catch { throw new InputError("path", "not found"); }
if (!isInside(rootReal, real)) throw new InputError("path", "symlink escapes the allowed directory");
return real;
}04Traps and tests
| Trap | Example | Fix |
|---|---|---|
| Validate, then decode | %2e%2e%2f passes a .. check, then becomes ../ | Decode once before validating; refuse residual encoding |
| Prefix check without separator | /srv/data-evil/x starts with /srv/data | path.relative, or append the separator before comparing |
Python $ anchor | re.match(r'^[a-z]+$', 'abc\n') matches | re.fullmatch or \Z; in JavaScript $ without the m flag is safe |
| Catastrophic backtracking | /^(a+)+$/ on 30 a characters and a ! takes seconds | Avoid nested quantifiers, bound input length first, or use a linear engine (RE2) |
| Unicode confusables | Cyrillic а in pаypal survives NFKC | Restrict identifiers to one script, or ASCII, as the username parser does |
| Time of check to time of use | A symlink is swapped after realpath | Open with O_NOFOLLOW, or openat2 with RESOLVE_BENEATH on Linux, or keep the directory unwritable by others |
A perfectly valid comment can still contain < and '. Validate for business rules at the edge, and encode or parameterise at every sink. Treat any HTML or SQL safety you get from validation as a bonus, never as the defence.
import { describe, it, expect } from "vitest";
import { mkdtemp, mkdir, writeFile, symlink } from "node:fs/promises";
import { tmpdir } from "node:os";
describe("username", () => {
it("canonicalises fullwidth and upper case to one form", () => {
expect(username.parse("ALICE_01", "u")).toEqual({ ok: true, value: "alice_01" });
});
it("refuses reserved names however they are spelt", () => {
expect(username.parse("admin", "u").ok).toBe(false);
});
it("refuses non-strings and oversized input before normalising", () => {
expect(username.parse(42, "u").ok).toBe(false);
expect(username.parse("a".repeat(10_000), "u")).toEqual({ ok: false, field: "u", reason: "too long" });
});
});
describe("resolveInside", () => {
it("allows files inside and refuses traversal, absolute paths and escaping symlinks", async () => {
const base = await mkdtemp(path.join(tmpdir(), "rv-"));
const root = path.join(base, "data");
await mkdir(root);
await writeFile(path.join(root, "ok.txt"), "x");
await writeFile(path.join(base, "secret.txt"), "s");
await symlink(path.join(base, "secret.txt"), path.join(root, "link.txt"));
await expect(resolveInside(root, "ok.txt")).resolves.toMatch(/ok\.txt$/);
await expect(resolveInside(root, "../secret.txt")).rejects.toThrow(/outside/);
await expect(resolveInside(root, path.join(base, "secret.txt"))).rejects.toThrow(/outside/);
await expect(resolveInside(root, "link.txt")).rejects.toThrow(/symlink/);
});
});05Extensions, follow-ups, how to say it
- Schema libraries: Zod, Valibot or Pydantic give the same parse-to-type shape for whole request bodies; keep custom canonicalisation in refinements so it runs before the type is produced.
- Unknown fields: reject or strip them (
strict()in Zod) so mass assignment ofisAdmin: truecannot ride along. - Numbers: parse with a strict pattern, then check
Number.isSafeIntegerand the range;parseInt('12abc')returns 12. - Identifiers across systems: store the canonical form and compare canonical to canonical, or two accounts can differ only by case or width.
Q1Why normalise before validating rather than after?+
Normalisation can change characters, including into ones your validator forbids (fullwidth solidus becomes / under NFKC). If you validate first, you approve a string that later turns into something you would have refused.
Q2NFC or NFKC for usernames?+
NFKC, because it folds compatibility characters such as fullwidth letters and ligatures into their plain forms, which is what you want for identifiers that must not collide visually. For free text you store and display, prefer NFC so you do not alter what the user wrote.
Q3Is a deny-list of `../` enough for paths?+
No. Encodings, absolute paths, Windows separators and symlinks all bypass it. Resolve and compare against the root instead.
"At the boundary I bound the length, decode exactly once, normalise, then validate the canonical value against an allow-list and return a branded type, so the rest of the code can only receive checked values. For paths I resolve against the root, compare with path.relative, then realpath and compare again for symlinks. Validation enforces business rules; injection safety still comes from encoding and parameterisation at each sink."
A download handler checks that `name` contains no `..`, then calls `decodeURIComponent(name)` and opens the file. What is the bug?