Design a log normaliser

Design the component that takes raw lines in JSON, CEF and syslog, detects the format, parses, maps fields into a common schema, handles failures without losing data and lets a new format be added without touching existing code.

● EasyFREEDesign problem·40 min read·Beginner → Advanced

parsingstrategyschemaTypeScript

01Understand the problem

✓What you will learn

Separate detection, parsing and mapping into stages
Add a format via a registry, not an if chain
Handle unparseable input as data, not exceptions
Keep the raw line attached for evidence

◎Interview radar

What they probe: whether the parser and the schema mapper are separate, how unknown formats are handled, how you add a format, and whether the output keeps a reference to the raw input. Where candidates slip: regex in one giant function, throwing on bad input, losing the original line.

Input: a raw line plus source metadata. Output: a normalised Event or a ParseFailure.
Support JSON, CEF and RFC 5424 syslog; new formats pluggable.
Map vendor fields to a common schema (actor.user, outcome, time).
Never throw on bad input; never drop the raw line.

02Interfaces, model and code

FigPipelineSchematic

ACTOR
SERVICE
STREAM
Live flow
Lookup / async

//A detector picks a parser; the parser produces a flat field map; a mapper turns fields into the schema. Failures at any stage become a typed failure carrying the raw line.

Contractsts

export interface RawLine { source: string; receivedAt: Date; text: string }
export type Fields = Record<string, string | number | boolean>;

export interface Event {
  class: "authentication" | "process" | "network" | "generic";
  time: Date;
  actor: { user?: string; ip?: string };
  target?: { host?: string; app?: string };
  outcome?: "success" | "failure";
  fields: Fields;          // everything else, flattened
  raw: RawLine;            // evidence
}
export interface ParseFailure { stage: "detect" | "parse" | "map"; error: string; raw: RawLine }
export type Result = { ok: true; event: Event } | { ok: false; failure: ParseFailure };

export interface Parser { readonly id: string; sniff(line: string): boolean; parse(line: string): Fields }
export interface Mapper { readonly forSource: string | "*"; map(fields: Fields, raw: RawLine): Omit<Event, "raw" | "fields"> }

Normaliser and two parsersts

export class Normalizer {
  constructor(private readonly parsers: readonly Parser[], private readonly mappers: readonly Mapper[], private readonly fallback: Mapper) {}

  normalize(raw: RawLine): Result {
    const parser = this.parsers.find((p) => p.sniff(raw.text));
    if (!parser) return fail("detect", "no parser matched", raw);
    let fields: Fields;
    try { fields = parser.parse(raw.text); } catch (e) { return fail("parse", String(e), raw); }
    const mapper = this.mappers.find((m) => m.forSource === raw.source) ?? this.fallback;
    try {
      const shaped = mapper.map(fields, raw);
      return { ok: true, event: { ...shaped, fields, raw } };
    } catch (e) { return fail("map", String(e), raw); }
  }
}
const fail = (stage: ParseFailure["stage"], error: string, raw: RawLine): Result => ({ ok: false, failure: { stage, error, raw } });

export class JsonParser implements Parser {
  id = "json";
  sniff(l: string) { return l.trimStart().startsWith("{"); }
  parse(l: string): Fields { return flatten(JSON.parse(l)); }
}

export class CefParser implements Parser {
  id = "cef";
  sniff(l: string) { return l.startsWith("CEF:"); }
  parse(l: string): Fields {
    // CEF:Version|Vendor|Product|Version|SignatureID|Name|Severity|Extension
    const [, vendor, product, version, sigId, name, severity, ext = ""] = l.split("|");
    const fields: Fields = { vendor, product, version, sigId, name, severity };
    for (const m of ext.matchAll(/(\w+)=((?:\\=|[^=])+?)(?=\s\w+=|$)/g)) fields[m[1]] = m[2].replace(/\\=/g, "=");
    return fields;
  }
}

A source-specific mapperts

export class OktaMapper implements Mapper {
  forSource = "okta-prod";
  map(f: Fields, raw: RawLine) {
    // new Date(undefined) is an Invalid Date, not an exception: validate, or broken events pass as ok
    const time = new Date(String(f["published"] ?? ""));
    if (Number.isNaN(time.getTime())) throw new Error("okta: published missing or not a timestamp");
    return {
      class: "authentication" as const,
      time,
      actor: { user: str(f["actor.alternateId"]), ip: str(f["client.ipAddress"]) },
      target: { app: "okta" },
      outcome: f["outcome.result"] === "SUCCESS" ? ("success" as const) : ("failure" as const),
    };
  }
}
const str = (v: unknown) => (v == null ? undefined : String(v));

▭On the whiteboard · Choices3 points

stages

detect → parse → map; failure typed per stage; raw always attached

extend

new format = new Parser in the list; new vendor = new Mapper; Normalizer untouched

safety

no throw escapes normalize(); DLQ gets ParseFailure with the raw line

“How to say it in the room

"Three stages behind three interfaces: a parser knows a syntax, a mapper knows a vendor, the normaliser just wires them and converts every failure into data with the raw line attached. Adding a format is adding a class to a list. The security property is that nothing is ever lost: a bad line becomes a typed failure that goes to the dead-letter queue."

Q1Order of parsers matters for sniffing. How do you keep it safe?+

Most specific first (CEF prefix, syslog PRI), JSON last; add a test that every fixture line is claimed by exactly one parser.

Q2How do you version mappings when a vendor changes field names?+

Mapper carries a version; the event records mapper: okta@3; old events stay queryable by their version and rules can target both during migration.

Test yourself

A line is valid JSON but the Okta mapper throws because `published` is missing. What should happen?

Finished this one?

Tick it and it shows as done on the roadmap and the course.