Queues and streams

Why security systems put a queue or a log between producers and consumers: buffering bursts, delivery guarantees and idempotency, poison messages and dead-letter queues, and how to secure the topic itself so it is not the easiest place to read or forge data.

● EasyFREEBy ·Design problem·12 min read·Beginner → Intermediate
message queueKafkastreamsat-least-onceidempotencydead-letter queue

01Queue or stream, and why have one

✓What you will learn
  • Tell a queue (work items, deleted when done) from a stream (an ordered, retained, replayable log)
  • Reason about at-most-once, at-least-once and effectively-once delivery, and make consumers idempotent
  • Handle poison messages with retries and a dead-letter queue
  • Secure the broker: authentication, per-topic authorisation, encryption, retention of sensitive data
◎Interview radar

What they probe: why the queue is there (bursts, decoupling, replay), what happens when a consumer crashes halfway, how duplicates are handled, who may produce to the topic that drives automated response. Where candidates slip: assuming exactly-once for free, no dead-letter queue, a broker where every service can write to every topic.

A message queue (Amazon SQS, RabbitMQ) holds work items. Several consumers compete for messages; each message goes to one consumer and is deleted when acknowledged. A stream or log (Apache Kafka, Amazon Kinesis) is an append-only sequence split into partitions. Consumers track their own offset (position), messages stay for a retention period, and any number of consumer groups can read the same data, including replaying it from the start.

QueueStream
After consumptionDeletedRetained until retention expires
ReadersCompeting consumers share the workMany independent consumer groups each see everything
OrderingBest effort (FIFO queues exist, with lower throughput)Strict within a partition
ReplayNoYes, by resetting an offset
Security exampleJobs to scan uploaded filesEvery audit event, read by the SIEM, the archive and a detection engine
iWhy security pipelines love logs

A detection rule with a bug can be fixed and re-run over the last seven days of retained events. A queue would have deleted them.

02Delivery guarantees and idempotency

GuaranteeHow it happensRisk
At most onceAcknowledge before processingCrash after ack loses the message
At least onceProcess, then acknowledgeCrash before ack means the message is redelivered: duplicates
Effectively onceAt least once plus an idempotent consumer, or transactional writesNeeds a stable message ID and a dedupe store

Most real systems choose at least once and make the consumer idempotent: processing the same message twice has the same effect as once. The usual tool is an idempotency key, a unique ID set by the producer, recorded by the consumer in the same transaction as the side effect.

Recording the message ID with the side effectts
interface Msg { id: string; kind: "isolate_host"; hostId: string }

const UUID_RE = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/;

/** Idempotent consumer: the dedupe insert and the request record commit together or not at all. */
export async function handle(msg: Msg, db: Db, edr: Edr): Promise<void> {
  if (!UUID_RE.test(msg.id)) throw new Error("bad message id");          // permanent error: goes to DLQ
  await db.transaction(async (tx) => {
    const fresh = await tx.insertIfAbsent("processed_messages", msg.id);  // unique key on id
    if (!fresh) return;                                                   // duplicate delivery: no-op
    await tx.insert("isolation_requests", { hostId: msg.hostId, msgId: msg.id });
  });
  await edr.isolate(msg.hostId);   // the EDR call is idempotent per host: isolating twice is safe
}

interface Db { transaction(fn: (tx: Tx) => Promise<void>): Promise<void> }
interface Tx { insertIfAbsent(t: string, k: string): Promise<boolean>; insert(t: string, row: object): Promise<void> }
interface Edr { isolate(hostId: string): Promise<void> }
Sizing a streamtxt
audit events          5,000 / s x 1 kB          = 5 MB/s
retention             7 days = 604,800 s       -> 5 MB/s x 604,800 ≈ 3.0 TB
replication factor 3                             ≈ 9 TB of broker disk
throughput per partition  ~10 MB/s comfortable  -> 1 partition would do
partitions chosen     12, for consumer parallelism and future growth
partition key         tenant ID, so one tenant's events stay ordered
▭On the whiteboard · Core3 points
01
producer
sets message ID, authenticates to broker, writes to its own topic only
02
broker
replicated partitions, retention by policy, ACLs per topic
03
consumer
process then ack, dedupe by ID, retries then DLQ

03Deep dive: poison messages and back-pressure

A poison message is one the consumer can never process: malformed JSON, an unexpected schema, a payload that crashes the parser. With naive retry, it blocks the partition forever or loops endlessly. The standard answer is bounded retries with backoff, then a dead-letter queue (DLQ): a separate topic where failed messages wait for a human or a fixed consumer.

  1. Validate against a schema (JSON Schema, Avro, Protobuf) before doing anything else.
  2. On a transient error (timeout to a dependency), retry with exponential backoff, a few times.
  3. On a permanent error, or after the retry budget, write the message plus the error to the DLQ and move on.
  4. Alert on DLQ growth. A DLQ nobody watches is a slow way to lose data.

Back-pressure is a security feature

When producers outpace consumers, the queue grows instead of the consumers falling over. During an incident, log volume can jump tenfold; a buffered pipeline keeps every event and catches up later. Size retention for the longest consumer outage you plan to survive, not the average lag.

!Untrusted content in messages

If a queue carries data from users or the internet (uploaded file names, email subjects, log lines from compromised hosts), the consumer must treat it as hostile input: cap sizes, validate, escape when rendering. The broker does not sanitise anything.

04Securing the broker itself

ControlWhat it meansWhy
AuthenticationmTLS or SASL for every producer and consumer, no anonymous listenerAnyone on the network could otherwise read or write
Per-topic authorisationProducer may write only its own topics; consumers read only what they needA compromised web app must not forge messages on the topic that triggers host isolation
Encryption in transit and at restTLS on every listener, encrypted disksStreams hold copies of sensitive data for days
Message size limitsBroker and consumer caps (Kafka defaults to about 1 MB per message)Huge messages are a cheap denial of service
Retention by data classShort retention for personal data topicsDeleting one person's data from an immutable log is hard; crypto-shredding with per-user keys is the usual workaround
Signed messages for high-impact actionsProducer signs, consumer verifiesDefends against a broker or topic compromise

05Summary, follow-ups and self-test

▭On the whiteboard · Remember3 points
01
choose
queue for work items, stream for events many readers need or replay
02
deliver
at least once + idempotent consumer keyed by message ID
03
protect
schema check, retries then DLQ, per-topic ACLs, retention by data class
“How to say it in the room

"I'd put a replicated stream between the collectors and everything that reads security events, partitioned by tenant so ordering holds where it matters, with seven days' retention so I can replay after a bad rule or a consumer outage. Delivery is at least once, so consumers dedupe by message ID in the same transaction as their side effect. Bad messages get a few retries and then go to a dead-letter queue we alert on. Every client authenticates with mTLS and can only write its own topics, because the topic that triggers automated response is a target in its own right."

Q1How do you keep events in order for one host but still scale?+

Partition by host ID: all events for a host go to one partition and stay ordered, while different hosts spread across partitions and consumers.

Q2A consumer was down for two days. What do you need to have designed?+

Retention longer than two days, enough consumer capacity to catch up faster than new data arrives, and alerting on consumer lag long before retention runs out.

Q3How do you delete one user's personal data from a Kafka topic?+

You usually cannot rewrite the log cheaply. Keep retention short for such topics, or encrypt each user's fields with a per-user key and delete the key (crypto-shredding). Compacted topics with tombstones work for keyed state.

Test yourself

A consumer processes a message, calls an external API, and crashes before acknowledging. With at-least-once delivery, what happens and what protects you?

Finished this one?
Tick it and it shows as done on the roadmap and the course.