Mistakes that ship vulnerabilities

Models learn from public code, tutorials and Stack Overflow answers, which optimise for "it works" and are frequently insecure. They also optimise for the request in front of them: ask for an endpoint that returns an alert and you get one that returns any alert. Two patterns dominate.

Trusting generated authorisation checks

The check exists and reads well, but checks the wrong thing: a role from the request body, a isAdmin claim from an unverified token, a middleware that is registered after the route it should protect.

Missing object-level authorisation (IDOR)

// Generated: authenticated, not authorised. Any user can read any tenant's alert.
app.get("/api/alerts/:id", requireLogin, async (req, res) => {
  const alert = await db.alert.findUnique({ where: { id: req.params.id } });
  res.json(alert);
});

// Fixed: the query is scoped to the caller; missing and forbidden look the same.
app.get("/api/alerts/:id", requireLogin, async (req, res) => {
  const alert = await db.alert.findFirst({
    where: { id: req.params.id, tenantId: req.session.tenantId },
  });
  if (!alert) return res.status(404).end();
  res.json(alert);
});

Catch it: the review question is "whose object is this, and where is that checked?" for every handler that takes an ID. Automate with an authorisation test matrix (user A requests user B's object, expect 404) in CI, and a lint rule or repository wrapper that refuses unscoped queries on tenant tables.

When a model hits an error it often suggests the change that makes the error go away. Those changes are usually security controls being switched off.

// Generated
const resetToken = Math.random().toString(36).slice(2);
const cipher = createCipheriv("aes-256-cbc", key, Buffer.alloc(16, 0)); // static IV, no MAC

// Fixed
const resetToken = randomBytes(32).toString("base64url");               // CSPRNG, 256 bits
const nonce = randomBytes(12);
const cipher = createCipheriv("aes-256-gcm", key, nonce);              // AEAD, unique nonce

SQL built with strings. Models concatenate or template user input into queries because so much example code does. Parameterise always, and let a query builder or ORM do it.

// Generated
const rows = await db.query("SELECT * FROM alerts WHERE host = '" + host + "'");

// Fixed
const rows = await db.query("SELECT * FROM alerts WHERE host = $1", [host]);

Regex DoS. Nested quantifiers like ^(\w+\s?)*$ backtrack exponentially on a crafted string, and a single request can pin a CPU. Log parsers and input validators generated by a model are a frequent source. Catch it with ReDoS linters, length limits before matching, and linear-time engines (RE2) for untrusted input.

Outdated APIs. Training data lags. Expect deprecated functions (crypto.createCipher), old library versions with known CVEs, and auth flows that are no longer recommended (OAuth implicit flow). Dependency scanning and a pinned, current lockfile catch most of it.

Licence contamination. A model can reproduce substantial chunks of copyleft code. Shipping it inside a commercial product can create obligations you did not intend. Use tools with code-reference filtering, run licence scanning, and treat large verbatim-looking blocks with suspicion.

Over-broad IAM generated for convenience. Ask for a policy that lets a Lambda read a bucket and you may get s3:* on *. It works, so it ships.

// Generated
{ "Effect": "Allow", "Action": "s3:*", "Resource": "*" }

// Fixed: one action, one bucket prefix
{ "Effect": "Allow", "Action": "s3:GetObject",
  "Resource": "arn:aws:s3:::cyart-evidence/tenant-*/reports/*" }

Swallowed errors. try { ... } catch { } or catch (e) { return true } hides failures and, in a security check, fails open. Lint for empty catches and review every catch in an auth path.

Tests that assert the bug. Generate tests from the code and they faithfully lock in its behaviour. If the handler returns another tenant's alert, the generated test asserts status 200. Tests must come from the spec, and must include the negative and cross-tenant cases.