CYART // SCHEMA · THREAT-MODEL6 NODES · 6 LINKSMITIGATEWHAT IS LEFTDATA · Assets — what we protectDATA01Assetswhat we protectEDGE · Entry points — APIs, logins, uploadsEDGE02Entry pointsAPIs, logins, uploadsCONTROL · Trust boundaries — internet | VPC | DBCONTROL03Trust boundariesinternet | VPC | DBTHREAT · Threats — STRIDE per boundaryTHREAT04ThreatsSTRIDE per boundarySERVICE · Controls — prevent · detect · respondSERVICE05Controlsprevent · detect · respondResidual riskaccepted + owner

LLM security evaluation and red teaming

Threat modelling LLM features, building security eval suites that account for non-determinism, measuring attack success rates, automated and human red teaming with tools such as garak and PyRIT, regression gates in CI, and production monitoring.

🔒 PROBy ·Lesson·11 min read·Advanced → Staff
AI red teamingLLM evaluationattack success rategarakPyRITMITRE ATLAS

01Threat modelling an LLM feature

✓What you will learn
  • Threat model an LLM feature by assets, untrusted inputs, capabilities and harms
  • Build security evals that measure attack success rate despite non-determinism
  • Run automated and human red teaming and use frameworks such as MITRE ATLAS and the OWASP LLM Top 10
  • Gate releases with regression evals and monitor production for abuse

Testing starts with knowing what you are defending. An LLM feature's threat model has the same parts as any other, with two differences: the component making decisions is probabilistic, and every piece of text it reads is a potential input channel.

ElementQuestionsExample answers for an email assistant
AssetsWhat could be stolen, changed or abused?Mailbox contents, contacts, ability to send as the user, API spend
Untrusted inputsWhich text can a third party influence?Inbound email bodies, attachments, linked web pages
CapabilitiesWhat can the model cause to happen?Draft and send email, create calendar events, render Markdown
HarmsWhat is the worst realistic outcome?Silent exfiltration of mail, sending phishing from the user's account, runaway cost
FigTrust boundaries in an LLM app
CYART // SCHEMA · THREAT-MODEL6 NODES · 6 LINKSMITIGATEWHAT IS LEFTDATA · Assets — what we protectDATA01Assetswhat we protectEDGE · Entry points — APIs, logins, uploadsEDGE02Entry pointsAPIs, logins, uploadsCONTROL · Trust boundaries — internet | VPC | DBCONTROL03Trust boundariesinternet | VPC | DBTHREAT · Threats — STRIDE per boundaryTHREAT04ThreatsSTRIDE per boundarySERVICE · Controls — prevent · detect · respondSERVICE05Controlsprevent · detect · respondResidual riskaccepted + owner
  • EDGE
  • SERVICE
  • DATA
  • CONTROL
  • THREAT
  • Live flow
  • Lookup / async
//Draw the boundaries as you would for any system: user input, retrieved content and tool outputs cross into the model's context; tool calls cross out of it. Every crossing is a place to test and a place to put a control.

This is an advanced module

AI & LLM security engineering is part of the Pro pass. 4 more sections — the deep dives, the final design, follow-ups and the test — unlock with it.

Already a cyart.io or studio.cyart.io member? Your pass code is in your Studio account.