LLM security evaluation and red teaming
Threat modelling LLM features, building security eval suites that account for non-determinism, measuring attack success rates, automated and human red teaming with tools such as garak and PyRIT, regression gates in CI, and production monitoring.
AI red teamingLLM evaluationattack success rategarakPyRITMITRE ATLAS
01Threat modelling an LLM feature
✓What you will learn
- Threat model an LLM feature by assets, untrusted inputs, capabilities and harms
- Build security evals that measure attack success rate despite non-determinism
- Run automated and human red teaming and use frameworks such as MITRE ATLAS and the OWASP LLM Top 10
- Gate releases with regression evals and monitor production for abuse
Testing starts with knowing what you are defending. An LLM feature's threat model has the same parts as any other, with two differences: the component making decisions is probabilistic, and every piece of text it reads is a potential input channel.
| Element | Questions | Example answers for an email assistant |
|---|---|---|
| Assets | What could be stolen, changed or abused? | Mailbox contents, contacts, ability to send as the user, API spend |
| Untrusted inputs | Which text can a third party influence? | Inbound email bodies, attachments, linked web pages |
| Capabilities | What can the model cause to happen? | Draft and send email, create calendar events, render Markdown |
| Harms | What is the worst realistic outcome? | Silent exfiltration of mail, sending phishing from the user's account, runaway cost |
FigTrust boundaries in an LLM app
- EDGE
- SERVICE
- DATA
- CONTROL
- THREAT
- Live flow
- Lookup / async
This is an advanced module
AI & LLM security engineering is part of the Pro pass. 4 more sections — the deep dives, the final design, follow-ups and the test — unlock with it.
Already a cyart.io or studio.cyart.io member? Your pass code is in your Studio account.