Day 6 of 960 minCipher CadetHands-on lab
Day 6: Trust Boundaries and Threat Modelling for AI Apps
Taught by Mr Elite Updated
By the end of today you can
- Mark trust boundaries on a diagram of an LLM application
- List attack surfaces as entry points where input can get in
- Apply a four-step method: assets, entry points, threats, defences
- Set tool rules in the lab so a hijacked agent cannot do harm
Warm-up
Answer from memory first, then tap to check. Recalling yesterday is what makes it stick.
What is the difference between direct and indirect prompt injection?
Direct injection comes from the user typing to the model. Indirect injection hides in content the model fetches, such as a web page or file.
Who actually runs a tool when a model asks for it?
The back end code runs it. The model only requests it, so the back end is the place to check and limit the request.
Imagine you are handed a diagram of a company chatbot and asked one question: where could this go wrong? Without a method you stare at the boxes and guess. With a method you can walk the diagram line by line and produce a list in twenty minutes.
Today is that method. You will draw lines on a diagram where control changes hands, list every way input gets in, and attach threats and defences to each. Along the way you will pull together everything from Days 1 to 5, because a good threat model uses all of it.
Trust boundaries: where control changes hands
A trust boundary is a line on a diagram where data or control passes between parts that deserve different levels of trust. On one side you control the code. On the other side someone else might control what arrives.
In a chatbot, the obvious boundary sits between the user's browser and your back end. The user can type anything. Less obvious ones sit between the back end and the model, between the model and its tools, and between the tools and the outside world, such as a web page the agent fetches.
Here is a plain text sketch of a small support bot. The dashed marks show where I would draw boundaries. Read it left to right and notice how many times data changes hands before anything happens.
[User browser] -- B1 --> [Back end] -- B2 --> [Model]
| |
| +-- B3 --> [Tools: email, fetch page]
| |
+-- B4 --> [Document store] +-- B5 --> [Outside web]- Trust boundary
- A line where data moves between parts with different levels of trust, so it needs checking.
- Asset
- Anything worth protecting, such as customer data, a system prompt, an API key or the company's reputation.
- Threat
- Something bad that could happen to an asset.
- Defence
- A control that stops a threat or reduces the harm.
To spot a boundary on any diagram, ask three questions about each arrow. Who wrote the data on it? Could that person be a stranger? Does the receiving part treat it as an instruction or as plain content? If the answers are stranger, yes and instruction, you have found a weak boundary.
Quick check
A model reads a web page it fetched and obeys a command hidden in it. Which boundary was crossed without a check?
Outside content, which an attacker can write, flowed into the place where the model takes instructions.
Attack surface: every place input can get in
Attack surface means every place where input can enter your system. The bigger the surface, the more doors an attacker can try. Your job is to list the doors honestly, including the ones nobody thinks of as doors.
For an LLM application, typical entry points are the chat box, uploaded files, fetched web pages, retrieved documents, tool results, emails the assistant reads, and even the system prompt if someone else can edit it. Any text an outsider can write and the model can read is an entry point.
The hidden doors are the ones that matter most. A user types in the chat box on purpose, so you expect trouble there. Nobody expects trouble from a PDF in a shared folder, yet the model reads it with the same attention as a user message.
The four-step method
I use a simple method with four questions, asked in order. It is a light version of what professionals do, and it fits on one page. The order matters because each step feeds the next.
A useful test at the end is to pick any threat and ask whether the defence still works if the model is completely fooled. If the answer is no, the defence is a hope, not a control, and you should replace it.
Watch me do it: a mini threat model
I will model a small bot that answers questions from company documents and can send a summary email. Everything here is a paper exercise on your own machine, so you touch no real system. Open a plain text editor and follow along.
ASSETS
A1 Customer documents
A2 System prompt
A3 Email sending abilityENTRY POINTS
E1 Chat box (user text)
E2 Retrieved documents (written by staff and others)
E3 Tool result from the email serviceTHREATS AND DEFENCES
T1 (E1, A2) User tricks bot into revealing system prompt
D1 Keep no secrets in the prompt; filter output
T2 (E2, A3) Document says 'email this to attacker'
D2 Back end asks human to approve every email
T3 (E2, A1) Bot quotes another customer's record
D3 Retrieval checks the user's access firstgrep -c "^T[0-9]" threat-model-demo.txtWhat just happened: in three short blocks I turned a vague worry into a checkable list. Note that D2 and D3 live in the back end, not in the prompt. That is the pattern to copy.
Look at how the threats were built. T1 starts at E1 and aims at A2. T2 and T3 both start at E2, because one door can lead to several assets. E3 has no threat yet, which tells me the list is unfinished. A tool result can carry hostile text too, so a real model would get a fourth threat there.
Quick check
Which defence is strongest for T2, a document telling the bot to send email?
A back end approval step works even if the model has been fooled.
Your turn: tool rules in the lab
Now we put the idea to work. The lab below gives you recorded traces of an agent that was hijacked by a web page. Your job is to set tool rules, which are boundary controls, and replay the traces to see whether the hijack still works. It runs in the SecurityElites practice lab, so you attack nothing real.
Before you touch any setting, use the method from earlier. The assets are whatever the tools can reach. The entry point is the fetched page. The threat is injection leading to excess permissions. The defence is the rule you are about to choose for each tool.
Your turn
Whose Instruction Was That?
This lab turns trust boundaries into settings you can change. The agent can fetch pages, read files, send email, run code and open tickets, and a fetched page once took control of it. You decide which tools stay on, which need approval and which are blocked, then replay the traces to test your choices.
What to try
What to notice
- The danger is highest where a tool that reads outside content sits beside a tool that sends data out.
- Blocking everything stops the attack but also stops the agent doing its job, so you are balancing safety and usefulness.
- The rule works at the back end, so the model being fooled no longer matters as much.
If it did not work
Listing only the chat box as an entry point
Walk every arrow on the diagram. Files, web pages, retrieved documents and tool results are all entry points.
Writing 'improve the prompt' as the defence for everything
Choose a back end control such as approval, permission checks, output filtering or removing the tool.
Writing threats too vaguely, such as 'hacking'
Name the entry point, the asset and what happens, in one sentence.
Putting secrets in the system prompt
Assume it can be extracted. Keep keys and private data out of it.
Recap
- A trust boundary is where control changes hands, and data crossing it needs checking.
- Attack surface is every place input can enter, including files, pages and tool results.
- The method is assets, entry points, threats, defences, in that order.
- Three common AI threats are injection, leakage and excess permissions.
- Defences belong in the back end, because the model cannot police itself.
- A threat model is a short, living list, not a long report.
Cheat sheet
Trust boundary- Line where data passes between parts with different trust
Attack surface- All the places input can get in
Asset- Something worth protecting
Injection- Someone steers the model with crafted text, directly or indirectly
Leakage- Secrets or private data come out of the system
Excess permissions- The model or its tools can do more than the task needs
Four steps- Assets, entry points, threats, defences
Before the next day
Mission: One-page threat model for your Day 4 chatbot
Take the chatbot diagram you made on Day 4 and write a one-page threat model for it. This pulls Days 1 to 5 together and becomes a main input to your capstone on Day 9. Use only your own diagram and a text editor.
About 30 min
You are done when: A one-page day6-threat-model.txt with a marked diagram, at least six threats and one defence for each.
Want more? Rank your threats by likelihood and impact from 1 to 3 each, multiply the scores, and sort the list by the result.
Quiz
4 questions. Your score is saved with your progress.
1. Why is a fetched web page an attack surface for an AI agent?
Anything an outsider can write and the model can read is an entry point, which is the basis of indirect injection.
2. What is the correct order of the four-step method?
You need to know what you protect and how input arrives before you can sensibly list threats and controls.
3. Which is an example of excess permissions?
The tool gives far more power than the task needs, so any successful injection can do serious harm.
4. Why prefer back end defences over prompt instructions?
A prompt is a request the model may ignore. A back end check enforces the rule regardless.
Finish the day
Mark the day complete to save your progress.
Next: Day 7 turns the threat model into a plan. You will learn what red teaming is and write a scope and test plan, so the threats you listed today become tests you can run safely.
Your notes
Private to this device. Write what you want to remember from today.