Day 6 of 960 minCipher CadetHands-on lab

Day 6: Trust Boundaries and Threat Modelling for AI Apps

Taught by Mr Elite Updated

By the end of today you can

  • Mark trust boundaries on a diagram of an LLM application
  • List attack surfaces as entry points where input can get in
  • Apply a four-step method: assets, entry points, threats, defences
  • Set tool rules in the lab so a hijacked agent cannot do harm

Warm-up

Answer from memory first, then tap to check. Recalling yesterday is what makes it stick.

What is the difference between direct and indirect prompt injection?

Direct injection comes from the user typing to the model. Indirect injection hides in content the model fetches, such as a web page or file.

Who actually runs a tool when a model asks for it?

The back end code runs it. The model only requests it, so the back end is the place to check and limit the request.

Imagine you are handed a diagram of a company chatbot and asked one question: where could this go wrong? Without a method you stare at the boxes and guess. With a method you can walk the diagram line by line and produce a list in twenty minutes.

Today is that method. You will draw lines on a diagram where control changes hands, list every way input gets in, and attach threats and defences to each. Along the way you will pull together everything from Days 1 to 5, because a good threat model uses all of it.

Trust boundaries: where control changes hands

A trust boundary is a line on a diagram where data or control passes between parts that deserve different levels of trust. On one side you control the code. On the other side someone else might control what arrives.

In a chatbot, the obvious boundary sits between the user's browser and your back end. The user can type anything. Less obvious ones sit between the back end and the model, between the model and its tools, and between the tools and the outside world, such as a web page the agent fetches.

Here is a plain text sketch of a small support bot. The dashed marks show where I would draw boundaries. Read it left to right and notice how many times data changes hands before anything happens.

text
[User browser] -- B1 --> [Back end] -- B2 --> [Model]
                              |                 |
                              |                 +-- B3 --> [Tools: email, fetch page]
                              |                                  |
                              +-- B4 --> [Document store]        +-- B5 --> [Outside web]
B1 to B5 are trust boundaries. Each arrow that crosses one is a place to ask what could arrive and who wrote it.
Trust boundary
A line where data moves between parts with different levels of trust, so it needs checking.
Asset
Anything worth protecting, such as customer data, a system prompt, an API key or the company's reputation.
Threat
Something bad that could happen to an asset.
Defence
A control that stops a threat or reduces the harm.

To spot a boundary on any diagram, ask three questions about each arrow. Who wrote the data on it? Could that person be a stranger? Does the receiving part treat it as an instruction or as plain content? If the answers are stranger, yes and instruction, you have found a weak boundary.

Quick check

A model reads a web page it fetched and obeys a command hidden in it. Which boundary was crossed without a check?

Attack surface: every place input can get in

Attack surface means every place where input can enter your system. The bigger the surface, the more doors an attacker can try. Your job is to list the doors honestly, including the ones nobody thinks of as doors.

For an LLM application, typical entry points are the chat box, uploaded files, fetched web pages, retrieved documents, tool results, emails the assistant reads, and even the system prompt if someone else can edit it. Any text an outsider can write and the model can read is an entry point.

The hidden doors are the ones that matter most. A user types in the chat box on purpose, so you expect trouble there. Nobody expects trouble from a PDF in a shared folder, yet the model reads it with the same attention as a user message.

The four-step method

I use a simple method with four questions, asked in order. It is a light version of what professionals do, and it fits on one page. The order matters because each step feeds the next.

A useful test at the end is to pick any threat and ask whether the defence still works if the model is completely fooled. If the answer is no, the defence is a hope, not a control, and you should replace it.

Watch me do it: a mini threat model

I will model a small bot that answers questions from company documents and can send a summary email. Everything here is a paper exercise on your own machine, so you touch no real system. Open a plain text editor and follow along.

text
ASSETS
A1 Customer documents
A2 System prompt
A3 Email sending ability
Step 2: assets, numbered so we can refer back to them.
text
ENTRY POINTS
E1 Chat box (user text)
E2 Retrieved documents (written by staff and others)
E3 Tool result from the email service
Step 3: every place text can enter the model's context.
text
THREATS AND DEFENCES
T1 (E1, A2) User tricks bot into revealing system prompt
   D1 Keep no secrets in the prompt; filter output
T2 (E2, A3) Document says 'email this to attacker'
   D2 Back end asks human to approve every email
T3 (E2, A1) Bot quotes another customer's record
   D3 Retrieval checks the user's access first
Step 4: each threat names its entry point, its asset and one defence.
bash
grep -c "^T[0-9]" threat-model-demo.txt
Counts lines that start with T and a digit. On Windows PowerShell use: (Select-String "^Td" threat-model-demo.txt).Count

What just happened: in three short blocks I turned a vague worry into a checkable list. Note that D2 and D3 live in the back end, not in the prompt. That is the pattern to copy.

Look at how the threats were built. T1 starts at E1 and aims at A2. T2 and T3 both start at E2, because one door can lead to several assets. E3 has no threat yet, which tells me the list is unfinished. A tool result can carry hostile text too, so a real model would get a fourth threat there.

Quick check

Which defence is strongest for T2, a document telling the bot to send email?

Your turn: tool rules in the lab

Now we put the idea to work. The lab below gives you recorded traces of an agent that was hijacked by a web page. Your job is to set tool rules, which are boundary controls, and replay the traces to see whether the hijack still works. It runs in the SecurityElites practice lab, so you attack nothing real.

Before you touch any setting, use the method from earlier. The assets are whatever the tools can reach. The entry point is the fetched page. The threat is injection leading to excess permissions. The defence is the rule you are about to choose for each tool.

Your turn

Whose Instruction Was That?

This lab turns trust boundaries into settings you can change. The agent can fetch pages, read files, send email, run code and open tickets, and a fetched page once took control of it. You decide which tools stay on, which need approval and which are blocked, then replay the traces to test your choices.

Open the lab in a new tab

What to try

What to notice

  • The danger is highest where a tool that reads outside content sits beside a tool that sends data out.
  • Blocking everything stops the attack but also stops the agent doing its job, so you are balancing safety and usefulness.
  • The rule works at the back end, so the model being fooled no longer matters as much.

If it did not work

Listing only the chat box as an entry point

Walk every arrow on the diagram. Files, web pages, retrieved documents and tool results are all entry points.

Writing 'improve the prompt' as the defence for everything

Choose a back end control such as approval, permission checks, output filtering or removing the tool.

Writing threats too vaguely, such as 'hacking'

Name the entry point, the asset and what happens, in one sentence.

Putting secrets in the system prompt

Assume it can be extracted. Keep keys and private data out of it.

Recap

  • A trust boundary is where control changes hands, and data crossing it needs checking.
  • Attack surface is every place input can enter, including files, pages and tool results.
  • The method is assets, entry points, threats, defences, in that order.
  • Three common AI threats are injection, leakage and excess permissions.
  • Defences belong in the back end, because the model cannot police itself.
  • A threat model is a short, living list, not a long report.

Cheat sheet

Trust boundary
Line where data passes between parts with different trust
Attack surface
All the places input can get in
Asset
Something worth protecting
Injection
Someone steers the model with crafted text, directly or indirectly
Leakage
Secrets or private data come out of the system
Excess permissions
The model or its tools can do more than the task needs
Four steps
Assets, entry points, threats, defences

Before the next day

Mission: One-page threat model for your Day 4 chatbot

Take the chatbot diagram you made on Day 4 and write a one-page threat model for it. This pulls Days 1 to 5 together and becomes a main input to your capstone on Day 9. Use only your own diagram and a text editor.

About 30 min

You are done when: A one-page day6-threat-model.txt with a marked diagram, at least six threats and one defence for each.

Want more? Rank your threats by likelihood and impact from 1 to 3 each, multiply the scores, and sort the list by the result.

Quiz

4 questions. Your score is saved with your progress.

1. Why is a fetched web page an attack surface for an AI agent?

2. What is the correct order of the four-step method?

3. Which is an example of excess permissions?

4. Why prefer back end defences over prompt instructions?

Finish the day

Mark the day complete to save your progress.

Go to Day 7

Next: Day 7 turns the threat model into a plan. You will learn what red teaming is and write a scope and test plan, so the threats you listed today become tests you can run safely.

Tell someone you are doing this:WhatsAppXLinkedIn

Your notes

Private to this device. Write what you want to remember from today.