Day 9 of 960 minCipher CadetHands-on lab

Day 9: Capstone: Threat Model, Test and Report

Taught by Mr Elite Updated

By the end of today you can

  • Draw an architecture diagram and mark trust boundaries for a small AI assistant
  • Write a threat model with one threat per boundary
  • Review a RAG ingestion plan in the lab and record what you found
  • Write a short report with three ranked fixes and three next learning goals

Warm-up

Answer from memory first, then tap to check. Recalling yesterday is what makes it stick.

Why do you record a baseline refusal before testing a bypass?

It proves the bot refused the plain request, so a later success shows your bypass changed the outcome.

How do you rate the severity of a finding?

By impact and ease, then explain the rating in one sentence.

You have spent eight days learning pieces: tokens, system prompts, tools, boundaries, test plans. Today you put them together the way a real engagement ends, with a document someone else can read and act on.

Picture a team about to connect company documents to an AI assistant. They ask you one question: is this safe to ship? A pile of notes will not answer it. A short report with a diagram, a list of threats and three ranked fixes will. That is what you build today.

Choose your target and map it

First we need something to assess. Today our target is a small fictional assistant: a company helpdesk bot that answers staff questions from internal documents using retrieval (RAG, short for retrieval-augmented generation, where the app fetches relevant document chunks and pastes them into the prompt). It matches today's lab, so your report and your lab work join up.

Everything here is paper and practice labs. You will not test any real system. A threat model is a structured way of asking what could go wrong before it does, and a diagram is where it always starts, because you cannot protect parts you have not named.

Ingestion
The step where documents are collected, split and stored in the index the assistant searches.
Trust boundary
A line where data passes between parts that deserve different levels of trust.
Finding
One problem you can describe, show evidence for, rate and fix.
Threat model
A written list of what could go wrong in a system, tied to the places in the design where it could happen.

Now the architecture. For the helpdesk bot I use six parts: the staff user, the web chat front end, the application server, the model API, the document index, and the document sources (shared drive, wiki). Draw them as boxes with arrows showing the flow of one question, then the flow of documents into the index.

text
[Staff user] -> [Chat web page] -> [App server] -> [Model API]
                                      |
                                      v
                               [Document index] <- [Shared drive, wiki]
A plain-text diagram. Copy it into your editor and adjust it if your design differs.

Mark trust boundaries and list threats

Earlier in the course we learned that trouble lives where data crosses a line. On your diagram, draw a dashed line wherever data moves between parties with different trust. For our bot there are four: user to web page, app server to model, document sources into the index, and index content into the prompt.

Why these four? The user is anyone who can type, so their words are untrusted. The model API is run by someone else, so what you send it leaves your control. Documents come from many authors, so they are untrusted too. And retrieved text lands in the prompt right next to your own rules, which is the most surprising crossing of all.

For each boundary, ask one question: what could go wrong here? A simple method is to ask about three things: can someone say what they should not (spoofing or injection), can someone see what they should not (disclosure), and can something be made to do what it should not (misuse). Write one threat per boundary, in one sentence.

text
B1 User -> chat page: user types an injection to override the rules
B2 App server -> model: system prompt leaks in the reply
B3 Sources -> index: someone plants instructions in a document
B4 Index -> prompt: the bot shows a document the user may not read
Four boundaries, one threat each. Your wording can differ, but keep one line per boundary.

Quick check

A staff member edits a wiki page to include hidden instructions for the bot. Which boundary is crossed first?

Watch me do it: draft the report skeleton

Before the lab, I set up the file that holds everything. A report is easier to fill in than to write from blank. Follow along on your own machine, using only your own files.

text
DAY 9 REPORT: Helpdesk RAG assistant (fictional)
1. Architecture diagram
2. Trust boundaries
3. Threat model (one threat per boundary)
4. Test results (Day 8 table + today's lab)
5. Ranked fixes (1 to 3)
6. Next learning goals (3)
The report skeleton. Plain text is fine; clarity matters more than formatting.

What just happened: you turned eight days of scattered notes into a structure with a slot for every claim. Anything you cannot place under a heading is probably not yet a finding. Anything with a heading but no content is a gap you now know about.

Lab: review a RAG ingestion plan

Now we test the design. The lab called What Goes in the Index? gives you a company ingestion plan with seven decisions. Three of them let the assistant repeat planted content or show people documents they should not see. Your job is to find which ones and say why, which is exactly a design review, the cheapest and best moment to catch these problems.

Quick check

Why is reviewing an ingestion plan useful before any chatbot exists?

Write findings and rank fixes

A good finding has five parts: what, where, evidence, severity, fix. Keep each to a line or two. Here is the shape using a made-up example. Notice that the evidence line names something a reader could go and look at.

text
Finding: Wiki pages are indexed without review
Where: Boundary B3 (sources to index)
Evidence: Lab plan, decision about auto-importing all pages
Severity: High. Easy to do, and the bot repeats the text to everyone
Fix: Index only approved folders and record who added each item
One finding. Write yours from the lab decisions you flagged.

Ranking is a decision, and being willing to make it is what makes a report useful. A team with a week to spare will do your first fix and probably your second. Make sure those two matter.

Review the course and choose what next

Look back over nine days. You learned how text becomes tokens, how a context window and system prompt shape answers, how applications and agents connect to tools, where trust boundaries sit, and how to plan and run tests safely. Today you joined those into one deliverable.

Add section 6. Pick three goals from these routes: agents and tool abuse, RAG security in more depth, defensive labs where you fix what you broke. Write one sentence on why you chose each and one small first step, such as a lab to complete this week.

text
6. Next learning goals
- Agents and tool abuse: I want to see how tool permissions fail. First step: finish one agent lab this week.
- RAG security in depth: B3 and B4 gave me my best findings. First step: re-read my flagged decisions.
- Defensive labs: I want to practise fixing. First step: pick one finding and write its fix as a task.
An example of section 6. Replace the reasons and steps with your own.

Your turn

What Goes in the Index?

This lab puts the design-review half of red teaming into practice. Instead of attacking a bot, you read a plan and judge it against the boundaries you drew, which is where many real RAG problems are cheapest to prevent.

Open the lab in a new tab

What to try

What to notice

  • Which of your four boundaries each risky decision touches
  • How a harmless-looking convenience choice becomes a risk
  • Whether your three flagged items share a common cause

If it did not work

Writing a report that lists attacks but no evidence

For every finding, point to the exact lab decision or recorded prompt and result that supports it.

Marking boundaries on the diagram but not tying threats to them

Number each boundary and label each threat with the same number.

Giving every finding the same severity

Rate by impact and ease, then add a one-sentence reason so the ranking can be challenged.

Leaving fixes vague, such as 'make it more secure'

State a specific change someone could put on a task list, for example 'index only approved folders'.

Recap

  • A threat model starts with a diagram and trust boundaries, then one question per boundary
  • Design reviews catch RAG problems before any attack is needed
  • Content that enters the index can be repeated or exposed to the wrong people
  • A finding needs what, where, evidence, severity and fix
  • Ranking three fixes is more useful than listing many
  • Learning continues with agents, RAG depth, tool abuse and defensive labs

Cheat sheet

Trust boundary
Where data crosses between parts with different trust
Ingestion
Collecting and storing documents in the searchable index
Finding format
What, where, evidence, severity, fix
Severity
Impact and ease, with a one-sentence reason
Report order
Diagram, boundaries, threats, results, fixes, next goals

Before the next day

Mission: Finish your capstone report

Pull everything into one file called day9-report.txt so that a reader who was not on this course can understand what you built, tested and recommend. This is the proof of your nine days of work.

About 45 min

You are done when: A file day9-report.txt containing diagram, boundaries, threat model, results, three ranked fixes and three learning goals.

Want more? Add a short paragraph on what you could not test and what you would test next given more time.

Quiz

4 questions. Your score is saved with your progress.

1. Why do you mark trust boundaries before listing threats?

2. A planted instruction in an indexed document is dangerous mainly because:

3. Which fix best belongs at the top of a ranked list?

4. What makes a finding believable to a reader?

Finish the day

Mark the day complete to save your progress.

What next: There is no Day 10, so your next step is your own. Pick one of your three goals, such as agents and tool abuse, deeper RAG security or defensive labs, and start it this week while today's report is fresh.

Tell someone you are doing this:WhatsAppXLinkedIn

Your notes

Private to this device. Write what you want to remember from today.